Boring architecture beats clever architecture, every time

Clever architecture is a loan against future velocity. The twelve questions we answer before the first commit, and the Postgres pattern we use instead of adding a queue.

Boring architecture beats clever architecture, every time

Every architecture review at NetPy starts with a question that makes new engineers visibly uncomfortable: what is the least interesting thing that could possibly work?

Not the most scalable. Not the most elegant. The least interesting. If two designs both survive the next eighteen months, the one with fewer moving parts wins — and it is not close. The clever design is a loan against future velocity, and the interest rate is set by whoever is on call at 2am in month seven, when nobody remembers why the event bus exists and the consumer lag graph is a straight line upward.

We wrote this checklist after paying that interest a few times too many. It is what we run before the first commit on a new build, and what we re-run whenever someone proposes adding a component to an existing one.

The cleverness is not the problem, the coupling is

A distributed system is not risky because it is distributed. It is risky because every new component is a new boundary where state can diverge, and divergent state is where debugging stops being reading and starts being archaeology.

Consider a three-service integration we inherited on a client project: an API, a worker, and a cache invalidation listener, all passing customer records around as events. On paper, clean separation. In practice, one failed invalidation meant a customer saw stale pricing for eleven minutes — long enough that support had logged fourteen tickets before anyone looked at a dashboard. The root cause was four lines of code. Finding it took two engineers most of a day, because they first had to establish which of three systems was lying.

The fix was not better observability. It was deleting the listener. The API now writes the record and the cache key in the same transaction and invalidates inline. p99 write latency went from 180 ms to 210 ms. That 30 ms bought us the ability to reason about correctness by reading one file.

That trade — 30 ms of latency for the ability to hold the whole path in your head — is almost always worth taking at the scale most Indian SaaS products actually operate at.

Four properties that make an architecture "boring"

When we say boring, we mean something specific and testable, not "old" or "enterprise-y":

  • One place where state lives. Postgres, usually. If there is a second source of truth, name it out loud and say what happens when it lags.
  • A failure list you can say from memory. "If the DB is down, the app is down" is a complete sentence. "If the queue is backed up but the API is healthy, some notifications are delayed up to N minutes" is also complete. "It depends" is not.
  • One runtime per deploy target. Mixing Node and Python inside a single service means two dependency trees, two debuggers, two sets of vulnerabilities.
  • A rollback that is a command, not a procedure. If reverting requires a data migration reversal, you do not have a rollback; you have a hope.

None of these are constraints on ambition. They are constraints on ambiguity.

The twelve questions, before the first commit

This is the checklist, near-verbatim from our internal doc. Any "no" is not a veto — it is a conversation with a named owner.

  1. Can one engineer hold the request path in their head, end to end?
  2. What is the blast radius of a bad deploy — one tenant, one region, the whole site?
  3. What happens, precisely, when each third-party dependency returns 500 for ten minutes?
  4. Have we restored from backup in the last 90 days, and did we time it?
  5. How many new moving parts does this add, and what does each one cost per month in money and attention?
  6. Is every state-changing operation idempotent under retry?
  7. What is p99 at 5× current traffic, and what breaks first?
  8. Who is on call for this, and do they know it?
  9. What does a schema migration rollback look like six weeks in?
  10. How will we know it is broken — a number, an alert, and a threshold, not "users will complain"?
  11. Could we delete this in a year without rewriting the product?
  12. If the new technology fails us, what is the exit, and how long does it take?

Question 7 kills the most proposals, and it should. "We need Kafka" is a different claim from "we need to move 2 million events a day and replay them." Only one of those is a requirement; the other is a preference with an operational budget attached.

The queue we did not add

The most common instance of premature cleverness we see is the message queue introduced for background work that a database could have handled. Here is the boring version, and we have shipped it at least four times:

# Postgres as the queue. One table, one index, no broker.
# Claim a job atomically; SKIP LOCKED lets N workers pull without contention.
CLAIM = """
WITH next AS (
    SELECT id FROM jobs
     WHERE run_at <= now()
       AND status = 'queued'
     ORDER BY run_at
     FOR UPDATE SKIP LOCKED
     LIMIT 1
)
UPDATE jobs SET status = 'running', started_at = now(), attempts = attempts + 1
 WHERE id = (SELECT id FROM next)
RETURNING id, kind, payload, attempts;
"""

def drain(worker_id: str, batch: int = 20) -> int:
    done = 0
    with db.transaction() as tx:
        while done < batch:
            job = tx.execute(CLAIM).fetchone()
            if job is None:
                break
            try:
                HANDLERS[job["kind"]](job["payload"])
                tx.execute("UPDATE jobs SET status='done', finished_at=now() WHERE id=%s", (job["id"],))
            except Exception as exc:
                # Exponential backoff with a hard cap: 30s, 2m, 8m, 32m, 2h, then park.
                delay = min(30 * pow(4, job["attempts"]), 7200)
                tx.execute(
                    "UPDATE jobs SET status='queued', run_at=now() + %s * interval '1 second',"
                    " last_error=%s WHERE id=%s",
                    (delay, str(exc)[:500], job["id"]),
                )
            done += 1
    return done

Two details do the real work. FOR UPDATE SKIP LOCKED makes claiming safe with any number of workers and no coordination service. The partial index on (run_at) WHERE status = 'queued' keeps the claim query flat as the table grows — without it, throughput collapses to a few hundred jobs a minute once you pass a million rows.

Measured on a 4 vCPU / 16 GB instance: 1,200–1,800 jobs per minute with six workers, claim latency under 8 ms. We have run this past 900k queued rows with the index and no tuning. Where does it stop being the right answer? When you need fan-out to dozens of heterogeneous consumers, replay from a retention window, or sustained rates above roughly 5k jobs a minute. That is the moment to reach for a broker — not before, and not because the diagram looks better.

Reversibility decides how much deliberation a choice deserves

The second filter is not "how good is this decision" but "how expensive is it to undo." We sort every architectural choice into two piles.

Two-way doors — cheap to reverse, decide fast, decide alone: ORM choice, HTTP framework, CSS approach, logging library, most third-party SDKs, column names. Argue for ten minutes, not three days.

One-way doors — expensive or impossible to reverse: multi-tenancy model (shared tables versus schema-per-tenant), public API resource shapes, event payload schemas already consumed by a client, the auth and session model, the primary key strategy, and anything a customer's integration depends on. These deserve a written decision record, a second reviewer, and a "what if we are wrong" paragraph.

Most teams invert the effort. They spend a week on the framework and an afternoon on tenancy, then live with the tenancy decision for four years. On Confidyn, the CA-firm inbox product, the tenant boundary was decided in a morning and documented in a page — that page has since saved us every time someone proposed a query that crossed it.

Where the cleverness budget goes instead

Boring architecture is not an argument against ambition. It is an argument for spending novelty in one place at a time, behind an interface you can replace.

Our rule: one new thing per quarter, with a named owner, an interface, and a deletion date for the prototype. If the experiment works, it replaces the boring thing behind a stable boundary. If it does not, we delete it and nobody's product code moves. This is how we have adopted vector search, LLM extraction, and offline sync in MilkPe without any of them becoming load-bearing in a way we could not reverse.

It also keeps hiring honest: a new engineer can ship to production in week one only if the system is small enough to read, and onboarding speed is the cheapest scaling lever a services company has.

What this looks like in practice

On a typical NetPy engagement the checklist changes the first sprint, not the architecture diagram. We usually remove one proposed component, add a backup restore test, add an idempotency key to two endpoints, and write down three one-way doors. The output of a review is a page, not a deck.

We do the same review for clients running Tally on NetPyCloud — where the boring answer is almost always "one well-sized box, a tested backup, and a restore procedure someone has actually executed" — and for teams we train through the FDE program. The checklist is not proprietary; it is the thing we would want a client's own engineer to hold us to.

You can read more about how we scope and run builds on netpy.in.

If a proposal cannot survive twelve questions, it will not survive production either. The questions cost an hour. The alternative costs a quarter and someone's weekend.