Scale the Pattern, Not the Instance

Sat, 20 Jun 2026 00:00:00 +0000

TL;DR

Most production fixes patch the occurrence in front of you (reboot this box, clean these rows, bump this instance), which clears today and leaves the class of problem intact to recur. The fix that holds lives one level up, where one change covers every future occurrence. The skill is telling a recurring class from a true one-off before you’ve been burned three times.

A development cluster ran on shared-tenancy EC2, the cheap tier with no dedicated host. Prod deploys had to clear dev first, so when a box wedged (SSH hung, the health check flatlined, steal time climbing on whatever still answered) the deploy queue stalled behind it. The team read it as capacity and moved up an instance size. It wedged again, so they went up another, now triple the bill and still wedging. Capacity was never it. A noisy neighbor on the oversubscribed host was starving the tenants, and a bigger seat at the same crowded table changes nothing. The unblock path made it worse: EU developers couldn’t reboot the box themselves (correctly, they shouldn’t hold that access), so a US infra engineer got paged in the middle of the night to stop/start a single dev box. What finally held wasn’t a bigger box. It was a pipeline any developer could run to relaunch a wedged instance on a fresh physical host.

One box wedged; the pipeline covered every box that ever would. That distance, between fixing the occurrence in front of you and retiring the class it belongs to, is the whole article. The dev cluster is the easy version, because the fix was operational and nobody’s data path ran through it. The expensive version is when the strain sits in your data layer and the instance control is right there in the console. The reflex is the same either way: fix the thing that’s strained, which clears the symptom by end of afternoon and is exactly why it keeps beating the slower question of what class of problem just announced itself.

Just scale up, just scale out

A Kafka consumer group on the orders topic starts lagging, a few hundred thousand messages behind by mid-morning and never catching up overnight. The handler does the obvious thing per message: deserialize, look up the customer, write a row, commit the offset. The team bumps the workers a size and goes from six partitions to twelve. Lag clears in an hour. Three weeks later it’s back; twelve isn’t enough, so they go to twenty-four and up another size. Relief, a quiet stretch, recurrence one size up. The cycle has a rhythm once you’ve seen it.

By the second recurrence the fix is obvious to anyone who’s met it before: stop processing one message at a time, batch the poll and the downstream write, commit once for the batch. The obviousness is the point, because the team didn’t reach for it. They reached for partitions and worker size, twice, because those are the controls the dashboard puts a slider on, and a change to the handler is the one that isn’t. The occurrence got fixed each time. The class, a handler that pays a fixed round trip per message however many workers run in parallel, stayed exactly as expensive as it started.

This is the same argument Part I of the query series made under the heading “Why Metal Doesn’t Help,” where adding a third read replica made replication lag worse rather than better. The hardware treated an occurrence, this system is slow today, and never touched the class, the write pattern that produced the lag. That post stayed inside Postgres. The shape isn’t a Postgres thing, and the rest of this is where it shows up.

The same mistake, four domains

The reboot pipeline and the Kafka lag are the same move pitched at different altitudes. Each problem has a fix that clears the occurrence in front of you and a fix that retires the whole category it belongs to. The first one you reach for by reflex, because the occurrence is what’s paging you. The second covers every occurrence that hasn’t happened yet, including the ones produced by a code path nobody on the current team remembers writing.

Domain	Fixing the instance	Fixing the class
Dev fleet	stop/start the box that wedged	a pipeline any dev runs against any box
Kafka lag	drain today’s backlog	a handler that batches every message it will see
Orphaned data	`UPDATE` the rows that broke the report	a foreign key that makes the orphan impossible to write
Slow reads	tune the query a reviewer caught	a log threshold that surfaces every N+1, not the one noticed

The first two rows you’ve already seen in this post. The other two are where the idea earns its rent on a database team.

Orphaned rows are the cleanest example, because the instance fix is so easy to reach for. A child row points at a parent that’s gone, a report breaks, and you write the UPDATE that reconciles it. It works. You’ll write a near-identical one next quarter when a different code path produces the same shape, because nothing stopped the shape from being written in the first place. The class fix is the foreign key (or the CHECK, or the NOT NULL) that makes the orphan unrepresentable. It covers every write that will ever touch the table: the migration script, the one-off backfill, the new service that integrated last week and was never code-reviewed against this invariant. The cost is real and worth stating, since it’s the reason people leave the constraint off. It validates on every write, and it will reject a messy bulk load you’d otherwise have let slide. That friction is the feature.

The slow-read row is the same trade one layer up. You can fix the N+1 a reviewer happened to catch, or you can turn the catching into infrastructure: log queries-per-request and alert when a single request crosses some threshold, so the next N+1 announces itself in CI instead of waiting for a customer to sit through a slow page. The pull request fixes one endpoint. The threshold catches the shape wherever it surfaces next, including in code no reviewer reads closely.

Not every invariant fits in a constraint or a metric. Some live only in a reviewer’s head: this table is append-only, that column is denormalized on purpose and both copies have to move together, this service owns the writes and everyone else reads. The class fix there is weaker but still real. Write the invariant down where the next change will run into it (a schema-conventions doc, an ADR, a comment on the migration that introduced the rule) and make checking it a standing line item in review, not a thing one careful person happens to catch. Enforcement that leans on people is the most fragile kind, which is the whole reason to push everything you can into the constraint or the CI check first. What’s left over is what review and documentation are for.

One more thing before the table hardens into a rule. Some instance fixes are cheap and reversible; some carry a cost the class fix doesn’t, and Kafka partitions are the second kind.

Partitions are hard to take back

Adding partitions to a keyed topic reshuffles the key-to-partition mapping. Confluent’s docs spell this out: Kafka maps a keyed message to a partition by hash(key) % partition_count, and changing the count means messages with the same key can start landing on a different partition, breaking the per-key ordering you may be relying on (Confluent, “Choose and Change the Partition Count”). And there is no supported operation to reduce the count afterward. If you scale out partitions to paper over a per-message handler, you’ve taken on an ordering hazard and a topology you can’t cleanly walk back, to avoid a change you’ll likely make anyway.

Finding the level where the fix generalizes

Fix the class, then. But which problems are a class? An instance of a class and a genuine one-off look identical the first time, and you have to bet on which you’re holding before you’ve watched it recur. The reboot pipeline was a wasted week if exactly one box ever wedges. The foreign key is bureaucratic drag if the orphan came from a bug already fixed and the shape can’t come back. Generalize too early and you’ve built a framework for a population of one, which is its own well-documented way to lose a sprint.

The heuristic that sorts them is unglamorous. Count. How many times has this exact shape cost someone an afternoon, and is the count rising? One wedge, ever, is a pet. The third overnight page to reboot a different box is a class wearing the disguise of three unrelated incidents, and the disguise is the whole problem: each one gets closed individually by whoever caught it, so the pattern never accumulates into a number anyone acts on.

Profiling is how you make that count honest instead of anecdotal. The CPU-pegged case sorts pg_stat_statements by total_exec_time, which floats the high-frequency cheap query above the rare expensive one. That’s the same act in a different tool: looking past the single event that hurt once to find the shape that repeats. Once you can see the shape, what the class fix is depends on the shape. Per-unit overhead wants batching. A hot lookup that runs identically a million times a day wants a cache. A predicate the planner can’t seek on wants a SARGable rewrite or the right index, the full treatment of which is in non-SARGable predicates. An ORM issuing a query per row wants one set-based query. The level you fix at is what’s shared, not the tool you fix with.

Managed services force the question by removing the instance fix entirely. On your own metal you can bump the box one more size, which is exactly what lets a recurring problem hide for years. Hit a vendor’s instance ceiling and there’s no next size, no knob on their storage engine, no replication setting to reach into. The only layer left under your control is your own load, so finding the class stops being the disciplined choice and becomes the only one. The cloud-bill leaks audit runs on the same fact: the spend is yours, and nobody’s coming to optimize it for you.

This is not 'always generalize'

Lifting a fix to the class level has costs. A foreign key validates on every write and blocks messy backfills. A batched handler trades per-item latency for throughput and hands you partial-failure handling that a per-item loop answered for free. A query-count alert is one more thing that pages. The argument is narrow: when a shape recurs and the count is climbing, fix where it generalizes. When the population is genuinely one, don’t build the framework.

When fixing the one instance is right

The argument inverts the moment the population really is one. A box that wedged once during a bad maintenance window is a pet. Reboot it, move on, and don’t build a pipeline for a host that won’t repeat. A startup with eighteen months of runway should fix the instance every time the class fix costs an SME afternoon it can’t spare, and write the class fix down as known debt rather than pretend it’s done. And when you need relief now and the general fix is a week out, fix the instance to buy the week, then put the class fix on the calendar instead of the backlog.

The trap is that last case going quiet. The cleanup query that bought a quarter is still the team’s monthly ritual two years on, the constraint never landed, and the cost has compounded the whole time in afternoons nobody bothered to total. Buying time is fine. Forgetting you borrowed it is the failure.

The tell

The signal that a team is patching instances instead of fixing classes is a particular déjà vu in the incident log: the same shape of problem, handled fresh each time, by whoever happened to catch it. Orphaned rows reconciled by three different people in three quarters with three slightly different queries. A reboot runbook followed forty times and automated zero. Ask what the recurring shape is and who owns the fix for the category. If the answer is a stack of individually-closed tickets and no name, the class is unowned, which almost always means it’s still arriving.