Shorts on EXPLAIN ANALYZE

Cross-Database Joins: The Coupling You Can't See Until You Split

Sat, 11 Jul 2026 00:00:00 +0000

TL;DR

A join across two schemas on the same MySQL server costs nothing to write, which is exactly why nobody counts it. Every one of those joins is a hard coupling that pins both schemas to one instance, so the day you need to split for scale, the joins are the wall. And the referential integrity you assumed the boundary carried was never enforced to begin with.

Somewhere in the codebase, since roughly the first month, there is a query like this:

1
2
3
4


SELECT li.order_id, li.sku, li.qty, inv.status, inv.amount_due
FROM orders.line_items li
JOIN billing.invoices inv ON inv.order_id = li.order_id
WHERE li.created_at >= '2026-07-01';

Two schemas, orders and billing, one MySQL server, single-digit milliseconds for three years. In that time roughly four hundred more queries have grown the same shape: application code, a dozen dbt models, cron reports, the admin dashboard. Nobody wrote a design doc for any of them, because MySQL lets you qualify a table with schema.table and join across the boundary as if it weren’t there. Then the primary crosses 90% on the capacity dashboard, the obvious relief is to lift billing onto its own instance, and the migration plan dies in review on one line: four hundred joins stop compiling the moment the schemas live on different servers.

The FDW that looks like a fix

The escape hatch is obvious: split anyway and put a foreign data wrapper between the instances so the joins keep working across the wire. The syntax survives the split. The performance does not. Cross-server joins don’t push down; postgres_fdw only sends a join to the far side when both tables live on the same foreign server (the docs are explicit), so across two instances the join executes locally, pulling rows over the network first. Svix hit exactly this in October 2025: a join that failed to push down pulled both tables in their entirety, moved millions of rows where thousands were expected, and timed out.

So the FDW gives you back the SELECT and takes away the reason you split. Every cross-boundary query drags its working set back over the network to be joined next to the primary anyway.

An FDW join across instances is a distributed join, not a local one

If EXPLAIN (VERBOSE) shows the join in the local plan rather than in the Remote SQL, the far table is shipping over in full. Treat every cross-instance FDW join as a network transfer sized by the larger table, not by your WHERE clause.

Free to write, fake underneath

A MySQL “database” and a MySQL “schema” are the same object: a namespace on one server. There is no cross-database join inside one server, only a fully-qualified one, and the qualifier costs a few keystrokes. Postgres draws the line elsewhere. Schemas in one database join freely; separate databases can’t be joined at all without FDW or dblink, so a team that split into Postgres databases hits the wall on day one, where at least it’s visible.

The worse fact: MySQL foreign keys can cross schemas. REFERENCES billing.invoices(order_id) is legal DDL, and almost nobody writes it, because the schemas belong to different teams and nobody wants their writes gated on another team’s table. So the boundary those four hundred joins cross has no enforced referential integrity and never did. The failure modes are the ones you’d predict from dropping a foreign key, except no FK was ever declared: orphaned line items with no cascade to catch them, ID-type drift that turns the join into a silent cast (BIGINT on one side, CHAR(36) UUID after a migration on the other), backfills that assume the far side exists.

This is a same-server, cross-namespace problem

Cross-schema joins inside one Postgres database, or on a MySQL server you never intend to split, are not the anti-pattern. The coupling bites when scale forces the namespaces onto different instances. If that day never comes, the free join is genuinely free; the trap is assuming that’s your situation when the growth curve says otherwise.

Recovery has the same hole. Each database backs up independently, two --single-transaction dumps, each consistent, each catching a different instant, and the skew surfaces only when you restore one side. Roll billing back to 04:00 while orders keeps its afternoon of writes and every cross-schema join now compares two timelines. No constraint exists to fail, so nothing errors; the afternoon’s orders point at invoices the restore erased. (Postgres PITR replays WAL for the whole cluster, never one database, so an independent restore was never on offer.) The honest unit of backup and restore is the set of databases your queries treat as one, and four hundred joins say that set has two members.

Find the joins before you need the split

The coupling lives in query text, not the catalog: no pg_constraint row, no KEY_COLUMN_USAGE entry. But it always leaves a fingerprint, because at least one table in the query has to be namespace-qualified. On a running MySQL server, the digest table preserves the qualifiers:

1
2
3
4
5


SELECT SCHEMA_NAME, DIGEST_TEXT, COUNT_STAR, SUM_ROWS_EXAMINED
FROM performance_schema.events_statements_summary_by_digest
WHERE SCHEMA_NAME = 'orders'
 AND DIGEST_TEXT LIKE '%billing.%'
ORDER BY COUNT_STAR DESC;

Two caveats: DIGEST_TEXT truncates at performance_schema_max_digest_length (1024 bytes by default), and the digest table resets on restart, so it shows recent traffic, not history. Pair it with a grep of the repos (application code, ORM models, dbt sources) for the billing. qualifier. Datadog ran this inventory across 30-plus teams when they unwound a shared database in 2025; mapping the cross-boundary queries came before any carving. In Postgres the hunt is easier: the cross-instance dependency is already wearing a postgres_fdw label, listed in pg_foreign_table, with query text in pg_stat_statements.

What the fixes cost

The cheapest move is writing the coupling down where a machine can read it: COMMENT 'references billing.invoices.order_id, service-owned, no FK across instances' on the column. Readable by humans and by schema-reading assistants, enforced by nothing. If the boundary has to actually hold, a scheduled reconciliation job anti-joins the two sides and reports the orphans the missing FK would have rejected at write time (the scale-the-class fix applied to integrity). Eventually-consistent by construction, and someone has to own the alert. For the analytics joins, a consolidated replica (logical replication or CDC into one reporting instance) makes the join legal again, at the price of a pipeline to run and a lag window to reason about. For the transactional joins, the durable answer is the one Datadog and the AWS decomposition guidance both land on: replace the query with a service call. That’s the real unwind and the expensive one, a rewrite of every call site the join touched.

When the split itself arrives, replication makes it survivable, and the order of operations matters more than the tooling. Write-path joins and cross-schema transactions get rewritten first, while everything still shares a server; after cutover there is no atomicity across the boundary. Then the future cluster comes up as a replica of the whole instance, billing reads migrate at leisure, and writes cut over in a short freeze once GTIDs match. GitHub ran this playbook in 2021: SQL linters to stop new cross-boundary queries, then a write-cutover that moved 130 tables with sub-second downtime. Read joins stay gradual. A multi-source channel keeps feeding orders into the new cluster read-only, so stragglers join against a lagged copy while the digest counts drain to zero; give that channel a deadline, it’s the weld on life support. One trap: a read join that feeds a write decision (check invoice status, then update the order) is a cross-schema transaction wearing a SELECT, and the lagged copy turns it into a race. The digest table won’t flag it; only the call sites do.

One more consumer of the missing metadata: an AI assistant scoped to one connection (the default MCP posture) either declares the column unjoinable or, handed a shared server, cheerfully reproduces the free cross-schema join with no signal that it just crossed a service boundary. The column comment is the only place the relationship exists in a form the model reads.

The join that costs nothing to write is the one that costs the most to remove. Every schema.table qualifier typed without a second thought is a vote to keep both schemas on one server forever, cast by someone who had no idea there was an election.

How Teams Actually Finish What They Start, Part V: The Sprint as a Working Set

Tue, 21 Apr 2026 00:00:00 +0000

TL;DR

For teams whose week is shaped by inbound work, the sprint should hold only what is being worked on now plus what gets pulled next. No forward estimates, no velocity commitments. Priority lives in labels; the team pulls from the labeled backlog as in-flight work completes.

Tuesday at 2pm. Sprint planning. The team has been here for ninety minutes. Eighteen tickets on the board, points being argued about (was this a 5 or an 8 last quarter?). A senior engineer flags they have to leave for an interview at 3. The product manager wants to commit to 42 points so the velocity curve in the leadership deck stays smooth. Three operational tickets came in during the meeting itself. Nobody has touched code today. The sprint will start tomorrow with eighteen tickets the team has not properly looked at, plus the three that arrived during planning, plus whatever arrives over the next two weeks. The planning meeting was the work today.

Better grooming doesn’t fix it

The standard fixes target the planning meeting: better grooming, T-shirt sizing instead of points, async estimation in Slack. Each saves twenty minutes and leaves the underlying mistake intact. The mistake is the sprint trying to be a committed plan for two weeks of work the team has not done yet, on a team whose two weeks are not predictable. No amount of grooming makes the unpredictable predictable. The fix is a sprint that admits it.

The sprint as a working set

A different mode: the sprint holds only what is being worked on right now, plus the highest-priority items the team will pull next. That is it. The sprint stops being a two-week forecast or a velocity commitment. The board reflects reality rather than narrating it.

The mechanics follow. The backlog holds everything, labeled by priority. The manager keeps the labels current, and high-priority items rise to the top of the filtered view. The sprint holds only tickets currently in progress plus immediate next pulls. Engineers pull from the labeled backlog into the sprint as their current work completes. There is no forward estimation at planning, because points are written post-hoc (see Part IV). Planning becomes a short check-in: the team confirms priorities, surfaces blockers, and returns to work.

Engineers file their own tickets when they discover work. A bug found while shipping a feature. A refactor that surfaces during code review. A dependency that needs chasing. The IC who found it writes the ticket. “Someone will write this up later” becomes nobody, and the work disappears from the tracker without disappearing from reality. The team’s tracker has to hold the team’s actual work; if the work is not in the tracker, the work does not exist for planning purposes.

The responder rotation absorbs incoming interruption tickets (Part III) so the sprint is not churned by every Slack message and every cross-team request. The sprint is what the team is building. The responder column is what arrives. The two queues stay separate, and the sprint stays small.

The sprint is not a copy of the backlog

The pressure to grow the sprint is constant. Leadership wants velocity numbers, the team wants to look ambitious, every new priority feels like it should land “in this sprint.” It should not. The sprint is what the team is doing now plus what they will pull next. If the sprint contains tickets nobody has looked at, the discipline has slipped, and the velocity that comes out the other side is fiction.

In the tracker

Jira gives you priority fields, labels, components, ranks, epics, and themes. The working-set sprint needs three things from the tracker: a backlog the manager can prioritize, a way to see the labeled top of the backlog, and a sprint board that shows what the team is doing right now. The rest is decoration.

A workable setup: priority lives on a single field or a single label, picked once and used consistently. A saved filter (JQL or board view) shows the labeled high-priority backlog, and that filter is the team’s entry point when their current ticket closes. The sprint board shows in-progress and next-up tickets only. Standups walk that board ticket by ticket. Estimation columns are optional; if used, they are filled in after the ticket closes, not before.

Use one priority mechanism, not three

Jira lets you mix priority field, labels, components, and rank order. Pick one. A label like priority:p0 works. So does the built-in priority field. Mixing them means engineers pull from one filter while the manager updates another, and the team works on the wrong tickets while the tracker says everything is fine.

When forward sprints work

A team with a stable, well-scoped backlog and predictable interruptions can run forward sprints with point commitments. Some maintenance teams have this. Some platform teams whose remit has been narrowed and frozen do too. For everyone else, the working-set sprint is the one that matches reality. Part IV’s measurement discipline is how the team finds out which side it is on; the working-set sprint is what the team does with the answer.

What changes

Sprint planning gets short. Velocity stops being a fiction the leadership deck has been running on. With the board reflecting actual work, the rest of the cadence gets honest too. Standups walk the board ticket by ticket and finish in fifteen minutes. Retros talk about what actually happened, not about the gap between estimate and reality. And reviews compare engineers on the same shapes of work, with the data Part IV produced. The tracker stops being a stage for the planning ritual and becomes a tool for getting work done.

How Teams Actually Finish What They Start, Part IV: Point After the Fact

Fri, 03 Apr 2026 00:00:00 +0000

TL;DR

Forward estimation breaks for any team whose week is shaped by work that arrives. The discipline that scales is to point after the fact: when the ticket closes, the person who did the work writes down what it took. Over a quarter the team has real data about where time actually goes.

Standup Monday. A 5-point ticket lands in the engineer’s column. Wednesday the engineer is still on it, two production fires deep, the original scope half-discovered. By Friday it ships. The retro looks at velocity. The 5 stays a 5, the team’s data says the engineer did 5 points of work, and the next sprint’s planning uses that data to size the next ticket of the same shape. The bug compounds.

Better estimates don’t fix it

The obvious move is to estimate better: planning poker, three-point estimates, finer story points, more grooming up front. None of it works on a team whose week is shaped by inbound work. The variance is not in how the engineer reads the ticket. It is in what arrives between Monday and Friday. A ticket scoped honestly Monday gets eaten by an unrelated incident Wednesday. A 5-point ticket stays 5 points until the dependency the engineer didn’t know about turns it into 13. Forward estimation is trying to predict the team’s week, and the team’s week is not predictable.

Point what you did

The fix is mechanical. When the ticket closes, the person who did the work writes down what it took. Over a quarter the team has real data: which categories of work consume the most time, which engineers carry which kinds of load, where the same shape of problem keeps eating a day each time. The cost is near zero (the ticket is closed; the person who did the work is sitting there). Bottlenecks surface fast: a category that always takes three times what its siblings take is a place to invest, and the data makes it visible without anyone having to argue for it.

The rule has to be load-bearing in the workflow, not aspirational. A ticket cannot move to Done without a points value. Without that constraint, the data has gaps, and the gaps are not the random kind.

Half-pointed data is worse than no data

Without a load-bearing constraint, easy tickets get pointed and hard tickets get closed in a rush. The long-tail work the team most needs to see disappears from the record. The team trusts the partial data anyway and reaches the wrong conclusions about its own capacity.

What the data enables

Operation-heavy teams can drop forward sprint commitments entirely. The manager sets priorities through labels on tickets and epics. The team pulls from the top of the labeled backlog as engineers free up. After-the-fact points accumulate over the quarter and give the team a real baseline: capacity, distribution, ticket-shape patterns. Forecasting becomes a quarter-long view rather than a sprint-long commitment.

The same data benchmarks people. Recurring work converges in shape over a quarter. Responder rotations cluster around the same handful of incident types. Refactors pattern-match to a few common shapes. When ten engineers have closed the same kind of ticket, the spread of points across people becomes visible. Reviews stop being a debate about effort and become a comparison against work everyone has done. The conversation shifts from “I think Alice ships fast” to “Alice closes the same shape of ticket in 6 points where the team median is 8.”

Compare same shapes, not raw velocity

The benchmark only works for like-for-like work. A frontend ticket and a database investigation are not on the same scale. Group tickets by shape (incident type, refactor pattern, investigation category) before comparing engineers, and ignore raw point totals across the team.

When forward estimation works

Forward estimation works when the team’s work is genuinely stable: same shape every week, predictable interruptions, no unscoped depth. Some maintenance teams have this. Most product teams and most operational teams do not. The point of measuring after the fact is to find out which side your team is actually on, and to design the cadence around the answer.

What changes over a quarter

A team that points after the fact has a quarter’s worth of evidence about itself: which work consumes time, who is faster on what, where the bottlenecks live. A team that points before the fact has a quarter’s worth of guesses, refined and re-litigated each sprint. Retros become data-driven instead of opinion-driven. Reviews compare engineers against the same shapes of work. And when the team asks for headcount, the ask is grounded in a category that has been swallowing unbudgeted time, not a hunch.

The Hello-World Procurement Problem: Why LLM Tooling Gets Bought Wrong

Sun, 21 Dec 2025 00:00:00 +0000

TL;DR

A CTO declares “full agentic” off a vendor demo. Without an SME watching the rollout, corruption ships and surfaces a year later when a customer reports a wrong number. With an SME, the work is information infrastructure first (so agents have enough context to make high-probability decisions) and guardrails for the cases where context isn’t enough.

A CTO sits through a vendor demo. A sales engineer types “show me the top ten customers by revenue last quarter” into a prompt and a working SQL query materializes in 30 seconds, runs against a sample dataset, returns plausible numbers. The CTO declares the company is going full agentic. Procurement closes the contract by Friday.

Procurement closes on Friday. The first agents reach production Tuesday. By the end of the quarter, three internal dashboards are running LLM-generated SQL, two of them against the company’s order data. The SQL passes the lint check (EXPLAIN runs, the result set has the right columns), and the dashboards display plausible numbers. Whether the numbers match what they should be is a question nobody on the team is positioned to answer.

Without SMEs

If the agent-generated SQL looks like gold to everyone in the room, the Rounders rule applies: if you can’t spot the sucker in your first half hour at the table, you are the sucker. Without someone in the room who’d catch the polysemic tier column or the undocumented soft-delete convention buried in three tables, the team is approving plausibility on a system optimized to produce it.

If the produced code looks good to you, you’re probably not the SME.

The corruption rate observed in the demo is a lower bound for what the tool produces against real data, often by a multiple. The realities catalogued in What AI Gets Wrong About Your Database (undocumented conventions, polysemic columns, business logic in tribal knowledge, ten-year-old codebases with three “current” patterns) are exactly the regions of input space where the model’s training distribution is sparse and contradictory. Demos run in the dense-distribution sweet spot. Production runs the inverse on every axis.

With nobody positioned to measure the gap, nothing flags it. Corruption is silent by construction. It doesn’t surface as one identifiable bug; it surfaces as drift across many places at once, traced back to LLM-generated code or queries whose authors can’t reconstruct what the model meant. By the time the rate is visible, the corruption has been propagating for weeks or months. The team has too many simultaneous issues to triage one at a time. Backups have rolled past the worst of the window.

The detection mode is external. A customer reports a number that doesn’t match what they expected. An analyst running LLM-powered queries on the company’s data publishes a report that contradicts internal numbers. A regulator asks a question and the answer doesn’t match the previous quarter’s filing. Whatever surfaces it, the failure is now a public one, and the team learning the failure mode is the same team trying to contain it.

With SMEs

The CTO’s declaration doesn’t change. The job changes. With an SME watching the rollout, the work is infrastructure first.

Agents make high-probability decisions when their inputs are dense. That means the schema is documented, polysemic columns are tagged, conventions are written down somewhere the model can reach, the dataset the agent runs against mirrors production rather than a curated subset. The realities that make a mature codebase mature (patterns evolved over years, decisions encoded in column names, exceptions buried in tribal knowledge) are exactly the inputs the agent doesn’t have unless someone puts them there. The SME’s first job is documenting what currently lives in heads. Without that, the agent operates in the sparse regions of its training distribution, and the floor on its corruption rate stays high regardless of how the harness is tuned.

Guardrails are the second piece, for the cases where dense inputs still aren’t enough. Decompose work into chunks small enough to verify. Route checkpoints between chunks to the SME whose domain it is. Audits produce a failure-rate number against ground truth, not a yes/no. Recovery drills test rolling back six months of LLM-generated changes, because that’s the realistic detection horizon for silent corruption. The point is to catch the cases where the agent’s confidence and its accuracy are decoupled, which is where most of the corruption lives.

Both pieces have to be in place before the deployment goes wide. Once the rate is visible from outside, the SME bench is already triaging incidents instead of building infrastructure, and the architecture won’t grow either piece on its own.

When this doesn’t apply

Small teams. The buyer is the SME, or one degree away. The infrastructure question gets answered by the same person making the rollout call.
Bounded, low-stakes use cases. Personal productivity tooling, draft generation, internal-only knowledge work where corruption is recoverable.
Mature vendor categories. Office suites, established CI/CD platforms, well-trodden CRM tooling. The failure modes are known and the buyer has reference points. New categories are where the asymmetry lives, and that’s exactly where LLM tooling sits in 2026.

The bill arrives later

The productivity dividend the CTO booked off the demo is real, in the sense that the deal closed and the harness shipped. The bill arrives a quarter or two later when a customer surfaces a number that doesn’t match the books, the auditor asks how the model arrived at it, and the team learns the failure mode in public instead of in QA.