Somewhere in the codebase, since roughly the first month, there is a query like this:
| |
Two schemas, orders and billing, one MySQL server, single-digit milliseconds for three years. In that time roughly four hundred more queries have grown the same shape: application code, a dozen dbt models, cron reports, the admin dashboard. Nobody wrote a design doc for any of them, because MySQL lets you qualify a table with schema.table and join across the boundary as if it weren’t there. Then the primary crosses 90% on the capacity dashboard, the obvious relief is to lift billing onto its own instance, and the migration plan dies in review on one line: four hundred joins stop compiling the moment the schemas live on different servers.
The FDW that looks like a fix
The escape hatch is obvious: split anyway and put a foreign data wrapper between the instances so the joins keep working across the wire. The syntax survives the split. The performance does not. Cross-server joins don’t push down; postgres_fdw only sends a join to the far side when both tables live on the same foreign server (the docs are explicit), so across two instances the join executes locally, pulling rows over the network first. Svix hit exactly this in October 2025: a join that failed to push down pulled both tables in their entirety, moved millions of rows where thousands were expected, and timed out.
So the FDW gives you back the SELECT and takes away the reason you split. Every cross-boundary query drags its working set back over the network to be joined next to the primary anyway.
EXPLAIN (VERBOSE) shows the join in the local plan rather than in the Remote SQL, the far table is shipping over in full. Treat every cross-instance FDW join as a network transfer sized by the larger table, not by your WHERE clause.Free to write, fake underneath
A MySQL “database” and a MySQL “schema” are the same object: a namespace on one server. There is no cross-database join inside one server, only a fully-qualified one, and the qualifier costs a few keystrokes. Postgres draws the line elsewhere. Schemas in one database join freely; separate databases can’t be joined at all without FDW or dblink, so a team that split into Postgres databases hits the wall on day one, where at least it’s visible.
The worse fact: MySQL foreign keys can cross schemas. REFERENCES billing.invoices(order_id) is legal DDL, and almost nobody writes it, because the schemas belong to different teams and nobody wants their writes gated on another team’s table. So the boundary those four hundred joins cross has no enforced referential integrity and never did. The failure modes are the ones you’d predict from dropping a foreign key, except no FK was ever declared: orphaned line items with no cascade to catch them, ID-type drift that turns the join into a silent cast (BIGINT on one side, CHAR(36) UUID after a migration on the other), backfills that assume the far side exists.
Recovery has the same hole. Each database backs up independently, two --single-transaction dumps, each consistent, each catching a different instant, and the skew surfaces only when you restore one side. Roll billing back to 04:00 while orders keeps its afternoon of writes and every cross-schema join now compares two timelines. No constraint exists to fail, so nothing errors; the afternoon’s orders point at invoices the restore erased. (Postgres PITR replays WAL for the whole cluster, never one database, so an independent restore was never on offer.) The honest unit of backup and restore is the set of databases your queries treat as one, and four hundred joins say that set has two members.
Find the joins before you need the split
The coupling lives in query text, not the catalog: no pg_constraint row, no KEY_COLUMN_USAGE entry. But it always leaves a fingerprint, because at least one table in the query has to be namespace-qualified. On a running MySQL server, the digest table preserves the qualifiers:
| |
Two caveats: DIGEST_TEXT truncates at performance_schema_max_digest_length (1024 bytes by default), and the digest table resets on restart, so it shows recent traffic, not history. Pair it with a grep of the repos (application code, ORM models, dbt sources) for the billing. qualifier. Datadog ran this inventory across 30-plus teams when they unwound a shared database in 2025; mapping the cross-boundary queries came before any carving. In Postgres the hunt is easier: the cross-instance dependency is already wearing a postgres_fdw label, listed in pg_foreign_table, with query text in pg_stat_statements.
What the fixes cost
The cheapest move is writing the coupling down where a machine can read it: COMMENT 'references billing.invoices.order_id, service-owned, no FK across instances' on the column. Readable by humans and by schema-reading assistants, enforced by nothing. If the boundary has to actually hold, a scheduled reconciliation job anti-joins the two sides and reports the orphans the missing FK would have rejected at write time (the scale-the-class fix applied to integrity). Eventually-consistent by construction, and someone has to own the alert. For the analytics joins, a consolidated replica (logical replication or CDC into one reporting instance) makes the join legal again, at the price of a pipeline to run and a lag window to reason about. For the transactional joins, the durable answer is the one Datadog and the AWS decomposition guidance both land on: replace the query with a service call. That’s the real unwind and the expensive one, a rewrite of every call site the join touched.
When the split itself arrives, replication makes it survivable, and the order of operations matters more than the tooling. Write-path joins and cross-schema transactions get rewritten first, while everything still shares a server; after cutover there is no atomicity across the boundary. Then the future cluster comes up as a replica of the whole instance, billing reads migrate at leisure, and writes cut over in a short freeze once GTIDs match. GitHub ran this playbook in 2021: SQL linters to stop new cross-boundary queries, then a write-cutover that moved 130 tables with sub-second downtime. Read joins stay gradual. A multi-source channel keeps feeding orders into the new cluster read-only, so stragglers join against a lagged copy while the digest counts drain to zero; give that channel a deadline, it’s the weld on life support. One trap: a read join that feeds a write decision (check invoice status, then update the order) is a cross-schema transaction wearing a SELECT, and the lagged copy turns it into a race. The digest table won’t flag it; only the call sites do.
One more consumer of the missing metadata: an AI assistant scoped to one connection (the default MCP posture) either declares the column unjoinable or, handed a shared server, cheerfully reproduces the free cross-schema join with no signal that it just crossed a service boundary. The column comment is the only place the relationship exists in a form the model reads.
The join that costs nothing to write is the one that costs the most to remove. Every schema.table qualifier typed without a second thought is a vote to keep both schemas on one server forever, cast by someone who had no idea there was an election.
