Featured image of post Cross-Database Joins: The Coupling You Can't See Until You Split

Cross-Database Joins: The Coupling You Can't See Until You Split

The escape hatch for splitting two joined schemas is a foreign data wrapper, and postgres_fdw will keep the syntax compiling while it ships the far table across the network in full to join it locally. One team expected thousands of rows and moved millions.

TL;DR
A join across two schemas on the same MySQL server costs nothing to write, which is exactly why nobody counts it. Every one of those joins is a hard coupling that pins both schemas to one instance, so the day you need to split for scale, the joins are the wall. And the referential integrity you assumed the boundary carried was never enforced to begin with.

Somewhere in the codebase, since roughly the first month, there is a query like this:

1
2
3
4
SELECT li.order_id, li.sku, li.qty, inv.status, inv.amount_due
FROM orders.line_items li
JOIN billing.invoices inv ON inv.order_id = li.order_id
WHERE li.created_at >= '2026-07-01';

Two schemas, orders and billing, one MySQL server, single-digit milliseconds for three years. In that time roughly four hundred more queries have grown the same shape: application code, a dozen dbt models, cron reports, the admin dashboard. Nobody wrote a design doc for any of them, because MySQL lets you qualify a table with schema.table and join across the boundary as if it weren’t there. Then the primary crosses 90% on the capacity dashboard, the obvious relief is to lift billing onto its own instance, and the migration plan dies in review on one line: four hundred joins stop compiling the moment the schemas live on different servers.

The FDW that looks like a fix

The escape hatch is obvious: split anyway and put a foreign data wrapper between the instances so the joins keep working across the wire. The syntax survives the split. The performance does not. Cross-server joins don’t push down; postgres_fdw only sends a join to the far side when both tables live on the same foreign server (the docs are explicit), so across two instances the join executes locally, pulling rows over the network first. Svix hit exactly this in October 2025: a join that failed to push down pulled both tables in their entirety, moved millions of rows where thousands were expected, and timed out.

So the FDW gives you back the SELECT and takes away the reason you split. Every cross-boundary query drags its working set back over the network to be joined next to the primary anyway.

An FDW join across instances is a distributed join, not a local one
If EXPLAIN (VERBOSE) shows the join in the local plan rather than in the Remote SQL, the far table is shipping over in full. Treat every cross-instance FDW join as a network transfer sized by the larger table, not by your WHERE clause.

Free to write, fake underneath

A MySQL “database” and a MySQL “schema” are the same object: a namespace on one server. There is no cross-database join inside one server, only a fully-qualified one, and the qualifier costs a few keystrokes. Postgres draws the line elsewhere. Schemas in one database join freely; separate databases can’t be joined at all without FDW or dblink, so a team that split into Postgres databases hits the wall on day one, where at least it’s visible.

The worse fact: MySQL foreign keys can cross schemas. REFERENCES billing.invoices(order_id) is legal DDL, and almost nobody writes it, because the schemas belong to different teams and nobody wants their writes gated on another team’s table. So the boundary those four hundred joins cross has no enforced referential integrity and never did. The failure modes are the ones you’d predict from dropping a foreign key, except no FK was ever declared: orphaned line items with no cascade to catch them, ID-type drift that turns the join into a silent cast (BIGINT on one side, CHAR(36) UUID after a migration on the other), backfills that assume the far side exists.

This is a same-server, cross-namespace problem
Cross-schema joins inside one Postgres database, or on a MySQL server you never intend to split, are not the anti-pattern. The coupling bites when scale forces the namespaces onto different instances. If that day never comes, the free join is genuinely free; the trap is assuming that’s your situation when the growth curve says otherwise.

Recovery has the same hole. Each database backs up independently, two --single-transaction dumps, each consistent, each catching a different instant, and the skew surfaces only when you restore one side. Roll billing back to 04:00 while orders keeps its afternoon of writes and every cross-schema join now compares two timelines. No constraint exists to fail, so nothing errors; the afternoon’s orders point at invoices the restore erased. (Postgres PITR replays WAL for the whole cluster, never one database, so an independent restore was never on offer.) The honest unit of backup and restore is the set of databases your queries treat as one, and four hundred joins say that set has two members.

Find the joins before you need the split

The coupling lives in query text, not the catalog: no pg_constraint row, no KEY_COLUMN_USAGE entry. But it always leaves a fingerprint, because at least one table in the query has to be namespace-qualified. On a running MySQL server, the digest table preserves the qualifiers:

1
2
3
4
5
SELECT SCHEMA_NAME, DIGEST_TEXT, COUNT_STAR, SUM_ROWS_EXAMINED
FROM performance_schema.events_statements_summary_by_digest
WHERE SCHEMA_NAME = 'orders'
  AND DIGEST_TEXT LIKE '%billing.%'
ORDER BY COUNT_STAR DESC;

Two caveats: DIGEST_TEXT truncates at performance_schema_max_digest_length (1024 bytes by default), and the digest table resets on restart, so it shows recent traffic, not history. Pair it with a grep of the repos (application code, ORM models, dbt sources) for the billing. qualifier. Datadog ran this inventory across 30-plus teams when they unwound a shared database in 2025; mapping the cross-boundary queries came before any carving. In Postgres the hunt is easier: the cross-instance dependency is already wearing a postgres_fdw label, listed in pg_foreign_table, with query text in pg_stat_statements.

What the fixes cost

The cheapest move is writing the coupling down where a machine can read it: COMMENT 'references billing.invoices.order_id, service-owned, no FK across instances' on the column. Readable by humans and by schema-reading assistants, enforced by nothing. If the boundary has to actually hold, a scheduled reconciliation job anti-joins the two sides and reports the orphans the missing FK would have rejected at write time (the scale-the-class fix applied to integrity). Eventually-consistent by construction, and someone has to own the alert. For the analytics joins, a consolidated replica (logical replication or CDC into one reporting instance) makes the join legal again, at the price of a pipeline to run and a lag window to reason about. For the transactional joins, the durable answer is the one Datadog and the AWS decomposition guidance both land on: replace the query with a service call. That’s the real unwind and the expensive one, a rewrite of every call site the join touched.

When the split itself arrives, replication makes it survivable, and the order of operations matters more than the tooling. Write-path joins and cross-schema transactions get rewritten first, while everything still shares a server; after cutover there is no atomicity across the boundary. Then the future cluster comes up as a replica of the whole instance, billing reads migrate at leisure, and writes cut over in a short freeze once GTIDs match. GitHub ran this playbook in 2021: SQL linters to stop new cross-boundary queries, then a write-cutover that moved 130 tables with sub-second downtime. Read joins stay gradual. A multi-source channel keeps feeding orders into the new cluster read-only, so stragglers join against a lagged copy while the digest counts drain to zero; give that channel a deadline, it’s the weld on life support. One trap: a read join that feeds a write decision (check invoice status, then update the order) is a cross-schema transaction wearing a SELECT, and the lagged copy turns it into a race. The digest table won’t flag it; only the call sites do.

One more consumer of the missing metadata: an AI assistant scoped to one connection (the default MCP posture) either declares the column unjoinable or, handed a shared server, cheerfully reproduces the free cross-schema join with no signal that it just crossed a service boundary. The column comment is the only place the relationship exists in a form the model reads.

The join that costs nothing to write is the one that costs the most to remove. Every schema.table qualifier typed without a second thought is a vote to keep both schemas on one server forever, cast by someone who had no idea there was an election.

SELECT insights FROM experience WHERE downtime = 0; -- Ruslan Tolkachev