An equipment-auction platform parses dealer feeds overnight; a nightly job writes each asset’s price. A refactor of that parser, agent-written and clean in review, declares asset IDs as int where the feed carries bigint. The largest dealer’s IDs sit past what 32 bits hold, so each one silently truncates to a smaller number, and that smaller number is some other asset’s ID. A $47,250 excavator’s price lands on a $3,100 flatbed trailer. Nothing objects: valid ID, plausible price, every row well-formed. The prices feed the valuation model, so every night’s mismatches flow into comparables, reserve suggestions, and settlement reports. It runs for forty days before a seller disputes a valuation against their own sale records.

The code fix is one line. The data is the problem: restore to when? The replica has the mispriced rows, and so does the delayed replica four hours behind it. Every nightly backup for forty days faithfully includes them. Binlog retention is fourteen days. And the moment before the corruption is also forty days of legitimate bids, sales, and settlements nobody will throw away. What actually worked was replaying all forty nightly runs and re-deriving everything downstream, and it was possible only because the raw feed was still there to replay.

Every layer in the stack was sized for a failure you notice

The standard disaster-recovery stack answers one failure class well: loss. A crashed instance, a dead disk, a dropped table, a region going dark. Detection is effectively instant, so a 30-day retention window feels generous, RPO and RTO become the numbers you argue about, and a hot replica reads as a safety net. Corruption breaks the assumption all of that was sized under, because nothing about it announces itself.

And week six is the normal case, not the pessimistic one. Plausible-but-wrong data trips no alert: right shape, valid types, clean plans. Corruption is silent by construction; it surfaces from outside, a customer disputing a number or a reconciliation refusing to tie out, on the timeline of whoever notices, not the one your retention policy was built around.

The reflex answer is point-in-time recovery: replay the binlog or WAL to the instant before the bad write and you’ve surgically removed it. Retention gets in the way first. The archive is sized in days, because it’s a storage line nobody wants to grow (MySQL’s binlog_expire_logs_seconds, a PostgreSQL WAL archive, both usually a few days to two weeks), and week-six corruption is a month past the binlogs that held the offending statement. And even with the archive intact, a point-in-time restore is a full rewind: it removes the bad write and every legitimate write after it. Corruption found three days in sits comfortably inside every retention window, and the restore still doesn’t happen, because three days of a busy system’s writes is more than anyone will throw away to fix one column. Retention decides whether PITR is possible; the writes that landed since decide whether it’s acceptable, and that window closes within hours.

The defenses stack, they don’t substitute

The layers look like an escalation ladder, and in cost they are: each rung up takes more to build and run. But they don’t stand in for each other: a production system taking writes it can’t afford to lose ends up needing all four, because each one recovers something the others structurally cannot. Managed databases ship the lower rungs as checkboxes, automated backups, PITR, vault-locked copies, and no provider ships the top one, because it’s made of your schema and your write path. That’s why every team has the first three and almost nobody has the fourth: the log, and the repair tooling around it, built before the incident that needs them. If the mantra is move fast and break things, this is the rung that covers the breaking.

Level 1: Backups that restore

The floor, and the rung every shop already has, which is why it gets mistaken for the whole ladder. A nightly dump answers “the disk died, restore last night,” and against corruption it does nothing: the bad write sits in last night’s backup and every one after it. The dump is also only half the mechanism: a backup that has never been restored is a hope, not a capability, and the distance between the two grows silently. Versions drift, runbooks go stale, the engineer who knew the procedure cold leaves.

GitLab’s January 31, 2017 outage is what that looks like. An engineer wiped the data directory on the primary instead of a secondary, and out of five backup and replication techniques deployed, none were working reliably or set up in the first place: pg_dump silently failing on a version mismatch, alert emails bouncing, Azure snapshots never enabled. They survived on an LVM snapshot a staging environment happened to take six hours earlier, and that was the easy failure class, instant detection, obvious cause, a clean state to restore to.

The shape that holds up is continuous automated restore-verification: on a schedule, restore last night’s backup to a scratch instance, check row counts and checksums, run an application-level smoke query, and alert loudly when any of it fails. That converts “the backup job exited 0” into “the backup restores,” which are different claims, and GitLab is the distance between them. It also turns restore time into a number you’ve measured on a quiet Tuesday: RTO you can quote, not RTO you hope for.

The backup itself is usually a chain, not one file: a weekly full, differentials or incrementals between them, and the binlog or WAL archive on top, which is what point-in-time recovery replays to reach an instant rather than last night. The chain is how RPO stays small without paying full-dump cost every night, and it raises the bar for the verification job: restoring the whole chain, full plus diffs plus log replay, because one broken link invalidates every restore point behind it. The log archive carries the retention limit from the section above: days, not weeks, never week six.

Level 2: The copy nobody can delete

Every other rung on this ladder answers to the credentials that run production, and the account that can write the database can usually delete its backups too. That is its own failure class: the deliberate one, where ransomware, a leaked key, or the same runaway agent that damaged the database also reaches for the recovery material. GitLab’s backups rotted by accident; this rung exists for the version done on purpose.

The mechanism is a retention lock: once a week, a full backup lands in an object-storage bucket under a policy (S3 Object Lock and its equivalents) that refuses deletion for at least 60 days, no matter whose credentials ask, including yours. It also quietly stretches the corruption horizon, because when the nightly rotation has long since expired, the locked weekly from before the bad write is often the only prior-value source left.

Level 3: Delayed replication

Regular replication doesn’t earn a rung of its own, even though every DR checklist lists it. A hot replica is the availability layer: the disk dies, you fail over, and the app is back before most users notice. It is not a replacement for backups, because DROP DATABASE replicates with the same fidelity as an INSERT, and the dropped database is gone on the replica within seconds of being gone on the primary. What earns replication a place on this ladder is the delay.

A standby held deliberately behind the primary is the window-bound rung. Run it four hours back (MySQL’s CHANGE REPLICATION SOURCE TO SOURCE_DELAY, PostgreSQL’s recovery_min_apply_delay) and you have a four-hour head start to stop a bad write before it applies downstream. Against a TRUNCATE on the wrong table at 2am, caught by the pager at 2:05, that is a genuine, cheap defense. Against corruption found at week six it does nothing; four hours against a six-week horizon rounds to zero.

Level 4: The event log

At the top rung the task changes shape. With a full rollback off the table, the job becomes “find every row this bad write touched, decide the correct value for each, and repair it in place while the system keeps taking writes.” Selective recovery depends on something none of the lower rungs had to provide: provenance.

Provenance answers “who wrote this row, from which job, deploy, or agent run, and when.” With it, the six-week-old corruption becomes a query: every row that actor wrote in that window is the blast radius. Without it you can’t scope the damage at all. You’re left diffing against a backup that also contains the corruption, or squinting at which of ten million rows “look wrong,” and for plausible-but-wrong data every one of them looks fine.

The event log that makes this tractable is one the application owns, covering the main entities. Every meaningful change writes an event in the same transaction as the change itself: which entity, what action, which actor, meaning the service, job, or agent identity rather than only a user, when, and a payload carrying the before and after values plus the request or job run that produced the write. And the log is immutable: events are inserted, never updated, so a later bad write can append wrong events but cannot rewrite the history already there. Because the application writes it, it records intent (this price came from that dealer feed, parsed by that run) rather than raw row deltas, it’s queryable in plain SQL, and its retention is your decision instead of a database setting quietly expiring your recovery material. The audit columns most schemas have, created_at and updated_at, say when a row changed and nothing about what changed it. The instinct is to widen them: add updated_by, then a reason column, then history bolted onto every hot table. Even fully decorated, a row holds only its last writer and no prior values, and every table pays the write cost separately. The entity table’s job is current state. Attribution and history belong in the event log, one place serving every table.

In table form, the basic version is small:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

CREATE TABLE events (
    id                 BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    entity_id          BIGINT UNSIGNED NOT NULL,
    action_id          SMALLINT UNSIGNED NOT NULL, -- 'price_updated' via lookup table, or an ENUM
    payload            JSON            NOT NULL,  -- before/after values, request id
    actor_id           SMALLINT UNSIGNED NOT NULL, -- service, job, or agent identity, via lookup or could be varchar
    created_at         TIMESTAMP       NOT NULL DEFAULT CURRENT_TIMESTAMP,
    KEY idx_entity (entity_id),
    KEY idx_actor_time (actor_id, created_at),
    FOREIGN KEY (action_id) REFERENCES actions (id),
    FOREIGN KEY (actor_id)  REFERENCES actors (id)
);

On PostgreSQL, JSON becomes JSONB and AUTO_INCREMENT becomes an identity column; nothing else changes. The split is deliberate: whatever the recovery queries filter on (entity, actor, time) stays a real column, and everything else rides in one JSON payload whose shape can vary per action without a migration. The action and the actor are IDs into small lookup tables, or enums where you prefer them, so the set of things that can happen to your data, and the set of writers that can make them happen, stays a constrained list instead of a schema hiding in free text. There is no updated_at, because nothing here is ever updated, and the cleanest way to keep that true is privileges: the application role gets INSERT and SELECT on this table and nothing else.

An event log in this shape also collapses a piece of infrastructure many teams already run. The transactional-outbox pattern exists to get events out of the database reliably: write an outbox row in the same transaction, let CDC (Debezium is the standard name) tail it into a retained topic, publish downstream. An entity event log written in-transaction is that outbox, one worth keeping instead of pruning. Tailed by the same connector, the same rows serve integration today and blast-radius queries at week six.

Agents change this calculus, just not where people look first. The repair mechanics stay the same: a fat-fingered UPDATE from a human at 4pm and an agent-generated one from a backfill are both scoped by provenance and repaired forward. What agents move is the two inputs that set the detection horizon, and they move both in the wrong direction. Volume goes up, because the cost of writing a migration collapsed while the cost of reviewing one didn’t, so more writes land with less scrutiny on each. Plausibility goes up too, because the model’s failure mode is confident, well-formed, semantically wrong output that reads as correct in review and against a clean EXPLAIN. Together they push detection past whatever your retention was sized for, which is why the event log shifts from good practice to a requirement the day agents join your write path.

The rollback tooling has to be ready before week six

Knowing which rows to fix is half the job. Generating the statements that fix them is the other half, and that tooling has to exist and be rehearsed in advance, because week six is not when you want to be writing your first replay script.

The event log gives that tooling two repair modes, and both are worth scripting ahead of time. Replay runs the events forward again through corrected code: the auction platform’s forty nights of re-parsed feeds were exactly this, re-deriving everything downstream from retained inputs. Inversion goes the other way: read the prior values out of the log for the blast-radius rows and emit reviewed UPDATEs that put them back. Neither needs a special tool: a query and a generator over a schema you own, built and drilled on a quiet week instead of improvised against the engine’s internals mid-incident.

When the log doesn’t reach back far enough, or the bad write predates it, the fallback is the scratch-instance pattern, and it works on any stack: PITR-restore a snapshot to a throwaway instance at the moment before the bad write, read the prior values for just the blast-radius rows, and apply the repair to production as ordinary reviewed UPDATEs while it keeps serving. It only works if snapshot retention still reaches past the bad write; the policy-locked weekly copy from Level 2 is usually what closes that gap, and the Level 1 restore rig is already most of the machinery it needs. Engine-side flashback exists but is narrow: MariaDB’s mysqlbinlog --flashback can invert ROW binlog events for DML; upstream MySQL and PostgreSQL ship nothing equivalent.

Written out, the recovery is a pipeline: a provenance query yields the blast radius as rows and their prior values, a generator turns that into inverse statements, a human reviews and applies them, and the system keeps taking writes throughout. The provenance query is not exotic. The auction platform’s blast radius starts as one WHERE clause:

1
2
3
4
5

SELECT e.entity_id, e.payload
FROM events e
JOIN actors a ON a.id = e.actor_id
WHERE a.name = 'feed-parser'
  AND e.created_at >= '2026-05-25';

That covers what the bad parser wrote directly. The valuations, comparables, and settlement reports computed from those prices don’t need the table to trace them, because the pipeline already knows its own order: valuations read prices, reports read valuations. The derived half of the repair is replay, re-running the downstream jobs on corrected inputs, which is exactly what the forty nights were. Retention is the precondition: the event log, CDC topic, or binlog archive has to reach back across the detection horizon, which for corruption means months. Months of history on a busy table is what time-based partitioning exists for: recent partitions on fast storage, older ones on something slower and cheaper instead of dropped to save space. The mechanics of partitioning you don’t have to babysit are their own subject.

Lay the four rungs against the two failure classes and the ladder collapses into one table.

No single row covers both columns at every horizon, which is why the layers read like an escalation but land as a stack. A replica copies every mistake the primary makes with perfect fidelity, so it protects availability and never data. Failover, never a good prior state. Backups invert that trade: they hold a good prior state, bought with the downtime and the writes-since-last-backup it costs to get there. The event log answers the one thing neither can: a backup or replica restore takes the database as a single unit, so neither can excise one bad transaction and keep the weeks of good ones interleaved around it.

How large each layer has to be isn’t a taste question. It falls out of two numbers the business either names or discovers mid-incident. RPO, the data it can afford to lose, sets backup frequency and how far back the archive reaches. RTO, the downtime it can afford while restoring, sets how often the restore is rehearsed and how much replica topology sits ready to fail over. Corruption adds a third number in the same family: how late the business can afford to find out, which sizes the delayed replica’s window and the event log’s retention. Availability commitments follow from those measured numbers. The common failure is the reverse order, a contract promising 99.95% over a DR posture that’s whatever happened to already be running. A team that has never put figures on its RPO and RTO has set both to whatever the current setup delivers, and reads off the real values during the incident that tests them.