The checkpoint-age line on the InnoDB dashboard has been climbing for two hours. It opened the evening at a comfortable third of the redo log’s capacity and it is now past three-quarters, closing on the point where InnoDB stops being polite about flushing. A batch reconciliation that normally runs Sunday afternoon got kicked off by a failed deploy’s retry logic, and it is rewriting a wide table row by row. Nobody is looking at the graph. When checkpoint age crosses log_max_modified_age_sync, InnoDB flips to synchronous flushing, write latency goes vertical, and the job that generated the pressure is now blocked behind the flushing it caused. The page fires a few minutes later, on the symptom, connections piling up against a server that is already stalling.

The remediation here is boring and it has existed since MySQL 8.0.30, when innodb_redo_log_capacity became dynamic. A single SET GLOBAL innodb_redo_log_capacity = ... resizes the redo log online, no restart, and you can watch the resize settle through Innodb_redo_log_resize_status. A larger redo log raises the sync threshold, so the same checkpoint age is now a smaller fraction of capacity, and the burst gets room to drain before InnoDB ever reaches for the aggressive flush. Done in the first ten minutes of that climb, the whole incident is a non-event nobody wakes up for. The signal you would trigger on, checkpoint age against the sync point, is the same signal a health check would sample. Which is the part worth sitting with: if you run Consul, or Nomad, or a Kubernetes cluster with liveness probes that actually mean something, you already own every piece needed to make that SET GLOBAL fire on its own, deterministically, before the page.

The 2026 move is to give it a brain

Given the vendor cycle we are all living through, the reflex is that this is the poster use case for an autonomous ops agent. Wire it to the metrics, give it a query(sql) tool and a shell, let it reason about the pressure and choose a remediation. And to be fair to the model, a decent one will surface SET GLOBAL innodb_redo_log_capacity among its suggestions without much prompting. Generation was never the missing piece.

The problem is that this class of page has exactly one correct action and no judgment in it. Wrapping one deterministic action inside a component that samples from a distribution over possible actions buys you nothing and adds a floor of fabricated actions underneath. The agent that resizes the redo log today can, on a similar-looking graph next week, decide the real fix is to kill the connection that “looks like” the offender, or reach for a config it half-remembers from a different engine version. An agent earns its keep when the next step is genuinely ambiguous and the value is in the reasoning. Here the reasoning is a threshold comparison and a lookup. A watch handler that runs one vetted script does the identical job with none of the confident-misattribution surface an LLM brings to a decision it was never needed for. The reliability question is not whether the model can find the fix. It is where the floor on wrong actions sits, and for a one-lever page the floor of a deterministic handler is zero.

What the watch and the handler actually do

Consul’s health checks are the wiring. You register a script check against the database service: a small query that runs every N seconds and returns critical when checkpoint age passes your threshold. That query hits information_schema.innodb_metrics for log_lsn_checkpoint_age (the counter is off by default, so it needs innodb_monitor_enable), or it parses the LOG section of SHOW ENGINE INNODB STATUS. When the check flips to critical, a watch of type check fires. The watch invokes a handler, and Consul writes the check’s current state to that handler as JSON on stdin: node, check ID, status, the check’s own output. The handler is an ordinary executable. It reads the JSON, confirms the condition still holds, runs the one thing it exists to run, and records that it did.

For the redo log that handler is close to trivial. Parse stdin, see the checkpoint-age check critical, issue the SET GLOBAL, poll Innodb_redo_log_resize_status until the resize completes. The grow is a bounded step, 10 or 20 percent, never a doubling, and it happens only after the handler checks the volume it is about to grow into: redo log files sit on the same disk as the data files and binlogs, and growing the log into a nearly full filesystem trades a flushing stall for a full-disk stop, which is strictly worse. Then come the two steps that keep the automation honest, and each is one curl. A POST to Jira’s REST API creates a ticket in the team’s morning queue: write burst pushed checkpoint age to the sync point at this timestamp, capacity grown from X to Y, the offending statement digest attached if the check captured one. A second webhook drops a low-urgency line in the team channel, so the responder learns about it over coffee instead of from the graph. The remediation bought the night. The ticket is the handoff that decides, in daylight, whether this was a runaway job to fix or a capacity trend to plan for.

There is a shortcut worth checking before wiring any of that: if your monitoring already alerts on configuration drift, running values diverging from what my.cnf or the parameter group declares, the resize surfaces there on its own. SET GLOBAL changes the runtime value and leaves the configured one untouched, so the next drift report carries the change whether or not anyone filed anything, and reconciling it (persist the new capacity or revert it) is exactly the morning decision the ticket existed to force. A drift alert with an owner can replace the Jira call. What it cannot carry is the trigger, so the handler still logs why it fired.

Disk filling from binary logs is the same shape with a sharper edge. A disk-free check goes critical, and the naive handler purges the oldest binlogs to claw back space. The safe handler does not purge blind, and it does not need a topology registry to see the cluster, because replication already knows its own shape. A script on the primary reads SHOW REPLICAS for the registered replicas, connects to each one for the binlog file it still needs (the Source_Log_File in SHOW REPLICA STATUS, formerly Master_Log_File), and recurses if a replica has replicas of its own. Only then does it run PURGE BINARY LOGS TO the lowest file anything still needs, never a file at or beyond it. Never rm, never find -delete against the binlog directory; that path corrupts the index and takes the server’s own bookkeeping out from under it. PURGE is the only handle that keeps binlog.index honest.

The third example is the strangest-looking page of the set: the abandoned transaction. A deploy tears down an application pod mid-transaction, the oom-killer takes a batch worker, someone Ctrl-Cs a migration script on a bastion. The client process is gone, but the server-side connection survives, holding a transaction open with nothing left to execute. InnoDB’s purge threads cannot reclaim undo history past the read view of the oldest open transaction, so the history list length starts climbing, version chains grow, reads on hot rows walk more and more history, and the undo tablespaces swell. The slow query log stays empty the whole time, because the transaction is not running anything. The signature is exact, though: a row in information_schema.innodb_trx whose trx_started is older than anything legitimate in the workload, attached to a session whose current statement is NULL (sys.session shows this directly; in the processlist it is a connection sleeping mid-transaction). A check on trx_rsseg_history_len in innodb_metrics goes critical, the handler runs that lookup, and the remediation is KILL on the thread. Waiting does not substitute: wait_timeout will reap the connection eventually, but its default is eight hours, and the history the transaction pins keeps costing you the entire time. The guard worth building in is trx_rows_modified. Near zero, the kill is free. Large, and the rollback the kill triggers is its own I/O event, at which point the handler’s right move is to page a human instead.

The same shape without a mesh

Consul is the concrete example, not the requirement. The pattern is a watcher on a pressure signal bound to a handler that runs one action, and every serious stack has it. Nomad and Kubernetes give you liveness checks and operators. Sensu-style monitoring has handlers as a first-class concept. A systemd timer running a guard script is the degenerate case that still counts.

In AWS the wiring is a CloudWatch alarm on FreeStorageSpace, replica lag, or freeable memory, routed through EventBridge to a Lambda. The Lambda connects and runs a scoped remediation: kill sessions idle-in-transaction past a hard age, trigger a managed failover, or grow the instance. RDS storage autoscaling is the fully managed instance of exactly the redo-log move from the opener, a watcher on free storage that grows the volume within a ceiling you set, no page, someone else owning the handler. The mechanism is identical; the only thing that changed is who wrote the script.

Postgres has its own version worth wiring. An abandoned replication slot retains WAL indefinitely and overrides every other retention setting until the disk is gone. A check on pg_replication_slots for a slot that is active = false while its retained WAL climbs is a clean critical signal, and the remediation is pg_drop_replication_slot() once the slot is confirmed dead. Since Postgres 13 there is a built-in preventive form, max_slot_wal_keep_size, which invalidates a slot that exceeds the cap rather than letting it fill the primary.

The line you don’t cross

The set that is safe to automate shares a property: the correct action does not depend on judgment about which side of a fault is healthy. Growing a redo log or a data volume that resizes online. Purging binlogs on a target chosen from live replica positions. Advancing or dropping a slot confirmed abandoned. Killing sessions that match a precise runaway signature, not sessions that merely look busy. The set to keep human is the mirror image. Deciding which node survives a partition is a failover call, and a handler that automates it will happily promote the wrong side of a split brain. Dropping data, resharding, anything where the right move depends on believing one replica over another, stays a person’s call. This is the same seam the blog keeps returning to: deterministic narrow tools beat autonomous ones precisely where a wrong action is expensive and the right one is unambiguous, and a prompt is not a guardrail around that seam.

Three failure modes come with the pattern, and pretending they don’t is how the pattern turns into a slow-motion outage.

The first is masking. A handler that resizes the redo log every night, or purges binlogs every morning, converts a capacity-planning signal into silence. The write volume that keeps tripping the check is telling you something, and a handler that quietly absorbs it every time makes sure nobody hears it. This is why the evidence step is load-bearing rather than nice-to-have. Every firing leaves a ticket in the morning triage queue, a metric increment, a log line with the before-and-after, so a human reviews the trend even though no human handled the night. Three redo-log resizes in a month is a capacity conversation; the tickets are what make it visible as one. A remediation that fires and leaves no trace is the outage you get to have twice.

The second is flapping. A check that oscillates across its threshold will invoke the handler on every crossing, and a handler that is not idempotent (or is expensive) turns that into a storm. The handlers that hold up are idempotent by construction, rate-limited with a cooldown, and wrapped in a circuit breaker that stops after N firings in an hour and pages a human instead. A remediation that has fired six times in an hour is not remediating anything; it is annotating a problem that needs a person.

The third is the credential surface. A handler that holds SUPER or REPLICATION_ADMIN and fires on a threshold is a standing operational risk, an account that can resize, purge, and kill, sitting next to a trigger. Scope it to the single grant its one operation needs, and keep the redo-log handler’s credentials disjoint from the binlog-purge handler’s. The blast radius of a compromised or buggy handler should be exactly the one lever it was built to pull.

None of this needs a model. The redo-log handler is a page of bash and a SET GLOBAL you already know by heart. It will not reason about your incident, and for this class of page that is the entire point: it pulls one boring lever the same way every time, and it files the ticket so you find out on Monday that Saturday almost happened.