Postgres, Kafka, Elasticsearch on EC2: Keep the Data on EBS, Not the Instance

Sat, 10 Oct 2026 00:00:00 +0000

TL;DR

For a self-managed storage system on EC2 (a relational database, a Kafka broker, an Elasticsearch node, a Redis replica), moving EBS volumes is far faster and cheaper than draining a node and refilling its replacement over the network: a resize or a replacement reattaches the data, and an OS upgrade swaps only the root volume while the data volumes stay attached. The price is a network hop on every fsync, one EBS ceiling per instance that every volume shares, and snapshot hydration inside the DR plan’s RTO.

This is the first of two posts on running self-managed stateful systems on EC2 (PostgreSQL and MySQL, Kafka, Elasticsearch and OpenSearch, Redis with persistence on) with storage and compute separated. This one covers why; Part 2 covers how the DR side gets built on AWS.

The two storage options behave very differently. Instance-store NVMe is physically attached to the host, fast, and ephemeral: per the AWS docs, its data does not survive a stop, a hibernation, a host retirement, or an automatic recovery, and the blocks are cryptographically erased. EBS is network-attached block storage with its own lifecycle. A volume outlives the instance it’s attached to, detaches and reattaches, snapshots incrementally to S3, and changes size and performance while mounted. When the data is on the instance, every instance event becomes a data event. When it’s on EBS, most of them become a reattach.

Why not the managed service

The managed services are the obvious alternative: RDS or Aurora, Amazon MSK, Amazon OpenSearch Service, ElastiCache. Aurora is the extreme form of this post’s argument (the SIGMOD 2017 paper pushes redo processing into a storage service so compute holds almost no state), and for plenty of teams it’s the correct call. The reasons to stay on EC2 are concrete. I/O-heavy workloads cost far more on a managed service, and AWS’s own pricing admits it: Aurora I/O-Optimized (May 2023) drops per-I/O charges and is pitched at clusters where I/O exceeds 25 percent of the Aurora bill. A multi-cloud estate, or a team that won’t be locked to one vendor, keeps the engine, its tooling, and the runbooks portable; EBS is AWS-specific, but the same split works on GCP Persistent Disk or Azure Managed Disks. Then there are builds the service doesn’t offer (Percona Server, a patched PostgreSQL, Elasticsearch past 7.10, the last version Amazon OpenSearch Service runs), plugins outside the allow-list, kernel tuning, and constraints a managed service can’t accommodate. Without one of those, the managed service usually wins.

Reattach the volume instead of draining the node

The alternative to moving a volume is draining a node, and every system has its own version. Elasticsearch excludes the node from allocation and relocates its shards to the rest of the cluster. Kafka reassigns the broker’s partitions with a plan the operations docs say has to be written by hand for a decommission, usually under a replication throttle (the docs’ example is 50 MB/s between brokers) so the move doesn’t starve producers. Zalando’s Kafka team put the cost at around 6–7 hours of rebalancing for one broker lost with its disk. A relational database builds a new replica and fails over to it. Redis does a full sync, snapshotting the whole dataset on the primary and shipping it. Every one of them copies the node’s whole dataset across the network while the cluster carries the extra load, and a rolling replacement of Elasticsearch or Kafka nodes on local disk pays it twice per node: once to drain the old node, once to fill the new one.

Moving the volume copies nothing. Stop the process cleanly, detach, attach to the new instance, start. PostgreSQL and MySQL come up without recovery after a clean shutdown, and an unclean stop costs only a log replay.

OS patching doesn’t even need a new instance once the OS lives on its own volume. EC2’s root volume replacement swaps the root volume of a running instance for one built from a new AMI and reboots it. Per the docs, the data EBS volumes stay attached, instance-store data survives (temp directories included), and the instance stays on the same host with the same network interfaces, IP addresses, and DNS name. To the rest of the cluster the node was briefly away and came back on the same address with the same data. Two conditions apply. The data directory must not sit on the root volume, which the replacement detaches. And the AMI must match the instance’s architecture, so a move to Graviton is still a new instance and a volume move.

Resizing is the same move: an r7i.4xlarge that ran out of memory becomes an r7i.8xlarge and the data stays put. Compute gets sized for CPU and memory, never for disk. For Redis, memory includes persistence headroom: the admin guide warns that a write-heavy instance can use up to twice its normal memory while the forked child writes an RDB or rewrites the AOF.

A Kafka broker that comes back on its old volumes, under the same broker identity, holds every segment it had and fetches only what was produced while it was gone. Zalando’s result was a lost broker recovered “in a number of minutes without copying data,” and an upgrade of 15 brokers in two hours, 42 times faster than before. An Elasticsearch node that returns with its data inside the delayed-allocation window gets its shards back “with the minimum of network traffic,” per the delayed allocation docs. The default is one minute, shorter than most instance launches, so a planned swap raises it first.

Redis is the odd one: a replica shut down cleanly keeps its replication offset in its snapshot file and can resume where it left off, while one restarting from its append-only log reloads from the volume and takes a full sync anyway.

Storage changes are online. Elastic Volumes grows size, IOPS, and throughput on a volume while it serves traffic. Nothing shrinks, so oversizing is a one-way door.

Engine upgrades are the same root swap, because the binaries live on the root volume and the data doesn’t. Bake the new PostgreSQL, MySQL, Kafka, Elasticsearch, or Redis version into the AMI, replace the root volume, and the process starts on the new version against the same data directory. For a minor version that’s the whole job. A major version also converts the data on disk, which is where the rollback below comes in. The one move that does need a copy is crossing Availability Zones, since an EBS volume lives in one: snapshot it, create a volume from the snapshot in the target AZ, attach.

Major versions get the rollback neither relational engine provides. The pg_upgrade docs say that in link mode, once the new cluster has started, the old one is unsafe and must be restored from backup. MySQL 8.0 to 8.4 upgrades the data dictionary in place on first start, and the downgrade docs allow going back only by logical dump and load or replication. Root volume replacement can keep the old root volume instead of deleting it, so the rollback is swapping that volume back in and restoring the data volume from a snapshot taken after a clean shutdown. Upgrade a pre-warmed copy of the data volume, not the original, or the rollback waits on hydration.

Each of these moves has one trap on the PostgreSQL side.

Warning

Attaching a PostgreSQL data volume to an instance with a newer C library can silently corrupt every text index on it. PostgreSQL collates through glibc by default, and glibc 2.28 rewrote the sort order for most locales; per the PostgreSQL wiki, B-tree indexes on text, varchar, char, and citext built under the old rules must be reindexed before the instance takes production traffic. Amazon Linux 2 ships glibc 2.26 and AL2023 ships 2.34, and AL2 reached end of support on June 30, 2026, so “new AMI, reattach the volume” this year crosses that line. The symptom is a UNIQUE index admitting duplicates and lookups missing rows that exist. Pin the OS major version of database AMIs or put the REINDEX in the cut-over plan, and consider ICU collations, which carry a version PostgreSQL can check. MySQL ships its own collations and doesn’t inherit this from the OS, though its collations do move between MySQL versions, a different trap.

Split the I/O, then hit the instance ceiling

Once the data is on volumes, nothing says it has to be on one. Separate volumes give each write stream its own IOPS budget and its own queue.

The relational engines have the finest controls. PostgreSQL can put WAL on one volume, the data directory on another, and give a hot table, a large index, or the current partitions of a time-series table a tablespace of their own. MySQL goes further: binlog, redo, undo, and doublewrite can each sit on their own volume, and individual tables or partitions can live outside the data directory. The win is isolation as much as aggregate. A commit waits on a log fsync, and on its own volume that fsync doesn’t queue behind a checkpoint flushing gigabytes of dirty pages. A durable MySQL commit syncs both redo and binlog, and separate volumes keep the two from contending.

Kafka and Elasticsearch moved in opposite directions on the same idea. Kafka can spread a broker’s log across several directories; per the hardware docs, partitions are assigned round-robin across directories and each sits entirely in one, so an uneven partition mix means uneven disks. JBOD under KRaft left early access in Kafka 3.8 (July 2024). Elasticsearch deprecated multiple data paths in 7.13 and recommends one filesystem spanning several disks or one node per data path, which on EC2 means one data volume per node and more nodes.

The split also enables tiering. Cold partitions go on baseline gp3, and provisioned io2 pays only for hot data. PostgreSQL can move a partition to another tablespace, but the move rewrites it under an exclusive lock, so it suits partitions that no longer take writes. MySQL decides placement when the partition rotation creates the partition; moving it later means a stopped server. Kafka and OpenSearch take the tier off block storage entirely. Kafka’s tiered storage, production-ready since 3.9 (November 2024), copies completed log segments to S3 and keeps only the tail on the broker, though not for compacted topics. OpenSearch’s remote-backed storage (2.10) does the same for segments and translog.

Temp needs no durability, which makes it the natural tenant of local NVMe on instance types that have it. Sort spills, hash joins, and the scratch files of online schema changes run at local-disk speed and stay off the EBS budget, and both PostgreSQL and MySQL can point their temporary files at the instance store. The PostgreSQL docs warn that a tablespace on temporary storage risks the cluster. One holding only temp files is the survivable exception, provided its directory is recreated on every boot, because instance store comes back empty after any stop.

Note

The engines fail in opposite directions when that boot step is missing, tested on PostgreSQL 17 and MySQL 8.4 with the temp directory wiped between restarts. MySQL refuses to start, which is loud. PostgreSQL starts normally, fails to create temporary tables, and quietly writes sort spills to the data volume instead, the exact I/O the split was meant to keep off it. Nothing in the startup log says so.

Then the ceiling. Every volume on an instance shares that instance’s EBS bandwidth, and it sits lower than the volumes’ combined limits suggest. Since September 2025 one gp3 volume provisions up to 80,000 IOPS and 2,000 MiB/s, and one io2 Block Express volume 256,000 IOPS and 4,000 MiB/s. Per the instance tables, an r7i.8xlarge gets 40,000 IOPS and 1,250 MB/s in total: one maxed gp3 already exceeds the instance, and a second volume buys isolation, not headroom. Only from r7i.24xlarge (120,000 IOPS, 3,750 MB/s) do several volumes add up to more than one. Sizes through 4xlarge advertise a burst they hold for 30 minutes a day before dropping to baseline, the same trap as gp2 burst balance. Splitting scales I/O and nothing else. A hot row, a replica applier that can’t parallelize the primary’s writes, PostgreSQL waiting on its WAL write lock, a hot Kafka partition with one leader: none improve with a third volume, and the headroom before sharding or adding nodes ends at a number in the instance-type table.

Latency is the other price, and how much of it a system pays depends on whether it waits for the disk. AWS describes gp3 as single-digit millisecond and io2 Block Express as under 500 microseconds on average for a 16 KiB I/O; local NVMe has no network hop in the path. The relational engines wait: every commit pays a WAL or redo fsync. Group commit amortizes it across concurrent transactions, so most OLTP never notices, but small serial commits will see their p99 move. Kafka mostly doesn’t wait. Its docs recommend the default flush settings, which “disable application fsync entirely,” because “a failed node will always recover from its replicas,” so EBS latency stays out of the produce path and the volume’s job is throughput. Redis sits between them: its default once-a-second fsync runs in the background, but per the latency guide a slow one eventually blocks the main thread. Where p99 is set by fsync time and nothing can hide it, a primary on local NVMe with synchronous replicas on NVMe is the better design, and the re-seed window is the price of the latency.

The DR plan is a snapshot schedule and a hydration rate

EBS snapshots are incremental and copy across regions, which supports two shapes of standby in the second region. Cold standby keeps only snapshots; failover creates volumes, launches instances, and attaches. Warm standby keeps volumes already restored from the latest snapshot, with no compute running. Both cost a fraction of a running replica, and both depend on things the cost estimate doesn’t show, starting with whether the snapshot is a usable copy at all.

A snapshot is only a backup if it’s consistent, and splitting the log from the data breaks that by default. Snapshot the volumes one at a time and each captures a different instant. The PostgreSQL docs require simultaneous snapshots when data files and WAL sit on different disks, and on MySQL a binlog captured ahead of the redo log produces replicas that diverge from their first minute. EBS can snapshot several volumes at the same instant, which gives crash consistency: what a power cut would leave, and what both engines recover from by design. Knowing where that instant sits in the log stream, so a restore can replay forward from it, takes the engine’s own backup hooks around the snapshot.

Some systems don’t accept a volume snapshot as a backup at all. Elasticsearch’s snapshot docs call its snapshot API “the only reliable and supported way to back up a cluster” and rule out restoring from filesystem-level copies. Kafka’s cross-region answer is cluster-to-cluster mirroring with MirrorMaker 2, since one broker’s volume holds only that broker’s replicas. Redis treats its RDB file as the DR artifact. For these, EBS carries a node through instance events and the system’s own mechanism carries DR. Redis’s replication docs also hold the sharpest version of the instance-store risk: a primary that restarts empty is still a primary, and replicas that sync from it “will effectively destroy their copy of the data.”

Warm standby isn’t warm, either. A volume restored from a snapshot is usable at once, but its blocks stay in S3 until first read, and every page arrives with the enthusiasm of a shared drive on hotel Wi-Fi. RTO includes that hydration. It gets paid in time, by reading every block before cutover, or in money, since AWS sells faster initialization. And it starts over on every refresh, so a standby is warm only if something reads it after each one.

The standby doesn’t have to be provisioned for production while it waits. Kept at baseline performance it costs little beyond its gigabytes, and failover attaches compute and raises the volumes to production performance online. The raise isn’t instant: the modification docs put performance changes at “a few minutes to a few hours,” longer on a volume that was never fully hydrated. Through the change the volume serves at no less than its old performance. That fits an RTO of hours. An RTO of minutes means paying for production performance all the time.

Recovery point is the other number. Snapshots alone lose everything written since the last one, plus the copy time. Shipping the log continuously to the DR region (WAL archiving on PostgreSQL, binlog streaming on MySQL) shrinks that to the shipping lag, and a restore becomes snapshot plus replay. None of it helps with a write that was wrong when it landed, which every snapshot preserves faithfully; that needs a different layer.

Separation covers instance failure, host failure, and planned work, and not the storage layer. In the April 2011 us-east-1 event, a misrouted network change set off a re-mirroring storm inside EBS that left about 13 percent of volumes in one AZ stuck for days, beyond reach of any instance replacement. Replacement also depends on the EC2 control plane: in the October 2025 us-east-1 outage, new launches failed from 2:25 to 10:36 AM PDT while running instances kept running. A recovery that starts with a launch is only as available as the launch API. A replica already running elsewhere removes that dependency, but it’s compute paid for around the clock, and if the DR side serves no reads it does nothing until the day it’s needed, which is the cost a volume-based standby exists to avoid.

Part 2 turns the DR half into an implementation outline: which AWS services schedule, copy, and restore the snapshots, how log shipping or the system’s own backup API fits alongside them, the hydration options and what they cost, and how a restore gets drilled against a stopwatch.