Database Fundamentals on EXPLAIN ANALYZE

Hash Join vs Merge Join vs Nested Loop: When Each One Wins

Tue, 10 Nov 2026 00:00:00 +0000

TL;DR

A join algorithm is the procedure a database uses to match the rows of one table with the rows of another, and PostgreSQL has three: nested loop, hash join and merge join. This post covers what each one needs, how to recognise each in a query plan (the database’s printout of how it runs a query), where the planner, the part that picks the plan, switches from one to the next, and what makes it switch at the wrong point; MySQL, which has two of the three, gets its own section. Every number comes from runs on PostgreSQL 18.6 and MySQL 8.4.13.

	Nested loop	Hash join	Merge join
How it works	For each row of the first input, look up matching rows in the second	Load one input into a hash table in memory, then check every row of the other against it	Walk two inputs sorted on the join column side by side
What it needs	An index on the second table’s join column	Memory for the hash table (`work_mem` in PostgreSQL, `join_buffer_size` in MySQL)	Both inputs in join-column order, from an index or a sort
Fastest here	Under about 5,000 customers: 68 ms against 381 ms for hash at 1,000, with merge join tied because it walks the same index	Past about 5,000: 360 ms against 1,219 ms for nested loop at 20,000	Two big tables in key order: 1,084 ms against 1,621 ms hash and 3,029 ms nested loop
How it fails	Cost grows with every outer row: 13.5 s at 200,000 customers, against 0.86 s for hash	Starts by reading the whole second table, so it took 268 ms even for 10 customers	Without an index to supply the order, a sort comes first
PostgreSQL 18	Chosen by cost	Chosen by cost	Chosen by cost
MySQL 8.4	Chosen whenever an index fits the join	Used only when no index fits the join	Does not exist

All of it is measured below except the sort in merge join’s failure row, which is documented behavior these tests never triggered.

So here is how a join works, in outline. Looking up each row of one table in the other is the natural first answer, and with an index on the second table’s join column (a sorted copy of that column with pointers back to the rows) it is a good one, because each lookup goes straight to the few rows that match. But every lookup is a separate trip, into the index and then out to wherever those rows sit in the table, and a join that needs a large share of the table makes one such trip per row. Past some point it is cheaper to read the whole table once, front to back, and match rows as they go by. Databases do that in two ways: hold one side in a lookup table in memory and stream the other side past it, or read both sides in the same sorted order and pair them off. Counting the lookup, that makes three algorithms, called nested loop, hash join and merge join, and before each query PostgreSQL’s planner, the part that picks how a query runs, estimates the rows each step will produce, prices the options and runs the cheapest.

A quick reminder about query plans

The planner’s choice is printed in the query plan, the tree of steps the database runs to answer a query. EXPLAIN shows the plan, and EXPLAIN ANALYZE runs the query and adds what actually happened. Each step is a node, indented under the node that uses its output, and each carries two estimates: rows, how many rows the planner expects it to produce, and cost, a number in arbitrary units that only means something next to the cost of another plan. A join is a node with two children, the two inputs it matches. The habit this post leans on is reading the estimated rows against the actual ones, because the planner chose its join from the estimate; the how-tos for PostgreSQL and MySQL cover the rest of the output (MySQL calls its planner the optimizer).

Three ways to join two tables

Each of the three needs something different from its inputs, and what it needs decides how its cost grows as the inputs grow. That growth is what the planner is really comparing when it prices them.

The tests ran on PostgreSQL 18.6 and MySQL 8.4.13 in Docker with default settings, each container limited to 4 CPUs and 4 GB of memory, on the same tables as the two EXPLAIN posts: 200,000 customers and 5,000,000 orders, each order belonging to a random customer, with an index on orders.customer_id. A third table, shipments, holds one row for each of the 4,950,073 shipped orders, keyed by order id and written in order id order. PostgreSQL can split one join across several processes, and it can compile parts of a query at run time (JIT); I turned both off, as in the EXPLAIN post, so parallel joins are simplified away here. Every join is also an inner join on one equality, such as o.customer_id = c.id; outer joins and other conditions change which of the three are allowed, which this post leaves aside. Timings are medians of five runs (three for the biggest joins) with the tables already read once. Run-to-run noise between sessions was up to about 20 percent, so smaller differences mean nothing.

The query for most of this post totals the orders of the first N customers:

1
2
3
4


SELECT count(*), sum(o.total)
FROM customers c
JOIN orders o ON o.customer_id = c.id
WHERE c.id <= 1000;

Nested loop

A nested loop is the lookup from the outline, and what it needs is an index on the second input’s join column, because it searches that input once for every row of the first. Without one, each search would have to read the whole orders table. With 1,000 customers PostgreSQL picks it (plan trimmed to the join and its two inputs):

1
2
3
4
5
6
7


 Nested Loop (cost=0.85..93322.60 rows=24625 width=6) (actual time=0.010..54.458 rows=24899.00 loops=1)
 Buffers: shared hit=8343 read=19558
 -> Index Only Scan using customers_pkey on customers c (cost=0.42..29.66 rows=985 width=4) (actual time=0.004..0.068 rows=1000.00 loops=1)
 Index Cond: (id <= 1000)
 -> Index Scan using orders_customer_id_idx on orders o (cost=0.43..94.45 rows=26 width=10) (actual time=0.003..0.053 rows=24.90 loops=1000)
 Index Cond: (customer_id = c.id)
 Execution Time: 55.356 ms

The first child is called the outer side, and here it produces 1,000 customers. The second child, the inner side, runs once per customer (loops=1000), and each run uses the index to find that customer’s 25 or so orders. So the work grows with the outer side: twice the customers, twice the index lookups. Buffers counts pages, the 8 KB blocks PostgreSQL stores a table in, and this join touched 27,901 of them, because each customer’s orders are scattered across the table and almost every order lives on a different page.

Hash join

A hash join needs no index at all. What it needs is memory, for a hash table: a lookup structure, keyed on the join column, that it builds from one input before reading the other. Raise the limit to 20,000 customers and the plan changes to one:

1
2
3
4
5
6
7


 Hash Join (cost=804.18..145898.06 rows=489805 width=6) (actual time=1.398..387.738 rows=500284.00 loops=1)
 Hash Cond: (o.customer_id = c.id)
 -> Seq Scan on orders o (cost=0.00..131968.48 rows=5000048 width=10) (actual time=0.066..179.694 rows=5000000.00 loops=1)
 -> Hash (cost=559.28..559.28 rows=19592 width=4) (actual time=1.328..1.328 rows=20000.00 loops=1)
 Buckets: 32768 Batches: 1 Memory Usage: 960kB
 -> Index Only Scan using customers_pkey on customers c (cost=0.42..559.28 rows=19592 width=4) (actual time=0.005..0.658 rows=20000.00 loops=1)
 Execution Time: 407.853 ms

Read it from the bottom. The Hash node collects the 20,000 customer ids into the hash table, which took 960 kB and a millisecond. Then Seq Scan on orders reads all 5,000,000 orders once, and for each one the join checks the hash table for its customer_id. Nothing runs more than once (loops=1 everywhere), and the index on orders.customer_id is not used at all. Most of the cost is that one scan of the orders table, and it is the same whether the hash table holds 20 customers or 20,000: with only 10 customers a forced hash join still took 268 ms. The memory limit comes from work_mem in PostgreSQL and join_buffer_size in MySQL, and Batches: 1 means the hash table fit inside it. That line comes back below.

Merge join

A merge join needs both inputs sorted on the join column, either read from an index in that order or sorted first, and in return it reads each input once. The customers-and-orders query cannot get its orders in that order cheaply, as the next section shows. The shipments join can, because both it and orders have primary keys on the order id:

1
2
3


SELECT count(*), sum(s.weight_g)
FROM orders o
JOIN shipments s ON s.order_id = o.id;

1
2
3
4
5


 Merge Join (cost=12.71..366817.13 rows=4950090 width=4) (actual time=0.017..967.736 rows=4950073.00 loops=1)
 Merge Cond: (o.id = s.order_id)
 -> Index Only Scan using orders_pkey on orders o (cost=0.43..129849.15 rows=5000048 width=8) (actual time=0.004..188.127 rows=5000000.00 loops=1)
 -> Index Scan using shipments_pkey on shipments s (cost=0.43..162601.78 rows=4950090 width=12) (actual time=0.011..337.965 rows=4950073.00 loops=1)
 Execution Time: 1083.770 ms

Both inputs come out of their primary key indexes already in order id order. The join reads one row from each side. If the ids match, that is a result row; if not, it moves the side with the smaller id forward. Each table is read once, front to back, with no hash table and no sort, and the Merge Cond line gives it away.

This is the case merge join exists for: two big tables, every row needed, both already in key order. Forcing the other two algorithms onto the same join shows why the planner picked it:

The hash join has to load 4,950,073 shipments into its hash table before it can match anything, and at the default work_mem that table did not fit, so PostgreSQL split it into 64 batches written to temporary files (Batches: 64 under the Hash node). The nested loop is less awful than its 4,950,073 lookups suggest, because it scans shipments in order id order, so consecutive lookups into the orders primary key land on the same index pages.

Where the choice goes wrong, and two fixes that backfire

The planner prices the three from estimates, and on the customers-and-orders query the break-even point it worked from was about a third of the one measured. To find the real one, I ran that query at 17 sizes, from 10 customers to all 200,000, and forced each algorithm in turn with enable_hashjoin, enable_mergejoin and enable_nestloop set to off for the session.

Customers	Orders joined	Nested loop	Hash join	Merge join	Planner’s pick
100	2,446	0.6 ms	288 ms	0.6 ms	nested loop
1,000	24,899	68 ms	381 ms	58 ms	nested loop
2,000	50,076	126 ms	312 ms	118 ms	hash join
5,000	125,075	311 ms	306 ms	297 ms	hash join
10,000	250,098	589 ms	323 ms	604 ms	hash join
20,000	500,284	1,219 ms	360 ms	1,451 ms	hash join
50,000	1,250,570	3,014 ms	450 ms	3,083 ms	hash join
200,000	5,000,000	13,451 ms	863 ms	12,511 ms	hash join

The two cost shapes from the plans explain the whole table. Nested loop pays per customer, about 0.06 ms for each one’s index lookup and page reads, so its line is straight. Hash join pays mostly for one scan of the orders table, so its line is flat. They cross near 5,000 customers, which is 125,000 orders, 2.5 percent of the table. At 200,000 customers the nested loop does 200,000 lookups to fetch every row of a table that a single scan reads in under a second.

Merge join tracks nested loop here, and for the same reason. To get orders in customer_id order it walks the same index, fetching scattered pages one at a time. It came out slightly ahead between 1,000 and 4,000 customers, probably because it walks the index once instead of starting a new search for each customer, and it stops as soon as the customer side runs out, so it never reads the rest of the orders index.

The planner is the surprise. It switched to hash join at 1,721 customers, when its own estimate reached about 1,695, and at 2,000 customers that choice cost 312 ms against 126 ms for the nested loop it rejected. It had the rows right. What it overpriced was the pages: the default random_page_cost = 4 tells it that a page fetched by an index lookup costs four times as much as a page read by a sequential scan, which is roughly right when the page has to come from a disk. In these runs every page was already in memory, so the lookups were much cheaper than the planner assumed.

The crossover moves with the cache

The orders table is 640 MB, which fits in the container’s 4 GB but not in PostgreSQL’s 128 MB shared_buffers, so most pages came from the operating system’s cache rather than disk. The jump in nested loop time from 300 customers (1.9 ms) to 1,000 (68 ms) is the point where the pages it touches stop fitting in shared_buffers: the nested loop plan above shows 19,558 pages read from outside it. On a server where those reads go to disk, every nested loop lookup costs more, the crossover moves toward fewer customers, and the planner’s early switch looks much better. Lowering random_page_cost toward 1.1 is the usual advice for SSDs, though the setting applies to every plan on the server at once.

That miss cost about two and a half times the time at 2,000 customers. A wrong row estimate errs in the other direction: it can keep a nested loop running long past the line. The PostgreSQL EXPLAIN post showed one source of wrong estimates in this data: the planner treats country and city as unrelated, so it multiplies their selectivities, the fractions of rows each filter keeps. Every city belongs to exactly one country, so the second filter removes nothing the first did not already remove. Here are customers in four countries and those countries’ 40 cities, 16,000 customers in total:

1
2
3
4
5


SELECT count(*), sum(o.total)
FROM customers c
JOIN orders o ON o.customer_id = c.id
WHERE c.country IN ('C00','C01','C02','C03')
 AND c.city IN ('C00-0','C00-1', /* ...all 40 cities of those countries... */ 'C03-9');

1
2
3
4
5
6
7


 Nested Loop (cost=0.53..122891.10 rows=32325 width=6) (actual time=0.043..1090.030 rows=399572.00 loops=1)
 Buffers: shared hit=119775 read=329022
 -> Seq Scan on customers c (cost=0.10..5274.10 rows=1293 width=4) (actual time=0.019..15.588 rows=16000.00 loops=1)
 Rows Removed by Filter: 184000
 -> Index Scan using orders_customer_id_idx on orders o (cost=0.43..90.70 rows=26 width=10) (actual time=0.004..0.066 rows=24.97 loops=16000)
 Index Cond: (customer_id = c.id)
 Execution Time: 1109.897 ms

The planner expected 1,293 customers, under its 1,700 switch point, and picked a nested loop. It got 16,000, and the inner side ran 16,000 times. The tell is on the outer side: estimated rows=1293, actual rows=16000.00, and an inner side whose loops matches the actual figure, not the estimate.

Countries	Estimated customers	Actual customers	Planner’s pick	Nested loop	Hash join
2	333	8,000	nested loop	598 ms	373 ms
3	735	12,000	nested loop	855 ms	344 ms
4	1,293	16,000	nested loop	999 ms	375 ms
5	2,001	20,000	hash join	1,206 ms	358 ms
6	2,873	24,000	hash join	1,442 ms	380 ms

The table is the crossover chart again, seen through the estimate. At two to four countries the estimate stays under about 1,700 and the planner keeps the nested loop, up to 2.7 times slower than the hash join. At five countries the estimate crosses 1,700, and the planner switches even though the estimate is still ten times too low. A wrong estimate costs nothing while it stays on the right side of the line.

The fix is to tell the planner the columns are related:

1
2


CREATE STATISTICS customers_country_city (dependencies) ON country, city FROM customers;
ANALYZE customers;

With that, the estimate for four countries went from 1,293 to 15,820 customers, the planner picked the hash join on its own, and the query took 376 ms. MySQL has no equivalent for two columns that share no index (its histograms cover one column each), and with an index on the join column it would not change the join method anyway, for the reason in the MySQL section below.

The quicker reaction to a slow Nested Loop is to turn nested loops off and see what happens. That is a good experiment and a bad fix, because the setting changes every query on the connection, and a nested loop that was slow for 16,000 rows is still the right plan for 16. Christophe Pettus puts it plainly (July 2026): “Diagnose with the switch, then fix the statistics and set it back.”

Turning off nested loops is a test, not a fix

SET enable_nestloop = off in one session is the quickest way to check whether a hash join would be faster. Leaving it off in postgresql.conf, or in a connection pool’s startup settings, makes every small lookup join in the application pay for a full table scan. With only nested loops off, the 100-customer query took 237 ms as a hash join instead of 0.6 ms. Fix the estimate, then put the setting back.

The other obvious fix is memory. Batches above 1 under a Hash node means the hash table spilled to temporary files, as it did in the shipments join, and the natural response is to give it room. I raised work_mem from 1 MB to 1 GB on that join with the hash join forced, and did the same with MySQL’s join_buffer_size:

PostgreSQL `work_mem`	Batches	Hash join	MySQL `join_buffer_size`	Temp files	Hash join
1 MB	128	1,446 ms	256 KB (default)	256	1,893 ms
4 MB (default)	64	1,503 ms	4 MB	128	1,417 ms
64 MB	4	2,286 ms	64 MB	8	3,076 ms
1 GB	1	2,425 ms	1 GB	0	3,349 ms

On both engines, the join got slower as it got more memory. In PostgreSQL the single in-memory hash table peaked at 297,571 kB, and the query took 60 percent longer than with 64 small batches. In MySQL the fully in-memory join took more than twice as long as the run that wrote 128 temporary files. I repeated the PostgreSQL runs in a different order, ten runs per setting in all, and the ranking held.

The explanation that fits is the CPU cache. Each probe of a hash table lands at a random spot, and a 290 MB table is far larger than the processor’s cache, so nearly every probe waits on main memory, while a batch of a few megabytes stays in cache for as long as it is being probed. I did not measure cache misses, so treat that as the likely reason rather than a proven one. There is also a caveat that cuts the other way: the temporary files here never reached a disk, since the container had memory to spare and the operating system kept them in its cache. On a server short of memory, where those writes really hit storage, batching costs more.

Small spills still cost something. MySQL’s default 256 KB join buffer is tiny: in the customers-and-orders join it spilled from 8,000 customers on, with 24 temporary files, and took 804 ms against 549 ms with a 4 MB buffer. So a few megabytes helped and a gigabyte hurt. Batches: 64 in a plan says the hash table spilled, which by itself says little about why a query is slow, and Part IV of the queries series covers why raising work_mem server-wide is risky anyway.

MySQL has two of the three, and picks one

MySQL added hash joins in 8.0.18, and in 8.0.20 they replaced the block nested loop, the older method for joining without an index. There is no merge join. The 8.4 manual says when the hash join is used: “for any query for which each join has an equi-join condition, and in which there are no indexes that can be applied to any join conditions.” With an index on orders.customer_id, the optimizer chose the nested loop at every size I tried. To get a hash join I had to hide the index with hints:

1
2
3


SELECT /*+ JOIN_ORDER(c, o) NO_INDEX(o orders_customer_id_idx) */ count(*), sum(o.total)
FROM customers c JOIN orders o ON o.customer_id = c.id
WHERE c.id <= 20000;

Customers	Nested loop (optimizer’s pick)	Hash join (forced)
1,000	121 ms	548 ms
5,000	584 ms	558 ms
8,000	913 ms	829 ms
20,000	2,142 ms	836 ms
50,000	1,084 ms	869 ms
200,000	2,876 ms	1,237 ms

The crossover lands near 5,000 customers again, the same place as in PostgreSQL. The difference is that MySQL never makes the switch on its own. Both plans at 20,000 customers:

1
2
3
4
5
6
7
8


-> Nested loop inner join (cost=990339 rows=981020) (actual time=0.21..2149 rows=500284 loops=1)
 -> Covering index range scan on c using PRIMARY over (id <= 20000) (cost=8022 rows=39992) (actual time=0.148..2.01 rows=20000 loops=1)
 -> Index lookup on o using orders_customer_id_idx (customer_id=c.id) (cost=22.1 rows=24.5) (actual time=0.099..0.107 rows=25 loops=20000)

-> Inner hash join (o.customer_id = c.id) (cost=19.8e+9 rows=981020) (actual time=2.75..884 rows=500284 loops=1)
 -> Table scan on o (cost=1.39 rows=4.96e+6) (actual time=0.0958..502 rows=5e+6 loops=1)
 -> Hash
 -> Covering index range scan on c using PRIMARY over (id <= 20000) (cost=8022 rows=39992) (actual time=0.149..1.39 rows=20000 loops=1)

The cost MySQL printed for the hash join it was forced into is 19.8e+9, against 990,339 for the nested loop. Those are 20,000 times apart, and the runtimes go the other way, 884 ms against 2,149. Whatever the optimizer’s model of a hash join is, it is not one it uses to compete with an index.

What MySQL does change at larger sizes is the join order, which table goes on the outer side. From 50,000 customers on, it starts from the orders table, scanning all 5,000,000 rows once and looking up each order’s customer by primary key. That is still a nested loop, but its cost now grows with the orders scanned instead of the customers selected, which is why 50,000 customers ran faster than 20,000. At 200,000 customers it took 2,876 ms against 1,237 ms for the hash join. (MySQL’s optimizer, to its credit, is very consistent.)

For a reporting query that joins a large share of two big tables, the lever in MySQL 8.4 is an index hint like the one above. Merge join is not available, so the shipments join ran either as a nested loop (2,614 ms) or as a hash join (1,417 to 3,349 ms depending on the buffer).

What to check when a join is slow

Start by naming the join. Each algorithm leaves the same marks in every plan above:

Algorithm	PostgreSQL plan	MySQL plan
Nested loop	`Nested Loop`, with `loops=` on the inner child equal to the outer side’s rows	`Nested loop inner join`, with an `Index lookup` child and its `loops=`
Hash join	`Hash Join` and `Hash Cond`, over a `Hash` child showing `Buckets`, `Batches` and `Memory Usage`	`Inner hash join (o.customer_id = c.id)`, over a `Hash` child
Merge join	`Merge Join` and `Merge Cond`, over two inputs in join-column order	does not exist

Then read its outer side. If estimated rows and actual rows differ by more than a few times, the planner may have chosen for a size it never got, and the inner side’s loops will match the actual number. In PostgreSQL, rerun the query in one session with SET enable_nestloop = off or enable_hashjoin = off. If the other plan is much faster, the algorithm choice was the problem, and the estimate is what to fix: ANALYZE, a higher statistics target on the column, or CREATE STATISTICS for columns that move together. In MySQL the same test is a NO_INDEX hint on the inner table, and with an index on the join column a hint is also the only way to get a hash join.

If the estimate is right and the join is still slow, look at how many rows go into it. A nested loop with 200,000 loops, or a hash join building on millions of rows, is often a query asking for more than the screen shows. Batches above 1 comes last: in these runs a few more megabytes helped and a gigabyte hurt.

Clustered vs Non-Clustered Index: The Table Is the Index in MySQL

Mon, 09 Nov 2026 00:00:00 +0000

TL;DR

A clustered index decides the order in which a table’s rows are stored on disk, and with it how many reads a query for related rows has to make. This post covers how InnoDB, MySQL’s storage engine, lays out a table, what every other (non-clustered) index costs and how it finds a row, and why the order of the primary key decides the reads. Then it measures what keeping related rows together costs on every insert, how much free room a rebuild can leave for new rows, and how PostgreSQL, which has no clustered index, gets close, on MySQL 8.4.13 and PostgreSQL 18.6.

The comparison first, on a clients-and-assets schema where each client owns about 50 assets. Every term in it is built up in the sections below, and every cell is measured there:

	MySQL, clustered on `asset_id`	MySQL, clustered on `(client_id, asset_id)`	PostgreSQL heap	PostgreSQL after `CLUSTER`	PostgreSQL covering index
One client’s assets, disk limited to 3,000 reads/s	15.0 ms, 45 pages	0.53 ms, 1.7 pages	14.8 ms, 46 pages	1.02 ms, 3.1 pages	0.60 ms, 1.7 pages
Same, after 30% more rows arrived	20.4 ms	1.07 ms	not run	5.22 ms	not run
One asset by `asset_id`, pages touched	4.0	8.4	4.0	not run	not run
Inserting the next 1M assets	34.5 s	105.7 s	not compared	not compared	not compared
Space it costs	baseline	2.2x the default table after inserts	baseline	none	a 398 MB index instead of 34 MB
Rows stay in client order	no	always	no	only until new rows arrive	the index always does

PostgreSQL’s insert times are left out, because two engines inserting through different code paths is not a fair race. The comparison that matters there is MySQL against itself, with only the primary key changed.

So here is how InnoDB stores a table, in outline. The simplest way to build a table is to put each new row wherever there is room, usually at the end of a file, and keep every index beside it as a sorted list of values, each with the address of its row. Inserts are cheap that way, and every index works alike. The catch shows up once the table is bigger than memory. Rows that a query wants together (all the assets of one client, say) arrived at different times and were written to different places, so each one the index finds can be a separate read from disk. PostgreSQL stores its tables like that, in what it calls a heap. InnoDB, the storage engine MySQL uses by default, made the other choice: it stores the rows inside the primary key’s index, sorted by the primary key, so rows whose keys are close sit next to each other on disk. That index is the clustered index. Because rows can be kept in only one order, a table gets one, and every other index stores a primary key value where PostgreSQL’s would store an address. Choosing the primary key is choosing which rows share a place on disk.

A quick reminder about indexes and pages

An index is a sorted copy of one or more columns that lets the database find rows without reading the whole table. MySQL and PostgreSQL both build theirs as B-trees: a tree of pages, where the top page says which page below it holds which range of values, down to the bottom pages (the leaves) that hold the entries themselves. A page is the unit the database reads from disk, 16 KB in InnoDB and 8 KB in PostgreSQL, and reading one row means reading the whole page it is on. So the number of pages a query touches is most of what it costs, and it is the number this post counts.

The term clustered index comes from SQL Server, where it names an index whose leaves are the table’s rows instead of pointers to them. Here is what that means for one client’s assets in each engine. The left half is what the rest of the post builds up, one piece at a time:

The rows live inside the primary key

In InnoDB the primary key does not point at the rows, because its leaf pages are the rows. Here is the table most people would write first, with an auto-increment asset_id as the primary key and an index on client_id for looking up a client’s assets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


CREATE TABLE assets (
 asset_id bigint NOT NULL AUTO_INCREMENT,
 client_id int NOT NULL,
 asset_type varchar(16) NOT NULL,
 serial char(16) NOT NULL,
 name varchar(64) NOT NULL,
 value decimal(12,2) NOT NULL,
 acquired_at datetime NOT NULL,
 notes varchar(100) NOT NULL,
 PRIMARY KEY (asset_id),
 KEY assets_client_id_idx (client_id)
);

The MySQL manual says it directly: “Each InnoDB table has a special index called the clustered index that stores row data. Typically, the clustered index is synonymous with the primary key.” In B-tree terms, the leaf pages of the primary key’s tree hold whole rows, sorted by asset_id, and the pages above them hold only enough of the key to find the right leaf. A new asset gets the next id, so it goes onto the last leaf page, and the table ends up in arrival order. That keeps inserts cheap. It also means that client 621’s 50 assets, added at 50 different moments over the years, sit on 50 different pages, each next to whatever other clients’ assets arrived around the same time.

A MySQL table always has a clustered index

Leave out the primary key and InnoDB picks one anyway. It uses the first UNIQUE index whose columns are all NOT NULL, and if there is none, it generates a hidden 6-byte row id that “increases monotonically as new rows are inserted”, per the same manual page. The rows are still clustered, just by an order nobody chose.

The index on client_id is the other kind. A non-clustered index, which MySQL calls a secondary index, is a B-tree of its own, and the manual describes what its leaves hold: “each record in a secondary index contains the primary key columns for the row, as well as the columns specified for the secondary index.” So it does not know where a row is. It knows the row’s primary key, and getting from there to the row takes a second search, down the clustered index.

To see what those two searches cost, I loaded both engines with the same data: MySQL 8.4.13 and PostgreSQL 18.6 in Docker with default settings. A clients table has 100,000 rows. An assets table has 5,000,000, each belonging to a random client, so a client’s assets arrive scattered over time, the way real data grows. The timed runs fetch the assets of 1,000 random clients with the container held to 1 GB of memory and 3,000 disk reads per second, roughly the baseline of an AWS gp3 volume, so that pages really come from disk. Page counts are averages over the same 1,000 clients. The test tables were loaded with asset_id values already assigned in increasing order, the order an AUTO_INCREMENT column produces, and the plans below name the test copies (a_by_id, a_by_client, assets_orig).

Now fetch one client’s assets:

1

SELECT asset_id, asset_type, name, value FROM assets WHERE client_id = 621;

1

-> Index lookup on a_by_id using assets_client_id_idx (client_id=621) (cost=55 rows=50) (actual time=0.0522..0.181 rows=50 loops=1)

The plan names one step, and both searches are hiding inside it. The secondary index’s entries for client 621 are 50 pairs of (621, asset_id). To get name and value, InnoDB takes each asset_id and searches the clustered index for it: 50 separate searches, ending on 50 different pages. Averaged over 1,000 clients, that was 45 pages read from disk per lookup, for 50 rows.

PostgreSQL ends up in the same place by a different road. Its table is a heap, and its index on client_id holds row addresses instead of keys:

1
2
3
4
5
6
7


 Bitmap Heap Scan on assets_orig (cost=4.83..206.34 rows=51 width=47) (actual time=1.328..7.858 rows=50.00 loops=1)
 Recheck Cond: (client_id = 621)
 Heap Blocks: exact=50
 Buffers: shared hit=8 read=45
 -> Bitmap Index Scan on assets_orig_client_id_idx (cost=0.00..4.82 rows=51 width=0) (actual time=0.703..0.704 rows=50.00 loops=1)
 Index Cond: (client_id = 621)
 Buffers: shared read=3

Heap Blocks: exact=50 says the 50 addresses pointed to 50 different heap pages, and read=45 is the pages that had to come from outside PostgreSQL’s memory. The heap and the auto-increment clustered index have the same problem: both keep rows in arrival order, and this query wants them in client order. The difference is that InnoDB lets you pick the order. Sort the clustered index by client instead, and the same 50 assets sit together:

Choosing the order on purpose

Since a table gets one clustered index, the decision worth making is which order its rows are in, and the answer comes from how the application reads them. Assets belong to clients, and the application mostly reads them client by client: show me this client’s laptops, total this client’s asset value. So the schema is two tables. clients keeps its own simple key, and assets puts client_id first in its primary key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


CREATE TABLE clients (
 client_id int NOT NULL,
 name varchar(32) NOT NULL,
 region varchar(16) NOT NULL,
 created_at datetime NOT NULL,
 PRIMARY KEY (client_id)
);

CREATE TABLE assets (
 asset_id bigint NOT NULL AUTO_INCREMENT,
 client_id int NOT NULL,
 asset_type varchar(16) NOT NULL,
 serial char(16) NOT NULL,
 name varchar(64) NOT NULL,
 value decimal(12,2) NOT NULL,
 acquired_at datetime NOT NULL,
 notes varchar(100) NOT NULL,
 PRIMARY KEY (client_id, asset_id),
 UNIQUE KEY assets_asset_id_uq (asset_id)
);

The columns are the same as before, and only the last two lines changed. The primary key is now (client_id, asset_id), so the clustered index sorts rows by client first, and asset_id puts each client’s assets in the order they arrived. Every asset of client 621 ends up next to the others, on one or two pages, whenever it was added.

In a real schema assets.client_id would also be a foreign key to clients, and the primary key that starts with client_id already provides the index the foreign key needs. The test tables left the foreign key out, so the insert timings further down do not include that check. asset_id alone is still unique and still auto-increments, and the UNIQUE KEY is not optional. Without it, MySQL 8.4 refuses the table:

1

ERROR 1075 (42000): Incorrect table definition; there can be only one auto column and it must be defined as a key

An AUTO_INCREMENT column has to be the first column of some index, and in the primary key it is second. The same query now reads the clustered index directly:

1

-> Index lookup on a_by_client using PRIMARY (client_id=621) (cost=6.07 rows=50) (actual time=0.0123..0.0199 rows=50 loops=1)

using PRIMARY means the search went straight into the clustered index, found the first row for client 621, and read forward through the next 49, which were on the same page or the one after it. That was 1.7 pages from disk on average, against 45.

On a disk that allows 3,000 reads a second, every page read costs a third of a millisecond, so the page count turns straight into time: 15.0 ms on average for the default table, 0.53 ms for the one clustered by client.

A composite primary key can still look like overthinking it. The default table already had an index on client_id, the query used it, and with the whole table in memory the gap almost disappears: 0.66 ms against 0.25 ms in a warm run. That is why this kind of change looks pointless on a laptop (I have dismissed it that way myself). The index on client_id finds the right rows in both tables. What it cannot do is put them on the same page; only the clustered index decides that. The difference shows once the data no longer fits in memory and each of those 45 pages is a trip to disk.

The composite key also changes every other index on the table, because each one carries a copy of the primary key and ends each lookup with a search of the clustered index. Look up one asset by its id on the client-clustered table:

1

SELECT client_id, asset_type, name, value FROM assets WHERE asset_id = 4995192;

EXPLAIN shows key: assets_asset_id_uq, the unique index, with type: const, one row. What it does not show is that the unique index only holds asset_id and the primary key (client_id, asset_id). InnoDB finds the entry, reads client_id out of it, and then searches the clustered index for (621, 4995192), two trees instead of one. A lookup by id is fast either way, and the extra pages only matter for an application that does millions of them.

Size is the other cost. I added an index on acquired_at to both MySQL tables, which at that point held the same 6.5 million rows. With asset_id as the primary key, each entry holds a date and an 8-byte id. With (client_id, asset_id), each entry also carries the 4-byte client_id, and the index came out 22 percent bigger for the same dates. The manual’s advice follows from that: “If the primary key is long, the secondary indexes use more space, so it is advantageous to have a short primary key.” A 16-byte UUID primary key is copied into every index the same way, which is part of what random UUIDs cost.

What every other index pays	MySQL, key `asset_id`	MySQL, key `(client_id, asset_id)`	PostgreSQL heap
One asset by `asset_id`, pages touched per lookup	4.0	8.4	4.0
Index on `acquired_at`	133 MB	162 MB	139 MB, whatever the key
Index on `client_id`	149 MB	not needed, the key starts with it	34 MB

PostgreSQL’s indexes store a 6-byte row address no matter what the primary key is, which is why its acquired_at index does not change with the key. Its index on client_id is under a quarter the size of MySQL’s, because PostgreSQL (since version 13) stores a repeated value once with a list of row addresses, while InnoDB stores a full (client_id, asset_id) entry for each of the 5 million rows.

The price: every new row goes into the middle

What the composite key saves on reads, the table pays back on every insert. An auto-increment primary key puts every new row at the end of the table, on the last page, which is the cheapest place to write. Clustering by client means a new asset for client 621 has to go into client 621’s page, somewhere in the middle. If that page is full, InnoDB splits it into two half-full pages to make room, which is a page split. I inserted 1.5 million new assets for random clients into both tables, in batches of 10,000. Random clients is the worst case: if most new assets belong to recently created clients, and client_id is itself an auto-increment, the inserts land near the end of the index anyway, which I reasoned through but did not measure.

	Clustered on `asset_id`	Clustered on `(client_id, asset_id)`
First 500,000 inserts	14.5 s, 5,896 page splits	65.6 s, 65,232 page splits
Next 1,000,000 inserts	34.5 s, 12,393 page splits	105.7 s, 24,424 page splits
Clustered index after both	1,167 MB	2,601 MB
After `OPTIMIZE TABLE`		1,337 MB, in 39 s

The first batch was the worst, because the table had just been loaded in sorted order and every page was nearly full, so almost every new row split a page. Once the pages were half empty there was room, and the second batch split far fewer, but it was still three times slower than appending at the end. The table also more than doubled in size, since all those split pages were half empty. OPTIMIZE TABLE rebuilds the table packed again. For InnoDB it runs as online DDL, and the manual says “an exclusive table lock is only taken briefly during the prepare phase and the commit phase”, so reads and writes keep going while it works.

A rebuild can also leave room for the rows that will arrive later. innodb_fill_factor sets how full InnoDB packs each page when it builds or rebuilds an index in sorted order, and the manual notes that even the default of 100 “leaves 1/16 of the space in clustered index pages free for future index growth.” I rebuilt the client-clustered table at three settings and inserted the same 500,000 assets into each:

1
2
3


SET GLOBAL innodb_fill_factor = 70;
ALTER TABLE assets FORCE; -- OPTIMIZE TABLE rebuilds the same way
SET GLOBAL innodb_fill_factor = 100;

Client-clustered table	Size before	500,000 inserts	Page splits	Size after
Loaded in sorted order, as above	900 MB	58.0 s	65,232	2,184 MB
Rebuilt, `innodb_fill_factor = 100`	1,028 MB	42.8 s	65,057	2,181 MB
Rebuilt, `innodb_fill_factor = 80`	1,210 MB	33.9 s	3,752	1,267 MB
Rebuilt, `innodb_fill_factor = 70`	1,378 MB	31.8 s	547	1,378 MB

At 70 the new assets fit into the room the rebuild left, and the table stayed the size it started. The client lookup barely noticed the emptier pages: 1.6 to 1.9 pages read from disk, against 1.6 when packed. The space is paid up front instead of after the splits, and it only lasts as long as the room does. These inserts added 10 percent more rows into 30 percent of free space, so a table that keeps growing needs the rebuild repeated before the room runs out. innodb_fill_factor is a global setting that applies to every sorted index build on the server, which is why the example sets it back afterwards. (These runs came from a fresh container, so the packed table’s 58.0 s is a little faster than the 65.6 s above. The page splits and sizes matched exactly.)

Cluster on what you read together, not on what is random

Grouping by client works because a client’s assets are read together. A clustered key with no useful order, such as a random UUID, gets all of the insert cost here and none of the read benefit: every new row lands on a random page and splits it, and nothing is ever read in that order.

PostgreSQL: sort it once, or copy the columns into an index

PostgreSQL has no clustered index and no way to declare one. Its table is a heap, where a new row goes wherever there is free space, and every index, the primary key included, stores the row’s ctid: a 6-byte address made of the page number and the position on that page. That is the address the client_id index handed back 50 times in the plan above. PostgreSQL offers two ways to get close to the client-clustered layout, and neither one is a clustered index.

The first is the CLUSTER command, which rewrites the table sorted by an index:

1

CLUSTER assets USING assets_client_id_idx;

It took 23.5 seconds on 5 million rows, and afterwards one client’s assets came back in 3.1 pages from disk and 1.02 ms under the disk limit, close to MySQL’s client-clustered table. The PostgreSQL documentation is clear about what it is: “Clustering is a one-time operation: when the table is subsequently updated, the changes are not clustered.” New rows still go wherever the heap has room, usually the end. While it runs, CLUSTER holds an ACCESS EXCLUSIVE lock, which “prevents any other database operations (both reads and writes) from operating on the table until the CLUSTER is finished.”

After I added 10 percent more assets, the clustered PostgreSQL table read 4.7 pages per client, and after 30 percent, 14.8 pages and 5.22 ms under the disk limit. The same inserts left MySQL’s client-clustered table at 2.4 pages, since InnoDB put each new asset on its client’s page. That is the trade from the previous section showing up from the other side: InnoDB pays on every insert to keep the order, and PostgreSQL pays nothing on insert and loses the order.

The second option is a covering index, an index that carries the columns the query needs so that the table is never read:

1

CREATE INDEX assets_client_cov ON assets (client_id) INCLUDE (asset_id, asset_type, name, value);

1
2
3
4


 Index Only Scan using assets_orig_cov on assets_orig (cost=0.56..5.45 rows=51 width=47) (actual time=0.049..0.061 rows=50.00 loops=1)
 Index Cond: (client_id = 621)
 Heap Fetches: 0
 Buffers: shared hit=4 read=4

Index Only Scan with Heap Fetches: 0 means the answer came entirely from the index, whose entries are sorted by client_id, so a client’s 50 entries sit together. It read 1.7 pages from disk and took 0.60 ms, the same as MySQL’s clustered table, and since an index is kept in order on every insert, it does not wear off the way CLUSTER does. It cost a 398 MB index in place of the 34 MB plain one, and it only helps queries that ask for the included columns. Heap Fetches also rises when rows change until VACUUM catches up, which One Column Added to a SELECT measures.

How to choose the clustered key

In MySQL, the primary key is the most important index decision on a table, because it is also the table’s physical order and part of every other index. An auto-increment id is the right default when rows are mostly written once and read one at a time, or in the order they arrived. A composite key that leads with the owner, such as (client_id, asset_id) or (tenant_id, order_id), pays off when the application nearly always reads a parent’s children together and the table does not fit in memory, and it costs slower inserts, a bigger table between rebuilds, and slightly bigger secondary indexes. Rebuilding with innodb_fill_factor below 100 trades some of that size up front for far fewer page splits. Measure the read you care about in pages, not milliseconds on a warm laptop.

In PostgreSQL there is no such choice to make, and the table stays in arrival order. If one access pattern dominates, a covering index gives the clustered read without the decay, at the price of the index’s size. CLUSTER is the one-off option for a table that is mostly read and can be locked for the rewrite.

How to Read EXPLAIN ANALYZE: MySQL

Wed, 04 Nov 2026 00:00:00 +0000

TL;DR

EXPLAIN ANALYZE runs your query and shows each step MySQL took to answer it, how long each step took, and how many rows it expected at each step next to how many it got. The habit this post teaches is to find the one slow step first and read only its numbers before changing anything. The output tells you where the time went, not why, and MySQL leaves out some things PostgreSQL shows, so the fix still takes some thinking.

Plain EXPLAIN shows what MySQL plans to do with your query without running it, so every number in it is a guess. EXPLAIN ANALYZE runs the query for real and prints the same plan with what happened at each step: rows out, time taken, and how many times the step ran. A plan can look sensible and still read the whole table, and only the second command shows that. You can paste the output into an AI and get a reading back, and it will often point at the right line. What it does not know is your system: which table is the busy one, what the query is for, whether a number that looks large is normal here. That judgment is yours, and this post is about having enough of it to check the reading rather than take it.

The part of MySQL that decides how to run a query is the optimizer, and it decides before reading any data. It lists the ways it could answer (read the whole table or use an index, which table to start a join with), estimates the cost of each, and picks the cheapest. Its estimates come from statistics InnoDB keeps about each table and its indexes, and from quick samples of an index taken while planning. When a column has no index, the optimizer knows very little about it and falls back on rough rules. That is why plain EXPLAIN answers instantly, and why it can be badly wrong.

The executor then runs the chosen plan and counts what happens. EXPLAIN ANALYZE prints those counts next to the optimizer’s guesses, and reading it mostly means comparing the two.

Every plan below comes from MySQL 8.4.13 in Docker with default settings, on a customers table of 200,000 rows (50 countries, 10 cities each) and an orders table of 5,000,000. Each order belongs to a random customer, so one customer’s orders are scattered across the table, the way real orders arrive. The PostgreSQL version of this post uses the same rows.

What the output is telling you

Start with the simplest slow query there is: one customer’s orders, before any index exists on customer_id.

1

SELECT id, created_at, total FROM orders WHERE customer_id = 4242;

EXPLAIN FORMAT=TREE prints the plan as a tree:

1
2


-> Filter: (orders.customer_id = 4242) (cost=529407 rows=496280)
 -> Table scan on orders (cost=529407 rows=4.96e+6)

Each line starting with -> is one step. MySQL calls a step an iterator, because each one hands rows, one at a time, to the step above it. The deeper a line is indented, the earlier it runs, so read from the innermost line out. Table scan on orders means read every row of the table from start to finish. Its rows go up to Filter, which is the WHERE condition, checked against each row. rows= is the estimate: about 4.96 million rows from the scan (4.96e+6 is engineering notation for 4,960,000), and 496,280 expected to match. cost is the optimizer’s price in its own units, not milliseconds. Now with EXPLAIN ANALYZE:

1
2


-> Filter: (orders.customer_id = 4242) (cost=529407 rows=496280) (actual time=16.7..652 rows=27 loops=1)
 -> Table scan on orders (cost=529407 rows=4.96e+6) (actual time=0.224..578 rows=5e+6 loops=1)

The first parenthesis is still the guess. The second is what happened. actual time=0.224..578 on the table scan is milliseconds until it produced its first row and its last row. rows=5e+6 is the rows it produced: all five million. loops=1 is how many times the step ran, which matters later. On the Filter, rows=27 is how many matched, against a guess of 496,280.

Two things are wrong here, and they are different problems. The guess was off by a factor of 18,000, which is a lot even for a guess. With no index on customer_id the optimizer had little to go on, and its guess works out to exactly a tenth of the table. And even a perfect guess would not have helped, because without an index there was no faster way to find 27 rows than reading all five million. That is what the table scan’s rows=5e+6 says, and on a slow query it is the first number worth finding.

An index is that faster way. It is a separate sorted structure, like the index at the back of a book, that maps each customer_id to where that customer’s rows are stored. After CREATE INDEX orders_customer_id_idx ON orders (customer_id):

1

-> Index lookup on orders using orders_customer_id_idx (customer_id=4242) (cost=28.7 rows=27) (actual time=0.044..0.0452 rows=27 loops=1)

One step, a lookup instead of a scan. The guess is now exactly right (rows=27), because the index gives the optimizer real numbers to work from, and the query takes 0.045 ms instead of 652.

EXPLAIN ANALYZE really runs the query

For a SELECT it takes as long as the query takes; the MySQL manual notes you can stop it with KILL QUERY or Ctrl-C. It also accepts some UPDATE and DELETE statements, so treat it as running them and wrap it in START TRANSACTION; and ROLLBACK;. Run long queries on a copy or a replica, not the server your users are on.

When the database guessed wrong

When the optimizer guesses wrong it can pick the wrong plan, so the estimate is worth checking on every slow step. This query joins the two tables (a join matches rows in one table to rows in another, here customers to their orders) and totals the orders of every customer in one city:

1
2
3
4


SELECT count(*), sum(o.total)
FROM customers c
JOIN orders o ON o.customer_id = c.id
WHERE c.country = 'C07' AND c.city = 'C07-3';

1
2
3
4
5


-> Aggregate: count(0), sum(o.total) (cost=75057 rows=1) (actual time=84.4..84.4 rows=1 loops=1)
 -> Nested loop inner join (cost=70156 rows=49011) (actual time=0.222..84 rows=9940 loops=1)
 -> Filter: ((c.city = 'C07-3') and (c.country = 'C07')) (cost=20384 rows=2001) (actual time=0.103..29.8 rows=400 loops=1)
 -> Table scan on c (cost=20384 rows=200070) (actual time=0.0927..22.6 rows=200000 loops=1)
 -> Index lookup on o using orders_customer_id_idx (customer_id=c.id) (cost=22.4 rows=24.5) (actual time=0.127..0.135 rows=24.9 loops=400)

Read it inside out. The deepest line, Table scan on c, reads all 200,000 customers. Filter keeps the ones in that city: it expected 2,001 and found 400. That is the thing to look for, estimated rows against actual rows on the same line. Neither column has an index, and the guess again works out to a tenth per condition: a tenth of a tenth of 200,070 is 2,001.

Nested loop inner join is one way to do a join. It has two children at the same depth: the first (the filtered customers) runs once, and for each row it produces, the second (find that customer’s orders through the index) runs once. That is what loops=400 on the Index lookup means. When loops is above 1, rows and actual time are averages per run, so multiply them back: 24.9 rows times 400 is the 9,940 rows the join produced, and 0.135 ms times 400 is 54 ms, most of the 84. The join’s own estimate was 49,011 rows against 9,940, the first mistake carried up the tree.

Here the wrong guess did no harm, because with 2,001 customers or 400, looking up each one’s orders in the index is a good plan. A gap matters when the right number would have made the optimizer choose differently, and the bigger the gap, the more likely that is. The next section has one where a wrong guess fed into a choice of index.

When the plan is fine but still slow

The first plan already showed the second lesson: a scan that read the whole table, given away by rows=5e+6 on the Table scan line. Here it is in a bigger tree to practice on, the 100 largest orders since November 1 with each customer’s location:

1
2
3
4
5
6


SELECT o.id, o.total, c.country, c.city
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.created_at >= '2024-11-01'
ORDER BY o.total DESC
LIMIT 100;

1
2
3
4
5
6


-> Limit: 100 row(s) (cost=2.29e+6 rows=100) (actual time=789..789 rows=100 loops=1)
 -> Nested loop inner join (cost=2.29e+6 rows=4.96e+6) (actual time=789..789 rows=100 loops=1)
 -> Sort: o.total DESC (cost=529519 rows=4.96e+6) (actual time=789..789 rows=100 loops=1)
 -> Filter: (o.created_at >= TIMESTAMP'2024-11-01 00:00:00') (cost=529519 rows=4.96e+6) (actual time=585..675 rows=608001 loops=1)
 -> Table scan on o (cost=529519 rows=4.96e+6) (actual time=0.114..572 rows=5e+6 loops=1)
 -> Single-row index lookup on c using PRIMARY (id=o.customer_id) (cost=0.997 rows=1) (actual time=0.0054..0.00541 rows=1 loops=100)

Each step’s time includes the time of the steps under it, so to see what one step cost by itself, subtract its children’s finishing times from its own. Sort finished at 789 ms and its child at 675, so sorting cost 114 ms. Filter finished at 675 and the scan under it at 572, so checking the dates cost 103. Table scan on o cost 572 ms by itself, more than everything else put together. It read five million rows to keep 608,001. The customer lookups ran 100 times at 0.0054 ms each, half a millisecond in all.

Two more things are visible once you know where to look. The Sort produced its first row at the same moment as its last (789..789), because a sort cannot hand anything up until it has seen all its input. And the Filter produced its first row at 585 ms, because the newest orders sit at the end of the table and the scan reaches them last.

There was an index on created_at while this ran, and MySQL did not use it. It estimated about 1.27 million matching rows (the real number was 608,001), and priced 1.27 million trips from the index back to the table as dearer than one pass over the whole table. The guess was off by half; the plan alone cannot say whether the right number would have changed the choice. An index that holds every column the query needs never makes those trips. After CREATE INDEX orders_created_total_cust_idx ON orders (created_at, total, customer_id):

1
2
3
4
5
6


-> Limit: 100 row(s) (cost=1.49e+6 rows=100) (actual time=183..183 rows=100 loops=1)
 -> Nested loop inner join (cost=1.49e+6 rows=1.28e+6) (actual time=183..183 rows=100 loops=1)
 -> Sort: o.total DESC (cost=258614 rows=1.28e+6) (actual time=183..183 rows=100 loops=1)
 -> Filter: (o.created_at >= TIMESTAMP'2024-11-01 00:00:00') (cost=258614 rows=1.28e+6) (actual time=0.0673..73.8 rows=608001 loops=1)
 -> Covering index range scan on o using orders_created_total_cust_idx over ('2024-11-01 00:00:00' <= created_at) (cost=258614 rows=1.28e+6) (actual time=0.0669..54.9 rows=608001 loops=1)
 -> Single-row index lookup on c using PRIMARY (id=o.customer_id) (cost=0.862 rows=1) (actual time=0.00143..0.00145 rows=1 loops=100)

Covering index range scan means MySQL read only the part of the index from November 1 onward, and found everything it needed there. 183 ms instead of 789. The sort is now the slowest step at 109 ms, and that part is the price of the question: finding the 100 largest of 608,001 orders means looking at all 608,001. The new index costs something too, 152 MB on disk and extra work on every insert, so it is worth it for a query that runs often and maybe not for a nightly report.

Next thing to learn: counting what MySQL read

PostgreSQL’s plan prints how many pages each step read. MySQL’s does not, and it does not say when a sort spilled to disk either. Session status counters fill part of the gap: run FLUSH STATUS, then the query, then SHOW SESSION STATUS LIKE 'Handler_read%'. For the one-customer query without an index, Handler_read_rnd_next was 5,000,001, one per row scanned. With the index, it was 0, and Handler_read_next was 27.

What the warning signs mean

Table scan under a Filter that keeps a small share of its rows means MySQL read the whole table to find a few. The usual fix is an index on the WHERE column. If one exists and goes unused, check whether the condition wraps the column in a function, which hides it from the index; Non-SARGable Predicates explains that one.

Index range scan or Index lookup where you might have had Covering index range scan means MySQL found the rows in the index, then went back to the table for each one to fetch the other columns. Counting 25,182 orders for a range of customers took 2.3 ms as a covering scan. Summing their total, a column not in the index, took 95 ms. Adding the missing column to the index is the usual fix, with the size cost above.

Nested loop inner join with a large loops on its second child means one lookup repeated many times. Fine when the repeated step is an index lookup, as in both joins above. When it is a Table scan, the join column has no index and the whole table is read once per loop.

A Sort with a large input means every row has to be read before the first one comes out. If the sort is the slow step, ask whether the query needs all those rows. The tree will not tell you whether it spilled to disk.

The first five things to look at

The top line’s last time: the number you want smaller.
The slowest step. From the top, subtract each step’s children from its own finishing time until you find the step that is slow by itself.
On that step, actual rows against the estimate. A big difference means the optimizer guessed wrong.
Under that step, a Table scan whose rows is close to the table’s size, when only a few rows survive. That means a missing index.
Rows handled against rows needed. Thousands sorted for a page of 100 means the query asks for more than the question.

When you are ready for a real one, A Query on a 3-Billion-Row Table Took 100 Minutes reads a production MySQL plan end to end the same way.

The ways MySQL reads a table

Every leaf of the tree is one of a short list of access methods, and the name says how much of the table or index MySQL expects to touch. The older EXPLAIN output, the one without FORMAT=TREE that most tutorials still show, names the same methods in its type column, so both are listed. Rows are ordered from cheapest to most expensive. The names below were taken from MySQL 8.4.13 on a small separate schema, since the post’s own fixture does not produce all of them.

What the tree says	`type` in classic EXPLAIN	What MySQL does	What it usually means
`Rows fetched before execution`	`const`	Looked up one row by primary key or unique index while planning; the row is a constant by the time the query runs	As good as it gets
`Single-row index lookup on t using PRIMARY (id=...)`	`eq_ref`	For each row from the previous step, one lookup that returns at most one row	The normal shape of a join on a primary key
`Index lookup on t using k (col=...)`	`ref`	Finds every row with one value through a non-unique index, then fetches each row from the table	Fine when `rows` is small; expensive when one value matches thousands
`Covering index lookup on t using k (col=...)`	`ref` + `Using index`	Same lookup, but every column the query needs is in the index, so the table is never read	Cheaper than a plain lookup; adding a column to the SELECT can turn it back into one
`Index lookup ... (col=... or NULL)`	`ref_or_null`	A lookup that also fetches the rows where the column is NULL	A `col = x OR col IS NULL` condition
`Index range scan on t using k over (a < col < b)`	`range`	Walks the part of the index between two bounds, fetching rows from the table	Normal for date ranges; watch `rows` against what the query returns
`Covering index range scan on t using k over (...)`	`range` + `Using index`	The same walk without touching the table	The cheapest range
`Sort-deduplicate by row ID` over two range scans	`index_merge`	Runs two index scans and combines their row IDs	An `OR` across two indexed columns; often slower than it looks
`Index scan on t using k`	`index`	Reads the whole index in order and fetches every row from the table	Usually an `ORDER BY` on an indexed column with no filter; expensive when the table is large
`Covering index scan on t using k`	`index` + `Using index`	Reads the whole index without the table	Cheaper than a table scan because the index is smaller, still every row
`Table scan on t`	`ALL`	Reads every row in the table	The one to look for when a query is slow and only a few rows survive the `Filter` above it
`Zero rows (Impossible WHERE)`	`Impossible WHERE`	The optimizer proved no row can match and reads nothing	A condition that contradicts itself

A few more names appear above the leaves and describe what happens to rows once they are read. Filter: keeps the rows that pass a condition the index could not check. Nested loop inner join runs its second child once per row from the first, which is where loops comes from. Inner hash join reads the smaller side once into memory and probes it, and the classic output shows it as Using join buffer (hash join). Sort: orders rows in memory or in a temporary file, which the plan does not distinguish. Limit:, Group aggregate: and Aggregate: do what their names say, and Materialize writes an intermediate result to a temporary table that a Table scan above it then reads.

The test data was generated once with PostgreSQL (the script is at the end of the PostgreSQL post), exported to CSV, and loaded into these tables on MySQL 8.4:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


CREATE TABLE customers (id int PRIMARY KEY, country varchar(8) NOT NULL,
 city varchar(16) NOT NULL, created_at datetime NOT NULL);
CREATE TABLE orders (id bigint PRIMARY KEY, customer_id int NOT NULL,
 status varchar(16) NOT NULL, created_at datetime NOT NULL,
 total decimal(10,2) NOT NULL, note text);
LOAD DATA INFILE '/var/lib/mysql-files/customers.csv' INTO TABLE customers
 FIELDS TERMINATED BY ',' (id, country, city, created_at);
LOAD DATA INFILE '/var/lib/mysql-files/orders.csv' INTO TABLE orders
 FIELDS TERMINATED BY ',' (id, customer_id, status, created_at, total, note);
ANALYZE TABLE customers, orders;

How to Read EXPLAIN ANALYZE: PostgreSQL

Sun, 01 Nov 2026 00:00:00 +0000

TL;DR

EXPLAIN ANALYZE runs your query and shows each step PostgreSQL took to answer it, how long each step took, and how many rows it expected at each step next to how many it got. The habit this post teaches is to find the one slow step first and read only its numbers before changing anything. The output tells you where the time went, not why, so the fix still takes some thinking.

Plain EXPLAIN shows what PostgreSQL plans to do with your query without running it, so every number in it is a guess. EXPLAIN ANALYZE runs the query for real and prints the same plan with what happened at each step: rows out, time taken, and how much of the table was read. A plan can look sensible and still read the whole table, and only the second command shows that. You can paste the output into an AI and get a reading back, and it will often point at the right line. What it does not know is your system: which table is the busy one, what the query is for, whether a number that looks large is normal here. That judgment is yours, and this post is about having enough of it to check the reading rather than take it.

The part of PostgreSQL that decides how to run a query is the planner, and it decides before reading any data. It lists the ways it could answer (read the whole table or use an index, which table to start a join with), estimates the cost of each, and picks the cheapest. Its estimates come from statistics that the ANALYZE command collects, such as how many rows a table has and which values are common. It never looks at the data itself, which is why plain EXPLAIN answers instantly and why it can be wrong.

The executor then runs the chosen plan and counts what happens. EXPLAIN ANALYZE prints those counts next to the planner’s guesses, and reading it mostly means comparing the two.

Every plan below comes from PostgreSQL 18.6 in Docker with default settings, on a customers table of 200,000 rows (50 countries, 10 cities each) and an orders table of 5,000,000. Each order belongs to a random customer, so one customer’s orders are scattered across the table, the way real orders arrive. I turned off JIT compilation and parallel workers to keep the output short. The MySQL version of this post uses the same rows.

What the output is telling you

Start with the simplest slow query there is: one customer’s orders.

1

SELECT id, created_at, total FROM orders WHERE customer_id = 4242;

Plain EXPLAIN prints this:

1
2


 Seq Scan on orders (cost=0.00..145287.00 rows=26 width=22)
 Filter: (customer_id = 4242)

A plan with one step. Seq Scan (sequential scan, or table scan) means read every row of the table from start to finish. Filter is the WHERE condition, checked against each row. rows=26 is the estimate: the planner expects 26 matches. cost is the planner’s price in its own units, not milliseconds. Now with EXPLAIN ANALYZE:

1
2
3
4
5
6


 Seq Scan on orders (cost=0.00..145287.00 rows=26 width=22) (actual time=3.082..144.209 rows=27.00 loops=1)
 Filter: (customer_id = 4242)
 Rows Removed by Filter: 4999973
 Buffers: shared hit=8694 read=74093
 Planning Time: 0.091 ms
 Execution Time: 144.249 ms

The first parenthesis is still the guess. The second is what happened. actual time=3.082..144.209 is milliseconds until the step produced its first row and its last row. rows=27.00 is the rows it produced, so the guess of 26 was close. loops=1 is how many times the step ran, which matters later.

The next two lines are the price. Rows Removed by Filter: 4999973 says PostgreSQL looked at all 5,000,000 rows and threw away all but 27. Buffers counts pages: PostgreSQL stores a table in 8 KB pages of dozens of rows each, and always reads whole pages. shared hit is pages already in its memory cache (shared buffers), and read is pages it asked the operating system for. Together they are 82,787 pages, the whole table. Execution Time is the total, 144 ms.

The estimate was right and the query was still slow, because without an index there was no faster way to find 27 rows than reading everything. An index is a separate sorted structure, like the index at the back of a book, that maps each customer_id to the pages holding that customer’s rows. After CREATE INDEX ON orders (customer_id):

1
2
3
4
5
6
7
8
9


 Bitmap Heap Scan on orders (cost=4.63..107.58 rows=26 width=22) (actual time=0.006..0.014 rows=27.00 loops=1)
 Recheck Cond: (customer_id = 4242)
 Heap Blocks: exact=27
 Buffers: shared hit=30
 -> Bitmap Index Scan on orders_customer_id_idx (cost=0.00..4.63 rows=26 width=0) (actual time=0.003..0.003 rows=27.00 loops=1)
 Index Cond: (customer_id = 4242)
 Index Searches: 1
 Buffers: shared hit=3
 Execution Time: 0.019 ms

Now the plan has two steps, called nodes. A line starting with -> is a node that feeds rows to the node above it, and the deeper it is indented, the earlier it runs. Read innermost first.

The innermost node, Bitmap Index Scan, looked up 4242 in the index (Index Cond), found 27 entries, and read 3 index pages. It hands the row locations up to Bitmap Heap Scan, which fetches those rows from the table (PostgreSQL calls a table’s storage the heap). Heap Blocks: exact=27 says the 27 rows sat on 27 different pages. A parent’s time and buffers include its children’s, so the top Buffers: shared hit=30 is 3 index pages plus 27 table pages. Thirty pages instead of 82,787, and 0.019 ms instead of 144.

EXPLAIN ANALYZE really runs the query

For a SELECT it takes as long as the query takes. For UPDATE, DELETE or INSERT the change really happens, so wrap it in BEGIN; and ROLLBACK;, as the PostgreSQL documentation suggests. Run long queries on a copy or a replica, not the server your users are on.

When the database guessed wrong

When the planner guesses wrong it can pick the wrong plan, so the estimate is worth checking on every slow step. This query joins the two tables (a join matches rows in one table to rows in another, here customers to their orders) and totals the orders of every customer in one city:

1
2
3
4


SELECT count(*), sum(o.total)
FROM customers c
JOIN orders o ON o.customer_id = c.id
WHERE c.country = 'C07' AND c.city = 'C07-3';

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


 Aggregate (cost=5137.70..5137.71 rows=1 width=40) (actual time=91.548..91.554 rows=1.00 loops=1)
 Buffers: shared hit=3861 read=8553 written=5
 -> Nested Loop (cost=4.63..5136.69 rows=200 width=6) (actual time=0.110..90.336 rows=9940.00 loops=1)
 Buffers: shared hit=3861 read=8553 written=5
 -> Seq Scan on customers c (cost=0.00..4274.00 rows=8 width=4) (actual time=0.010..6.803 rows=400.00 loops=1)
 Filter: ((country = 'C07'::text) AND (city = 'C07-3'::text))
 Rows Removed by Filter: 199600
 Buffers: shared hit=1274
 -> Bitmap Heap Scan on orders o (cost=4.63..107.58 rows=26 width=10) (actual time=0.044..0.205 rows=24.85 loops=400)
 Recheck Cond: (c.id = customer_id)
 Heap Blocks: exact=9940
 Buffers: shared hit=2587 read=8553 written=5
 -> Bitmap Index Scan on orders_customer_id_idx (cost=0.00..4.63 rows=26 width=0) (actual time=0.014..0.014 rows=24.85 loops=400)
 Index Cond: (customer_id = c.id)
 Index Searches: 400
 Buffers: shared hit=785 read=415
 Execution Time: 91.591 ms

Read it inside out. Seq Scan on customers reads all 200,000 customers and keeps the ones in that city. It expected 8 and found 400. That is the thing to look for: estimated rows against actual rows on the same line.

The planner assumes country and city are unrelated, so it multiplies their odds: 200,000 customers, times 1 in 50 for the country, times 1 in 500 for the city, is 8. In this data the city already decides the country, which the planner had no way to know, and the mistake carries up the tree: Nested Loop expected 200 rows and produced 9,940.

Nested Loop is one way to do a join: for each row from its first child (a customer), it runs its second child once (find that customer’s orders through the index). That is what loops=400 on the orders side means. When loops is above 1, rows and actual time are averages per run, so multiply them back: 24.85 rows times 400 is the 9,940, and 0.205 ms times 400 is 82 ms, most of the 91. Buffers are already totals.

Here the wrong guess did no harm, because with 8 customers or 400, looking up each one’s orders in the index is a good plan. A gap like this matters when the right number would have made the planner choose a different plan, and the bigger the gap, the more likely that is. Uniqueness and Selectivity shows one that went from 50 ms to 50 minutes.

Next thing to learn: teaching the planner about related columns

CREATE STATISTICS customers_country_city (dependencies) ON country, city FROM customers, then ANALYZE customers, tells PostgreSQL the two columns move together. The estimate went from 8 to 398 and the query got no faster, because the plan did not need to change. Fixing an estimate and fixing a query are different jobs.

When the plan is fine but still slow

The first plan already showed the second lesson: a correct estimate and a slow query, given away by Rows Removed by Filter and Buffers. Here it is in a bigger tree to practice on, a query listing every order since November 1 with its customer’s location, largest total first:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


 Sort (cost=234697.30..236254.94 rows=623058 width=24) (actual time=512.409..573.290 rows=608001.00 loops=1)
 Sort Key: o.total DESC
 Sort Method: external merge Disk: 20848kB
 Buffers: shared hit=14992 read=69072, temp read=4525 written=4535
 -> Hash Join (cost=6751.00..161952.56 rows=623058 width=24) (actual time=239.832..395.547 rows=608001.00 loops=1)
 Hash Cond: (o.customer_id = c.id)
 Buffers: shared hit=14989 read=69072, temp read=1919 written=1919
 -> Seq Scan on orders o (cost=0.00..145287.00 rows=623058 width=18) (actual time=219.531..257.878 rows=608001.00 loops=1)
 Filter: (created_at >= '2024-11-01 00:00:00+00'::timestamp with time zone)
 Rows Removed by Filter: 4391999
 Buffers: shared hit=13715 read=69072
 -> Hash (cost=3274.00..3274.00 rows=200000 width=14) (actual time=20.133..20.134 rows=200000.00 loops=1)
 Buckets: 262144 Batches: 2 Memory Usage: 6749kB
 Buffers: shared hit=1274, temp written=438
 -> Seq Scan on customers c (cost=0.00..3274.00 rows=200000 width=14) (actual time=0.010..6.891 rows=200000.00 loops=1)
 Buffers: shared hit=1274
 Execution Time: 584.068 ms

The lines that jump out are Sort Method: external merge Disk: 20848kB and Batches: 2. Both mean a spill: the step needed more working memory than PostgreSQL gives one step by default (4 MB), so it wrote part of its work to a temporary file and read it back. It is tempting to start there. Find the slow step first.

Each node’s time includes its children’s, so to see what one node cost by itself, subtract its children’s finishing times from its own. Sort finished at 573 ms and its child, the Hash Join, at 396, so the sort cost 178 ms. The Hash Join (a join that loads one table into memory, the Hash node, and then matches the other table’s rows against it) finished at 396, minus 258 for one child and 20 for the other, which leaves 118 ms. Seq Scan on orders cost 258 ms on its own, the biggest single share. It read the whole table and threw away 4,391,999 rows to keep 608,001.

That leaves two things to fix. The filter on created_at has no index, so I added an index on that column. Over four runs each, the query went from 450 to 570 ms down to 310 to 340 ms, with both spills still there. And the query asks for 608,001 sorted rows. If the screen shows the top 100, ask for 100, in a subquery with ORDER BY total DESC LIMIT 100 that runs before the join, so only 100 orders need a customer looked up. With the index that ran in 76 ms, and its sort used 36 kB of memory instead of a 20 MB file. Giving the original query 16 times more memory removed both spills and saved only 40 to 60 ms. The spills were real, and the smallest problem in the tree.

What the warning signs mean

Seq Scan with a large Rows Removed by Filter means PostgreSQL read the whole table to find a few rows. The usual fix is an index on the WHERE column. If one exists and goes unused, check whether the condition wraps the column in a function, which hides it from the index; Non-SARGable Predicates explains that one.

Index Only Scan with Heap Fetches above zero means the answer should have come from the index alone, but PostgreSQL still visited the table for some rows to check they are visible to your query. Counting 25,182 orders took 1.3 ms with Heap Fetches: 0, and 26.7 ms with Heap Fetches: 15709 after an UPDATE touched rows across the table. VACUUM brings it back down; One Column Added to a SELECT goes further.

Nested Loop with a large loops on its second child means one lookup repeated many times. Fine when the repeated step uses an index, as above. When it is a Seq Scan, the join column has no index and the whole table is read once per loop.

Sort with Disk, or Hash with Batches above 1, means a spill. If the subtraction says it is not where the time goes, leave it. If it is, send fewer rows into the sort or join. Raising the memory setting (work_mem) server-wide is risky, because every sort in every connection can use that much; Part IV of the queries series explains why.

The first five things to look at

Execution Time at the bottom: the number you want smaller.
The slowest step. From the top, subtract each step’s children from its own finishing time until you find the step that is slow by itself.
On that step, actual rows against the estimate. A big difference means the planner guessed wrong.
On that step, Rows Removed by Filter and Buffers. Millions removed, or pages near the table’s size, mean a missing index.
Rows handled against rows needed. Thousands sorted for a page of 100 means the query asks for more than the question.

When you are ready for a real one, A Query on a 3-Billion-Row Table Took 100 Minutes reads a production plan end to end the same way.

The ways PostgreSQL reads a table

Every leaf of the tree is one of a short list of scan types, and the name says how much of the table or index PostgreSQL expects to touch. Rows are ordered from cheapest to most expensive. The names were taken from PostgreSQL 18.6 on a small separate schema, since the post’s own fixture does not produce all of them.

What the plan says	What PostgreSQL does	What it usually means
`Index Only Scan using k on t`	Reads the index and never the table, because every column the query needs is in the index	The cheapest way to get rows; `Heap Fetches` above zero means the table had to be checked after all, and `VACUUM` brings it back to zero
`Index Scan using k on t` with `Index Cond`	Walks the index for the matching range, then fetches each row from the table in index order	The normal lookup for a primary key or a small range; expensive when the range is wide, because every row is a separate table read
`Bitmap Index Scan` under a `Bitmap Heap Scan`	Collects every matching row’s location from the index first, then reads the table pages in order, each once	What the planner picks when an index matches many scattered rows; `Recheck Cond` and `Heap Blocks` tell you how much of the table it touched
`BitmapOr` or `BitmapAnd` over two `Bitmap Index Scan`s	Combines the row locations from two indexes before reading the table	An `OR` or `AND` across two indexed columns
`Seq Scan on t` with `Filter`	Reads every row in the table and keeps the ones that pass the filter	The one to look for when a query is slow: `Rows Removed by Filter` close to the table’s size, for a handful of rows kept, means a missing index
`Seq Scan on t` with no `Filter`	Reads the whole table because the query asked for it	Expected under an aggregate or a join that needs every row; it is only a problem if the query did not need every row
`Result` with `One-Time Filter: false`	Reads nothing, because the planner proved no row can match	A condition that contradicts itself

The nodes above the leaves describe what happens to rows once they are read, and three of them decide most of a plan’s cost. Nested Loop runs its second child once for every row from the first, which is where loops on the inner side comes from, and it is right when the outer side is small. Hash Join reads the smaller side once into memory, shown as the Hash node under it, and probes it with every row of the other side; Batches above 1 means the hash did not fit in memory. Merge Join walks two inputs that are already in the same order, usually from an index, and needs no memory at all. Below those, Sort orders rows in memory or on disk, and the line under it says which. HashAggregate groups rows with a hash table and GroupAggregate groups rows that already arrive sorted. Limit stops pulling rows once it has enough. Append glues the parts of a UNION ALL together, and CTE Scan reads a WITH query that was run once and kept.

The test data, if you want to build it yourself on PostgreSQL 18:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


SELECT setseed(0.42);
CREATE TABLE customers (id int PRIMARY KEY, country text NOT NULL,
 city text NOT NULL, created_at timestamptz NOT NULL);
INSERT INTO customers
SELECT g, 'C' || lpad((g % 50)::text, 2, '0'),
 'C' || lpad((g % 50)::text, 2, '0') || '-' || ((g / 50) % 10),
 timestamptz '2022-01-01' + g * interval '7 minutes'
FROM generate_series(1, 200000) g;

CREATE TABLE orders (id bigint PRIMARY KEY, customer_id int NOT NULL,
 status text NOT NULL, created_at timestamptz NOT NULL,
 total numeric(10,2) NOT NULL, note text)
 WITH (autovacuum_enabled = false); -- vacuum runs only when I ask
INSERT INTO orders
SELECT g, 1 + floor(random() * 200000)::int,
 CASE WHEN random() < 0.01 THEN 'pending' ELSE 'shipped' END,
 timestamptz '2024-01-01' + g * interval '6 seconds',
 round((random() * 500)::numeric, 2),
 md5(g::text) || md5((g * 7)::text)
FROM generate_series(1, 5000000) g;
ANALYZE customers;
ANALYZE orders;

Random UUIDs as Primary Keys: The B-Tree Penalty

Fri, 22 Aug 2025 00:00:00 +0000

TL;DR

UUIDv4 primary keys are globally unique and coordination-free, and the cost is paid every time you write a row: random B-tree positions, page splits, secondary indexes bloated with 16- or 36-byte key copies, and a working set that stops fitting in the buffer pool once the table is large enough. UUIDv7 fixes the insert-locality problem (time-ordered, sortable) without changing storage size; the full fix is picking v7, storing as BINARY(16) or native uuid, and keeping UUIDs at the API boundary rather than internal to every join.

A table configured like this on day one looks unremarkable:

1
2
3
4


CREATE TABLE orders (
 id CHAR(36) PRIMARY KEY, -- UUIDv4, generated by the application
 ...
);

Inserts are fast, reads are fast, the ORM is happy. At 100,000 rows, it’s still fine. At 10 million, the nightly ingest job gets noticeably slower. At 200 million, inserts take 50 ms each instead of 2 ms, the buffer pool is constantly churning, and the secondary indexes are three to four times the size they’d be with a BIGINT primary key. Nothing about the schema changed. The table just got large enough for a design decision to start charging rent.

The obvious fix is “use BIGINT auto-increment.” That’s the right answer in a lot of cases and the wrong one in others; it reintroduces coordination requirements, leaks row counts through URL-exposed IDs, and doesn’t work for schemas that need to be generated offline or across shards. UUIDs exist because those constraints are real. The sharper question is: what exactly is UUIDv4 costing you at scale, and which of those costs have cheaper alternatives?

What random keys do to a B-tree

B-tree indexes are sorted structures. When the primary key is an auto-incrementing integer, every new row goes to the end. The rightmost leaf page is the only one that gets written to, and the rest of the index stays in cache undisturbed. Inserts are sequential and cheap.

UUIDv4 is random by design. Every new row lands at a random position in the B-tree. Instead of appending to one page, the engine has to:

Find the right page somewhere in the middle of the tree.
Load it into the buffer pool if it isn’t already (on a large table, it usually isn’t).
Split it if it’s full.
Write both halves back.

On a table with hundreds of millions of rows, the index doesn’t fit in memory, so most inserts trigger a random disk read before they can do anything else. The write amplification is real and measurable: factor of 5 to 10× versus sequential inserts isn’t unusual.

The damage doesn’t stop at the primary-key index. In InnoDB (MySQL), every secondary index includes a copy of the primary key at its leaves. A 36-byte CHAR(36) UUID embedded in every secondary index entry means larger indexes, more pages, more I/O compared to an 8-byte BIGINT. Secondary indexes on a UUID-keyed table are routinely 3–4× the size of the same indexes on a BIGINT-keyed table. Every lookup through a secondary index reads more pages to cover the same rows.

PostgreSQL handles storage differently. Its heap means the primary key is just another index, so the physical table isn’t ordered by it. The primary-key index still suffers the same random-insertion pathology, and the write amplification from random page loads still applies.

Page splits compound over time. When a new UUID lands in a full page, InnoDB splits the page in two, each roughly half full. Over millions of inserts, the index develops internal fragmentation: pages allocated but only partially used. The index is physically larger than it needs to be, and scans read more pages for the same row count. OPTIMIZE TABLE (MySQL) or REINDEX (PostgreSQL) can repack the index, but on a busy table it’s a maintenance window you have to schedule.

UUIDv7: the insert-locality fix

UUIDv7 is the version most new code should reach for when UUIDs are the right answer. It encodes a Unix millisecond timestamp into the high 48 bits, with random bits filling the rest. Two practical consequences:

Sortable. Sequential generation means new IDs land at the end of the B-tree, not scattered across it. Insert locality is close to a BIGINT’s. The pathological page-split behaviour of v4 goes away.
Time-parseable. The creation time is embedded in the ID, recoverable from the primary key alone: useful for log correlation, rough time-range filtering, and debugging without reaching for created_at.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


-- UUIDv7: time-ordered, so inserts are roughly sequential
-- PostgreSQL 18 ships a built-in uuidv7() function
CREATE TABLE orders (
 id UUID PRIMARY KEY DEFAULT uuidv7(),
 ...
);

-- Recover creation time from the ID - no created_at column needed
SELECT id,
 uuid_extract_timestamp(id) AS created_at,
 uuid_extract_timestamp(id)::date AS created_date
FROM orders
ORDER BY id DESC -- v7 sorts chronologically, newest first
LIMIT 10;

uuid_extract_timestamp() has existed in PostgreSQL since 17 but only returned a value for UUIDv1. PG 18 extended it to support v7 alongside the new uuidv7() generator. One caveat: calling it in a WHERE clause (WHERE uuid_extract_timestamp(id) >= '2026-04-01') is non-SARGable and forces a scan; see Non-SARGable Predicates. For indexed time-range filtering, keep a created_at column as the query target, or compare against a boundary UUID generated at the target timestamp.

MySQL 8 doesn’t ship a v7 generator or a timestamp extractor, so application-side generation is the norm there - libraries exist in every major language, and most modern ORMs default to v7 if you ask for UUIDs. Extraction is manual: for BINARY(16) storage (the recommended form), the first 6 bytes hold the millisecond timestamp.

1
2
3
4
5
6
7


-- MySQL: manually parse v7's timestamp prefix (BINARY(16) storage)
SELECT id,
 FROM_UNIXTIME(CONV(HEX(SUBSTRING(id, 1, 6)), 16, 10) / 1000) AS created_at,
 DATE(FROM_UNIXTIME(CONV(HEX(SUBSTRING(id, 1, 6)), 16, 10) / 1000)) AS created_date
FROM orders
ORDER BY id DESC -- v7 sorts chronologically
LIMIT 10;

For CHAR(36) storage, the extraction strips hyphens first: CONCAT(SUBSTRING(id, 1, 8), SUBSTRING(id, 10, 4)) gives the 12 hex characters of the timestamp prefix. If your v1 UUIDs were stored with UUID_TO_BIN(id, 1) (the swap flag that reorders bytes for v1 index locality), the byte layout differs and the substring offsets change. Most v7-generating libraries skip the swap because v7 is already time-ordered without it - check what yours does before trusting the extraction.

What v7 doesn’t change. It’s still 16 bytes on disk, and still 36 if you stored it as CHAR(36). The insert-locality win doesn’t come with a storage discount, so the overhead versus a BIGINT is the same as v4. The readable creation timestamp is usually a feature and occasionally a problem: in systems where row-creation time is sensitive (order IDs revealing traffic patterns to competitors, user IDs exposing signup timing), it’s the one property v4 had that v7 gives up.

CHAR(36) is the silent tax

The worst-case UUID storage, CHAR(36), is what most ORM-generated schemas default to, because it’s the portable representation. BINARY(16) in MySQL or the native uuid type in PostgreSQL cuts storage by more than half and keeps comparisons on fixed-width integers instead of strings. Pick the narrow form on day one; retrofitting it later is a full-table rewrite that touches every secondary index.

UUID-to-integer mapping: keep UUIDs at the edge

The other workable fix is structural: expose UUIDs externally, use integers internally. A single lookup table maps the external UUID to an internal BIGINT, and every other table in the database uses the BIGINT as its foreign key. The UUID lookup happens once (at the API boundary) and everything downstream is fast, compact, 8-byte integer joins.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


CREATE TABLE users (
 id BIGINT AUTO_INCREMENT PRIMARY KEY,
 external_id BINARY(16) NOT NULL UNIQUE, -- the UUID the outside world sees
 ...
);

-- Every other table references the BIGINT, not the UUID
CREATE TABLE orders (
 id BIGINT AUTO_INCREMENT PRIMARY KEY,
 user_id BIGINT NOT NULL REFERENCES users(id),
 ...
);

-- API request comes in with a UUID; one indexed lookup to resolve it
SELECT id FROM users WHERE external_id = UUID_TO_BIN('a1b2c3d4-...');
-- From here on, everything uses the BIGINT

The UUID column has a unique index, so the lookup is a single index seek, sub-millisecond regardless of table size. The rest of the schema gets 8-byte keys everywhere: smaller indexes, faster joins, no page splits, no secondary-index bloat. The external-facing API still uses UUIDs, so you don’t leak sequence information or row counts.

The trade-off is an extra layer of indirection. Every inbound request resolves the UUID before anything else; in practice this is negligible (one indexed lookup), but it means the schema has two identity systems to maintain. For long-lived OLTP applications where every join on every table pays the UUID cost, this structure is often worth the extra lookup.

When random UUIDs are actually fine

Not every schema needs to bend. Three cases where UUIDv4 as a primary key is a defensible choice:

Small tables that stay small. A configuration table, a lookup table, a feature-flag table. At 50,000 rows the page-split pathology doesn’t show up, secondary indexes are tiny, and the convenience of client-generated IDs outweighs any cost.

Write rates low enough that random I/O doesn’t matter. An admin tool recording 50 events per minute doesn’t care about write amplification. The index fits in cache, every page is warm, page splits happen rarely enough that fragmentation stays manageable. “Doesn’t survive scale” is only a problem at scale.

Information-leak concerns that outweigh performance. If hiding creation-order is a hard requirement (competitive, privacy, or security), v7’s embedded timestamp is a non-starter and v4 is the only UUID version that meets the requirement. Pay the write-amplification cost and use the UUID-to-integer mapping to contain the damage.

Why v4 keeps showing up as the default

Schema-reading assistants and scaffolding tools reinforce UUIDv4 as the default answer, and the reason is mostly inertial. Training corpora are heavy on examples where uuid4() is the canonical “globally unique ID” call; CREATE TABLE ... id UUID DEFAULT gen_random_uuid() appears in orders of magnitude more tutorials than the v7 equivalent. Asked for a new table schema, a model produces the v4 version because that’s what the surrounding code it learned from produced. B-tree locality and write amplification don’t show up in the DDL (they’re runtime properties of the key distribution) so the catalog gives no signal that v4 and v7 behave differently at 100M rows. Both look identical in information_schema: uuid or char(36), primary key, not null.

The fix is the same discipline this post already makes the case for, with one amplified beat: document the choice where a schema reader can find it. A comment on the PK column ('UUIDv7: time-ordered; required for insert locality' or 'UUIDv4: randomized; chosen to hide creation order') turns a silent convention into a machine-readable decision. The next reader (teammate or model) sees why the column is what it is and why the alternative was rejected. Without the comment, the next table scaffolded by an assistant inherits whichever version the training data sampled, and the schema drifts toward whichever default produces the most plausible-looking DDL.

The bigger picture

UUIDv4 is a tool that solved a coordination problem (distributed ID generation without central authority) and accidentally became the default for everything, including the cases where coordination wasn’t a problem and the cost of random writes is non-trivial. “Pick a UUID for your PK” is a decision most schemas make without ever being explicit about what they’re trading.

The decision matrix is short. Do you need globally unique, coordination-free IDs? If no, use BIGINT. If yes, use UUIDv7 and store it as BINARY(16) or native uuid, never CHAR(36). If v7’s embedded timestamp is a problem, use v4 but keep it at the API boundary and use integers inside the schema. Each of those decisions costs almost nothing on day one and saves a lot of rework at 100 million rows.

Database Deadlocks, Part 1: The Patterns

Thu, 13 Feb 2025 00:00:00 +0000

TL;DR

A deadlock is two transactions each holding a lock the other needs, caught in a cycle the engine breaks by killing one. The patterns are finite and repeatable: inconsistent lock ordering across workers, InnoDB gap locks under REPEATABLE READ, foreign-key shared locks on hot parent rows, unique-index conflicts, index-scan lock amplification, and parallelism patterns that only surface on replicas or under worker-pool load. This post is the patterns. Part 2 covers diagnosis, retry architecture, and prevention.

Deadlocks occupy a strange place in production operations. They’re rare enough that most engineers haven’t thought hard about them, frequent enough in high-concurrency workloads to show up as paging incidents, and subtle enough that the first instinct (“just retry”) is right often enough to keep the root cause hidden. The transaction that got killed was syntactically perfect. The one that survived was too. The bug wasn’t in either statement; it was in the order the two transactions touched rows.

That makes deadlocks harder to reason about than most database failures. The query text in the error log isn’t wrong. The lock it was waiting for isn’t held by a misbehaving process. The system is doing exactly what concurrency control says it should. The failure mode is the interaction between transactions, and those interactions are almost never visible from any single query.

This post covers the patterns: the shapes deadlocks take and why each one exists. The companion post covers reading the deadlock log end-to-end, retry architecture, hot-row isolation, SERIALIZABLE’s serialization failures, DDL migration windows, and NOWAIT / SKIP LOCKED as prevention primitives.

What a deadlock actually is

A deadlock is a cycle in the wait-for graph. Transaction A holds lock L1 and needs L2; transaction B holds L2 and needs L1. Neither can proceed. The only way out is to kill one: pick a victim, roll it back, release its locks, and let the other complete. Every modern relational database does this automatically, usually within hundreds of milliseconds.

The three preconditions are always the same:

Two or more transactions hold locks.
Each needs a lock the other holds.
The locks can’t be acquired atomically (there’s no single “grab both or grab nothing” operation).

Remove any one and the deadlock can’t form. In practice, that’s the shape of every prevention strategy: reduce the number of locks held concurrently, reduce the duration they’re held, or make the acquisition order consistent across all code paths.

Deadlock vs. lock wait timeout

A deadlock is a cycle. A lock wait timeout is a long queue: transaction A is waiting for transaction B, which is waiting for something reasonable, which is taking too long. No cycle, no victim selection, just a timer expiring. Both produce errors that look similar in application logs, but they’re entirely different failure modes with different fixes. innodb_lock_wait_timeout (MySQL) and lock_timeout (PostgreSQL) govern the second. The deadlock detector is a separate mechanism that fires independently of those timers.

The canonical lock-ordering deadlock

The single most common production deadlock is two transactions updating the same two rows in opposite orders:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


-- Transaction A
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

-- Transaction B (concurrent)
BEGIN;
UPDATE accounts SET balance = balance - 50 WHERE id = 2;
UPDATE accounts SET balance = balance + 50 WHERE id = 1;
COMMIT;

If A and B interleave such that A takes the row-level lock on row 1, B takes the row-level lock on row 2, and each then tries to grab the other row, the engine has a cycle. One gets killed.

sequenceDiagram
 participant A as Transaction A
 participant R1 as Row id=1
 participant R2 as Row id=2
 participant B as Transaction B

 Note over A,B: t=0, both transactions begin
 A->>R1: UPDATE (acquire X-lock)
 R1-->>A: granted
 B->>R2: UPDATE (acquire X-lock)
 R2-->>B: granted

 Note over A,B: t=1, each reaches for the other's row
 A->>R2: UPDATE (request X-lock)
 R2--xA: BLOCKED (held by B)
 B->>R1: UPDATE (request X-lock)
 R1--xB: BLOCKED (held by A)

 Note over A,B: Wait-for graph has a cycle: A → B → A
 Note over A,B: Detector fires; victim: Transaction B
 B->>B: ROLLBACK, release R2 lock
 R2-->>A: now granted
 A->>A: COMMIT

The key property is that neither transaction is wrong in isolation. Each acquires locks in an order that’s locally correct. The cycle forms in the global ordering across concurrent sessions, which no single query can see. That’s the defining shape of the pattern: correct code, in both cases, interacting at the transaction boundary. The fix is in how all code paths agree on an ordering, covered in Part 2: Consistent lock ordering.

InnoDB gap locks turn inserts into deadlock sources

MySQL’s default isolation level is REPEATABLE READ, and under that isolation level, InnoDB takes next-key locks: a row lock plus a gap lock on the range before it. The gap lock prevents other transactions from inserting into that range, which is how REPEATABLE READ keeps range queries consistent across re-execution.

The consequence: a SELECT ... FOR UPDATE or an UPDATE with a range predicate locks not just the matching rows, but the gaps between them. Two concurrent transactions that both try to insert into the same gap can deadlock without sharing a single row:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


-- Table: orders(id BIGINT PK, customer_id BIGINT, amount_cents BIGINT)
-- Index on customer_id
-- Existing rows: customer_id = 5 has orders with ids 100, 200, 300

-- Transaction A
BEGIN;
SELECT * FROM orders WHERE customer_id = 5 FOR UPDATE;
-- Takes next-key locks covering ids 100, 200, 300 AND the gaps between them,
-- AND the gap after 300 extending to the next customer_id.

-- Transaction B (concurrent)
BEGIN;
INSERT INTO orders (id, customer_id, amount_cents) VALUES (250, 5, 10000);
-- Blocks: tries to insert into a gap A has locked.

-- Transaction A
INSERT INTO orders (id, customer_id, amount_cents) VALUES (150, 5, 5000);
-- Deadlock if B has also started acquiring locks that A now needs.

Two transactions inserting into what looks like “different rows” can still cycle through gap locks. The failure mode is especially insidious because the EXPLAIN plan doesn’t mention gaps - only rows - and the lock information in SHOW ENGINE INNODB STATUS requires reading the next-key notation carefully.

The category is peculiar to InnoDB under REPEATABLE READ. PostgreSQL prevents phantom reads through MVCC snapshot isolation starting at its own REPEATABLE READ level (stricter than the SQL standard requires) without any range-locking mechanism, so the whole class of gap-lock deadlocks doesn’t exist on PG, at any isolation level. Under READ COMMITTED on MySQL, gap locks are disabled for most searches and index scans but retained for foreign-key and duplicate-key checking, which is the first lever most teams reach for once this pattern dominates their deadlocks, though it doesn’t eliminate the gap-lock category entirely. The isolation-level trade-off and the “avoid range locks on write paths” refactor both live in Part 2: Isolation-level trade-offs.

Unique-index deadlocks are a category of their own

The detailed patterns are covered in Uniqueness and Selectivity, but the shape worth naming here: when InnoDB detects a duplicate-key error on an INSERT, it acquires a shared lock on the conflicting index record before raising the error. Under REPEATABLE READ that shared lock is next-key (record + gap). Under READ COMMITTED, gap locks are mostly disabled, but duplicate-key checking is one of the documented exceptions where gap locking still occurs, so dropping isolation alone doesn’t eliminate the category. Two concurrent transactions inserting toward the same unique key end up holding shared locks and waiting for each other to release: a deadlock caused entirely by the uniqueness check, not by the rows the application thought it was writing.

INSERT ... ON DUPLICATE KEY UPDATE behaves differently: on conflict it takes an exclusive lock instead of a shared one, because the statement is about to modify the row. This matters for reasoning about cycles. Two concurrent ODKU statements contend on exclusive locks (mutually exclusive, one always waits), whereas two concurrent plain INSERTs can both hold shared locks at once and then deadlock when either tries to upgrade. Blog posts and older documentation sometimes conflate the two; the locking rules are documented in the MySQL reference: Locks Set by Different SQL Statements in InnoDB.

The equivalent in PostgreSQL is less severe (the duplicate-key check doesn’t hold long-lived shared locks the same way) but INSERT ... ON CONFLICT with multiple unique indexes can still produce deadlocks when batches touch overlapping keys in different orders. The shape is the same across engines: the uniqueness check itself is what forces the extra locking, and the cycle forms when two sessions approach the same key from different batches.

Foreign keys take shared locks you didn’t ask for

Both MySQL and PostgreSQL acquire shared locks on the referenced row when you insert or update a row with a foreign key. The purpose is to prevent the referenced row from being deleted mid-transaction; you can’t have an orders.customer_id pointing to a customers.id that’s being concurrently deleted.

The side effect is that a high-write child table concentrates shared locks on hot parent rows:

1
2
3
4
5


-- customers has id = 42 (a frequently-used customer)
-- Many concurrent transactions inserting orders for customer 42:

INSERT INTO orders (customer_id, amount_cents) VALUES (42, 1000);
-- Takes a shared lock on customers(id=42)

Shared locks don’t block each other, so concurrent inserts coexist fine. What breaks is the interaction with any transaction that wants an exclusive lock on the parent row: an update to the customer’s name, a soft-delete, a trigger that updates a cached counter. Suddenly, dozens of shared-lock holders are blocking one exclusive-lock request, and if any of them start trying to acquire other locks (say, through a trigger cascade), a cycle can form.

The symptoms: deadlocks that mention tables far removed from the one the application thought it was touching. “Why is my UPDATE customers deadlocking against an INSERT INTO order_items?” Because the order_items insert took a shared lock on customers through the FK chain, and the UPDATE wanted exclusive on the same row.

This is one of the hardest patterns to diagnose on sight, because the offending query never references the contended table explicitly. Mitigations (narrowing FK scope, moving hot parent-row updates to side tables, isolation-level trade-offs) are covered in Part 2: Hot-row isolation.

Index scans lock more rows than queries return

Under InnoDB’s default REPEATABLE READ, an UPDATE with a WHERE clause on a non-indexed column acquires a record lock on every row it scans, not just the ones that match. The engine has to examine each row to check the predicate, and it takes a lock to guarantee the check is stable for the duration of the transaction.

1
2
3


-- Without an index on status, under REPEATABLE READ:
UPDATE orders SET priority = 'high' WHERE status = 'pending';
-- Locks every row in orders during the scan.

If the table has a million rows and only a thousand match, all million get locked for the duration of the update. Any concurrent transaction touching any of those rows has to wait, which inflates the wait-for graph and makes deadlocks more likely.

Under READ COMMITTED, InnoDB narrows this substantially: per the docs, it releases locks on non-matching rows after the WHERE evaluation and uses semi-consistent reads, returning the latest committed version of an already-locked row so the engine can check whether it matches the WHERE before deciding to wait. The net effect is much lower lock footprint and deadlock risk on the same query. PostgreSQL behaves similarly by default: only rows actually updated retain their locks. This is one of the few cases where the same underlying issue (an unindexed predicate) shows up as both a latency problem and a concurrency problem, and where the concurrency angle is specifically a REPEATABLE READ-on-InnoDB amplifier.

Secondary index locks on InnoDB

InnoDB takes locks on both the clustered index (primary key) and any secondary indexes touched by the query. A WHERE status = 'pending' using a status index locks the relevant index entries and the corresponding PK entries. Transactions that approach the same rows from different indexes (one via status, another via customer_id) can deadlock on the PK-side lock even though their index-side locks don’t overlap. This is the most common “why are these two queries deadlocking, they don’t even reference the same columns?” failure mode.

Parallelism-induced deadlocks

The lock-ordering patterns above assume two separate transactions from two separate sessions. Parallelism adds a few variants that don’t fit that frame; the cycle can form inside a single logical unit of work, or show up only on a replica that never issued the original statements.

Worker pools racing on a shared queue. The archetypal production pattern: N application workers pulling jobs from the same table (jobs, outbox, email_queue) and locking rows for processing. If every worker does SELECT ... FOR UPDATE on “the next available batch” without a deterministic ordering, two workers can grab overlapping row sets in opposite orders and cycle. This is the same lock-ordering cycle from earlier, distributed across workers that all look identical from a code-review perspective.

Intra-query parallel workers. PostgreSQL has a full parallel query executor (parallel sequential scans, bitmap heap scans, index and index-only scans (B-tree), parallel aggregates, parallel joins) that spawns worker processes to cooperate on a single query. MySQL has a much narrower feature: innodb_parallel_read_threads (added in 8.0.14) enables parallel scanning of the clustered index, used initially by CHECK TABLE and extended to unconditional SELECT COUNT(*) in 8.0.17. It is not general parallel query; MySQL does not parallelize arbitrary SELECTs, joins, or aggregates. In both engines, workers coordinating on a single query don’t deadlock among themselves in normal operation; the engine manages the shared lock state. What can happen is a parallel worker holds a lock an unrelated transaction needs, and the parallel query itself takes longer than a serial one would, widening the wait window. Usually not a direct deadlock source, but it changes the timing of existing ones.

Parallel replication on replicas. MySQL’s multi-threaded replica applies committed transactions in parallel. Transactions that committed serially on the source (no possibility of deadlock there) can deadlock on the replica because the applier threads are racing on rows the source never had concurrent writers on. The replica’s deadlock detector resolves them the same way it would a live deadlock, but they show up in the replica’s error log with no corresponding entry on the source. Since MySQL 8.0.27, replica_parallel_type=LOGICAL_CLOCK and replica_parallel_workers=4 are the defaults, and replica_parallel_type was deprecated in 8.0.29; LOGICAL_CLOCK is effectively the only supported mode going forward. The slave_* → replica_* rename happened in 8.0.26; older deployments and blog posts still use the legacy names. PostgreSQL 16+ introduced parallel apply for logical replication (streaming = parallel is the default on CREATE SUBSCRIPTION), which exposes the same class of apply-side cycles on a setup that historically didn’t have them: a surprise for teams upgrading from 15 and earlier.

Parallel/online DDL interacting with DML. Tools like pt-online-schema-change and gh-ost run concurrent DML against the table being altered (through triggers or a row-copy process). Under load, the trigger-installed writes and the copy process can both take locks on the same rows the application is updating, and the wait-for graph gains edges that wouldn’t exist during a normal workload. This rarely manifests as a hard deadlock (the tools are written defensively) but it does manifest as elevated deadlock rates during the migration window.

None of these are properties of the queries themselves. They’re properties of how work gets distributed across workers, processes, or replicas, which means they’re invisible to query-level review and only surface when the deadlock counter is watched over time. The primitives for fixing them (SKIP LOCKED, NOWAIT, advisory locks, DDL timeouts) are covered in Part 2: NOWAIT and SKIP LOCKED.

Engine-level differences that shape the patterns

The same pattern can deadlock on one engine and not the other. These differences are pattern-shaping; they change which of the above sections apply to your workload. Operational tuning (detector cost, wait timeouts, monitoring) is covered in Part 2: Monitoring.

Default isolation. PostgreSQL defaults to READ COMMITTED. MySQL defaults to REPEATABLE READ (with gap locks). The same application code has measurably different deadlock rates between the two because of this alone, before any other tuning.
Gap locks. Only InnoDB has them, and only under REPEATABLE READ (plus the foreign-key and duplicate-key exceptions that retain gap locking even under READ COMMITTED). PostgreSQL prevents phantom reads through MVCC at its own REPEATABLE READ (stricter than the SQL standard requires) without a range-locking mechanism, so the entire gap-lock deadlock category doesn’t exist on PG at any isolation level.
Lock granularity. PostgreSQL takes row-level (tuple) locks; InnoDB takes record locks on index entries (with next-key extension under REPEATABLE READ). The practical consequence is that InnoDB locks are more entangled with index choice than PostgreSQL’s; changing which index a query uses can change which rows and gaps get locked.
FK lock style. MySQL’s FK check holds a shared lock on the referenced row (next-key under REPEATABLE READ, and the docs list FK checking as one of the places gap locks persist even under READ COMMITTED). PostgreSQL takes a FOR KEY SHARE lock (added in 9.3 specifically to reduce FK lock contention vs. the older FOR SHARE). Hot parent rows are more contended under MySQL as a result.
Row-lock visibility. PostgreSQL row-level locks don’t show up in pg_locks. Per the docs, they’re stored on the tuple header on disk, not in shared memory. A process waiting for a row lock usually appears in pg_locks as waiting for the holder’s transaction ID, not the row. InnoDB’s performance_schema.data_locks exposes row-level lock state directly. More on this in Part 2.

Neither engine is “better.” The behaviors are different, and code that assumes one can deadlock mysteriously when moved to the other.

Why schema-reading assistants hit these patterns

Locking behavior has no syntax in the query text. A SELECT ... FOR UPDATE advertises the intent; a plain INSERT ... ON DUPLICATE KEY UPDATE or INSERT ... ON CONFLICT doesn’t. The shared next-key lock on a duplicate-key violation, the FK shared lock on the parent row, the gap-lock extension under MySQL’s default isolation are all implementation details of the storage engine. Schema-reading assistants read the catalog, which describes tables, columns, and constraints, and the codebase, which describes queries. Neither surfaces lock ordering, gap-lock semantics, or the difference between READ COMMITTED and REPEATABLE READ unless the prompt specifically includes them.

That’s why AI-generated UPSERT and batch-insert code deadlocks in production the way it does. The model reads INSERT ... ON DUPLICATE KEY UPDATE as an atomic upsert, not as “takes a shared next-key lock, possibly including a gap, before raising the duplicate-key error that the application will retry.” It generates batch INSERTs that process rows in whatever order the application supplies, not sorted by key: fine for a single writer, a lock-ordering cycle under any realistic concurrency. The patterns above are the ones the catalog and query text can’t warn about, and they’re the ones that arrive in production as “intermittent deadlocks under load” after passing every test that didn’t include a second concurrent worker. The fix lives one level up from the query (sorted batches, explicit lock ordering, retry loops) which is the subject of Part 2.

What’s in Part 2

The patterns are the first half. Turning them into working systems takes a different set of skills: reading the deadlock log to identify which pattern is firing, building retry logic that doesn’t mask the real bug, isolating hot rows before they become incident reports, and choosing the right tool for each (NOWAIT, SKIP LOCKED, advisory locks, counter tables, or the isolation-level change that eliminates the category entirely). PostgreSQL’s SERIALIZABLE/SSI produces serialization failures that look like deadlocks but aren’t; the difference matters for retry architecture. AUTO_INCREMENT and sequence-related locking have their own failure modes. DDL migrations on both engines introduce lock queues that manifest as deadlock-like incidents.

All of that is in Database Deadlocks, Part 2: Diagnosis, Retries, and Prevention.

Mental model for the patterns

Deadlocks are what consistent concurrency control does when two transactions make the engine choose between them. The database isn’t misbehaving; it’s refusing to let both of two contradictory orderings win. The error in the application log is a notification, not a fault.

That makes the diagnostic question concrete. Which pattern is firing? Every deadlock in production fits one of the shapes above: lock-ordering cycle, gap lock on a range, duplicate-key shared lock, FK shared lock on a hot parent, unindexed predicate lock amplification, worker-pool race, or replication-apply cycle. Identifying the pattern from the deadlock log narrows the fix enormously. “Two transactions deadlocked, retry the transaction” is true but useless. “Two workers took locks on the same jobs queue in different orders, switch to SKIP LOCKED” is a fix. The work is in the identification.