Concurrency on EXPLAIN ANALYZE

Database Deadlocks, Part 1: The Patterns

Sun, 19 Apr 2026 00:00:00 +0000

TL;DR

A deadlock isn’t a database bug — it’s two transactions each holding a lock the other needs, caught in a cycle the engine has to break by killing one. The patterns are finite and repeatable: inconsistent lock ordering across workers, InnoDB gap locks under REPEATABLE READ, foreign-key shared locks on hot parent rows, unique-index conflicts, index scans that lock more rows than they return, and parallelism patterns that only surface on replicas or under worker-pool load. This is Part 1 — the patterns themselves and why they happen. Part 2 covers diagnosis, retry architecture, and prevention.

Deadlocks occupy a strange place in production operations. They’re rare enough that most engineers haven’t thought hard about them, frequent enough in high-concurrency workloads to show up as paging incidents, and subtle enough that the first instinct — “just retry” — is right often enough to keep the root cause hidden. The transaction that got killed was syntactically perfect. The one that survived was too. The bug wasn’t in either statement; it was in the order the two transactions touched rows.

That makes deadlocks harder to reason about than most database failures. The query text in the error log isn’t wrong. The lock it was waiting for isn’t held by a misbehaving process. The system is doing exactly what concurrency control says it should. The failure mode is the interaction between transactions, and those interactions are almost never visible from any single query.

This post covers the patterns — the shapes deadlocks take and why each one exists. The companion post covers reading the deadlock log end-to-end, retry architecture, hot-row isolation, SERIALIZABLE’s serialization failures, DDL migration windows, and NOWAIT / SKIP LOCKED as prevention primitives.

What a deadlock actually is

A deadlock is a cycle in the wait-for graph. Transaction A holds lock L1 and needs L2; transaction B holds L2 and needs L1. Neither can proceed. The only way out is to kill one — pick a victim, roll it back, release its locks, and let the other complete. Every modern relational database does this automatically, usually within hundreds of milliseconds.

The three preconditions are always the same:

Two or more transactions hold locks.
Each needs a lock the other holds.
The locks can’t be acquired atomically (there’s no single “grab both or grab nothing” operation).

Remove any one and the deadlock can’t form. In practice, that’s the shape of every prevention strategy — either reduce the number of locks held concurrently, reduce the duration they’re held, or make the acquisition order consistent across all code paths.

Deadlock vs. lock wait timeout

A deadlock is a cycle. A lock wait timeout is a long queue — transaction A is waiting for transaction B, which is waiting for something reasonable, which is taking too long. No cycle, no victim selection, just a timer expiring. Both produce errors that look similar in application logs, but they’re entirely different failure modes with different fixes. innodb_lock_wait_timeout (MySQL) and lock_timeout (PostgreSQL) govern the second. The deadlock detector is a separate mechanism that fires independently of those timers.

The canonical lock-ordering deadlock

The single most common production deadlock is two transactions updating the same two rows in opposite orders:

-- Transaction A
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

-- Transaction B (concurrent)
BEGIN;
UPDATE accounts SET balance = balance - 50 WHERE id = 2;
UPDATE accounts SET balance = balance + 50 WHERE id = 1;
COMMIT;

If A and B interleave such that A takes the row-level lock on row 1, B takes the row-level lock on row 2, and each then tries to grab the other row, the engine has a cycle. One gets killed.

sequenceDiagram
    participant A as Transaction A
    participant R1 as Row id=1
    participant R2 as Row id=2
    participant B as Transaction B

    Note over A,B: t=0 — both transactions begin
    A->>R1: UPDATE (acquire X-lock)
    R1-->>A: granted
    B->>R2: UPDATE (acquire X-lock)
    R2-->>B: granted

    Note over A,B: t=1 — each reaches for the other's row
    A->>R2: UPDATE (request X-lock)
    R2--xA: BLOCKED (held by B)
    B->>R1: UPDATE (request X-lock)
    R1--xB: BLOCKED (held by A)

    Note over A,B: Wait-for graph has a cycle: A → B → A
    Note over A,B: Detector fires — victim: Transaction B
    B->>B: ROLLBACK, release R2 lock
    R2-->>A: now granted
    A->>A: COMMIT

The key property is that neither transaction is wrong in isolation. Each acquires locks in an order that’s locally correct. The cycle forms in the global ordering across concurrent sessions, which no single query can see. That’s the defining shape of the pattern: correct code, in both cases, interacting at the transaction boundary. The fix is in how all code paths agree on an ordering — covered in Part 2: Consistent lock ordering.

InnoDB gap locks turn inserts into deadlock sources

MySQL’s default isolation level is REPEATABLE READ, and under that isolation level, InnoDB takes next-key locks — a row lock plus a gap lock on the range before it. The gap lock prevents other transactions from inserting into that range, which is how REPEATABLE READ keeps range queries consistent across re-execution.

The consequence: a SELECT ... FOR UPDATE or an UPDATE with a range predicate locks not just the matching rows, but the gaps between them. Two concurrent transactions that both try to insert into the same gap can deadlock without sharing a single row:

-- Table: orders(id BIGINT PK, customer_id BIGINT, amount_cents BIGINT)
-- Index on customer_id
-- Existing rows: customer_id = 5 has orders with ids 100, 200, 300

-- Transaction A
BEGIN;
SELECT * FROM orders WHERE customer_id = 5 FOR UPDATE;
-- Takes next-key locks covering ids 100, 200, 300 AND the gaps between them,
-- AND the gap after 300 extending to the next customer_id.

-- Transaction B (concurrent)
BEGIN;
INSERT INTO orders (id, customer_id, amount_cents) VALUES (250, 5, 10000);
-- Blocks: tries to insert into a gap A has locked.

-- Transaction A
INSERT INTO orders (id, customer_id, amount_cents) VALUES (150, 5, 5000);
-- Deadlock if B has also started acquiring locks that A now needs.

Two transactions inserting into what looks like “different rows” can still cycle through gap locks. The failure mode is especially insidious because the EXPLAIN plan doesn’t mention gaps — only rows — and the lock information in SHOW ENGINE INNODB STATUS requires reading the next-key notation carefully.

The category is peculiar to InnoDB under REPEATABLE READ. PostgreSQL prevents phantom reads through MVCC snapshot isolation starting at its own REPEATABLE READ level (stricter than the SQL standard requires) without any range-locking mechanism — so the whole class of gap-lock deadlocks doesn’t exist on PG, at any isolation level. Under READ COMMITTED on MySQL, gap locks are disabled for most searches and index scans but retained for foreign-key and duplicate-key checking — which is the first lever most teams reach for once this pattern dominates their deadlocks, though it doesn’t eliminate the gap-lock category entirely. The isolation-level trade-off and the “avoid range locks on write paths” refactor both live in Part 2: Isolation-level trade-offs.

Unique-index deadlocks are a category of their own

The detailed patterns are covered in Uniqueness and Selectivity — Unique indexes concentrate deadlock pressure, but the shape worth naming here: when InnoDB detects a duplicate-key error on an INSERT, it acquires a shared lock on the conflicting index record before raising the error. Under REPEATABLE READ that shared lock is next-key (record + gap). Under READ COMMITTED, gap locks are mostly disabled — but duplicate-key checking is one of the documented exceptions where gap locking still occurs, so dropping isolation alone doesn’t eliminate the category. Two concurrent transactions inserting toward the same unique key end up holding shared locks and waiting for each other to release — a deadlock caused entirely by the uniqueness check, not by the rows the application thought it was writing.

INSERT ... ON DUPLICATE KEY UPDATE behaves differently: on conflict it takes an exclusive lock instead of a shared one, because the statement is about to modify the row. This matters for reasoning about cycles — two concurrent ODKU statements contend on exclusive locks (mutually exclusive, one always waits), whereas two concurrent plain INSERTs can both hold shared locks at once and then deadlock when either tries to upgrade. Blog posts and older documentation sometimes conflate the two; the locking rules are documented in the MySQL reference — Locks Set by Different SQL Statements in InnoDB.

The equivalent in PostgreSQL is less severe — the duplicate-key check doesn’t hold long-lived shared locks the same way — but INSERT ... ON CONFLICT with multiple unique indexes can still produce deadlocks when batches touch overlapping keys in different orders. The shape is the same across engines: the uniqueness check itself is what forces the extra locking, and the cycle forms when two sessions approach the same key from different batches.

Foreign keys take shared locks you didn’t ask for

Both MySQL and PostgreSQL acquire shared locks on the referenced row when you insert or update a row with a foreign key. The purpose is to prevent the referenced row from being deleted mid-transaction — you can’t have an orders.customer_id pointing to a customers.id that’s being concurrently deleted.

The side effect is that a high-write child table concentrates shared locks on hot parent rows:

-- customers has id = 42 (a frequently-used customer)
-- Many concurrent transactions inserting orders for customer 42:

INSERT INTO orders (customer_id, amount_cents) VALUES (42, 1000);
-- Takes a shared lock on customers(id=42)

Shared locks don’t block each other, so concurrent inserts coexist fine. What breaks is the interaction with any transaction that wants an exclusive lock on the parent row — an update to the customer’s name, a soft-delete, a trigger that updates a cached counter. Suddenly, dozens of shared-lock holders are blocking one exclusive-lock request, and if any of them start trying to acquire other locks (say, through a trigger cascade), a cycle can form.

The symptoms: deadlocks that mention tables far removed from the one the application thought it was touching. “Why is my UPDATE customers deadlocking against an INSERT INTO order_items?” Because the order_items insert took a shared lock on customers through the FK chain, and the UPDATE wanted exclusive on the same row.

This is one of the hardest patterns to diagnose on sight, because the offending query never references the contended table explicitly. Mitigations — narrowing FK scope, moving hot parent-row updates to side tables, isolation-level trade-offs — are covered in Part 2: Hot-row isolation.

Index scans lock more rows than queries return

Under InnoDB’s default REPEATABLE READ, an UPDATE with a WHERE clause on a non-indexed column acquires a record lock on every row it scans, not just the ones that match. The engine has to examine each row to check the predicate, and it takes a lock to guarantee the check is stable for the duration of the transaction.

1
2
3

-- Without an index on status, under REPEATABLE READ:
UPDATE orders SET priority = 'high' WHERE status = 'pending';
-- Locks every row in orders during the scan.

If the table has a million rows and only a thousand match, all million get locked for the duration of the update. Any concurrent transaction touching any of those rows has to wait — which inflates the wait-for graph and makes deadlocks more likely.

Under READ COMMITTED, InnoDB narrows this substantially: per the docs, it releases locks on non-matching rows after the WHERE evaluation and uses semi-consistent reads — returning the latest committed version of an already-locked row so the engine can check whether it matches the WHERE before deciding to wait. The net effect is dramatically lower lock footprint and deadlock risk on the same query. PostgreSQL behaves similarly by default: only rows actually updated retain their locks. This is one of the few cases where the same underlying issue — an unindexed predicate — shows up as both a latency problem and a concurrency problem, and where the concurrency angle is specifically a REPEATABLE READ-on-InnoDB amplifier.

Secondary index locks on InnoDB

InnoDB takes locks on both the clustered index (primary key) and any secondary indexes touched by the query. A WHERE status = 'pending' using a status index locks the relevant index entries and the corresponding PK entries. Transactions that approach the same rows from different indexes (one via status, another via customer_id) can deadlock on the PK-side lock even though their index-side locks don’t overlap. This is the most common “why are these two queries deadlocking, they don’t even reference the same columns?” failure mode.

Parallelism-induced deadlocks

The lock-ordering patterns above assume two separate transactions from two separate sessions. Parallelism adds a few variants that don’t fit that frame — the cycle can form inside a single logical unit of work, or show up only on a replica that never issued the original statements.

Worker pools racing on a shared queue. The archetypal production pattern: N application workers pulling jobs from the same table (jobs, outbox, email_queue) and locking rows for processing. If every worker does SELECT ... FOR UPDATE on “the next available batch” without a deterministic ordering, two workers can grab overlapping row sets in opposite orders and cycle. This is the same lock-ordering cycle from earlier — just distributed across workers that all look identical from a code-review perspective.

Intra-query parallel workers. PostgreSQL has a full parallel query executor — parallel sequential scans, bitmap heap scans, index and index-only scans (B-tree), parallel aggregates, parallel joins — that spawns worker processes to cooperate on a single query. MySQL has a much narrower feature: innodb_parallel_read_threads (added in 8.0.14) enables parallel scanning of the clustered index, used initially by CHECK TABLE and extended to unconditional SELECT COUNT(*) in 8.0.17. It is not general parallel query — MySQL does not parallelize arbitrary SELECTs, joins, or aggregates. In both engines, workers coordinating on a single query don’t deadlock among themselves in normal operation — the engine manages the shared lock state. What can happen is a parallel worker holds a lock an unrelated transaction needs, and the parallel query itself takes longer than a serial one would, widening the wait window. Usually not a direct deadlock source, but it changes the timing of existing ones.

Parallel replication on replicas. MySQL’s multi-threaded replica applies committed transactions in parallel. Transactions that committed serially on the source — no possibility of deadlock there — can deadlock on the replica because the applier threads are racing on rows the source never had concurrent writers on. The replica’s deadlock detector resolves them the same way it would a live deadlock, but they show up in the replica’s error log with no corresponding entry on the source. Since MySQL 8.0.27, replica_parallel_type=LOGICAL_CLOCK and replica_parallel_workers=4 are the defaults, and replica_parallel_type was deprecated in 8.0.29 — LOGICAL_CLOCK is effectively the only supported mode going forward. The slave_* → replica_* rename happened in 8.0.26; older deployments and blog posts still use the legacy names. PostgreSQL 16+ introduced parallel apply for logical replication (streaming = parallel is the default on CREATE SUBSCRIPTION), which exposes the same class of apply-side cycles on a setup that historically didn’t have them — a surprise for teams upgrading from 15 and earlier.

Parallel/online DDL interacting with DML. Tools like pt-online-schema-change and gh-ost run concurrent DML against the table being altered (through triggers or a row-copy process). Under load, the trigger-installed writes and the copy process can both take locks on the same rows the application is updating, and the wait-for graph gains edges that wouldn’t exist during a normal workload. This rarely manifests as a hard deadlock — the tools are written defensively — but it does manifest as elevated deadlock rates during the migration window.

None of these are properties of the queries themselves. They’re properties of how work gets distributed across workers, processes, or replicas — which means they’re invisible to query-level review and only surface when the deadlock counter is watched over time. The primitives for fixing them — SKIP LOCKED, NOWAIT, advisory locks, DDL timeouts — are covered in Part 2: NOWAIT and SKIP LOCKED.

Engine-level differences that shape the patterns

The same pattern can deadlock on one engine and not the other. These differences are pattern-shaping — they change which of the above sections apply to your workload. Operational tuning (detector cost, wait timeouts, monitoring) is covered in Part 2: Monitoring.

Default isolation. PostgreSQL defaults to READ COMMITTED. MySQL defaults to REPEATABLE READ (with gap locks). The same application code has measurably different deadlock rates between the two because of this alone — before any other tuning.
Gap locks. Only InnoDB has them, and only under REPEATABLE READ (plus the foreign-key and duplicate-key exceptions that retain gap locking even under READ COMMITTED). PostgreSQL prevents phantom reads through MVCC at its own REPEATABLE READ — stricter than the SQL standard requires — without a range-locking mechanism, so the entire gap-lock deadlock category doesn’t exist on PG at any isolation level.
Lock granularity. PostgreSQL takes row-level (tuple) locks; InnoDB takes record locks on index entries (with next-key extension under REPEATABLE READ). The practical consequence is that InnoDB locks are more entangled with index choice than PostgreSQL’s — changing which index a query uses can change which rows and gaps get locked.
FK lock style. MySQL’s FK check holds a shared lock on the referenced row — next-key under REPEATABLE READ, and the docs list FK checking as one of the places gap locks persist even under READ COMMITTED. PostgreSQL takes a FOR KEY SHARE lock (added in 9.3 specifically to reduce FK lock contention vs. the older FOR SHARE). Hot parent rows are more contended under MySQL as a result.
Row-lock visibility. PostgreSQL row-level locks don’t show up in pg_locks — per the docs, they’re stored on the tuple header on disk, not in shared memory. A process waiting for a row lock usually appears in pg_locks as waiting for the holder’s transaction ID, not the row. InnoDB’s performance_schema.data_locks exposes row-level lock state directly. More on this in Part 2.

Neither engine is “better.” The behaviors are different, and code that assumes one can deadlock mysteriously when moved to the other.

What’s in Part 2

The patterns are the first half. Turning them into working systems takes a different set of skills: reading the deadlock log to identify which pattern is firing, building retry logic that doesn’t mask the real bug, isolating hot rows before they become incident reports, and choosing the right tool for each — NOWAIT, SKIP LOCKED, advisory locks, counter tables, or the isolation-level change that eliminates the category entirely. PostgreSQL’s SERIALIZABLE/SSI produces serialization failures that look like deadlocks but aren’t — the difference matters for retry architecture. AUTO_INCREMENT and sequence-related locking have their own failure modes. DDL migrations on both engines introduce lock queues that manifest as deadlock-like incidents.

All of that is in Database Deadlocks, Part 2: Diagnosis, Retries, and Prevention.

Mental model for the patterns

The useful reframe is: deadlocks are what consistent concurrency control does when two transactions make the engine choose between them. The database isn’t misbehaving — it’s refusing to let both of two contradictory orderings win. The error in the application log is a notification, not a fault.

That makes the diagnostic question concrete. Which pattern is firing? Every deadlock in production fits one of the shapes above — lock-ordering cycle, gap lock on a range, duplicate-key shared lock, FK shared lock on a hot parent, unindexed predicate lock amplification, worker-pool race, or replication-apply cycle. Identifying the pattern from the deadlock log narrows the fix enormously. “Two transactions deadlocked, retry the transaction” is true but useless. “Two workers took locks on the same jobs queue in different orders — switch to SKIP LOCKED” is a fix. The work is in the identification.

Part 2 picks up there: how to read the log, how to design retries that don’t hide the pattern, and how to remove the categories that matter most from the workload entirely.

Database Deadlocks, Part 2: Diagnosis, Retries, and Prevention

Sun, 19 Apr 2026 00:00:00 +0000

TL;DR

Part 1 covered the patterns — lock ordering, gap locks, unique-index conflicts, FK shared locks, index-scan amplification, and parallelism cycles. This post is the operational half: reading the deadlock log to identify which pattern fired, designing retries that fail loudly instead of hiding the real bug, isolating hot rows before they become incidents, the SERIALIZABLE/SSI distinction that trips up every team moving from READ COMMITTED, DDL migration windows and lock_timeout idioms, NOWAIT / SKIP LOCKED as prevention primitives, and innodb_autoinc_lock_mode.

The patterns in Part 1 answer “why did this deadlock happen?” This post answers the next three questions every team ends up asking: “which pattern fired?”, “what should the application do when it does?”, and “how do we stop the ones that hurt most?”. Each has its own tooling, failure modes, and production idioms — and almost all of them are learned after the first incident, not before.

The trap worth naming up front: tuning retry logic is easy, reading a deadlock log is harder, and the second is what tells you whether the first matters. A team that only tunes retries is optimizing the symptom without ever seeing the cause. This post is structured in the opposite order — log-reading first, then retries, then the prevention patterns that remove entire categories from the workload.

Reading the MySQL deadlock log

SHOW ENGINE INNODB STATUS dumps the most recent deadlock in the LATEST DETECTED DEADLOCK section. The catch: only the most recent. On a busy system, deadlocks overwrite each other faster than someone can log in and copy the output. Before anything else, turn on innodb_print_all_deadlocks = ON in every production deployment — it writes every deadlock to the error log instead of a single overwriting slot. The volume is negligible, the diagnostic value is high, and there is no downside.

A representative entry looks like this:

*** (1) TRANSACTION:
TRANSACTION 4823941, ACTIVE 3 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 4 lock struct(s), heap size 1136, 3 row lock(s)
MySQL thread id 892, OS thread handle 0x7f..., query id 18293 ...
UPDATE orders SET status = 'shipped' WHERE id = 1001

*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 48 page no 112 n bits 144 index PRIMARY
of table `shop`.`orders` trx id 4823941 lock_mode X locks rec but not gap
waiting

*** (2) TRANSACTION:
TRANSACTION 4823942, ACTIVE 2 sec starting index read
UPDATE orders SET status = 'paid' WHERE id = 1002

*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 48 page no 112 n bits 144 index PRIMARY
of table `shop`.`orders` trx id 4823942 lock_mode X locks rec but not gap

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 48 page no 112 n bits 144 index PRIMARY
of table `shop`.`orders` trx id 4823942 lock_mode X locks rec but not gap
waiting

*** WE ROLL BACK TRANSACTION (2)

The parts that matter:

lock_mode vs. lock_type. X is exclusive, S is shared. locks rec but not gap is a pure record lock; locks gap before rec is a gap lock; the unadorned X under REPEATABLE READ is usually next-key (record + gap). Matching lock_mode S locks rec but not gap against Part 1’s unique-index section tells you immediately that this is a duplicate-key-on-insert deadlock.
index name. index PRIMARY vs. index idx_customer reveals whether the cycle formed on the clustered index or a secondary one. Two transactions approaching the same rows from different indexes is the “secondary index locks on InnoDB” pattern from Part 1 — the fix is usually consolidating access paths.
The query text. This is the last statement the transaction executed before the deadlock, not necessarily the one that caused it. A transaction holding locks from three earlier statements can deadlock on the fourth, and the log only shows the fourth. Cross-reference with the application’s structured logs to reconstruct the full transaction.
trx id is monotonically increasing and stable for the life of the transaction. Searching the general log or slow-query log for that trx id reconstructs the full statement sequence — but only if general-query logging is on for the window in question, which it usually isn’t.

performance_schema.data_locks and data_lock_waits give a real-time view of current locks and waits. Useful for catching a deadlock-adjacent pathology (long wait chains, hot rows) before the cycle forms:

SELECT
  bl.lock_type, bl.lock_mode, bl.object_name, bl.index_name,
  w.REQUESTING_ENGINE_TRANSACTION_ID AS waiting_trx,
  w.BLOCKING_ENGINE_TRANSACTION_ID AS blocking_trx
FROM performance_schema.data_lock_waits w
JOIN performance_schema.data_locks bl
  ON w.BLOCKING_ENGINE_LOCK_ID = bl.ENGINE_LOCK_ID
WHERE bl.OBJECT_SCHEMA = 'shop';

Reading the PostgreSQL deadlock log

PostgreSQL’s diagnostic story is narrower by design. Deadlocks are logged automatically when the cycle is detected — log_lock_waits = on logs any wait exceeding deadlock_timeout (default 1s), which catches the wait-chain escalation before the detector fires. There’s no equivalent to SHOW ENGINE INNODB STATUS; everything lives in postgresql.log or the extensions you’ve installed.

A representative deadlock entry:

ERROR:  deadlock detected
DETAIL:  Process 14234 waits for ShareLock on transaction 89234;
         blocked by process 14235.
         Process 14235 waits for ShareLock on transaction 89233;
         blocked by process 14234.
         Process 14234: UPDATE accounts SET balance = balance - 100 WHERE id = 2
         Process 14235: UPDATE accounts SET balance = balance + 50 WHERE id = 1
HINT:  See server log for query details.
CONTEXT:  while updating tuple (0,18) in relation "accounts"

The ShareLock on transaction X wording is PostgreSQL-specific — one transaction is waiting to see the commit status of another (a row-lock wait manifests as waiting on the holder’s transaction ID). The tuple identifier (0,18) points to the exact physical row (page 0, tuple 18 in the heap), which is useful for reproducing the scenario but changes as rows are updated (MVCC creates new versions at new (page, tuple) locations).

For real-time inspection, pg_locks joined against pg_stat_activity shows live lock state:

SELECT
  a.pid, a.usename, a.state,
  a.wait_event_type, a.wait_event,
  l.mode, l.locktype, l.relation::regclass,
  pg_blocking_pids(a.pid) AS blocked_by,
  LEFT(a.query, 80) AS query
FROM pg_stat_activity a
LEFT JOIN pg_locks l ON l.pid = a.pid AND NOT l.granted
WHERE a.state != 'idle'
ORDER BY a.xact_start;

pg_blocking_pids(pid) returns the array of PIDs blocking a given transaction. Walking it recursively reconstructs the live wait-for graph — the same data the deadlock detector uses, just before a cycle forms. For hot systems, pg_stat_statements combined with pg_stat_activity snapshots at regular intervals builds a picture of which statements accumulate the most wait time, which is almost always the right first place to look.

Row locks are invisible in pg_locks

PostgreSQL’s row-level locks (the result of SELECT ... FOR UPDATE, FK checks, and plain UPDATE/DELETE) are stored on the tuple itself — in the xmax system column — not in the shared lock table. They don’t show up in pg_locks. The only way to see them is through the pgrowlocks extension, which scans the heap directly:

1
2

CREATE EXTENSION pgrowlocks;
SELECT * FROM pgrowlocks('accounts');

This is the single biggest difference between PG and InnoDB lock introspection, and the reason PG operators often feel blind to row-level contention until a cycle actually forms.

PostgreSQL SERIALIZABLE: serialization failures are not deadlocks

Under SERIALIZABLE isolation, PostgreSQL uses Serializable Snapshot Isolation (SSI) — an optimistic mechanism based on SIREAD predicate locks that track read-write dependencies between transactions. SSI cannot deadlock by design: it never blocks on lock acquisition. What it does is abort one transaction with a serialization failure when it detects a dangerous read-write dependency cycle that would violate serializability.

The two failure codes look similar but have fundamentally different semantics:

40001 serialization_failure — SSI detected a dependency cycle and aborted the transaction before it could commit a non-serializable result. The transaction did nothing wrong; the combination of its operations with a concurrent transaction would have produced an inconsistency. Retrying is always safe and usually succeeds (the concurrent transaction will have committed or aborted, so the second attempt doesn’t see the same conflict).
40P01 deadlock_detected — a cycle in the wait-for graph was broken by killing a victim. Retrying may or may not succeed depending on what caused the cycle. If the cycle was deterministic (two code paths with inconsistent ordering), it will keep recurring.

The practical consequence for retry architecture: an application running under SERIALIZABLE must handle 40001 — it’s not a deadlock, it’s the normal failure mode of SSI, and retries are the only recovery path. An application running under READ COMMITTED never sees 40001. An application that handles 40001 identically to 40P01 is correct but coarse. The right granularity is: always retry 40001 (the workload’s own correctness guarantee assumes this); retry 40P01 with caution and escalate on repeat.

def retry_on_conflict(fn, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return fn()
        except psycopg.errors.SerializationFailure:
            # 40001: always retry. SSI guarantees make this safe.
            backoff(attempt)
        except psycopg.errors.DeadlockDetected:
            # 40P01: retry with caution; log for root-cause analysis.
            log_deadlock_for_analysis(...)
            backoff(attempt)
    raise TransactionRetryExhausted

Retry architecture that doesn’t hide the bug

Every database driver documentation says “deadlocks happen, retry the transaction.” That’s true. It’s also incomplete. The dangers are subtle enough that a naive retry loop becomes part of the problem:

Retries without backoff make the cycle worse. The condition that caused the deadlock (contention on a hot key set) is still in effect when the retry runs. A tight retry loop turns one deadlock into a thundering herd — all victims retry simultaneously, all hit the same contention, all deadlock again. Use exponential backoff with full jitter, capped at a few hundred milliseconds.

Retries mask lock-ordering bugs. If an application deadlocks 10x/minute but retries successfully, the operator sees no failures — but the underlying transactions are doing up to 20x the work (original + retry). The deadlock rate itself is a metric worth tracking, not just the post-retry error rate. When the rate grows, the fix is diagnosing the pattern, not tuning the retry limit.

Retries aren’t always safe. A transaction that sent an external notification, wrote to a message queue, or called a non-idempotent HTTP endpoint before the deadlock can’t be blindly retried — the external side effect already happened. Retries belong on database-only transactions, or on transactions where the external calls are idempotent and tolerant of duplicate execution. The boundary is architectural, not a library setting.

Retries need a budget. If a transaction can’t complete after ~3 attempts, the problem is no longer transient contention — it’s either a systemic hot spot or a bug that retries will never resolve. Escalate (alert, fail the request, enqueue for manual review), don’t loop forever.

The retry pattern that works in production:

for attempt in range(3):
    try:
        with db.transaction():
            do_work()
        return
    except (DeadlockError, SerializationFailure) as e:
        metrics.increment("db.retry", tags={"error": e.code})
        time.sleep(random.uniform(0, 0.1 * 2**attempt))
raise RetryBudgetExhausted()

Three attempts, exponential backoff up to ~400ms, metrics emitted on every retry, hard failure past the budget. The metric matters as much as the retry — without it, the team never learns which transactions are retrying frequently and why.

Idempotency is a transaction-shape property

A transaction is safe to retry iff re-running it produces the same observable state. That includes downstream side effects. A transaction that writes to a table AND sends a webhook is not safe to retry even if both operations are internally correct — the second attempt sends a duplicate webhook. The fix is the outbox pattern: write the webhook-send intent to a table in the same transaction, then have a separate worker process the outbox with its own idempotency guarantees. This is a non-negotiable part of building deadlock-retry-safe systems at scale.

Consistent lock ordering: the highest-leverage fix

Part 1’s lock-ordering deadlock — two transactions updating the same set of rows in opposite orders — is the single most common production pattern and the one with the highest-leverage fix. If every code path that writes to a set of tables acquires locks in the same order, the wait-for graph literally cannot form a cycle on those rows. The engine still takes the locks, still holds them for the duration of the transaction, but the second transaction waits cleanly for the first instead of grabbing a lock the first will need.

The rule is: sort the rows by a stable key — usually the primary key — in application code, before any SQL is issued. Lock acquisition order in both engines is determined by the order the engine processes rows, which for most write patterns is the order the application submitted them. Sort once up front, and N workers all doing the same thing can’t cycle because they all approach the row set from the same end.

The three batch shapes that matter in practice, and where the ordering actually happens:

1. Loop of per-row UPDATEs — the classic batch worker:

# Sort in the application; the iteration order IS the lock acquisition order.
rows.sort(key=lambda r: r.id)
for row in rows:
    cursor.execute(
        "UPDATE accounts SET balance = balance + %s WHERE id = %s",
        (row.delta, row.id),
    )

Each UPDATE locks its target row at execution time; the loop order determines acquisition order. No SQL-level ORDER BY involved — the fix lives in the .sort() call.

2. Bulk UPSERT — INSERT ... ON DUPLICATE KEY UPDATE (MySQL) or INSERT ... ON CONFLICT (PostgreSQL):

# Sort rows by the unique key BEFORE building the batch.
rows.sort(key=lambda r: r.id)
execute_values(
    cursor,
    "INSERT INTO accounts (id, balance) VALUES %s "
    "ON CONFLICT (id) DO UPDATE SET balance = accounts.balance + EXCLUDED.balance",
    [(r.id, r.delta) for r in rows],
)

The engine processes the VALUES list in order and acquires locks as it goes. Two concurrent batches sorted by the same key approach the key space from the same end; without the sort, one batch might process (1, 2, 3) while another processes (3, 2, 1) and they deadlock on the middle row. This is the exact shape of the bulk-UPSERT deadlocks called out in Part 1’s unique-index section.

3. Small multi-row transactions (the canonical bank transfer):

-- Always update the lower id first, regardless of transfer direction.
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

For 2–3 rows with per-row different values, the application computes sorted((X, Y)) and issues UPDATEs in that order. Same principle as the batch case, just smaller.

Where SELECT ... FOR UPDATE ORDER BY actually earns its keep. Most batches don’t need it — they control lock order through the submission order. The one shape where it’s the right answer is a single UPDATE statement over a derived table where the engine decides scan order and you can’t control it from outside:

UPDATE accounts a
SET balance = a.balance + u.delta
FROM (VALUES (1, 50), (2, 100), (3, 25)) AS u(id, delta)
WHERE a.id = u.id;

Here, sorting the VALUES list in application code doesn’t reliably control lock order — the planner picks the scan. A SELECT id FROM accounts WHERE id IN (...) ORDER BY id FOR UPDATE up front pre-acquires locks in deterministic order before the UPDATE runs. Or refactor into shape 1 or 2.

ORDER BY controls result order, not scan order

ORDER BY on a SELECT ... FOR UPDATE controls the result ordering, but lock acquisition happens during the scan. With a primary-key or unique-index predicate (WHERE id IN (...) on a PK), the planner does ordered index lookups and locks land in ORDER BY order in practice. For non-indexed predicates or range scans on non-unique columns, the planner may scan in a different order and sort results afterward — locks get acquired in scan order. Verify with EXPLAIN before relying on this pattern against non-PK predicates. Also: MySQL’s UPDATE ... ORDER BY syntax applies one SET clause to all matching rows — it doesn’t help when rows need different values.

This sounds trivial. It almost never is in practice — the ordering has to hold across every code path that writes to the same tables: the main request handler, backfill scripts, admin scripts, scheduled jobs, ORM bulk-save methods, and whatever migration scripts run during releases. One path that writes in a different order is enough to reopen the cycle. The durable fix is encoding the order in the access layer so individual query sites can’t diverge from it:

A repository function that always sorts by PK before writing.
A stored procedure or database function that owns the multi-row write.
A service method with the ordering baked in, and a lint rule or review check forbidding direct table writes from elsewhere.

Two places this invariant breaks without anyone noticing:

ORM bulk-save methods. ORMs hide whether they process in input order or reorder internally. Django’s Model.objects.bulk_update, SQLAlchemy’s bulk_update_mappings, ActiveRecord’s upsert_all, Hibernate’s batch inserts — some process in input order, some sort by PK internally, some chunk before doing either. If you can’t tell from docs, test it: two concurrent bulk-saves with opposing-ordered input lists will either deadlock (proving input order matters) or not (proving the ORM sorts internally). Either way, sorting the input collection before handing it to the ORM is cheap insurance.

Batches sourced from a SELECT in the same transaction. A common pattern: “grab N pending rows, process them.” If the feeder SELECT doesn’t have an ORDER BY, rows come back in scan order — non-deterministic across workers, which reopens the cycle. The fix is an explicit ORDER BY on the feeder query, not in the subsequent loop:

# Bad: scan order feeds the loop.
rows = cursor.execute("SELECT id, delta FROM pending WHERE status = 'ready' LIMIT 100")
for row in rows:  # Whatever the scan produced — varies across workers.
    ...

# Good: deterministic order, same across every worker.
rows = cursor.execute(
    "SELECT id, delta FROM pending WHERE status = 'ready' ORDER BY id LIMIT 100"
)
for row in rows:
    ...

The lint-rule angle matters more than it sounds. Deadlock-ordering violations are almost impossible to catch in code review — two PRs that each look correct in isolation can introduce inconsistent ordering when combined. The check that actually works is structural: no direct writes to tables X, Y from anywhere except the repository. Once that invariant is enforced, the ordering invariant follows.

Multi-table transactions need the same rule applied to table order. A transaction that updates users then orders in one code path, and orders then users in another, can deadlock through the FK chain even with per-table row ordering. The rule generalizes: sort rows within a table, and sort tables within a transaction, by a stable convention the whole codebase agrees on (alphabetical is fine; just pick one).

Isolation-level trade-offs

The isolation level a workload runs under determines which deadlock categories even apply. Most MySQL deadlock incidents stem from REPEATABLE READ’s gap locks — a category that doesn’t exist on PostgreSQL or on MySQL at READ COMMITTED. Changing the isolation level is the single highest-leverage tuning lever, and also the one with the most potential to quietly break application correctness.

Dropping MySQL from REPEATABLE READ to READ COMMITTED. Under READ COMMITTED, InnoDB still takes row locks but skips most gap locks (they exist only for unique-key and FK enforcement on inserts). Most OLTP workloads don’t need REPEATABLE READ’s range-consistency guarantee — most application code was designed around READ COMMITTED semantics anyway, because that’s what PostgreSQL and SQL Server default to. Teams migrating to READ COMMITTED typically see deadlock rates drop by an order of magnitude with no functional change.

Avoiding range locks on write paths. Independent of isolation level, replacing SELECT ... FOR UPDATE scans over ranges with point lookups by primary key removes the gap-lock surface entirely on the statements that do it. If a write path doesn’t need to lock “all orders for customer 5,” locking just the specific row it’s about to update is both faster and safer.

FK shared locks are shorter-lived under READ COMMITTED. The foreign-key shared-lock pattern from Part 1 (high-write child tables concentrating shared locks on hot parent rows) has a narrower window under READ COMMITTED simply because the lock lifespan is tied to the statement rather than the transaction. The cycle potential is still there, but the wait window is smaller.

Isolation change is a behavior change, not a tuning knob

READ COMMITTED eliminates most gap locks but also changes the visibility semantics of long-running transactions. Any code that relied on re-reading a row and getting the same result (transfer logic, inventory deductions, financial calculations) has to be re-examined. The safe migration is application-by-application, not database-wide. Run it in staging under production-like load and watch for subtle correctness regressions — “phantom” rows appearing inside a transaction that used to see a stable snapshot, inventory counts that shift mid-transaction, calculations that no longer match because an underlying row changed between reads.

Session-scoped change as a migration path. Both engines let you set isolation level per session or per transaction — SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED in MySQL, SET TRANSACTION ISOLATION LEVEL READ COMMITTED per transaction in PostgreSQL. The usual migration pattern is to start with the most contended code paths, move them to session-scoped READ COMMITTED, monitor for regressions, then expand the scope. A global flip from REPEATABLE READ to READ COMMITTED on a large, stable MySQL deployment is rarely the right first step.

Hot-row isolation: removing the pattern instead of retrying it

When the top N deadlocks on a production system concentrate on a small set of rows (a counter, a config row, an AUTO_INCREMENT source of truth), retries don’t converge. Every retry hits the same row, takes the same lock, cycles with the same peers. The fix is removing the row from the hot path, not tuning the retry layer.

Three patterns that work:

Counter tables with sharding — for extreme write-hot counters only. Reach for this only when the counter is taking thousands of writes per second against a single row and the simpler options below aren’t viable. For anything less, the queue pattern below or an external store (Redis atomic INCR, a time-series DB) is almost always the better answer — less complexity, no schema overhead, no sum-across-rows read cost. Sharded counters are the specialized escalation, not the default.

When it does fit the workload: N physical shards per logical counter, keyed on a compact integer (counter_id, shard_id) composite. Application code picks the shard — keeping the random choice out of SQL makes it portable across engines and testable independently:

CREATE TABLE counter_shards (
    counter_id BIGINT NOT NULL,   -- FK to a counters metadata table if you need names
    shard_id SMALLINT NOT NULL,   -- 0..N-1, fixed per-counter
    value BIGINT NOT NULL DEFAULT 0,
    PRIMARY KEY (counter_id, shard_id)
);

-- Seed the shards once per counter (e.g., when the counter is created):
INSERT INTO counter_shards (counter_id, shard_id, value)
VALUES (42, 0, 0), (42, 1, 0), (42, 2, 0), (42, 3, 0),
       (42, 4, 0), (42, 5, 0), (42, 6, 0), (42, 7, 0),
       (42, 8, 0), (42, 9, 0), (42, 10, 0), (42, 11, 0),
       (42, 12, 0), (42, 13, 0), (42, 14, 0), (42, 15, 0);

# Increment: application picks the shard. Portable, cheap, no SQL-side RAND().
shard = random.randrange(16)
cursor.execute(
    "UPDATE counter_shards SET value = value + 1 "
    "WHERE counter_id = %s AND shard_id = %s",
    (42, shard),
)

# Read: sum across shards.
cursor.execute(
    "SELECT COALESCE(SUM(value), 0) FROM counter_shards WHERE counter_id = %s",
    (42,),
)

16 shards turn one hot row into 16 warm rows. The contention surface scales with shard count. The read cost is one SUM across N rows instead of a single-row SELECT — usually acceptable for counter use cases; if not, cache the aggregate.

A common refinement is deriving the shard deterministically from the connection or worker ID (e.g., connection_id % 16) so each worker consistently hits the same shard. That improves InnoDB buffer-pool locality and reduces cross-shard interference, at the cost of slightly less even distribution if worker counts aren’t balanced.

Advisory locks for app-level serialization. Both MySQL and PostgreSQL support advisory locks — named locks that exist outside the table model and don’t take row locks. For operations that need to be serialized at the application level (leader election, rate limiting, config migration), advisory locks are dramatically cheaper than row locks and can’t participate in a table-lock cycle:

-- PostgreSQL: advisory lock keyed by a bigint.
SELECT pg_advisory_xact_lock(hashtext('refresh_cache:customer_42'));
-- Lock released at transaction end. Only one worker per key runs at a time.

-- MySQL equivalent:
SELECT GET_LOCK('refresh_cache:customer_42', 10);
-- Returns 1 if acquired, 0 on timeout. Must explicitly RELEASE_LOCK.

The caveats: advisory locks are application-layer discipline — they don’t enforce anything the database checks. Use them where the application chooses to serialize, not where correctness requires it.

Queue patterns instead of direct updates. For counter-like workloads, write intents to an append-only table and aggregate periodically:

INSERT INTO counter_events (counter_key, delta, created_at) VALUES (?, ?, NOW());
-- No contention: every insert creates a new row.

-- Periodic aggregation job:
INSERT INTO counter_totals (counter_key, total)
SELECT counter_key, SUM(delta) FROM counter_events
WHERE processed = FALSE
GROUP BY counter_key
ON CONFLICT (counter_key) DO UPDATE SET total = counter_totals.total + EXCLUDED.total;

Trades real-time accuracy for throughput. The right trade-off for page-view counters, metric accumulation, any workload where eventual consistency is acceptable.

Hot parent rows behind FK chains. Part 1 described how a high-write child table concentrates shared locks on a hot parent row, and how any exclusive-lock request on that parent (a name update, a soft-delete, a trigger-driven counter update) becomes a contention point. Two levers that work:

Move high-frequency parent-row updates to a side table. The last_activity_at timestamp, the cached counter, the updated_at that a trigger bumps on every child insert — none of these need to live on the parent table. Moving them to customer_activity(customer_id, last_seen_at) or customer_counters(customer_id, order_count) eliminates the exclusive-lock contention entirely. The parent row stops changing on hot paths, the shared locks from FK checks coexist fine, and the cycle potential disappears.
Narrow the FK scope where integrity can tolerate it. Not every child table needs an enforced FK to every parent — logs, events, and audit tables are often the biggest offenders, and often have the least need for strict integrity (an orphaned log row is rarely a correctness problem). Dropping the FK removes the shared-lock dependency entirely. This trades integrity for throughput — a decision that belongs with the team owning the data, not a default.

Under READ COMMITTED, both levers matter less because the FK shared locks release at statement end rather than transaction end. A workload that runs on REPEATABLE READ and can’t change isolation level (because of application semantics) gets the most benefit from these two fixes.

Long-running transactions are a deadlock amplifier

The longer a transaction holds locks, the wider the window for a cycle. Two patterns recur in production:

Application-layer long transactions. A transaction that opens at request start, makes several queries, calls an external API, then commits. The external call is where the transaction actually spends its time — seconds of open transaction holding row locks the whole time. Every concurrent transaction that touches those rows waits. Deadlock probability scales with transaction duration. The fix is the inverse of the outbox pattern: do the external call outside the transaction, passing in any needed IDs.

Idle-in-transaction sessions. A session that runs BEGIN, some writes, then stalls — idle but not committed. In PostgreSQL, this blocks vacuum on touched tables, bloats MVCC, and holds locks indefinitely. pg_stat_activity shows state = 'idle in transaction'. MySQL’s equivalent is a thread with an open transaction and no current query.

PostgreSQL has a first-class timeout for this; MySQL does not:

1
2
3

-- PostgreSQL: kill idle-in-transaction sessions after 5 minutes.
-- (Units required — bare integer would be interpreted as milliseconds.)
SET idle_in_transaction_session_timeout = '5min';

MySQL has no direct equivalent. wait_timeout and interactive_timeout govern idle connections, not sessions idle inside an open transaction — a connection that did BEGIN then stopped sending queries will hold its locks until the connection drops or the client commits. The production workaround is either a watchdog script (e.g., Percona’s pt-kill) that polls information_schema.innodb_trx and terminates transactions exceeding a duration threshold, or a connection pool with per-connection transaction lifetimes. Connection pools that acquire a connection, start a transaction, then return the connection to the pool without committing (rare but real) will produce sessions that live indefinitely otherwise.

Finding long-running transactions:

-- PostgreSQL
SELECT pid, usename, state, xact_start, now() - xact_start AS duration, query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL AND now() - xact_start > INTERVAL '30 seconds'
ORDER BY duration DESC;

-- MySQL
SELECT trx_id, trx_started, trx_mysql_thread_id, trx_rows_locked, trx_query
FROM information_schema.innodb_trx
WHERE TIMESTAMPDIFF(SECOND, trx_started, NOW()) > 30;

Alerting on any transaction exceeding 30s in an OLTP workload catches most of the long-transaction-induced deadlocks before they produce incidents.

Triggers and cascades are invisible lock sources

A trigger that updates a second table on every write to the first adds an edge to the wait-for graph that isn’t visible in the original query. ON DELETE CASCADE foreign keys behave similarly — one delete can take locks on every child row in the cascade, and if the cascade order differs between two concurrent deletes, they can deadlock through tables neither statement directly referenced.

This is the origin of the “why is my DELETE FROM users deadlocking against an INSERT INTO events?” question. The DELETE triggered a cascade to user_preferences, which had a trigger that updated a counter in tenants, which was locked by the INSERT. Four tables in the cycle, two in the application’s explicit query, and zero mention of the other two in any log entry until someone reads the DDL.

The operational pattern: when a deadlock log mentions a table the application’s code doesn’t explicitly reference, check (1) FK cascades on the tables that are in the query, (2) triggers on those tables, (3) generated columns that fire on update. All three are non-obvious lock sources, and all three are fixable — but only after they’re identified.

`innodb_autoinc_lock_mode` and the AUTO-INC lock

MySQL InnoDB’s AUTO_INCREMENT column has its own lock, historically a source of contention and occasional deadlock. The innodb_autoinc_lock_mode parameter controls the behavior:

Mode 0 (traditional) — table-level AUTO-INC lock held for the duration of the statement. Serialized across inserts. Safe for statement-based replication, terrible for concurrency.
Mode 1 (consecutive) — a lighter lock for simple inserts (single-row or known-row-count), and the traditional table lock for bulk inserts (INSERT ... SELECT). Was the default in 5.7.
Mode 2 (interleaved) — no AUTO-INC table lock; IDs are assigned per-row as needed, possibly interleaved across concurrent statements. Default in MySQL 8.0. Fastest, and correct for row-based replication (which is also the 8.0 default).

The mode-2 default in 8.0 eliminated a substantial source of historical deadlocks and contention — bulk inserts that used to serialize on the AUTO-INC lock now proceed in parallel. If you’re migrating from 5.7 to 8.0, this is a free win. If you’re still on binlog_format = STATEMENT (uncommon but not unheard of in legacy deployments), you cannot safely run mode 2 — the replica may generate different IDs than the source, corrupting the data. Switch to binlog_format = ROW first, then adopt mode 2.

DDL migration windows and `lock_timeout`

Online schema change isn’t deadlock-prone in the classical sense, but it interacts with deadlocks in a specific operational way: DDL takes heavy locks that queue behind ongoing DML, and while the DDL waits, every subsequent query on that table queues behind the DDL. In PostgreSQL, a DDL taking ACCESS EXCLUSIVE that waits for an existing long-running SELECT will cause every new SELECT to wait behind the DDL. The system grinds to a halt, and application logs fill with timeout errors that look like deadlocks but aren’t — it’s a queue, not a cycle.

The standard prevention idiom for PostgreSQL migrations:

1
2

SET lock_timeout = '2s';
ALTER TABLE orders ADD COLUMN notes TEXT;

If the ALTER can’t acquire its lock in 2 seconds, it fails instead of queueing. The migration tool catches the error and retries with backoff. This prevents the queue-behind-DDL outage entirely — the cost is that some migrations need multiple attempts to land, which is almost always the right trade-off.

MySQL’s equivalent tooling is pt-online-schema-change (Percona) and gh-ost (GitHub). Both create a copy of the table, stream writes to both via trigger or binlog, and swap at the end. They run concurrent DML against the original and the copy, so they inflate deadlock rates during the migration window — not because the tool is buggy, but because there are now more transactions touching the same rows. The operational practice: run migrations at low-traffic windows, watch the deadlock counter during the run (not just replica lag), and have a rollback path ready.

DDL inside transactions is engine-dependent

PostgreSQL supports transactional DDL — BEGIN; ALTER TABLE ...; COMMIT; is atomic. MySQL does not; every DDL statement implicitly commits the current transaction. A migration script that assumes it can roll back mid-migration works on PostgreSQL and silently half-applies on MySQL. Know which engine you’re writing migrations for.

`NOWAIT` and `SKIP LOCKED` as prevention primitives

Both engines support two SQL-level concurrency primitives that remove the need for application-layer deadlock handling in specific patterns:

SELECT ... FOR UPDATE NOWAIT — if the row is locked by another transaction, fail immediately with an error instead of waiting. Useful for user-facing paths where “I can’t get this resource right now” is a better UX than “wait 500ms and maybe deadlock anyway.” Also useful for detecting lock contention synthetically in tests.
SELECT ... FOR UPDATE SKIP LOCKED — if rows are locked by another transaction, skip them and return only rows the current transaction can lock. Transforms a contended queue-processor pattern into a lock-free one: N workers each grab a different set of rows, zero contention, zero deadlocks.

-- Queue processor: deadlock-free, contention-free.
SELECT * FROM jobs
WHERE status = 'pending'
ORDER BY priority, created_at
LIMIT 10
FOR UPDATE SKIP LOCKED;

-- Fast-fail acquisition: don't wait, fail now.
SELECT * FROM leader_election
WHERE resource_id = 'cache-refresh'
FOR UPDATE NOWAIT;

SKIP LOCKED arrived in PostgreSQL 9.5 and MySQL 8.0. Before those versions, queue-processor patterns required either advisory locks or application-level coordination (Redis, Zookeeper). Post-SKIP LOCKED, they can live entirely in the database with a single primitive. For any workload where workers pull from a shared queue, this is the pattern — not retry loops on FOR UPDATE.

Monitoring that actually catches regressions

The single most useful metric is deadlock rate over time. Not error rate, not retry rate — the raw count of deadlocks per minute or per thousand transactions. A workload with 0.1 deadlocks per thousand transactions is healthy; 10 per thousand is a paging threshold; 100 per thousand means retries aren’t converging and something is structurally wrong.

For MySQL: there’s no Innodb_deadlocks status variable. The correct source is performance_schema.events_errors_summary_global_by_error, which is enabled by default:

-- Cumulative deadlock count (compare over time windows).
SELECT SUM_ERROR_RAISED AS deadlock_count
FROM performance_schema.events_errors_summary_global_by_error
WHERE ERROR_NAME = 'ER_LOCK_DEADLOCK';

-- Lock-wait activity (useful for adjacent contention, NOT a deadlock counter):
SHOW GLOBAL STATUS LIKE 'Innodb_row_lock_waits';

-- Plus the error log, searchable for "LATEST DETECTED DEADLOCK"
-- once innodb_print_all_deadlocks=ON.

Innodb_row_lock_waits is commonly misread as a deadlock counter — the manual defines it as “the number of times operations on InnoDB tables had to wait for a row lock,” which is contention in general. Pair it with the events_errors_summary query, not in place of it.

For PostgreSQL:

-- pg_stat_database exposes per-database deadlock counter.
SELECT datname, deadlocks, xact_commit, xact_rollback
FROM pg_stat_database
WHERE datname = current_database();

Scrape both into Prometheus (mysqld_exporter and postgres_exporter both expose these), compute the rate, alert on sustained rises. Pair the deadlock rate with a retry rate from the application layer — a spike in one without the other means either the retry logic is broken or the workload shape changed. A spike in both means a real regression.

Beyond the rate itself, the top-K pairs of statements involved in deadlocks (extracted from innodb_print_all_deadlocks logs or PG’s deadlock log entries) identify exactly which code paths are fighting. This list rarely changes — the same two or three patterns account for most deadlocks in any given system. Fix those and the rate drops by an order of magnitude.

The mental model for Part 2

Part 1’s patterns answer why deadlocks happen. Part 2’s operations answer what to do about them — and the useful reframe is that the answer is almost never “tune the retry logic.” Retries are the recovery mechanism that keeps the application running while the actual fix lands. The actual fix is almost always one of:

Identify the pattern from the log. This is step zero; skipping it means you’re tuning blind.
Enforce consistent lock ordering at the access-layer level. Highest-leverage fix for the lock-ordering pattern; deterministically eliminates the cycle rather than shrinking its window.
Change the code path to use SKIP LOCKED / NOWAIT where the pattern matches (queue processors, resource acquisition).
Isolate hot rows (counter tables, shards, advisory locks, queue patterns; move high-frequency parent-row updates to side tables).
Shorten transactions. Move external calls out, enforce idle-transaction timeouts.
Drop isolation level where the workload allows it — session-scoped first, global only after regression testing. Eliminates the gap-lock category on MySQL.
Remove cascades/triggers from the hot path when they’re the hidden lock source.
Handle SERIALIZABLE’s 40001 as a normal event if you’re on SERIALIZABLE, and don’t confuse it with 40P01.
Plan DDL windows with lock_timeout and watch the deadlock counter through the migration.

Retries let the application survive while the fix is in flight. Monitoring tells you which fix to prioritize. Each of the above removes a category from the workload entirely — which is what “eliminating deadlocks” actually looks like in production. Not zero deadlocks forever, but a workload where the few remaining are rare enough that the retry layer handles them invisibly and the team’s attention can go elsewhere.

That’s the goal. Not a deadlock-free system — that’s a theoretical fiction at realistic concurrency — but a system where deadlocks are a managed failure mode rather than a recurring incident. The patterns in Part 1 tell you what to look for. The tooling and idioms here are how to do something about what you find.