PostgreSQL Didn't Fail. It Hit Its Limit.

Four limits, logged the way an incident channel actually reads: symptom, investigation, fix. None of these are exotic. They're the same handful of ceilings that show up again and again in production Postgres, and every one of them is survivable if you know where to look before the pager does.

SEV-1 INC-3104 orders-db / pgbouncer

New connections start timing out under normal traffic

Nothing about the query load looks unusual. CPU is calm, disk is calm, and yet the application layer is throwing connection timeouts. This is almost never a Postgres problem in the query-tuning sense — it's an arithmetic problem. max_connections was set once, for a service that has since grown five deploys and three new background workers past what that number ever assumed.

Check active vs. idle connections first. A pool full of connections sitting in idle in transaction is not capacity being used — it's capacity being held hostage by a client that opened a transaction and forgot to close it.
Count connections per service, not just in total. A single misbehaving worker fleet can quietly consume the majority of the pool while every other service starves for what's left.
Look for a missing pooler before reaching for a bigger max_connections. Each Postgres connection carries real backend memory; raising the ceiling without pooling just moves the wall further away, it doesn't remove it.
Check for connection storms on deploy. A rolling restart that opens hundreds of fresh connections at once can exhaust the pool in the same second every replica comes back healthy.

FixPgBouncer in transaction-pooling mode in front of the primary, a statement timeout so a stuck transaction can't hold a slot indefinitely, and an alert on pool saturation — not just on connection errors, which only fire after it's already too late.

SEV-2 INC-3117 events-db

Queries that were fast for months get steadily slower

No schema change, no traffic spike, no obvious culprit — just a slow, creeping decline in query latency over weeks. The table on disk has grown far larger than the number of live rows in it would explain. This is bloat, and it's usually autovacuum losing a race it was never tuned to win.

Compare table size to estimated live rows. A table that's 4x larger on disk than its row count and average row width would suggest is carrying dead tuples that were never reclaimed.
Check autovacuum's actual run history, not just whether it's enabled. A high-churn table with default vacuum thresholds can fall permanently behind the rate at which it's generating dead tuples.
Look for long-running transactions holding back the vacuum horizon. A single forgotten transaction open for hours can prevent autovacuum from reclaiming anything on tables it isn't even touching.
Check index bloat separately from table bloat. Indexes can bloat even when the underlying table is well-vacuumed, and a bloated index quietly degrades every query that relies on it.

FixPer-table autovacuum tuning for the highest-churn tables, an alert on transaction age so nothing can pin the vacuum horizon unnoticed, and a scheduled pg_repack pass on the worst offenders instead of waiting for a full table rewrite to become unavoidable.

SEV-1 INC-3129 primary-db / disk

Disk usage climbs steadily until the primary refuses writes

Disk utilization on the primary rises for hours with no corresponding growth in table data. Eventually writes fail outright because there's no room left to accept them. The tables aren't the problem — the write-ahead log is, and it's building up because something downstream stopped consuming it.

Check replication slots first. A logical or physical replication slot tied to a consumer that's disconnected — a downed replica, a stalled CDC pipeline — will hold WAL on disk indefinitely, because Postgres assumes that consumer is coming back for it.
Check archive_command health. If WAL archiving to object storage is failing silently, segments accumulate locally with no external signal until disk pressure makes it impossible to ignore.
Distinguish WAL growth from table growth early — they call for completely different responses, and treating a WAL problem as a table-bloat problem wastes the exact time you don't have.
Have a drop-the-slot runbook ready for the case where the disconnected consumer truly isn't coming back, understanding that this discards data that consumer will never receive.

FixAlerting on replication slot lag measured in bytes, not just replica health checks, plus a hard cap via max_slot_wal_keep_size so an abandoned slot can degrade a downstream consumer instead of taking down the primary.

SEV-2 INC-3142 read-replicas

Replicas fall minutes behind during a write-heavy batch job

A nightly batch job hammers the primary with writes, and read replicas that are normally near real-time start serving data that's several minutes stale. Nothing has crashed — which is exactly why it takes longer than it should to notice.

Confirm it's apply lag, not network lag. WAL can arrive at the replica quickly and still sit unapplied if the replica's own I/O or a long-running query on it is holding up recovery.
Check for a query on the replica blocking WAL apply. A long analytical query holding a lock that conflicts with an incoming WAL record will pause replay until that query finishes or is cancelled.
Look at replica hardware, not just the primary's. Replicas are often provisioned smaller than the primary on the assumption that they only serve reads — a write-heavy period asks them to keep up with primary-level I/O regardless.
Check whether anything downstream trusts replica freshness implicitly. A read-after-write flow routed to a lagging replica doesn't error — it just quietly returns the wrong answer.

FixReplica-lag-aware routing that falls back to the primary past a defined threshold, batch jobs scheduled against replica lag rather than a fixed clock time, and matching the replica's I/O provisioning to what write-heavy windows actually demand.