incident log / reliability notebook
SRE Core Areas: What Breaks Under Pressure
There's a kind of question that separates "I know the theory" from "I've been paged for this." It's not the vocabulary — it's the sequence of moves you make in the first five minutes of a production fire, and the decisions you make long before the fire starts.
By Vimal Maheedharan — Technical Architect & SRE Consultant · July 2026
Four scenarios, logged below the way an incident channel might read them — symptom, investigation, resolution. Not a postmortem template exactly, more a set of reasoning patterns worth keeping close, because the failures that matter most rarely announce themselves. They hide in saturation, in propagation delay, in a metric nobody's watching, in consistency nobody's testing.
SEV-1
INC-2201
payments-gateway
5xx errors appear only under peak load
Service behaves correctly at normal traffic, then starts throwing server errors the moment load spikes. Easy to mistake for a code bug — it usually isn't. It's a capacity bug wearing a code bug's clothes.
- Confirm the correlation first. Overlay error rate against request volume, not against deploy timestamps — ruling out "did we just ship something" early saves the whole investigation from going sideways.
- Check saturation before anything else. Connection pools, worker queues, CPU throttling, GC pauses — invisible at low concurrency, the entire story at high concurrency.
- Trace the slow requests specifically. Where does latency actually balloon — a query, a lock, an external call?
- Watch for locks that only bite under concurrency. A query that takes 200ms alone can take 8 seconds queued fifty-deep behind the same lock.
- Check the downstream dependency. A bank API or payment gateway rate-limiting you exactly when you need it most is a common, easy-to-miss culprit.
FixPool tuning, a missing circuit breaker, and a new alert on saturation metrics that fires before the error rate does — because by the time errors show up, you're already behind.
SEV-2
INC-2233
primary-db / replicas
A failover leaves reads stale for 90 seconds
Nothing breaks in the traditional sense — the service stays up. That's exactly why it's dangerous. Nobody pages you for data that's merely wrong.
- Expose replication lag as a first-class metric, with alerting tied directly to failover events — not discovered after the fact.
- Run synthetic canaries. Write a known value, read it back on a schedule, alert on mismatch.
- Route consistency-sensitive reads to primary — a balance check right after a transaction can't afford to trust eventual consistency to catch up in time.
FixTreat failover as an event that automatically tightens monitoring for a window afterward, rather than waiting for steady-state alerting to notice.
SEV-2
INC-2229
autoscaler
Autoscaling goes silent during a flash sale
A CPU-based autoscaling policy stops adding nodes exactly when a flash sale needs it most. Deceptive, because everything looks correctly configured on paper.
- Check whether metrics are actually arriving. Under heavy load, the metrics pipeline itself can fall behind — you can be flying blind while believing you're being monitored.
- Look for a hard ceiling. Node pool max size, cloud quota, no available instance type in the zone.
- Question the signal itself. If the real constraint is memory or I/O wait, a CPU-based policy will never fire, no matter how stressed the system is.
FixMulti-metric autoscaling (queue depth, custom app metrics) plus pre-scaling ahead of known peak events. Single-metric autoscaling is a fair-weather strategy.
SEV-1
INC-2214
edge-routing
A DNS change sends traffic to the wrong cluster
Production traffic gets misrouted after a routine DNS change. This is a contain-first, understand-later situation — the instinct to dig into "why" before "how do I stop it" is the wrong instinct here.
- Revert immediately — but don't trust DNS alone to save you. TTLs lie, propagation isn't instant. Flip routing at the load balancer or edge layer if speed matters more than elegance.
- Ask what state mutated on the wrong cluster before you caught it. Any writes accepted there are now a data-integrity question, not a routing question.
- Pause anything asynchronous — queues, background workers — on the wrong cluster until you're certain nothing partial got committed.
- Reconcile after containment. Look for orphaned records or duplicate transactions written during the misroute window.
FixStaged rollout and a health-check gate on any DNS change, plus lower TTLs on records that sit in front of anything critical.
Pattern across all four
The failures that matter most rarely announce themselves. They hide in saturation, in propagation delay, in a metric nobody's watching, in consistency nobody's testing. The job isn't memorizing incidents — it's building the reflex to ask "what would this look like before it becomes visible" every time you design a system.
Next in this series
Platform design trade-offs — multi-region rollouts, security guardrails at scale, and the tension between developer velocity and compliance.