Incident Patterns

AI Model Failure Patterns — Failure Patterns (P1)

Difficulty: Advanced

Key Points for AI Model Failure Patterns

AI failures are silent failures. The system returns 200 OK with confidently wrong answers, and nothing in your infrastructure monitoring catches it.
Training-serving skew is the most common root cause of AI production issues, where the model sees different data in production than it saw during training
Data quality failures cascade into model quality failures with a time delay, often days or weeks, making root cause analysis harder
Canary deployments for models must include model quality metrics like accuracy and prediction distribution, not just latency and error rate
Feedback loop delay makes AI incidents fundamentally harder to detect than traditional software failures. The bad model may run for days before anyone notices.

Incident Timeline for AI Model Failure Patterns

T+0d: Data pipeline ingests corrupted data source with missing values and changed schema
T+1d: Scheduled model retraining completes on corrupted data, model weights updated
T+1d: Model registry promotes retrained model, canary deployment begins
T+2d: Canary passes automated checks since latency and error rate look normal
T+3d: Full rollout to 100% traffic, model quality degradation begins but goes undetected
T+7d: Customer complaints spike as recommendation quality drops noticeably
T+8d: Investigation reveals training data corruption, model rolled back to previous version

Detection Signals for AI Model Failure Patterns

Model prediction distribution shift where outputs cluster differently than baseline
Feature importance change where a previously unimportant feature suddenly dominates
Online-offline metric divergence where model performs well on test data but poorly in production
Business metric degradation that correlates with model deployment timing
Increase in user-reported quality issues or feedback submission rate

Prevention Strategies for AI Model Failure Patterns

Implement data validation gates (Great Expectations, Deequ) at every pipeline boundary
Run automated model quality tests (accuracy, bias, calibration) on a held-out evaluation set before every deployment
Use shadow mode deployments that compare new model outputs against the production model before routing real traffic
Monitor business-level metrics (conversion rate, user engagement, support tickets) alongside infrastructure metrics
Maintain the ability to instantly rollback to the previous model version

Common Mistakes with AI Model Failure Patterns

Relying solely on infrastructure monitoring (CPU, memory, latency) for AI systems when the real failures are in output quality
Not validating training data schema and distribution before kicking off model retraining pipelines
Deploying a retrained model without comparing its quality metrics against the currently running production version
Having no rollback plan for model deployments, treating model updates like irreversible database migrations
Expecting immediate detection of AI quality issues when feedback loops inherently introduce multi-day delays

Related to AI Model Failure Patterns

Cascading Failure Patterns, Deployment Rollback Patterns

Anatomy of a Production Incident — Incident Response (P1)

Difficulty: Intermediate

Key Points for Anatomy of a Production Incident

The first five minutes determine whether an incident escalates or gets contained — fast acknowledgment is everything
Clear role assignment (incident commander, communicator, investigator) prevents chaos and duplication of effort
Every incident has a communication timeline that matters as much as the technical timeline
Rollback is almost always faster than forward-fixing under pressure — default to rolling back
Documentation during the incident (not after) produces the most accurate post-mortem timelines

Incident Timeline for Anatomy of a Production Incident

T+0m: Monitoring alert fires — error rate exceeds 5% threshold
T+2m: On-call engineer acknowledges alert, begins initial triage
T+5m: Incident declared, communication channel opened, stakeholders notified
T+10m: Root cause hypothesis formed — recent deployment suspected
T+15m: Rollback initiated to last known good version
T+20m: Error rate returns to baseline, monitoring confirms recovery
T+25m: All-clear communicated, incident moved to post-mortem phase

Detection Signals for Anatomy of a Production Incident

Error rate spike above baseline (>2x normal)
Latency P99 exceeding SLO threshold
Sudden drop in successful transaction volume
Customer support ticket volume spike

Prevention Strategies for Anatomy of a Production Incident

Implement canary deployments to catch issues before full rollout
Set up automated rollback triggers based on error rate thresholds
Maintain runbooks for common incident types
Practice incident response regularly with game days

Common Mistakes with Anatomy of a Production Incident

Too many people investigating simultaneously without coordination — creates noise and conflicting actions
Skipping the declaration step and trying to quietly fix the issue — delays stakeholder awareness
Attempting a forward fix under time pressure instead of rolling back to a known good state
Not communicating status updates regularly — silence breeds panic among stakeholders

Related to Anatomy of a Production Incident

Blameless Post-Mortem Guide, Deployment Rollback Patterns

Blameless Post-Mortem Guide — Post-Mortem Practice (P2)

Difficulty: Intermediate

Key Points for Blameless Post-Mortem Guide

Blameless does not mean accountabilityless — it means focusing on systems and processes, not individual fault
The five whys technique works best when you follow causal chains, not blame chains
Action items must be concrete, assigned, and time-bound — vague improvements never get done
Publishing post-mortems widely builds organizational learning and prevents duplicate failures
The quality of a post-mortem is measured by whether the same incident could happen again, not by its length

Incident Timeline for Blameless Post-Mortem Guide

Day 1: Incident resolved, initial timeline documented while memory is fresh
Day 2-3: Post-mortem document drafted with contributing factors and timeline
Day 3-5: Post-mortem review meeting with all involved parties
Day 5-7: Action items assigned with owners and deadlines
Day 14: First check-in on action item progress
Day 30: Action items completed and verified

Detection Signals for Blameless Post-Mortem Guide

Post-mortems being skipped or delayed beyond one week
Action items not being completed within 30 days
Same root cause appearing in multiple incidents
Team members reluctant to participate in post-mortems

Prevention Strategies for Blameless Post-Mortem Guide

Schedule post-mortems within 48 hours while context is fresh
Use a standard template to ensure consistency across incidents
Track action items in a shared system with due dates and owners
Review open action items in weekly engineering leadership meetings

Common Mistakes with Blameless Post-Mortem Guide

Treating the post-mortem as a formality rather than a genuine learning opportunity
Assigning action items without clear owners or deadlines, ensuring nothing gets done
Stopping at the proximate cause instead of digging into systemic contributing factors
Making the post-mortem document so long and detailed that nobody reads it

Cascading Failure Patterns — Failure Patterns (P0)

Difficulty: Advanced

Key Points for Cascading Failure Patterns

Cascading failures propagate through resource exhaustion, not through errors — a service that queues forever is more dangerous than one that crashes
Circuit breakers are the primary defense — they convert slow failures into fast failures that the system can handle
Bulkhead isolation limits the blast radius by ensuring one failing dependency cannot consume all shared resources
Load balancer health checks can accelerate cascades by concentrating traffic on fewer healthy instances
The most dangerous cascading failures cross team boundaries because no single team sees the full picture

Incident Timeline for Cascading Failure Patterns

T+0m: Database connection pool exhausted on primary database
T+1m: Application threads blocked waiting for connections, request queue grows
T+2m: Load balancer health checks start failing as response times exceed threshold
T+3m: Traffic redistributed to remaining healthy instances, overwhelming them
T+5m: All instances marked unhealthy, upstream services begin queuing
T+8m: Upstream services exhaust their own connection pools — cascade propagates
T+15m: Full system outage across all dependent services

Detection Signals for Cascading Failure Patterns

Connection pool utilization exceeding 75%
Thread pool saturation across multiple services simultaneously
Exponential growth in error rates across service boundaries
Upstream services showing increased latency without direct load increase

Prevention Strategies for Cascading Failure Patterns

Implement circuit breakers at every service boundary
Set aggressive timeouts — fail fast rather than queue indefinitely
Use bulkhead isolation to prevent one failure domain from affecting others
Design graceful degradation paths for every critical dependency
Load test with dependency failures, not just high traffic

Common Mistakes with Cascading Failure Patterns

Setting timeouts too high or not at all — letting blocked threads accumulate until the connection pool is exhausted
Not testing failure scenarios — only load testing the happy path misses the most dangerous failure modes
Using shared thread pools across all dependencies — one slow service blocks requests to all services
Ignoring partial failures — treating the system as either fully healthy or fully broken, with no graceful degradation
Retrying without backoff during a cascade — amplifying the load on an already overwhelmed service

Related to Cascading Failure Patterns

Thundering Herd & Retry Storms, Deployment Rollback Patterns

Chaos Engineering Principles — Chaos Engineering (P3)

Difficulty: Expert

Key Points for Chaos Engineering Principles

Chaos engineering is not about breaking things randomly — it is about forming hypotheses about system behavior and testing them scientifically
The steady-state hypothesis defines what 'working correctly' looks like in measurable terms before you inject any failure
Blast radius control is non-negotiable — start small and expand only after confidence grows
Game days are the team-level practice of chaos engineering — scheduled exercises where the team responds to simulated incidents
The value of chaos engineering compounds over time as each experiment reveals and fixes weaknesses before they cause real outages

Incident Timeline for Chaos Engineering Principles

Week 1: Define steady-state hypothesis and success metrics for the experiment
Week 2: Design experiment with minimal blast radius (single AZ, low traffic)
Week 3: Run experiment during business hours with the team ready to abort
Week 3: Analyze results — did the system behave as expected?
Week 4: Document findings, create action items for gaps discovered
Week 5-6: Implement fixes and re-run experiment to verify

Detection Signals for Chaos Engineering Principles

Systems that have never been tested for failure modes
Runbooks that haven't been validated in production
Recovery procedures that exist only in documentation
Teams that can't articulate their system's failure domains

Prevention Strategies for Chaos Engineering Principles

Start chaos experiments in non-production environments first
Always have a kill switch to abort experiments immediately
Run experiments during business hours when the team is available
Begin with known failure modes before exploring unknown ones
Get organizational buy-in before running chaos experiments

Common Mistakes with Chaos Engineering Principles

Running chaos experiments without a kill switch — if the experiment causes unexpected damage, you need to stop it immediately
Starting with large blast radius experiments instead of building confidence with small, contained tests
Treating chaos engineering as a one-time activity rather than an ongoing practice that evolves with the system
Not getting buy-in from leadership before running experiments — chaos engineering requires organizational support to be effective

Related to Chaos Engineering Principles

Cascading Failure Patterns, Blameless Post-Mortem Guide

Deployment Rollback Patterns — Incident Response (P1)

Difficulty: Intermediate

Key Points for Deployment Rollback Patterns

Rollback speed is the most important metric for deployment safety — if you cannot roll back in under 5 minutes, your deployment process is risky
Feature flags decouple deployment from release, letting you deploy code without activating it until you are ready
Database migrations must be backward-compatible — the old code must work with the new schema, because rollback means running old code
Automated canary analysis removes human judgment from the go/no-go decision during deployments
Blue-green deployments provide instant rollback by keeping the previous version running and switching traffic at the load balancer

Incident Timeline for Deployment Rollback Patterns

T+0m: New version deployed via canary to 5% of traffic
T+5m: Error rate on canary instances exceeds threshold (>1% 5xx)
T+6m: Automated rollback triggered, canary traffic redirected to stable version
T+8m: All traffic serving from last known good version
T+10m: Engineering team begins analyzing failed deployment logs
T+30m: Root cause identified, fix developed and tested

Detection Signals for Deployment Rollback Patterns

Error rate divergence between canary and baseline instances
Latency increase on recently deployed instances
Memory or CPU pattern change after deployment
Log volume anomaly on new version

Prevention Strategies for Deployment Rollback Patterns

Always maintain the ability to rollback within 5 minutes
Use feature flags to decouple deployment from release
Implement automated canary analysis with automatic rollback
Test rollback procedures regularly, not just deployments
Keep database migrations backward-compatible for safe rollbacks

Common Mistakes with Deployment Rollback Patterns

Deploying database migrations that break the previous version of the application — making rollback impossible
Not testing the rollback path — only testing the deploy path means you discover rollback is broken during an incident
Coupling multiple service deployments together — if one needs rollback, all must roll back in the correct order
Relying on manual rollback procedures during high-stress incidents when automated rollback should be the default

Related to Deployment Rollback Patterns

Anatomy of a Production Incident, Cascading Failure Patterns

LLM Production Incident Patterns — Incident Response (P0)

Difficulty: Expert

Key Points for LLM Production Incident Patterns

LLM provider outages are external dependency failures, similar to AWS outages. You have zero control over when they happen or how long they last.
LLM latency is measured in seconds, not milliseconds. Your circuit breakers, timeouts, and retry policies must be calibrated for this reality.
Cost runaway is a unique LLM incident type that doesn't exist with traditional APIs. A bug that sends long prompts in a loop can burn thousands of dollars in minutes.
Prompt injection incidents are security incidents that require the same response protocol as XSS or SQL injection, not just a bug fix.
Output quality degradation may not trigger any infrastructure alert. The API returns 200 OK while the model produces nonsensical answers.

Incident Timeline for LLM Production Incident Patterns

T+0m: LLM provider API returns 429 rate limit errors during peak traffic
T+1m: Application retry logic amplifies load, all LLM-powered features start failing
T+2m: No graceful degradation path exists, pages dependent on LLM responses show errors
T+5m: Customer support receives flood of complaints about broken AI features
T+10m: Engineering team manually disables LLM features via feature flags
T+15m: Non-AI fallback paths activated, user experience partially restored
T+30m: LLM provider restores service, gradual re-enablement of AI features begins

Detection Signals for LLM Production Incident Patterns

LLM API response time exceeding 10 seconds when normal P95 is 2-5 seconds
Rate limit (429) or server error (500/503) responses from LLM provider
Cost anomaly with sudden spike in per-minute API spending
Content quality degradation with user reports of nonsensical or irrelevant AI outputs
Prompt injection detection where outputs contain system prompt content or unexpected instructions

Prevention Strategies for LLM Production Incident Patterns

Design graceful degradation for every LLM-powered feature, decide what the user sees when the LLM is unavailable
Implement circuit breakers tuned for LLM API latency patterns (seconds, not milliseconds)
Configure multi-provider failover so you can route to Anthropic when OpenAI is down, and vice versa
Set rate limit budgets well below provider limits to leave headroom for traffic spikes
Cache aggressively using semantic caching for repeated similar queries, which can serve 30-60% of requests without an API call

Common Mistakes with LLM Production Incident Patterns

Building LLM-powered features with no degradation path, so when the provider goes down, the feature shows raw errors to users
Setting circuit breaker timeouts based on traditional API patterns (100-500ms) when LLM responses routinely take 2-10 seconds
Not implementing cost circuit breakers, allowing a runaway bug or traffic spike to generate an unbounded API bill
Treating LLM provider outages as rare black swan events when they actually happen multiple times per month across major providers
Not logging prompt inputs and model outputs, making it impossible to investigate quality incidents or prompt injection attacks after the fact

Related to LLM Production Incident Patterns

Cascading Failure Patterns, Thundering Herd & Retry Storms

Thundering Herd & Retry Storms — Failure Patterns (P0)

Difficulty: Advanced

Key Points for Thundering Herd & Retry Storms

A thundering herd happens when many clients simultaneously request the same resource — the classic example is cache expiry or restart
Retry storms are self-reinforcing: each retry adds load, which causes more timeouts, which causes more retries
Exponential backoff without jitter still causes synchronized retry waves — jitter is essential to spread the load
Retry budgets cap the total retry traffic at the system level, preventing any single client from overwhelming the backend
The safest default is to not retry at all — only add retries when you have confirmed they improve reliability with backoff and jitter

Incident Timeline for Thundering Herd & Retry Storms

T+0m: Cache server restarts, all cached entries lost (cold cache)
T+0m: Thousands of requests simultaneously miss cache and hit database
T+1m: Database CPU spikes to 100%, query latency increases 50x
T+2m: Application timeouts trigger automatic retries with no backoff
T+3m: Retry storm multiplies load by 3-5x, database becomes unresponsive
T+5m: All services dependent on this database begin failing
T+10m: Manual intervention required — retries disabled, cache warmed gradually

Detection Signals for Thundering Herd & Retry Storms

Sudden spike in database QPS after a cache restart or miss pattern
Retry rate exceeding 3x normal across multiple clients
Backend latency increasing despite no organic traffic growth
Load balancer reporting connection queue depth growing

Prevention Strategies for Thundering Herd & Retry Storms

Implement exponential backoff with jitter on all retry logic
Use cache stampede protection (lock-based or probabilistic early expiration)
Set retry budgets — limit total retries per time window
Implement circuit breakers that open before retry storms form
Design cache warming procedures for cold-start scenarios

Common Mistakes with Thundering Herd & Retry Storms

Using fixed retry intervals instead of exponential backoff — creates synchronized retry waves that hit the backend in bursts
Not adding jitter to backoff timers — all clients retry at the same calculated interval, recreating the thundering herd
Retrying non-idempotent operations — causing duplicate writes, double charges, or inconsistent state
Setting cache TTLs to the same value across all keys — causing mass expiration events that trigger thundering herds

Related to Thundering Herd & Retry Storms

Cascading Failure Patterns, Deployment Rollback Patterns

Cracking Walnuts

AI Model Failure Patterns

Cascading Failure Patterns

Thundering Herd & Retry Storms

Anatomy of a Production Incident

Deployment Rollback Patterns

LLM Production Incident Patterns

Chaos Engineering Principles

Blameless Post-Mortem Guide

Cracking Walnuts

Incident Patterns

AI Model Failure Patterns

Cascading Failure Patterns

Thundering Herd & Retry Storms

Anatomy of a Production Incident

Deployment Rollback Patterns

LLM Production Incident Patterns

Chaos Engineering Principles

Blameless Post-Mortem Guide

Cracking Walnuts

Incident Patterns

AI Model Failure Patterns

Cascading Failure Patterns

Thundering Herd & Retry Storms

Anatomy of a Production Incident

Deployment Rollback Patterns

LLM Production Incident Patterns

Chaos Engineering Principles

Blameless Post-Mortem Guide

AI Model Failure Patterns — Failure Patterns (P1)

Key Points for AI Model Failure Patterns

Incident Timeline for AI Model Failure Patterns

Detection Signals for AI Model Failure Patterns

Prevention Strategies for AI Model Failure Patterns

Common Mistakes with AI Model Failure Patterns

Related to AI Model Failure Patterns

Anatomy of a Production Incident — Incident Response (P1)

Key Points for Anatomy of a Production Incident

Incident Timeline for Anatomy of a Production Incident

Detection Signals for Anatomy of a Production Incident

Prevention Strategies for Anatomy of a Production Incident

Common Mistakes with Anatomy of a Production Incident

Related to Anatomy of a Production Incident

Blameless Post-Mortem Guide — Post-Mortem Practice (P2)

Key Points for Blameless Post-Mortem Guide

Incident Timeline for Blameless Post-Mortem Guide

Detection Signals for Blameless Post-Mortem Guide

Prevention Strategies for Blameless Post-Mortem Guide

Common Mistakes with Blameless Post-Mortem Guide

Related to Blameless Post-Mortem Guide

Cascading Failure Patterns — Failure Patterns (P0)

Key Points for Cascading Failure Patterns

Incident Timeline for Cascading Failure Patterns

Detection Signals for Cascading Failure Patterns

Prevention Strategies for Cascading Failure Patterns

Common Mistakes with Cascading Failure Patterns

Related to Cascading Failure Patterns

Chaos Engineering Principles — Chaos Engineering (P3)

Key Points for Chaos Engineering Principles

Incident Timeline for Chaos Engineering Principles

Detection Signals for Chaos Engineering Principles

Prevention Strategies for Chaos Engineering Principles

Common Mistakes with Chaos Engineering Principles

Related to Chaos Engineering Principles

Deployment Rollback Patterns — Incident Response (P1)

Key Points for Deployment Rollback Patterns

Incident Timeline for Deployment Rollback Patterns

Detection Signals for Deployment Rollback Patterns

Prevention Strategies for Deployment Rollback Patterns

Common Mistakes with Deployment Rollback Patterns

Related to Deployment Rollback Patterns

LLM Production Incident Patterns — Incident Response (P0)

Key Points for LLM Production Incident Patterns

Incident Timeline for LLM Production Incident Patterns

Detection Signals for LLM Production Incident Patterns

Prevention Strategies for LLM Production Incident Patterns

Common Mistakes with LLM Production Incident Patterns

Related to LLM Production Incident Patterns

Thundering Herd & Retry Storms — Failure Patterns (P0)

Key Points for Thundering Herd & Retry Storms

Incident Timeline for Thundering Herd & Retry Storms

Detection Signals for Thundering Herd & Retry Storms

Prevention Strategies for Thundering Herd & Retry Storms

Common Mistakes with Thundering Herd & Retry Storms

Related to Thundering Herd & Retry Storms

Cracking Walnuts

Incident Patterns

AI Model Failure Patterns

Cascading Failure Patterns

Thundering Herd & Retry Storms

Anatomy of a Production Incident

Deployment Rollback Patterns

LLM Production Incident Patterns

Chaos Engineering Principles

Blameless Post-Mortem Guide

AI Model Failure Patterns — Failure Patterns (P1)

Key Points for AI Model Failure Patterns

Incident Timeline for AI Model Failure Patterns

Detection Signals for AI Model Failure Patterns