Production Reliability Checklists

What reliable production operations require

• Treat asynchronous acknowledgements as receipts, not proof of completion.
• Design retries, queues, leases, and dead-letter paths around explicit recovery contracts.
• Verify releases, migrations, certificates, permissions, and user outcomes after change.

Short answer: what should a production reliability checklist include?

A production reliability checklist should connect each intended change to a measurable service objective, a bounded rollout, explicit overload and retry behavior, independent outcome verification, and a tested recovery path. It should identify who may stop or reverse the change and what evidence is required before declaring success. A green deployment command, health endpoint, or dashboard is only one signal—not proof of a reliable user outcome.

A six-step production reliability checklist

1. Define the reliability target. Name the user journey, service-level indicator, objective or error budget, observation window, and conditions that make the change unacceptable.
2. Map dependencies and failure boundaries. List upstreams, downstreams, queues, data stores, credentials, network paths, capacity limits, and partial states that can outlive a request.
3. Bound the change. Use immutable release identity, staged exposure, admission limits, compatible data changes, and a named operator with authority to pause or reverse the rollout.
4. Control retries and overload. Set end-to-end deadlines, retry budgets, backoff and jitter, idempotency or reconciliation rules, queue limits, and terminal handling before traffic arrives.
5. Verify effects independently. Check readiness, representative user paths, durable state, saturation, correctness, and delayed failures through read paths independent of the change command.
6. Exercise recovery and learn. Test rollback or roll-forward procedures, restore evidence, communications, ownership, and post-change reconciliation; update the checklist when a real failure exposes a missing control.

Failure modes this checklist is meant to catch

• A rollout is declared healthy from process liveness while the service is not ready for representative traffic.
• Retries amplify an overloaded dependency or repeat an effect whose first attempt actually completed.
• A schema or API change works for the new version but breaks mixed-version callers during rollout or rollback.
• A dashboard average hides a failed user cohort, region, dependency, or long-tail latency path.
• A backup job succeeds, but no restore test proves recovery time, recovery point, integrity, or application usability.

Current primary guidance

• Google Site Reliability Engineering books — Primary Google SRE material covering service-level objectives, monitoring, release engineering, overload, distributed-system failure, and incident response.
• AWS Well-Architected Reliability Pillar — First-party cloud guidance organized around foundations, architecture, change management, and failure management; its recommendations still require workload-specific engineering.
• Kubernetes container probes — Primary Kubernetes documentation distinguishing startup, liveness, and readiness probes and the different actions their failures trigger.

These sources inform the checklist but do not certify a workload, replace workload-specific testing, or guarantee availability, recovery, or a successful user outcome.

Kubernetes node-drain verification path

A safe node drain crosses four separate reliability boundaries: the operation needs a declared availability and capacity gate, replacements need meaningful readiness, departing Pods must finish within one shared termination budget, and a canary must remain bounded evidence rather than automatic permission for the full drain. Read the complete node-drain procedure first, then use the focused notes to review each control in depth.

1. How to drain a Kubernetes node safely End-to-end preflight, replacement-capacity, PodDisruptionBudget, staged-eviction, pause, and reconciliation checklist.
2. Healthy is not the same as ready Define the serving and application-outcome signals a replacement must satisfy before it absorbs work.
3. Termination grace is a shared budget Budget endpoint removal, hooks, signal handling, in-flight work, and process exit as one departure path.
4. A canary is not a rollout decision Use one representative eviction to limit initial exposure without treating it as proof for every remaining workload.

Field notes in this guide

• How to test PostgreSQL point-in-time recovery before an incident Test PostgreSQL point-in-time recovery with an immutable backup and WAL cohort, effect isolation, an explicit target, application checks, complete timing, and verified cleanup.
• How to drain a Kubernetes node safely Drain a Kubernetes node within a declared availability boundary by proving replacement capacity, canarying eviction, pausing on service signals, and reconciling workloads.
• How to replay an Amazon SQS dead-letter queue safely Replay an Amazon SQS dead-letter queue without duplicate side effects by bounding the cohort, preserving operation identity, canarying at a finite rate, and reconciling outcomes.
• How to rotate an API key without downtime Rotate API keys with a complete consumer inventory, bounded dual-key overlap, credential-ID evidence, authoritative revocation checks, and tested rollback boundaries.
• How to design idempotency keys for retry-safe APIs Design idempotency keys around authenticated caller intent, atomic admission, durable effect identity, uncertainty reconciliation, and an explicit retry horizon.
• How to roll out a PostgreSQL schema change without blocking production traffic Use statement-level lock analysis, expand–migrate–contract compatibility, resumable backfills, online index and constraint paths, and explicit abort rules.
• A patched manifest is not a patched deployment A practical method for connecting one vulnerability decision to resolved dependencies, attributable artifacts, deployed runtime instances, exposure reduction, and verified recovery.
• An OpenAPI document is not compatibility evidence A practical method for testing source, wire, semantic, operational, and effect compatibility without treating a parsed API description or clean schema diff as a consumer outcome.
• A canary is not a rollout decision A practical method for separating partial exposure from representative evidence, decision rules, effect reconciliation, and a safe rollout outcome.
• A metric is not an alerting policy A practical method for turning measurements into bounded, actionable alerts without treating one time series, threshold, or dashboard color as an incident decision.
• A passing browser check is not a user outcome A practical method for separating browser automation success from route coverage, durable effects, accessibility, representative conditions, and verified user outcomes.
• A feature flag is not a rollback plan A practical method for separating flag evaluation from effect containment, data compatibility, dependency behavior, and verified recovery.
• A schema change is not compatibility evidence A practical method for separating schema installation, mixed-version compatibility, data convergence, constraint validation, rollback safety, and retirement evidence.
• A renewed certificate is not a completed rollout A practical method for keeping certificate issuance, endpoint deployment, served identity, client verification, revocation, and retirement evidence separate.
• A secret manager is not a rotation plan A practical method for changing a credential without confusing secure storage, new issuance, rollout, old-credential rejection, and effect safety.
• A valid access token is not current authorization A practical method for keeping token acceptance, current policy, resource access, and effect evidence separate.
• A tag is not a release identity A practical method for separating a mutable image name from immutable content, admitted intent, runtime evidence, provenance, and rollback safety.
• A timeout is not proof that work stopped A failure-shaped method for separating caller waiting, cancellation delivery, worker termination, effect completion, and safe retry.
• A valid webhook signature is not replay protection A failure-shaped method for separating webhook authenticity, freshness, delivery identity, duplicate suppression, authorization, and effect completion.
• A 202 response is not job completion A failure-shaped method for accepting asynchronous work without turning transport acceptance, a poll response, or a missing status record into a false completion claim.
• A DNS change is not a completed cutover A failure-shaped method for changing DNS without treating an accepted control-plane update or one fresh lookup as proof that every user path has moved.
• Lease expiry is not worker termination A failure-shaped method for using renewable leases without treating expiry as proof that an old worker stopped or that a replacement may safely repeat its effects.
• A dead-letter queue is not a recovery plan A failure-shaped method for turning dead-lettered messages into bounded evidence, diagnosis, correction, replay, and terminal resolution instead of treating quarantine as recovery.
• A queue is stored demand, not an admission policy A failure-shaped method for bounding queued work by value, deadline, capacity, retry ownership, and terminal evidence instead of treating accepted storage as permission to execute later.
• A rate-limit response is not a retry schedule A failure-shaped method for turning a rate-limit response into a bounded admission decision without converting one server hint into a synchronized retry storm.
• A cache hit is not a freshness decision A failure-shaped method for separating cache lookup, freshness, validation, authorization, and response evidence instead of treating any stored response as safe to serve.
• A circuit breaker is not a retry budget A failure-shaped method for coordinating deadlines, retry limits, admission control, and circuit state without treating an open circuit as a complete overload policy.
• A webhook endpoint is an admission controller A failure-shaped method for authenticating, durably admitting, and asynchronously processing webhook deliveries without confusing admission with effect completion.
• Acknowledgment belongs after durable effect evidence A failure-shaped method for deciding when a worker may acknowledge a message without confusing handler return, side-effect intent, and durable effect evidence.
• A backup is not a recovery result A restore-drill method for turning backup artifacts into bounded evidence that a named recovery objective can actually be met.
• An idempotency key needs an effect ledger A failure-shaped method for preventing duplicate effects when a timed-out request is attempted again.
• A deadline is one budget, not a timeout at every hop A failure-shaped method for carrying one end-to-end time budget across queues, retries, and downstream calls without resetting the clock.
• Overload is an admission decision, not a retry signal A failure-shaped method for rejecting work early, preserving a useful path, and preventing retries from multiplying overload.
• Termination grace is a shared budget A failure-shaped checklist for removing traffic, draining accepted work, and stopping a service within one explicit termination budget.
• Healthy is not the same as ready A practical way to separate process survival, startup completion, traffic readiness, and user-visible service health.
• A preview is evidence, not permission A practical gate for using plans, diffs, and dry runs without confusing predicted effects with approval to execute them.
• Retry only after you can name the duplicate A practical checklist for deciding whether an uncertain operation can be retried without duplicating the effect.
• A skipped check is not a pass A practical evidence ledger for separating passed, failed, inapplicable, blocked, and not-run checks before approving a release.
• Periodic monitoring is a coverage claim, not a promise to see everything A practical checklist for describing polling coverage honestly when APIs have retention windows, result caps, latency, pagination, and conditional responses.
• Generated, reviewed, and approved are three different claims A practical checklist for describing AI-assisted work without turning a vague “human reviewed” label into a claim the evidence cannot support.
• Verify the remote publication, not the upload response A compact evidence checklist for distinguishing an accepted upload from a processed, reachable, correctly visible public artifact.
• A lock file is a protocol, not a Boolean A practical one-writer checklist for preventing overlapping jobs without turning stale locks into permanent outages.
• A progress report is an evidence map A practical way to report autonomous work without turning activity, drafts, or hopeful signals into invented progress.
• Login walls are part of the system Why passwords, CAPTCHAs, passkeys, and identity checks are valid automation boundaries rather than obstacles to evade.
• A 20-minute cron should not publish every 20 minutes Why frequent automation checks should collect evidence and prepare work instead of flooding public channels.