Alfred's Field Notes

Production reliability checklists

How to replay an Amazon SQS dead-letter queue safely

Bound the replay cohort, make the consumer duplicate-safe, canary at a finite velocity, and reconcile authoritative business effects before declaring recovery.

Field note · · Written by Alfred

Short answer

To replay an Amazon SQS dead-letter queue without causing duplicate side effects, treat redrive as a new delivery attempt—not as proof that the original work failed cleanly. Freeze the exact source queue, dead-letter queue, destination, message-age range, approximate message set, failed consumer version, and repaired consumer version. Before redrive, classify a privacy-safe sample, quarantine work that is malformed, expired, unauthorized, or no longer meaningful, and make the consumer claim a stable business-operation identity atomically before any irreversible effect.

Test the repaired path against duplicate delivery, timeout after the effect commits, partial failure, stale messages, and concurrent live traffic. Redrive a small canary at a finite velocity below verified consumer and dependency headroom. Pause on predefined error, latency, saturation, duplicate-claim, ordering, or unexplained-outcome signals. Finish by reconciling every selected logical operation as succeeded once, safely deduplicated, still pending, quarantined, or unknown. An empty DLQ is not recovery evidence, and Amazon SQS standard queues do not provide exactly-once delivery.

The operating rule is:

Redrive only a named message boundary into a duplicate-safe consumer, then reconcile business effects rather than trusting queue movement.

1. Freeze the incident and replay boundary

Do not begin with “redrive everything.” Record the state that makes this recovery distinct from live traffic:

incident_id:
source_queue_arn:
dead_letter_queue_arn:
selected_destination_arn:
queue_type: standard | fifo
message_age_start:
message_age_end:
approximate_selected_count:
failed_consumer_version:
repaired_consumer_version:
failure_causes_known:
oldest_selected_message_age:
source_and_dlq_retention:
live_traffic_state:
redrive_owner:
reconciliation_owner:

Amazon SQS moves a message to a configured DLQ after the source queue's redrive policy reaches maxReceiveCount. That threshold is a retry policy, not a diagnosis. A message can arrive because of a persistent data defect, a transient dependency failure, a consumer deployment bug, an undersized visibility timeout, an authorization change, or an effect that committed before acknowledgement failed.

Record the queue type and destination exactly. AWS permits redrive to the source queue or another queue of the same type. Choosing a different destination can isolate replay from live traffic, but it also changes routing and operational assumptions. Choosing the source queue reintroduces work beside new messages. Neither option is safe by default.

SQS queue counts are approximate. Preserve the best available boundary using timestamps, message attributes, producer operation IDs, incident timing, and a manifest of sampled or selected logical operations where feasible. Do not claim exact selection merely because a dashboard showed one number.

Retention is part of the boundary. AWS documents different age behavior for standard and FIFO queues when messages move to a DLQ. Check the current source and DLQ retention settings and the age visible for the queue type. Do not assume that redrive restores expired business authority or that a syntactically present message is still valid to execute.

2. Classify before replaying

Inspect a privacy-minimized sample large enough to expose distinct failure shapes. Avoid copying complete payloads into tickets, chat, logs, or dashboards. Use redacted schemas, safe operation identifiers, failure classes, and counts.

Classify messages into at least:

Quarantine rather than cycle malformed, unauthorized, obsolete, poison, and unknown work. Quarantine needs a restricted destination, owner, retention policy, and decision path; it must not become an unmonitored second DLQ.

Do not mutate the only copy of a failed message in place. If a transformation is necessary, record the original transport identity, transformation version, new message identity, stable business-operation identity, authorization basis, and reviewer. A corrected payload represents an explicit repair decision, not the unchanged original event.

3. Preserve one stable business-operation identity

An SQS MessageId identifies a transport message. A receive attempt changes receipt handles, and redrive can change transport-level details. Duplicate protection must therefore bind to the logical business effect, not to one receive attempt or a newly generated redrive identity.

Define an operation identity from authenticated producer intent or durable domain state:

operation_type:
producer_or_tenant_scope:
business_subject:
stable_operation_id:
canonical_intent_digest:
authorization_version_or_reference:
required_postcondition:
reversal_or_compensation_path:

Examples might be one invoice issuance, one entitlement grant, one export request, or one notification obligation. The exact fields are application-specific. Do not put personal data or secrets into metric labels or broadly visible deduplication keys.

Before an irreversible side effect, atomically claim the scoped operation in durable storage. A minimal effect ledger can hold:

scope + stable_operation_id
intent_digest
state: claimed | executing | succeeded | failed_terminal | uncertain | quarantined
first_message_id
latest_message_id
attempt_count
consumer_version
external_effect_id
last_authoritative_observation
updated_at

A conditional write or unique constraint must decide the winner before the effect. “Check whether it happened, perform it, then write success” races when two consumers receive duplicates. If the effect is external, carry the stable operation ID into the provider's idempotency facility or resource metadata when supported. If the provider times out after accepting the operation, reconcile by that identity before any second execution.

AWS states that standard queues use at-least-once delivery and that a message copy can be delivered again. An SQS visibility timeout temporarily hides a received message; it does not delete it, cancel a worker, or prove that no effect occurred. The consumer needs duplicate-safe effect semantics independently of the redrive control.

4. Repair and test the complete consumer path

Do not redrive into an unproven fix. Bind the candidate to an immutable consumer build, configuration, schema versions, permissions, dependency versions, and feature flags. Test the complete path from receive through authoritative postcondition and delete.

At minimum, exercise:

  1. the same logical operation delivered twice under the same message ID;
  2. the same logical operation delivered under different message IDs;
  3. two consumers race to claim one operation;
  4. the effect commits and the consumer times out before deleting the message;
  5. the effect commits but local success recording fails;
  6. processing exceeds the initial visibility timeout;
  7. visibility extension fails while the worker continues;
  8. a stale message is valid JSON but no longer authorized;
  9. a partial batch contains repairable and quarantined items;
  10. live and replay traffic target the same business subject concurrently;
  11. the dependency accepts an idempotency identity but returns an uncertain result;
  12. a poison message reaches the destination and would return to the DLQ; and
  13. the repaired build rolls back while replay is active.

Match visibility timeout to the measured processing path and extend it when the application can prove the worker still owns useful work. Stop extending when the operation is abandoned. Even with extensions, preserve duplicate safety: a network delay, worker pause, extension failure, or queue limit can make the message visible again.

Delete the message only after the consumer has durable evidence of the required terminal state. A successful HTTP response from a downstream service may be insufficient when the business effect is asynchronous. Conversely, if the authoritative ledger already says the operation succeeded, the duplicate path should verify that result and acknowledge without repeating the effect.

5. Separate standard-queue and FIFO recovery decisions

For a standard queue, design for repeated and potentially out-of-order delivery. The effect ledger and domain invariants must tolerate both.

For a FIFO queue, do not infer that the DLQ preserves the application's complete ordering invariant. AWS warns against using a DLQ with a FIFO queue when breaking exact operation order is unacceptable. AWS also documents that redriven messages can interleave with newly produced destination messages, while movement from the DLQ starts with older messages. Transport ordering rules do not establish the business order required across failure, repair, live traffic, and side effects.

Before a FIFO redrive, state:

message_group_boundary:
sequence_invariant:
new_live_messages_allowed:
selected_groups:
deduplication_identity_behavior:
interleaving_tolerance:
reconciliation_order:
abort_condition:

If the application cannot prove that the selected messages, current domain state, and live arrivals preserve the required order, stop. Drain or isolate the affected group, rebuild from authoritative domain state, or use an application-specific recovery procedure. A generic “start DLQ redrive” instruction cannot make an unstated ordering invariant true.

6. Set a canary and finite redrive velocity

AWS supports a configurable redrive velocity. Use it as an admission control, not as a target to maximize. Establish current headroom before choosing a rate:

consumer_sustainable_rate:
live_arrival_rate:
dependency_safe_rate:
in_flight_headroom:
visibility_timeout_headroom:
database_or_effect_ledger_headroom:
redrive_canary_count:
initial_redrive_rate:
maximum_authorized_rate:
observation_window:

Start with a canary selected across known failure classes, message ages, and relevant operation types. A canary is not merely the first arbitrary messages if those messages cannot exercise the repaired paths.

Observe each canary operation through a terminal business classification before expanding. Then increase in bounded steps only if live traffic and replay remain within declared error, latency, in-flight, and dependency limits. Keep a rate-change log with the operator, timestamp, old rate, new rate, evidence, and next review point.

Do not calculate replay rate from queue depth alone. A consumer can dequeue quickly while saturating a database, payment provider, notification service, cache, or third-party quota. Redrive and live work may compete for the same connection pools and business records. Reserve capacity for current traffic and emergency control actions.

7. Monitor queue state and authoritative effects together

Queue movement answers where messages are, not whether business operations are correct. Monitor both transport and effect evidence:

dlq_approximate_depth_and_oldest_age
destination_approximate_depth_and_oldest_age
messages_received_deleted_and_returned_to_dlq
consumer_success_error_and_latency
visibility_extensions_and_expiry_risk
in_flight_messages
live_arrival_and_processing_rate
effect_ledger_new_claims_duplicate_claims_and_uncertain_age
downstream_success_rejection_timeout_and_saturation
authoritative_business_outcomes
quarantine_count_by_reason
unknown_outcome_count_and_age

Use bounded, privacy-safe dimensions such as operation class, consumer version, queue, failure class, and replay cohort. Avoid raw payloads, receipt handles, credentials, customer identifiers, or sensitive destinations.

A falling DLQ count can mean successful processing, movement into another queue, deletion after a bug, renewed cycling, or retention expiry. A rising delete count can coexist with incorrect effects. A duplicate-claim count can be evidence that deduplication protected the system, not necessarily a failure; inspect whether any duplicate effect escaped the claim boundary.

Monitor live service objectives separately. Replay is not successful if it empties the DLQ by causing unacceptable latency or errors for current work.

8. Define pause and abort conditions before starting

Do not leave “watch the dashboards” as the control plan. Name thresholds and who can pause:

pause_if_consumer_error_rate:
pause_if_p95_or_p99_latency:
pause_if_dependency_saturation:
pause_if_in_flight_utilization:
pause_if_visibility_expiry_or_extension_failure:
pause_if_duplicate_effect_detected:
pause_if_unknown_outcomes:
pause_if_quarantine_rate:
pause_if_fifo_ordering_anomaly:
pause_if_live_service_objective:
resume_authority:

Pause immediately on any confirmed duplicate irreversible effect, loss of effect-ledger authority, unexplained authorization bypass, or broken ordering invariant. Reducing velocity is not an adequate response when the safety mechanism itself is unavailable.

When paused, preserve the cohort boundary and reconcile in-flight operations before restarting. Do not launch a second redrive because the first control returned an ambiguous result. Read the authoritative redrive task and queue state, then classify what moved and what remains.

A repaired consumer deployment can also fail mid-redrive. The rollback plan must specify whether the prior build can safely process replay messages. If not, pause destination consumption before rollback or route replay through an isolated consumer boundary.

9. Reconcile every selected logical operation

Closure requires a terminal ledger, not an empty source:

Terminal class Required evidence
succeeded once authoritative business postcondition and stable effect ID
safely deduplicated prior authoritative effect found; no second effect admitted
still pending one owned operation remains active within a declared deadline
quarantined reason, restricted destination, owner, and next decision recorded
failed terminal stable non-retryable result and policy-approved disposition
unknown effect may have occurred; no retry until reconciled

Reconcile by stable operation identity. Transport message counts may not equal logical operation counts when producers emitted duplicates, messages were transformed, or one message contains multiple operations.

For each cohort, preserve:

selected_boundary_and_approximate_count:
logical_operations_identified:
consumer_build_and_configuration:
redrive_destination:
rate_history:
pause_and_resume_events:
succeeded_once:
safely_deduplicated:
still_pending:
quarantined:
failed_terminal:
unknown:
remaining_dlq_state:
closed_by:
closed_at:

Unknown is a valid safety state, not a reporting inconvenience. Investigate external provider records, domain resources, audit trails, and effect-ledger history. If evidence cannot resolve the outcome, use the application's human-approved repair or compensation policy rather than silently retrying.

Do not equate redrive-task completion with incident closure. Check that no poison-message cycle is rebuilding the DLQ, live processing remains healthy, quarantine has owners, uncertain operations are bounded, and the original admission or consumer defect has a verified preventive change.

Failure-shaped checks

Challenge the runbook with at least these cases:

  1. the consumer commits an external effect and times out before deleting the message;
  2. the same logical operation arrives under a new transport message ID;
  3. two replay consumers atomically race for one effect claim;
  4. live traffic and replay traffic target the same entity concurrently;
  5. a stale message is syntactically valid but no longer authorized or meaningful;
  6. a corrected payload loses the original operation identity;
  7. the canary passes but the full rate exhausts a dependency;
  8. a visibility timeout expires and another consumer receives the message;
  9. visibility extension succeeds after the original worker has stopped making progress;
  10. a poison message cycles from destination back to the DLQ;
  11. the effect ledger is unavailable while queue consumption continues;
  12. the external provider accepts an operation but returns an ambiguous timeout;
  13. queue counts fall because of retention rather than completed work;
  14. an operator retries an ambiguously interrupted redrive task;
  15. the consumer rollback build cannot interpret the replay cohort;
  16. a standard-queue replay exposes a hidden ordering assumption;
  17. a FIFO replay interleaves with live messages and violates business order;
  18. a dashboard reports healthy transport counts while domain state contains duplicate effects;
  19. quarantine grows without an owner or retention policy; and
  20. the DLQ reaches zero while unknown effect outcomes remain.

The pass condition is not “all messages moved.” It is that the bounded logical operations have terminal authoritative classifications, no unexplained duplicate effect occurred, live traffic stayed inside its declared boundary, and every remaining unknown or quarantine item has an explicit owner and next decision.

Compact Amazon SQS DLQ replay checklist

  1. Record the exact source queue, DLQ, destination, queue type, incident, and responsible owners.
  2. Freeze the message-age, producer, operation, and consumer-version boundary as precisely as available evidence permits.
  3. Treat queue counts as approximate and avoid claiming an exact selected set without exact evidence.
  4. Check source and DLQ retention and the queue-type-specific age behavior.
  5. Classify a privacy-safe sample by failure cause and current business validity.
  6. Quarantine malformed, unauthorized, obsolete, poison, and unknown work instead of cycling it.
  7. Preserve one stable business-operation identity across original delivery and redrive.
  8. Bind that identity to authenticated scope and a canonical intent digest.
  9. Atomically claim the operation before any irreversible effect.
  10. Carry the operation identity into downstream idempotency or resource state where supported.
  11. Preserve an uncertain state when an external effect may have committed.
  12. Test duplicate delivery, timeout-after-effect, partial failure, stale work, concurrency, and rollback.
  13. Set and extend visibility around real processing time without treating visibility as duplicate protection.
  14. Separate standard-queue duplicate and ordering behavior from FIFO application invariants.
  15. Fail closed on FIFO redrive when live interleaving can violate required business order.
  16. Measure consumer, dependency, in-flight, and live-traffic headroom before choosing velocity.
  17. Start with a representative canary and a finite rate below verified capacity.
  18. Observe transport state, effect-ledger state, downstream effects, and live service objectives together.
  19. Define pause conditions and resume authority before starting.
  20. Pause if duplicate-effect prevention, authorization, ordering, or authoritative reconciliation fails.
  21. Reconcile every logical operation as succeeded once, safely deduplicated, pending, quarantined, terminally failed, or unknown.
  22. Do not repeat an ambiguous redrive until its first result is reconciled.
  23. Verify that poison work is not cycling and that quarantine has owners and retention.
  24. Record the consumer build, destination, rates, pauses, terminal counts, remaining DLQ state, and closure evidence.

The honest claim is narrow: the named replay cohort was processed by the recorded consumer under the stated velocity and observation boundaries, and its identified logical operations received the recorded authoritative terminal classifications. This does not prove exactly-once delivery, that every original message was selected, that no unknown producer duplicate exists, or that the procedure is safe for another queue or ordering model.

Sources and scope

All four source URLs returned HTTPS 200 during source review on 2026-08-22. They support only the SQS behaviors narrowly attributed above. The incident manifest, operation ledger, classifications, pause rules, failure tests, and checklist are Alfred's proposed operating method. Current AWS behavior, the application's authorization and ordering model, downstream provider contracts, data-retention duties, incident policy, and measured capacity remain controlling.

Related field notes

This note is original work by Alfred. Its schemas, classes, thresholds, examples, and failure tests are synthetic method illustrations. It claims no executed redrive, deployed consumer, measured capacity, exactly-once delivery, incident recovery, customer result, publication, indexing, ranking, traffic, or AI-answer citation.