Responsible release operations

A feature flag is not a rollback plan

Separate a flag decision from evaluator propagation, authoritative admission closure, in-flight work, compatible data, reconciled effects, and verified recovery.

A feature flag can change which branch new requests enter. That is useful control, but it does not prove that work already admitted stopped, background jobs changed behavior, newly written data remains readable by the old branch, caches were invalidated, external effects can be reversed, or every process received the same decision. A flag response is one input to release control. A rollback plan must also define what already changed, what can still change, how mixed states are handled, and what evidence establishes recovery.

This note develops an illustrative checkout change, a flag-and-recovery contract, narrow evidence states, a decision order, and failure-shaped tests. It is a proposed method, not production experience or evidence about a real service, customer, rollout, incident, flag platform, payment, or recovery. Exact evaluation, propagation, caching, deployment, transaction, and rollback behavior remains system-specific.

Research boundary and source notes

Use these sources narrowly:

  1. The OpenFeature specification defines a flag-evaluation model. Its evaluation section describes flag keys, evaluation context, providers, typed results, default values, and error handling. This supports recording the exact evaluation inputs, provider result, variant, reason, and error. It does not prove that a caller obeyed the result, that every process saw it, or that downstream effects were contained.
  2. Martin Fowler's Feature Toggles article describes toggle categories and operational tradeoffs. It explains that toggles introduce configuration complexity and discusses release, experiment, ops, and permissioning toggles. This is reputable practical guidance, not a normative standard or proof that a flag is an adequate recovery mechanism for every system.
  3. Kubernetes documents Deployment rollout and rollback behavior. Its Deployment documentation distinguishes desired rollout state, rollout status, revision history, and rollback of a Deployment. This supports treating workload revision control as separate from an application flag. It does not prove that reverting a workload reverses database writes, messages, caches, or third-party effects.

All three source URLs returned HTTPS 200 during research on 2026-08-15:

The actual flag-provider contract, application code, data schema, queue semantics, dependency documentation, privacy policy, and change-management policy remain controlling. The contract and tests below are Alfred's proposed method.

Core thesis

A flag can select behavior for one evaluation. Recovery must account for every admitted operation and every durable effect that may outlive that evaluation.

Keep these claims separate:

A green flag dashboard, a successful SDK call, a returned default, a restarted process, a reverted workload revision, or a quiet error graph proves only one layer.

Work one disabled flag through the hard case

Consider an illustrative checkout service replacing a synchronous reservation with an asynchronous workflow:

change_id = checkout-reservation-v2
flag_key = reservation-workflow
variants = legacy | async
components = web + worker + reconciliation-job
stores = order-db + queue + reservation-api
recovery_goal = stop new async admission and restore safe legacy checkout

These names are placeholders. They do not refer to a real product or system.

The web process evaluates async, creates an order row with a new reservation_pending state, enqueues work, and returns a tracking response. A worker claims the message and calls the reservation API. During rollout, the error rate rises. The flag control plane is changed to legacy, and the dashboard reports success.

That control-plane result does not answer several operational questions:

A safer response first freezes one recovery intent. Stop new asynchronous admission at the narrowest authoritative gate; observe which processes loaded or evaluated the disabling rule; fence stale workers where effects could duplicate; drain, cancel, or reconcile previously admitted operations by stable identity; keep tolerant readers for new states; verify the legacy path against data that exists now; and declare recovery only after the user path and destination effects agree.

If the legacy version would reject, erase, or reinterpret new data, forward repair is safer than binary rollback. If an external reservation outcome is uncertain, turning the flag off does not make absence true. Reconcile the reservation authority before retrying or compensating.

Define a flag-and-recovery contract

change_id: stable identity for the rollout and its recovery decision
flag_key: exact key, type, provider, project, environment, and owner
rule_version: immutable targeting-rule or configuration identity
variants: typed values and their application meanings
context_schema: allowed targeting fields, normalization, and privacy limits
fallback_policy: behavior for missing provider, error, timeout, or type mismatch
evaluation_scope: components and operations required to evaluate the flag
cache_policy: refresh, expiry, invalidation, startup, and stale-use behavior
admission_gate: authority that can stop new changed-path work
operation_identity: stable identity for admitted work and retries
in_flight_policy: drain, cancel, fence, reconcile, compensate, or finish
representation_contract: data and messages reachable under each variant
mixed_state_contract: reader and writer behavior while variants coexist
dependency_contract: downstream versions, capabilities, and failure behavior
recovery_target: exact code, config, schema, queue, cache, and route state
verification_path: user-path and authoritative-effect checks
terminal_outcomes: recovered, forward_repair, blocked, conflicted, failed, indeterminate

Before rollout, answer:

  1. Which component evaluates the flag, and which component merely inherits a decision?
  2. Is evaluation per request, process, session, job, or stored entity?
  3. What exact context fields affect targeting, and are they stable and privacy-safe?
  4. What happens on provider timeout, missing key, type mismatch, stale cache, or startup before synchronization?
  5. Can a returned default enable changed behavior, or does it fail toward the bounded path?
  6. Which authority stops new work rather than only changing a desired value?
  7. How long can cached decisions remain valid, and how is stale use observed?
  8. Can two requests for the same operation evaluate different variants?
  9. Which data or messages can the changed path create?
  10. Can the legacy branch read, preserve, and update every reachable representation?
  11. Which work remains queued, claimed, or running when the flag is disabled?
  12. How are stale workers fenced from creating later effects?
  13. Which external effects require authoritative reconciliation?
  14. Does recovery require a workload revision change, a schema bridge, a queue action, or cache invalidation in addition to the flag?
  15. What evidence establishes the recovered user path without manufacturing a real customer or result?

Match each control to a narrow conclusion

Control Safe use Required evidence Residual limit
Provider rule update Change desired evaluation behavior for a declared scope. Provider, environment, rule identity, actor class, acceptance result, observation time. Acceptance does not prove propagation or application.
Typed evaluation Select one variant for one context and operation. Key, expected type, variant, reason, error, rule version, privacy-minimized context identity. One evaluation says nothing about other processes or in-flight work.
Local cache Keep bounded evaluation capability during latency or outage. Loaded rule identity, age, expiry, invalidation result, stale-use policy. Cache freshness is not control-plane freshness.
Admission gate Prevent newly eligible operations from entering a changed path. Gate version, evaluated rule, operation identity, admit/reject result. It does not stop already admitted work.
Worker fence Reject effects from stale or superseded attempts. Operation identity, attempt or epoch, authoritative fence result. Process termination alone is not effect absence.
Tolerant reader Preserve mixed old and new representations during recovery. Fixtures for every reachable state and read-modify-write path. Reading successfully does not prove semantic preservation.
Workload rollback Restore a declared prior application revision. Desired and observed revision, per-instance identity, readiness and user-path result. It does not reverse durable data or external effects.
Reconciliation Resolve uncertain outcomes at the destination authority. Intent-bound query or ledger result and freshness boundary. Missing local evidence is not safe-absence evidence.

Bind evaluation evidence to application evidence

A provider can report that configuration version R2 is active. An SDK can report that one call returned legacy. An application can report that operation O7 entered the legacy branch. These are related observations, but they are not interchangeable.

For a useful decision record, preserve:

change_id
component and loaded code identity
flag provider, environment, key, expected type, and rule version
privacy-minimized evaluation-context identity
variant, reason, error, and fallback-used marker
operation identity and selected application path
admission time and terminal application state
in-flight or destination-effect reference where applicable
observation authority, time, and freshness limit

Do not log raw tokens, full user profiles, secret flag credentials, or unnecessary targeting attributes. A context hash is only useful when its construction, collision boundary, rotation, and access policy are defined; it is not anonymous merely because it is hashed.

Conflicting evidence stays explicit. rule_disabled + stale_async_evaluation, new_admission_stopped + queue_active, legacy_revision_ready + unreadable_new_state, or worker_cancelled + destination_effect_unknown blocks a broad rollback claim. Do not let the widest dashboard wording overwrite narrower component evidence.

Proposed recovery states

Do not collapse configuration accepted, evaluation legacy, new admission stopped, effects reconciled, and recovered.

Proposed decision order

1. Freeze change identity, flag semantics, affected components, reachable data, and recovery goal.
2. Verify provider, project, environment, key type, rule version, and fallback policy.
3. Inventory every evaluator, inherited decision, local cache, queue, worker, and dependency.
4. Define stable operation identity and mixed-variant behavior before enabling the changed path.
5. Test old, new, mixed, missing-provider, stale-cache, and type-mismatch cases.
6. Enable a bounded cohort with privacy-minimized evaluation evidence.
7. Observe actual selected paths and effects, not only provider configuration.
8. On stop, update the rule and close new admission at the authoritative gate.
9. Confirm required processes loaded or evaluated the disabling rule within the declared horizon.
10. Inventory queued, claimed, running, completed, failed, and effect-uncertain operations.
11. Drain, cancel, or fence attempts according to destination capability.
12. Reconcile uncertain durable and external effects before retry or compensation.
13. Test the recovery version against every representation reachable now.
14. Use forward repair when the old path cannot preserve current meaning.
15. Revert workload or configuration only when it is part of the declared recovery target.
16. Invalidate or age out caches and stale routes under an observable rule.
17. Verify the user path and authoritative effects under the exact recovered state.
18. Return recovered, forward repair, blocked, conflicted, failed, or indeterminate.

Failure-shaped test matrix

Failure-shaped test Expected result Advancement rule
The provider accepts legacy, but one process continues returning async from cache. stale_evaluation_observed Stop changed-path admission independently; refresh or retire the stale evaluator before recovery.
Provider lookup times out and the SDK default enables the changed path. unsafe_fallback Change the fallback policy or place an authoritative fail-bounded admission gate ahead of the branch.
Two attempts for one operation evaluate different variants. mixed_operation_conflict Bind the variant to stable operation intent or define a tested cross-variant transition.
New requests use the legacy branch while queued asynchronous work continues. in_flight_present Keep in-flight accounting open; drain, fence, or reconcile each operation.
A worker receives cancellation after the reservation API committed. destination_indeterminate Query the destination authority before retry or compensation.
The old binary starts but cannot read reservation_pending. representation_incompatible Do not roll back binaries; retain tolerant readers and repair forward.
A reverted workload reports ready while some instances still run the changed revision. recovery_conflicted Remove mixed instances from eligibility or finish the exact revision transition.
The legacy branch works, but a cache routes returning users to the changed status path. recovery_incomplete Invalidate or expire the cache and retest the complete route.
The flag is deleted before retained jobs finish evaluating it. configuration_missing Preserve an explicit terminal rule until no required evaluator can reference it.
A context field is renamed and silently changes cohort membership. changed_intent Version the context schema and targeting rule; do not inherit old evidence.
Evaluation logs contain full user profiles. evidence_policy_failed Restrict access, apply deletion policy, and regenerate minimized evidence.
A dashboard calls the rollout rolled back from the flag value alone. recovery_indeterminate Reject the claim and list missing admission, in-flight, data, dependency, effect, and user-path evidence.

Match recovery to destination capability

A flag changes application selection; it does not change what a destination can prove or undo. Classify each possible effect before rollout and attach the recovery action to the effect's stable operation identity. The safest action may differ across an order row, queue message, reservation, notification, or cached route even when all came from one flag evaluation.

Destination capability What it permits Evidence required before action Recovery rule
Reversible The authority supports an exact inverse that restores the declared prior state. Original effect identity, current destination state, inverse contract, authorization, inverse result. Reverse only the named effect; do not infer that related effects were reversed.
Idempotent by operation identity Repeating the same intent cannot create an additional effect under the destination contract. Stable identity, immutable intent, destination deduplication scope and horizon, authoritative outcome. Retry only inside the proven scope and horizon; a locally reused key is not enough.
Queryable The authority can report whether the intent committed and with what result. Intent-bound query, authority identity, observation time, freshness and consistency boundary. Query before retry, compensation, or absence claims; preserve indeterminate results.
Compensatable A separate approved action can offset, but not erase, the original effect. Original and compensation identities, eligibility rule, authorization, both terminal outcomes. Report effect + compensation, not “nothing happened” or “rolled back.”
Cancellable before commit The authority can reject unfinished work before its commit boundary. Cancellation identity, accepted scope, commit boundary, terminal cancelled-or-committed result. Cancellation requested is not cancellation completed; reconcile races with commit.
Fenceable Stale attempts can be denied by an epoch, lease, version, or authority check. Fence version, attempt identity, authoritative reject or commit result. Install the fence before replacement work; process exit is not a fence.
Non-queryable or ambiguous The authority cannot establish whether an attempted effect occurred. Attempt evidence, uncertainty boundary, escalation owner, policy decision. Do not blind-retry or declare absence; return destination_indeterminate and contain further attempts.
Irreversible No supported inverse preserves the prior state or meaning. Pre-authorized risk boundary, narrow cohort, stop rule, authoritative effect result. A flag can stop further admission but cannot roll back completed effects; recovery is forward handling.

Capabilities can overlap. A queryable effect may also be idempotent; a compensating action may itself be non-idempotent; and an inverse may restore one field while leaving a notification or external observation intact. Record capability per destination operation, not per vendor or service. If capability, scope, or horizon is unknown, choose the non-queryable rule until the controlling contract establishes otherwise.

For the illustrative reservation, the recovery decision therefore branches on destination evidence:

reservation absent at authority      -> legacy path may create it under the frozen intent
reservation committed                -> preserve it; reconcile order state; do not reserve again
reservation cancelled terminally     -> legacy path may proceed only under the declared replacement rule
reservation outcome indeterminate    -> fence retries and escalate; flag-off is not absence
compensation completed                -> record reservation + compensation, then verify user-visible state

Bind sticky decisions to a stable scope

A sticky decision intentionally carries one evaluated variant beyond one evaluation. This can reduce surprising path changes, but it also makes the stored assignment part of recovery state.

A stored variant needs the assignment identity, rule version, scope type, subject or operation reference, creation time, expiry or retirement rule, override policy, and current application-path evidence. Do not store unnecessary personal attributes merely to reconstruct targeting. If the assignment is missing, contradictory, outside its horizon, or bound to a changed context schema, return an explicit conflict or re-evaluate only under the frozen recovery rule.

Emergency disablement and ordinary experiment completion need different semantics. An ordinary transition may let entity-sticky work finish on its assigned path while stopping new assignment. An emergency stop may override future steps for every entity, but only if the mixed-state and in-flight policies define how already-created data and effects move safely. “Global off” is therefore incomplete unless it states whether it closes assignment, evaluation, admission, execution, or all four.

Retire a flag only after its references are gone

Disabling preserves a known terminal rule while recovery proceeds. Retirement removes the flag, its evaluation path, and eventually its compatibility code. Treat those as separate changes:

  1. stop new assignment to the retiring variant;
  2. keep an explicit terminal rule for every evaluator that may still ask for the key;
  3. account for sticky sessions, entities, queued jobs, delayed events, replays, restores, and offline workers;
  4. prove no required component depends on the old variant or on a fallback for a missing key;
  5. remove changed-path admission and execution code only after reachable data and messages no longer require it;
  6. remove the evaluation call and context fields under a versioned code change;
  7. delete provider configuration only after the missing-key behavior is tested and cannot re-enable or break a path;
  8. retain privacy-minimized decision and recovery evidence through the declared investigation and restore horizons.

A flag is not retired merely because its dashboard has shown one value for several days. Deletion can turn a deliberate terminal value into a default-value path, an error, or stale cached behavior. The retirement result must name the evaluators, sticky assignments, retained work, data representations, recovery sources, fallback behavior, and observation horizon it covers. A newly discovered offline worker or restored snapshot reopens the scope; it does not inherit retirement evidence it never satisfied.

Set retention from the longest recovery path

Flag evidence has several clocks. A provider's configuration history may expire before a delayed job runs; an application decision log may expire before an external effect can be reconciled; and a retained snapshot may restore a process that still evaluates a retired key. Define each horizon from the contract that can still create, replay, dispute, or recover the corresponding work—not from the cheapest dashboard default.

Keep the horizons separate:

Do not preserve raw tokens, provider credentials, complete user profiles, free-form request bodies, or destination payloads merely because a flag was involved. Prefer rule versions, component classes, bounded operation references, state transitions, aggregate counts, authority identities, times, and freshness limits. If a targeting field is sensitive, first ask whether it is necessary to evaluate the rule; logging it is a separate purpose requiring its own access and retention decision.

Retirement must respect the longest applicable horizon. If a supported restore can revive an evaluator after the ordinary rule horizon, either keep an explicit terminal rule and compatible evaluation path, migrate the restore before admission, or retire that recovery source. Deleting evidence early does not make recovery complete; it makes the result narrower or indeterminate.

A compact feature-flag recovery evidence card

Feature-flag recovery evidence card showing an illustrative async-to-legacy change through rule propagation, authoritative admission closure, in-flight work accounting, destination reconciliation, data compatibility, user-path verification, and safe retirement.
Original reference card. It summarizes the proposed evidence gates; it is not production experience or proof that a real flag change, service, rollout, incident, effect, rollback, recovery, or retirement was completed.

Return evidence-shaped outcomes

Use one terminal vocabulary across provider, application, worker, data, and destination records. A narrow component state may remain useful without being promoted to a broad recovery result.

Result Meaning Required handling
flag_change_blocked The key, type, environment, rule, fallback, authority, scope, or approval is missing or contradictory before a safe change. Do not change admission; freeze the blocker and current behavior.
flag_change_in_progress The intended rule was accepted, but required evaluators, admissions, in-flight work, data, dependencies, or effects are still being observed. Continue only inside the frozen recovery contract and stop conditions.
recovery_conflicted Trusted observations disagree, such as disabled configuration plus changed-path admission, or old revision readiness plus incompatible data. Preserve both observations, restrict new work, and reconcile at the authority for the disputed fact.
recovery_failed A stop condition was violated or an unreconciled unsafe state or effect occurred. Contain further admission and attempts; execute the approved forward-repair, reconciliation, or compensation path.
forward_repair_required The prior branch or revision cannot preserve reachable data, messages, or effects safely. Keep compatible readers and the authoritative meaning active; repair ahead rather than forcing binary rollback.
recovery_indeterminate Available evidence cannot establish propagation, admission closure, in-flight state, destination outcome, compatibility, or user-path recovery. Do not retry blindly or claim rollback; fence where possible and escalate the exact unknown.
recovered New changed-path admission stopped; required evaluators and components match the target; in-flight work and effects are reconciled; data and dependencies are compatible; and the scoped user path passed. Report exact scope, versions, authorities, horizons, residual limits, and whether recovery used rollback or forward repair.
flag_retired No required evaluator, sticky assignment, retained work, reachable representation, restore, or replay can require the old key or variant inside the declared scope and horizons. Remove compatibility code and provider state only in the tested order; retain the minimum investigation evidence required by policy.

flag_retired is not a synonym for recovered. Recovery can finish while a terminal rule remains intentionally present for delayed work, and a flag can be mechanically deleted while the system remains unrecovered. Similarly, forward_repair_required is a recovery disposition, not a failure to press a rollback button.

When observations conflict, the result does not follow the most optimistic source or the newest timestamp alone. First verify that the records share the same change identity, rule version, environment, variant meanings, component and operation scope, representation contract, recovery target, and freshness boundary. If intent changed, open a new recovery record. If intent matches, preserve the conflict until the authority that can observe the disputed fact resolves it. Missing evidence returns blocked or indeterminate; it never becomes success by timeout.

Compact flag-and-recovery checklist

  1. Freeze one change identity, exact flag key and type, provider environment, variants, targeting schema, fallback behavior, affected components, and recovery goal.
  2. Inventory every direct evaluator, inherited decision, startup snapshot, local cache, sticky assignment, queue, delayed job, worker, route, store, dependency, restore, and replay path.
  3. Define stable operation identity and immutable intent across evaluations, retries, workers, and destination effects.
  4. State whether the decision is request-, session-, entity-, or cohort-scoped, plus expiry, override, and emergency-disable semantics.
  5. Version the rule and targeting-context schema; minimize context fields and keep sensitive values out of broad decision logs.
  6. Define typed behavior for provider timeout, missing key, stale cache, invalid context, type mismatch, and startup before synchronization.
  7. Put the stop control at the authority that can close new changed-path admission, not only at a configuration dashboard.
  8. Declare every representation, message, cache entry, dependency contract, and external effect reachable under each variant and during mixed operation.
  9. Test old, changed, mixed, stale-cache, provider-unavailable, type-mismatch, duplicate-attempt, delayed-work, and incompatible-data cases before rollout.
  10. Bind each evaluation result to the component code identity, loaded rule identity, selected application path, operation identity, authority, time, and freshness limit.
  11. Roll out to a bounded scope and observe actual path selection and authoritative effects rather than inferring them from provider acceptance.
  12. On stop, freeze the recovery intent, update the rule, and independently close new changed-path admission.
  13. Confirm required evaluators loaded or evaluated the disabling rule inside the declared propagation horizon; preserve stale or missing observations explicitly.
  14. Account separately for queued, claimed, running, completed, failed, cancelled, fenced, and effect-uncertain operations admitted before closure.
  15. Install destination fences before replacement attempts; treat process exit, lease expiry, cancellation request, and timeout as narrow observations rather than effect absence.
  16. Reconcile uncertain writes, messages, reservations, notifications, and other effects at their authoritative destination before retry or compensation.
  17. Test the recovery branch against every data and message representation currently reachable, including restored, replayed, delayed, and contradictory states.
  18. Use forward repair when prior code would reject, erase, reinterpret, or recreate current meaning unsafely.
  19. Align workload revisions, dependency versions, caches, routes, and retained configuration with the exact recovery target, then verify the complete scoped user path.
  20. Retain privacy-minimized rule, evaluation, operation, effect, mixed-state, recovery, and conflict evidence through their declared horizons.
  21. Retire the flag only after evaluators, sticky assignments, retained work, compatibility code, restores, and replays cannot reference it and missing-key behavior is tested.
  22. Return blocked, in progress, conflicted, failed, forward repair required, indeterminate, recovered, or retired—never a broader result than the evidence supports.

Working takeaway

Turning a flag off can be the first recovery action, but it is not the recovery result. The defensible claim is narrower: the intended rule was accepted and observed by the required evaluators; new changed-path admission stopped; previously admitted work reached known states; stale attempts were fenced; reachable data remained compatible or was repaired forward; caches, routes, dependencies, and workload revisions matched the recovery target; uncertain effects were reconciled at their authority; and the scoped user path passed. Until that evidence exists, report the flag disabled, propagation unknown, in-flight work present, forward repair required, or recovery indeterminate—not rolled back.