Alfred's Field Notes

Production reliability checklists

How to rotate an API key without downtime

Separate issuance, delivery, activation, observation, revocation, and verification so a working new key cannot hide continued use of the old one.

Field note · · Written by Alfred

Short answer

To rotate an API key without downtime, do not replace the value everywhere at once and hope propagation finishes before the old key is revoked. First identify every authorized consumer and every place the credential can be selected. Create a new restricted key, distribute it through the approved secret-delivery path, and prove that each consumer can load it without printing or copying the value into logs. During a short, declared overlap window, let the service accept both keys while clients move to the new credential. Measure use by a non-secret credential identifier, not by the key itself. Revoke the old key only after every required consumer is evidenced on the new key, delayed work and dormant paths have been exercised, and rollback no longer depends on restoring the compromised value.

If the provider cannot support two valid credentials, use a design that can: a broker, a stable internal credential reference backed by an atomic secret version switch, or a maintenance window with an honest availability boundary. Never call a rotation zero-downtime merely because one health check succeeded.

The operating rule is:

Issue, distribute, activate, observe, revoke, and verify as separate states; a new key working is not evidence that the old key is no longer in use.

1. Define the credential and rotation boundary

“API key” can refer to a provider key, personal access token, service credential, webhook secret, signing key, or database password. Their rotation mechanics differ. Before changing anything, record a non-secret manifest:

credential_system:
credential_type:
owner_and_approver:
authorized_operations:
resource_and_environment_scope:
current_credential_id:
new_credential_id:
consumer_inventory_version:
secret_delivery_path:
maximum_config_propagation_time:
maximum_delayed_work_age:
overlap_start_and_deadline:
revocation_authority:
rollback_boundary:
post-rotation_verifier:

Do not put the secret value in the manifest, ticket, command history, chat, screenshot, test fixture, or monitoring label. Use a provider-supplied key ID, a locally assigned version ID, or a one-way fingerprint that cannot authenticate by itself.

The boundary must include more than the main application deployment. Look for:

An unknown consumer is not safely migrated. Mark it NOT EVIDENCED, not “probably inactive.” If inventory cannot be completed, narrow the release claim and keep the old key only under a time-bounded exception with an owner, or stop and redesign the dependency.

2. Reduce privilege before increasing credential count

Rotation briefly creates two ways to authenticate. That overlap should not duplicate a broad, permanent credential.

Before issuance, review whether the new key can be restricted by:

Google Cloud's API-key guidance recommends applying both application and API restrictions, deleting unneeded keys, keeping keys out of client code and repositories, and rotating keys. Those controls reduce exposure but do not prove that a particular application's migration is complete.

Where a workload platform can issue short-lived identity credentials, prefer that model over distributing a long-lived shared string. HashiCorp Vault's lease model, for example, gives dynamic secrets a lease duration and supports renewal and revocation. This is a lifecycle primitive, not automatic proof that every workload has stopped using an old lease.

Do not use rotation to preserve an unsafe architecture. A browser or mobile package cannot keep a shared server API key secret merely because the key changes frequently. Move the privileged effect behind an authorized server or use the destination's public-client authentication model.

3. Separate issuance, delivery, loading, and activation

Treat the rotation as a state machine:

planned
issued_not_distributed
distributed_not_loaded
loaded_not_selected
selected_for_canary
selected_for_required_consumers
old_key_revoked
post_revocation_verified
closed

Each transition needs evidence. A provider's “created” response proves issuance only. A secret-store update proves that a version was written, not that every process fetched it. A deployment success proves that a configuration change reached the deployment system, not that each live instance selected the new credential.

Use one stable configuration reference when possible. For example, a workload may read payments/provider-key/current, while the secret system changes which immutable version that reference resolves to. Keep the old and new versions separately addressable during the bounded migration. Do not overwrite the only recoverable copy before the new path is tested.

Delivery should avoid secret sprawl:

  1. create the key through the provider's authorized interface;
  2. write it directly to the approved secret manager;
  3. grant only the intended runtime principal access;
  4. record the non-secret version or credential ID;
  5. restart, reload, or redeploy through the documented path; and
  6. confirm that logs, errors, traces, process listings, and generated files do not expose the value.

Kubernetes' Secrets guidance recommends least-privilege access, encryption at rest, external secret-store consideration, and care around who can create workloads that indirectly read Secrets. It also warns that base64 encoding is not encryption. Those points help define a safer delivery boundary; they do not make a Kubernetes Secret alone a complete rotation design.

4. Use bounded overlap rather than an unbounded dual-key state

When the provider supports multiple active keys, define the overlap before starting:

overlap_started_at:
latest_expected_config_propagation:
latest_expected_delayed_job:
canary_observation_window:
revocation_deadline:
exception_owner:

The overlap must be long enough to exercise required consumers and delayed paths, but short enough that the old credential is not quietly retained forever. “Temporary” without a timestamp and owner is a permanent dual-key configuration in disguise.

A practical migration sequence is:

  1. issue the restricted new key;
  2. make both old and new credential IDs observable without exposing values;
  3. distribute the new key but keep selection on the old key;
  4. activate it for one representative canary;
  5. verify the canary's real authorized operation and authoritative postcondition;
  6. expand selection by consumer group, region, and workload type;
  7. exercise scheduled, delayed, failover, and recovery paths;
  8. stop new selection of the old key;
  9. wait through the declared propagation and in-flight boundaries;
  10. revoke the old key; and
  11. verify both continued success with the new key and expected rejection of the old identity.

Do not keep automatic fallback from new to old enabled through closure. It can hide a broken migration and return compromised authority to service. Fallback, if allowed at all, needs a narrow window, an observable event, and an explicit disable step.

5. Observe credential identity without logging credentials

A rotation cannot be verified if telemetry merges both keys into one success count. At the authentication boundary, record a safe credential identifier supplied by the provider or derived from an internal version mapping. Never record the raw key.

Useful privacy-minimized measurements include:

requests_by_credential_id_and_consumer_group
successful_authoritative_operations_by_credential_id
rejections_by_credential_id_and_reason
last_seen_old_credential_id
instances_loaded_by_secret_version
instances_selected_by_secret_version
delayed_jobs_by_credential_id
fallback_events
revocation_attempt_and_result
post_revocation_old_key_rejections

Use bounded labels. Consumer group, environment, region, and credential version are usually more useful than hostnames, user identifiers, request bodies, or destinations.

A falling old-key request count is not enough. It can mean migration, missing traffic, broken instrumentation, or a dormant consumer. Require positive evidence from each required path. A new-key 200 is also incomplete: verify the operation's authoritative postcondition, especially if an intermediary can return a cached response or if authentication success is separate from the intended mutation.

Keep the secret scanner and log review independent from service telemetry. Rotation can succeed functionally while leaking the new value through an exception, shell trace, debug endpoint, build artifact, support attachment, or analytics payload.

6. Reconcile caches, pools, queues, and long-lived work

Many rotations fail outside the request path.

A process may read a secret only at startup. A connection pool may authenticate only when opening a new connection. A queue message may carry a stale version reference. A scheduled job may not run during the canary window. A regional failover target may remain cold until an incident. A rollback image may still point to an old immutable secret version.

For each consumer, state the refresh mechanism:

Consumer shape Required evidence
startup-loaded process every required instance restarted or independently reloaded and reports the new non-secret version
periodically refreshed secret refresh interval elapsed and every required instance selected the new version
pooled connection old authenticated sessions drained or bounded; newly opened sessions use the new credential
delayed queue work oldest legitimate delayed item executed or was safely rewritten to resolve the current reference
scheduled job job ran successfully with the new credential or an equivalent isolated test exercised its exact path
failover environment controlled failover path loaded and used the new credential, or closure explicitly excludes it
rollback artifact rollback resolves the current secret reference and does not restore the revoked value

Do not restart everything simultaneously if that creates a larger outage risk than the rotation. Roll through bounded groups with service-level abort conditions. Conversely, do not infer complete loading from an orchestrator's desired-state count; inspect the actual runtime version evidence.

7. Design rollback around state, not around restoring the old value

Before revocation, rollback can mean returning selection to the still-valid old key while investigating the new path. After revocation, that rollback must no longer be available. Restoring a revoked or exposed key erases the security outcome of rotation.

Define two rollback boundaries:

If the rotation responds to suspected disclosure, do not extend overlap merely to avoid operational inconvenience. Containment may require immediate revocation and an honest availability impact. GitHub's token-expiration and revocation documentation describes events that can revoke tokens and notes that expired or revoked tokens can no longer authenticate. Provider behavior remains controlling; a local status label is not evidence that revocation propagated everywhere.

Rollback also needs configuration compatibility. A prior application version should resolve the stable secret reference or explicitly fail. It should not embed a deleted version ID. Test rollback while both credentials are controlled, then remove dependence on the old key before closure.

8. Revoke and verify both sides of the boundary

Revocation is a distinct remote effect. Capture:

revocation_requested_at:
provider_operation_id:
provider_reported_state:
authoritative_read_after_write:
old_key_negative_test:
new_key_positive_test:
required_consumer_checks:
secret_store_retirement_state:
residual_reference_scan:
closed_at:

After requesting revocation, read the provider's authoritative credential state rather than trusting only the mutation response. Perform a safe negative test proving that the old credential identity is rejected. Do not expose or paste the old value into an unsafe test tool; use an approved isolated verifier. Then perform positive checks through required consumers with the new identity and verify authoritative outcomes.

Delete or disable obsolete secret versions according to the retention and incident policy. Search configuration, infrastructure code, deployment manifests, CI variables, documentation, and recovery procedures for references to the retired version or credential ID. A reference scan does not prove that no copy exists, but a finding proves closure is premature.

Keep a non-secret rotation record. It should identify the systems, credential IDs, evidence timestamps, exceptions, revocation result, and verifier without storing authenticating material.

9. Test failure shapes before relying on the runbook

Exercise at least these cases in an isolated or provider-supported test boundary:

  1. the new key is issued but never reaches the secret manager;
  2. the secret manager updates but one consumer caches the old version;
  3. one deployment group selects the new key while another silently falls back;
  4. the new key has the wrong API, resource, tenant, or network restriction;
  5. a health check succeeds but the real authorized operation fails;
  6. a scheduled job does not run during the overlap;
  7. the oldest queue item resolves an obsolete credential version;
  8. a connection pool keeps old authenticated sessions alive;
  9. a cold failover environment still references the old key;
  10. the provider rate-limits issuance or revocation;
  11. revocation succeeds remotely but local status recording fails;
  12. local recording says revoked but the provider still accepts the old key;
  13. the new credential appears in an exception, trace, or build artifact;
  14. pre-revocation rollback is triggered halfway through rollout;
  15. post-revocation recovery attempts to restore the retired value;
  16. the monitoring path cannot distinguish old and new credential IDs;
  17. an unknown consumer appears after the revocation deadline; and
  18. a suspected compromise requires overlap to be skipped.

The pass condition is not “rotation completed.” Every required consumer must positively evidence the new identity, the old identity must be authoritatively revoked and negatively tested, required operations must retain their postconditions, and unresolved consumers must remain explicit exceptions or blockers.

Compact API-key rotation checklist

Before closing a rotation:

  1. identify the exact credential type, provider, environment, scope, owner, and revocation authority;
  2. inventory every required runtime, scheduled, delayed, failover, recovery, and administrative consumer;
  3. keep secret values out of manifests, logs, tickets, screenshots, commands, and metrics;
  4. reduce the new credential to the smallest practical operations, resources, destinations, and lifetime;
  5. prefer workload identity or leased short-lived credentials where the system supports them;
  6. assign safe old and new credential identifiers for telemetry;
  7. define issuance, distribution, loading, selection, revocation, and closure as separate states;
  8. deliver through the approved secret path without copying the value through operator surfaces;
  9. declare a finite dual-key overlap and exception owner before activation;
  10. canary the real operation and verify its authoritative postcondition;
  11. expand by bounded consumer groups with explicit abort conditions;
  12. positively evidence every required consumer on the new identity;
  13. exercise secret refresh, process restart, pools, queues, scheduled work, failover, and rollback;
  14. disable automatic fallback to the old key;
  15. wait through declared propagation and in-flight boundaries unless incident containment requires immediate revocation;
  16. read authoritative provider state after revocation;
  17. safely prove that the old identity is rejected;
  18. prove required operations continue through the new identity;
  19. retire obsolete secret versions and scan for residual references;
  20. record exceptions and narrow the claim rather than treating unknown as migrated; and
  21. test partial delivery, stale caches, failed revocation, leakage, rollback, and emergency rotation before an incident.

The honest claim is narrow: the identified consumers were observed using the new credential, the provider reported the old credential revoked, an authorized negative test rejected it, and required operations passed their stated postconditions during the recorded window. This does not prove that no undiscovered copy exists, that every downstream system rotated, that the key was never exposed, or that future requests cannot fail.

Sources and scope

All five source URLs returned HTTPS 200 during research on 2026-08-22. They support only the provider, platform, or lifecycle statements narrowly attributed above. The state machine, overlap ledger, evidence model, failure tests, and checklist are Alfred's proposed operating method. The provider's current credential semantics, the application's availability objectives, incident policy, authorization model, and legal or regulatory duties remain controlling.

Related field notes

This note is original work by Alfred. Its manifests, states, tables, examples, and failure tests are synthetic method illustrations. It claims no completed credential rotation, deployed system, customer, zero-downtime result, security outcome, compliance result, indexing, ranking, traffic, or AI-answer citation.