Responsible operations

Termination grace is a shared budget

Make shutdown races visible by assigning routing, draining, settlement, and exit to one shared deadline.

Field note · Published August 13, 2026 · Written by Alfred

A service receives a stop signal, flips readiness, waits, and exits with code zero. The deployment still drops requests.

That sequence can look graceful while hiding three races: traffic may still be arriving, accepted work may have no completion deadline, and a pre-stop hook may have consumed the same grace period the application thought it owned.

A safer shutdown design treats termination as one shared budget. Routing withdrawal, connection draining, work settlement, telemetry flush, and process exit each need a bounded place in that budget and an observable outcome.

“Graceful” names an opportunity to settle work before a forced stop. It is not evidence that every request finished, every client observed the drain state, or the rollout preserved enough capacity.

Write one shutdown contract

Start by answering five separate questions:

When does the instance stop accepting new work?
What already-accepted work is allowed to finish?
Which component owns each deadline inside the termination window?
What evidence shows the process exited before forced termination?
What happens to work that cannot finish in time?

A compact contract can make those answers reviewable:

contract_name: api-termination
platform_grace_period: 45s
routing_state_change: mark instance not ready immediately
new_work_policy: reject new application work after drain begins
accepted_work_policy: finish idempotent requests until t+30s
stream_policy: send drain notice; close by t+32s
background_work_policy: stop leases; checkpoint or return unfinished jobs
telemetry_deadline: t+35s
application_exit_deadline: t+38s
forced_stop_reserve: 7s
unfinished_work_record: durable job or idempotency key plus terminal reason
success_evidence: no new work accepted after drain state; accepted-work outcomes reconciled; process exited before platform force
owner: service release workflow

These values are illustrative, not recommendations. Derive them from observed request duration, stream behavior, queue-lease rules, dependency latency, and the platform’s actual termination sequence.

The contract should describe failure as carefully as success. If an operation cannot finish before the drain deadline, state whether it is cancelled, checkpointed, returned to a queue, or reconciled through a stable key. “The process stops” is not an unfinished-work policy.

Name the states

One shutting_down=true flag hides too much. Use explicit states:

Serving: eligible for new traffic and accepting new application work.
Draining: no longer eligible for ordinary new traffic; bounded accepted work may continue.
Settling: no new work; outcomes, leases, checkpoints, and telemetry are being reconciled.
Exited: the process stopped itself before the platform’s forced deadline.
Forced: the platform ended the process before voluntary exit; unfinished effects require reconciliation.

Routing state and application admission state are related but not identical. Kubernetes documents that a terminating Pod’s endpoint can remain present while exposing termination state. For compatibility, the endpoint’s ready condition is false so ordinary load balancers do not use it for regular traffic; clients designed for draining can inspect the serving condition.

That does not make endpoint withdrawal instantaneous everywhere. A request may already be in flight. A client may hold a persistent connection. Different routing layers can observe state at different times. The application should therefore enforce its own bounded admission policy instead of assuming every upstream has already observed readiness.

The pre-stop hook spends the same budget

Kubernetes documents that the Pod termination grace-period countdown begins before a PreStop hook runs. The hook must complete before the stop signal is sent, but the container is still subject to the Pod’s overall termination grace period.

This makes a common timing diagram wrong:

pre-stop hook -> grace period -> application drain

The safer model is:

platform grace period:
  pre-stop hook
  + routing propagation allowance
  + accepted-work drain deadline
  + settlement and telemetry deadline
  + forced-stop reserve

Some phases can overlap, but overlap must be measured rather than assumed. A 20-second hook inside a 30-second grace period can leave little time for the application to receive its stop signal, settle work, and exit.

Budget backward from forced termination. Reserve time for voluntary exit and forced-stop diagnostics first. Then place telemetry, job settlement, stream closure, and request draining before that reserve. Avoid a chain of independent timeouts whose sum exceeds the platform deadline.

A compact shared-budget card

Shutdown-budget card placing routing withdrawal, application admission, accepted requests, streams, leased jobs, outcome reconciliation, bounded telemetry, and voluntary exit inside one forced-stop deadline — Original reference card. It proposes a shutdown contract; it is not evidence that work settled, a process exited voluntarily, or a rollout stayed healthy.

The same sequence is available as an original square SVG reference card. Keep its evidence-limit footer attached when reusing it. The diagram summarizes the contract; it does not establish platform timing, immediate routing withdrawal, completed work, or a successful deployment.

Separate workload classes

A generic server shutdown call cannot express every kind of accepted work.

Short requests

Stop admitting fresh application work when drain begins. Let already-accepted requests finish only until the declared deadline. Record whether each operation completed, was cancelled, or moved to a durable recovery path.

Persistent connections and streams

HTTP keep-alive connections, WebSockets, and long-lived RPC streams can outlive a request-oriented drain. Define how a client learns that draining has begun, when new work on an existing connection is refused, and when the connection will close. A readiness change by itself is not a stream protocol.

Queue workers

Stop acquiring new leases before settling existing ones. For each active job, declare whether the worker finishes, checkpoints, returns the lease, or allows it to expire. A retry is safe only when the queue and effect have an idempotency or reconciliation contract; worker disappearance does not create one.

Scheduled and local background work

Scheduled jobs need stable run identity or a durable completion marker if another worker may resume them. Fire-and-forget local tasks can be lost at process exit unless the application makes them durable before acknowledging the triggering request.

Treat these classes separately in the contract even when one process handles all of them.

Preserve evidence when the graceful path fails

Kubernetes’ termination flow has a forced end. After the grace period, remaining processes can be killed. The shutdown record must not translate “Pod disappeared” into “drain succeeded.”

Capture at least:

termination_started_at
routing_state_changed_at
application_admission_closed_at
in_flight_count_at_each_deadline
last_accepted_work_identifier
unreturned_queue_leases
stream_close_outcomes
telemetry_flush_outcome
voluntary_exit_at
forced_termination_observed
unfinished_work_reconciliation_state

Exit code zero is process evidence, not proof that every accepted effect reached its durable destination. Likewise, a forced stop should remain a distinct result even if a retry later recovers the user-visible operation.

A deployment-level claim also needs fleet evidence. Several instances can each follow their local shutdown contract while simultaneous termination overloads the capacity left behind. Observe traffic distribution, saturation, error rate, and unavailable capacity across the rollout—not only the logs of the departing instance.

Test the races deliberately

Use synthetic or controlled work and retain the expected evidence for each case:

Test	Expected evidence	Failure exposed
Send a short request just before drain begins	The accepted request finishes once and its outcome is recorded.	Accepted work is abandoned on readiness change.
Attempt new work after application drain state	The service rejects or redirects it according to the contract.	Readiness is the only admission control.
Keep a persistent connection open	The client receives the declared drain or close behavior before the application deadline.	Streams outlive the budget silently.
Run a request longer than the drain deadline	The request follows the declared cancellation or durable-resume path.	“Wait forever” collides with forced termination.
Make `PreStop` consume most of the grace period	The application reports reduced remaining time and preserves the forced-stop reserve, or the test fails visibly.	Hook time is treated as outside the budget.
Delay routing-state propagation	Application admission control still prevents unintended fresh work.	Traffic removal is assumed instantaneous.
Lose a queue worker during settlement	The lease expires or is returned, and retry does not duplicate the effect.	In-flight jobs have no recovery contract.
Make telemetry export hang	Export stops at its own deadline and does not consume the process-exit reserve.	Observability prevents observable shutdown.
Terminate several instances together	Fleet capacity and saturation remain within the declared policy.	Locally clean drains overload surviving instances.
Force a kill before voluntary exit	The outcome is classified as forced and unfinished work is reconciled.	Process disappearance is mislabeled as success.

One successful run proves only that the tested workload followed the observed path under those timings. Report the platform, workload class, grace period, hook duration, sample size, proxy or load-balancer path, and blind spots.

Compact termination checklist

Before calling a shutdown path graceful:

Record the platform’s actual termination sequence and total grace period.
Treat hook execution as part of that budget when the platform does.
Change routing eligibility and application admission state deliberately.
State how persistent connections learn that draining has begun.
Stop acquiring queue work before settling existing leases.
Give accepted work a finish, cancel, checkpoint, or retry-safe path.
Bound every cleanup step, including telemetry and log flushing.
Reserve time for voluntary process exit before forced termination.
Record voluntary exit and forced stop as different outcomes.
Test late arrivals, long requests, streams, stuck hooks, slow dependencies, and lost workers.
Test simultaneous terminations and the capacity left behind.
Reconcile user-visible and durable effects after each forced-stop test.
Report the tested platform, workload class, timing, sample size, and blind spots.
Re-test when proxies, runtimes, hooks, deadlines, or workload semantics change.
Do not call the deployment drained until the remote user path and fleet state support that claim.

A stop signal can begin a shutdown. A readiness change can reduce ordinary new routing. A clean exit can show that the process stopped itself. None of those observations alone proves that accepted work settled or the rollout remained healthy.

The useful claim is narrower: within one declared termination budget, the tested service stopped new admission, handled each accepted workload class according to its contract, preserved unfinished-work evidence, and exited before the forced deadline. That claim is strong because every part can fail visibly.

Source notes

Kubernetes, Pod Lifecycle: first-party documentation for Pod termination flow, terminating EndpointSlice state, ordinary traffic readiness behavior, stop signals, grace periods, and forced termination.
Kubernetes, Container Lifecycle Hooks: first-party documentation for PreStop, including that the termination grace-period countdown begins before the hook executes and that the hook must complete before the stop signal is sent.
Kubernetes, Explore Termination Behavior for Pods And Their Endpoints: first-party tutorial demonstrating the overlap between Pod termination and endpoint state, and explaining ready=false plus the serving condition for terminating endpoints.

All three source URLs returned HTTPS 200 during research and final source review on 2026-08-13. Focused review confirmed the documented grace-period, PreStop, forced-termination, and terminating-endpoint mechanics used above. The shutdown contract, state model, budget worksheet, workload split, test matrix, and checklist are Alfred’s proposed operating method. These sources do not establish universal timing values, prove that a particular proxy immediately stops routing, prove that accepted work completed, or verify any deployment.