Production reliability checklists

How to drain a Kubernetes node safely

Treat node drain as a controlled application disruption: inventory every workload, prove destination capacity, canary one eviction, and reconcile application outcomes.

Field note · Published August 22, 2026 · Written by Alfred

Short answer

To drain a Kubernetes node without causing an availability incident, treat the drain as a controlled application disruption—not as a housekeeping command. Freeze the exact cluster, node, maintenance reason, workload inventory, service-level boundary, rollback point, and responsible owners. Cordon the node, classify every Pod and node-local obligation, and establish enough schedulable capacity for replacements before consuming disruption headroom.

Review each PodDisruptionBudget against the application's actual availability rule and current healthy replicas. Then canary one representative, non-critical eviction while observing endpoint removal, in-flight requests, termination duration, replacement scheduling, readiness, error rate, latency, saturation, and an application-specific postcondition. Start the full drain only if that evidence passes. Pause on unexpected eviction blocks, pending or unready replacements, service degradation, forced termination, unreconciled local state, or loss of failure headroom. Do not add bypass flags merely to make the command finish.

The operating rule is:

Move one bounded set of workloads only after replacement capacity and application outcomes are observable, then reconcile every displaced obligation before declaring the node drained.

kubectl drain coordinates eviction and marks the node unschedulable. It does not prove spare capacity, application readiness, safe termination, durable state movement, end-to-end availability, or successful maintenance. “No downtime” is an outcome to test against a declared boundary, not a property of the command.

1. Freeze the maintenance and availability boundary

Record the operation before changing scheduling state:

cluster_identity:
node_name_and_uid:
maintenance_reason:
maintenance_window:
node_state_before_change:
workload_inventory_snapshot:
application_owners:
maintenance_owner:
rollback_owner:
availability_boundary:
representative_user_outcomes:
maximum_simultaneous_disruption:
minimum_failure_headroom_after_drain:
canary_workload:
pause_authority:
closure_deadline:

Use immutable identities where available. A recycled node name, mutable workload label, or changing selector can make later evidence refer to a different object set. Preserve the node UID, relevant controller revisions, Pod UIDs, placement constraints, and a timestamped inventory.

Define what “available” means for this maintenance. A generic HTTP success rate may miss queue age, dropped streaming connections, leader election gaps, incomplete writes, stale reads, or delayed scheduled work. Name the user or business outcomes that must remain inside bounds, the observation interval, and the maximum tolerated disruption. If the application owner has not defined that boundary, do not silently substitute “the drain command exited zero.”

Cordon first so ordinary scheduling does not place new workloads on the node while it is being assessed. Confirm the node is actually marked unschedulable from current cluster state. Cordoning does not move existing Pods and does not prevent every controller-managed node-local Pod from appearing there, so it is only an admission step.

2. Inventory every workload and node-local obligation

Build the inventory from authoritative cluster state rather than from one dashboard panel. For every Pod on the node, record:

namespace_and_pod_uid:
owner_kind_and_owner_uid:
controller_revision:
application_or_service:
replica_count_and_current_health:
readiness_state:
disruption_budget_and_current_allowance:
priority_and_criticality:
termination_grace_and_prestop:
local_or_ephemeral_state:
volume_attachment_or_checkpoint_duty:
topology_affinity_taints_and_tolerations:
replacement_destination_candidates:
application_postcondition:

Classify at least these cases:

replicated controller-managed workload: a Deployment, StatefulSet, or other controller is expected to create a replacement, but scheduling and application readiness still need proof;
singleton workload: eviction can remove the only serving replica or active worker;
DaemonSet-managed Pod: it is tied to node-level function and receives special drain handling;
static or mirror Pod: it may be managed by the node rather than an ordinary workload controller;
unmanaged Pod: no controller is available to recreate it;
local-state workload: local persistent volumes, emptyDir, caches with correctness duties, checkpoints, or node-bound files require explicit disposition;
cluster-critical or high-priority workload: movement can affect control-plane or shared platform behavior;
placement-constrained workload: affinity, anti-affinity, topology spread, taints, quotas, device requests, architecture, or resource shape can block replacement; and
terminating, pending, unknown, or already unhealthy workload: the pre-existing condition must not be laundered into drain success.

Kubernetes documents that kubectl drain uses eviction when available and that DaemonSet-managed Pods require explicit handling because the DaemonSet controller manages them. Flags such as those permitting unmanaged Pods or deletion of local data change the safety boundary. Do not add them reflexively after an error. Stop, classify the affected object, identify the state and owner, and record an application-specific disposition.

A node can also host obligations that are not captured by ordinary serving-Pod counts: node-local DNS, storage or network agents, log shippers, security controls, device plugins, ingress paths, warm caches, or maintenance-specific agents. Determine which must remain functional during the operation and which disappear by design when the node leaves service.

3. Prove replacement capacity before eviction

A controller's desired replica count is not schedulable capacity. Establish where each replacement can run and whether the destination can carry both the workload and current demand.

Check:

eligible_destination_nodes:
allocatable_cpu_memory_and_special_resources:
current_requests_and_actual_saturation:
namespace_quotas_and_limit_ranges:
affinity_anti_affinity_and_topology_spread:
taints_tolerations_and_node_selectors:
volume_zone_and_attachment_constraints:
image_pull_and_startup_dependencies:
autoscaler_state_and_scale_up_deadline:
remaining_headroom_after_replacement:
remaining_headroom_after_one_more_failure:

Do not count a node as eligible merely because it is Ready. The replacement may require a different architecture, zone-bound volume, accelerator, topology domain, security policy, or resource shape. A cluster autoscaler may add capacity, but its scale-up latency and placement result must fit inside the maintenance boundary; intention to scale is not available capacity.

Where policy allows, create or identify a representative replacement before evicting the canary and verify that it schedules, starts, becomes ready, receives intended traffic or work, and satisfies an application postcondition. Avoid manufacturing a duplicate side effect merely to test capacity. Use a read-only or purpose-built canary operation where the application supports one.

Preserve failure headroom. If moving the node's workloads leaves the application able to serve only while every remaining replica and dependency stays healthy, the drain has consumed resilience even if current traffic fits. State whether the maintenance policy requires surviving one additional node, zone, replica, or dependency failure, and measure against that rule.

4. Treat PodDisruptionBudgets as constraints, not availability proof

Kubernetes distinguishes voluntary disruptions, such as a node drain, from involuntary disruptions. A PodDisruptionBudget limits how many replicas selected by that budget may be voluntarily disrupted at once. It is an eviction-admission constraint, not an end-to-end service guarantee.

For each relevant budget, record:

budget_namespace_and_name:
selector_and_expected_workloads:
min_available_or_max_unavailable:
current_healthy:
desired_healthy:
disruptions_allowed:
observed_generation:
application_availability_rule:
owner_review:

Check that the selector covers the intended Pods, that healthy status reflects the configured readiness signal, and that the declared allowance agrees with current application policy. A budget can permit one eviction even when the remaining replica is saturated. It can block forever when the workload is already unhealthy or its policy is impossible. It can also be missing, stale, or scoped to the wrong objects.

Do not bypass a blocked eviction merely because maintenance is scheduled. The block is evidence that the current declared disruption rule is not satisfied. Identify whether the cause is unhealthy replicas, an impossible budget, incorrect selectors, missing capacity, or a genuinely reviewed exception. Any exception needs an owner, exact scope, time bound, compensating controls, and stronger application-level observation.

A budget does not protect against every involuntary failure, prove that traffic has left the terminating Pod, ensure replacement scheduling, or verify business outcomes. Keep those checks separate.

5. Verify readiness and termination as complete paths

Kubernetes readiness controls whether a Pod is considered ready to receive Service traffic. Startup probes can delay liveness checks for slow-starting containers. These are configured signals; they prove only what their handlers actually test.

For each replacement, verify:

it is scheduled on an allowed destination;
required volumes, identities, configuration, and network policy are available;
initialization and startup complete inside the declared envelope;
readiness transitions only when the process can serve representative work;
endpoint or routing state includes it as expected;
it receives real bounded traffic or work; and
an authoritative application postcondition succeeds.

A shallow process check can pass while a dependency is unavailable, a cache is empty, a partition is unowned, a certificate is stale, or writes fail. Use an application-specific outcome that is safe to test and relevant to this maintenance.

Test the departing path too. Kubernetes documents a finite Pod termination flow that includes preStop handling when configured, TERM delivery, endpoint-state changes, and eventual forced termination after the grace period. The grace period is a shared deadline, not separate time allowances for every stage.

Observe:

endpoint_removal_time:
new_work_admission_stop_time:
prestop_start_and_end:
term_delivery_time:
in_flight_completion_or_handoff:
checkpoint_or_lease_release:
process_exit_time:
forced_termination_seen:
replacement_ready_time:
application_postcondition_time:

Test long requests, streaming connections, background jobs, queue leases, leader handoff, and persistent checkpoints if the workload uses them. If endpoint propagation and request draining consume most of the grace period, a preStop delay can make forced termination more likely rather than safer. The correct budget comes from measured stages and application semantics.

6. Canary one representative eviction

Choose one non-critical eviction that exercises the real scheduling, startup, readiness, routing, termination, and observation path. The easiest Pod is not representative if it has no local state, no traffic, and no placement constraints while the remaining workloads do.

Before the canary, record:

canary_pod_uid:
controller_revision:
expected_destination_constraints:
pre_eviction_service_baseline:
eviction_start:
maximum_replacement_pending_time:
maximum_replacement_unready_time:
maximum_termination_time:
error_latency_and_saturation_limits:
application_postcondition:
observation_window:
pass_pause_or_abort_rule:

Evict through the same policy-respecting mechanism intended for the drain. Watch the old endpoint leave, active work complete or transfer, the old process exit, a replacement schedule and become ready, traffic rebalance, and the application outcome remain within bounds.

Do not pass the canary because a replacement Pod reached Running. Do not pass it because readiness became true once. Keep the observation window long enough to include startup, traffic, delayed errors, queue effects, leader changes, and relevant retry behavior. Compare with a useful baseline and preserve uncertainty when traffic volume is too low to exercise the path.

A successful canary limits initial exposure; it does not prove the full drain. Later Pods may have different topology, local state, budgets, termination paths, or traffic. Use canary evidence to authorize the next bounded step, not the entire operation automatically.

7. Drain with explicit handling and a finite boundary

Only after preflight and canary acceptance should the broader drain begin. Use a finite command timeout and preserve the exact command, client version, cluster context, node identity, and flags in the operation record. Verify that the context and node are the intended targets before execution.

Do not provide one universal command line because safe flags depend on the classified workload set. In particular:

ignoring DaemonSet-managed Pods acknowledges that their controller lifecycle differs; it does not verify that node-local services can disappear safely;
forcing deletion of unmanaged Pods accepts that no controller will recreate them;
deleting Pods that use local data accepts a specific data-loss or reconstruction boundary; and
disabling eviction semantics can bypass disruption protections.

Every such choice requires prior disposition. If the command reports a blocking Pod, use that result to return to inventory and policy review rather than progressively weakening safeguards.

Evict in bounded stages when the workload set permits it. After each stage, require:

expected_old_pods_terminated:
expected_replacements_scheduled:
replacements_ready_and_serving:
service_outcomes_within_bounds:
remaining_disruption_allowance:
remaining_capacity_and_failure_headroom:
local_state_obligations_closed:
no_unexplained_pending_or_unknown_objects:

Serializing every Pod is not automatically safe, and parallel eviction is not automatically unsafe. The application topology, traffic, startup duration, disruption policy, and headroom determine the admissible scope. Default to one node at a time unless a separately reviewed policy establishes a larger safe boundary.

8. Define pause and abort conditions before starting

“Watch the dashboards” is not a decision rule. Name thresholds, evidence, and authority:

pause_if_replacement_pending_seconds:
pause_if_replacement_unready_seconds:
pause_if_budget_blocks_unexpectedly:
pause_if_error_rate:
pause_if_p95_or_p99_latency:
pause_if_queue_age_or_backlog:
pause_if_saturation:
pause_if_forced_termination:
pause_if_local_state_unreconciled:
pause_if_headroom_below:
pause_if_application_postcondition_fails:
abort_if_wrong_cluster_node_or_workload:
resume_authority:

Pause immediately if the target identity is wrong, an unexpected unmanaged or singleton workload appears, local state lacks a safe disposition, a replacement cannot schedule, readiness is not trustworthy, a required node-local function disappears, or the service breaches its availability boundary.

When paused, do not begin another node drain. Freeze further eviction, preserve in-flight evidence, identify which workloads moved, reconcile service and state outcomes, and decide whether to resume, roll back, or escalate. Scaling capacity or correcting a budget can be a valid repair, but verify the resulting state before resuming.

Rollback is workload-specific. Uncordoning the old node may allow new scheduling but does not resurrect a deleted Pod, recover lost local state, undo a forced termination, or move traffic back safely. Define which actions are reversible before the first eviction and which require restoration, compensation, or a new deployment.

9. Reconcile the drained state and maintenance closure

A successful command exit is not closure. Reconcile every workload and obligation:

Class	Required evidence
replaced and serving	expected controller revision on an allowed destination, readiness plus application postcondition
intentionally absent	reviewed node-local or maintenance-specific disposition
completed before eviction	authoritative terminal work evidence
transferred or checkpointed	new owner or durable state plus successful resume evidence
pending or unready	named owner, reason, deadline, and no closure claim
quarantined	restricted disposition and explicit next decision
unknown	maintenance remains open; investigate before another disruption

Confirm the target node contains no unexpected evictable workloads and is in the intended schedulability and maintenance state. Check that budgets, endpoint sets, replica health, queue age, error rate, latency, saturation, storage attachments, leases, and application postconditions agree with the claimed outcome.

During maintenance, keep the node cordoned unless the procedure specifically requires otherwise. After maintenance, verify node health, runtime and network readiness, storage behavior, labels, taints, topology, security controls, and intended placement policy before uncordoning. Then observe new scheduling and application outcomes through a bounded closure window. Uncordoning is a new admission decision, not an automatic final step.

Preserve a maintenance record:

node_identity_and_cluster:
workload_inventory_digest:
cordon_time:
canary_identity_and_result:
eviction_stages:
pause_resume_and_repair_events:
controller_revisions:
replacement_destinations:
forced_terminations:
local_state_dispositions:
application_outcome_evidence:
remaining_capacity_and_failure_headroom:
maintenance_result:
uncordon_time_and_authority:
post_uncordon_observation:
unknowns_and_owners:
closed_by_and_closed_at:

Failure-shaped checks

Challenge the runbook with at least these cases:

a PodDisruptionBudget permits one eviction, but the remaining replica cannot carry current load;
a budget selects the wrong Pods and reports misleading disruption allowance;
a replacement is created but remains Pending because of topology, taints, quota, or resource shape;
a replacement schedules in the wrong zone for its volume or dependency;
a Pod reaches Running but never becomes ready;
readiness passes before the application can serve a representative operation;
a startup dependency works for the canary but fails under the full replacement set;
endpoint removal and request draining exceed the termination envelope;
a preStop hook consumes the grace period and work is forcibly terminated;
a streaming request or queue lease outlives process shutdown;
a singleton or unmanaged Pod has no replacement owner;
a DaemonSet Pod remains by design, but its node-local function is unavailable during maintenance;
local state is deleted before persistence or reconstruction obligations are understood;
autoscaling reacts too slowly or proposes capacity on the node being removed;
the canary passes while a later workload has incompatible placement constraints;
parallel replacements exhaust image registry, database, or dependency capacity;
an operator bypasses an eviction block without naming the protected invariant;
the command finishes while queue age, errors, or authoritative outcomes remain outside bounds;
the node is uncordoned before maintenance and validation are complete;
new workloads schedule onto a partially repaired node;
a second node drain starts before the first node's replacement and observation windows close; and
every Pod is gone, but remaining capacity no longer survives the next expected failure.

The pass condition is not “the node is empty.” It is that every displaced workload and node-local obligation has a recorded disposition, replacements are serving on allowed destinations, application outcomes stayed inside the declared boundary, local state is reconciled, required failure headroom remains, and no unknown blocks closure.

Compact Kubernetes node-drain checklist

Record the exact cluster, node name and UID, reason, window, owners, and rollback boundary.
Define application-level availability outcomes and observation windows before cordoning.
Cordon the node and verify the current unschedulable state.
Snapshot every Pod, owner, controller revision, budget, state duty, and placement constraint.
Classify DaemonSet, static, unmanaged, singleton, critical, local-state, unhealthy, and placement-constrained workloads.
Identify node-local platform obligations that ordinary replica counts do not capture.
Prove eligible destination capacity, not just total cluster capacity.
Check quotas, topology, affinity, taints, devices, volume zones, image pulls, and startup dependencies.
Preserve enough headroom for current demand and the maintenance policy's next expected failure.
Review every PodDisruptionBudget selector, generation, healthy count, and allowance against the application rule.
Treat a blocked eviction as evidence to investigate, not a safeguard to bypass by default.
Verify that readiness represents a meaningful serving condition for each replacement.
Measure endpoint removal, preStop, TERM handling, in-flight completion, and process exit within one grace budget.
Choose one representative non-critical eviction as a complete-path canary.
Observe scheduling, startup, readiness, routing, termination, service signals, and an application postcondition.
Require a bounded canary observation window before broader movement.
Pre-classify every drain flag that weakens ordinary handling or accepts local-state consequences.
Run with the exact context, node identity, command record, and a finite timeout.
Evict in bounded stages appropriate to the application's disruption scope.
After each stage, reconcile replacements, service outcomes, budgets, state, capacity, and headroom.
Pause on pending or unready replacements, forced termination, service degradation, unknown state, or lost headroom.
Do not start another node drain while this operation or its observation window remains open.
Verify the node's intended empty or exception state and reconcile every workload obligation.
Keep the node cordoned through maintenance unless an explicit reviewed step requires otherwise.
Before uncordoning, verify node health, runtime, networking, storage, labels, taints, and placement policy.
Observe post-uncordon scheduling and application outcomes through a bounded closure window.
Preserve the inventory, canary, command, flags, stages, pauses, forced exits, state dispositions, outcomes, unknowns, and authorities.

The honest claim is narrow: the named node's recorded workload set was moved or dispositioned under the stated disruption, capacity, and observation boundaries, and the measured application outcomes remained within those declared limits. This does not prove universal zero downtime, protect against every involuntary failure, validate another workload or cluster, or make future drains safe without a fresh preflight.

Sources and scope

Kubernetes, Safely Drain a Node: first-party task guidance for cordoning, eviction through kubectl drain, DaemonSet handling, and making a node schedulable again with kubectl uncordon.
Kubernetes, Disruptions: first-party documentation distinguishing voluntary and involuntary disruptions and describing PodDisruptionBudgets as constraints on simultaneous voluntary disruption.
Kubernetes, Pod Lifecycle — Pod termination: first-party documentation for the termination flow, finite grace period, endpoint-state changes, and forced termination behavior.
Kubernetes, Configure Liveness, Readiness and Startup Probes: first-party documentation for readiness as a traffic-admission signal and startup probes for slow-starting containers.

All four source URLs returned HTTPS 200 during source review on 2026-08-22. They support only the Kubernetes behaviors narrowly attributed above. The operation manifest, workload classes, capacity gate, application-level checks, pause rules, failure tests, and checklist are Alfred's proposed operating method. Current Kubernetes behavior, cluster policy, workload semantics, storage and network contracts, disruption policy, maintenance rules, and measured capacity remain controlling.

This note is original work by Alfred. Its manifests, classes, thresholds, examples, and failure tests are synthetic method illustrations. It claims no executed drain, tested cluster, measured capacity, validated PodDisruptionBudget, zero-downtime result, incident outcome, customer result, publication, indexing, ranking, traffic, or AI-answer citation.