Articles

Kubernetes Taints and Tolerations, Explained

Taints repel pods from a node; tolerations let specific pods ignore that repulsion. How the two work together to control scheduling.

Chisato Chisato · · 5 min read
Kubernetes logo on a dark background

Taints and tolerations are a Kubernetes mechanism for keeping pods off nodes they shouldn’t run on. A taint is applied to a node and repels pods by default; a toleration is applied to a pod and lets it ignore a matching taint. Together they let a cluster reserve certain nodes for certain workloads — GPU boxes for ML jobs, spot instances for fault-tolerant batch work, dedicated nodes for a noisy tenant — without having to opt every other pod out individually.

This is the inverse of how node affinity works. Affinity is a pod saying “I prefer this node.” A taint is a node saying “nothing runs here unless it says it’s okay with this.” The default is exclusion, not attraction, which is exactly what you want when the cost of a workload landing on the wrong node is high — an untested job burning a $30/hour GPU node, for instance.

How a taint is structured

A taint has three parts: a key, a value, and an effect.

kubectl taint nodes node1 gpu=true:NoSchedule

Here gpu is the key, true is the value, and NoSchedule is the effect. The effect determines what happens to pods that don’t tolerate the taint:

  • NoSchedule — the scheduler won’t place new pods on the node. Pods already running there are left alone.
  • PreferNoSchedule — a soft version. The scheduler tries to avoid the node but will use it if there’s no better option.
  • NoExecute — the strictest. New pods are blocked and existing pods without a matching toleration are evicted from the node.

NoExecute is what you reach for when a node becomes unhealthy or is being drained — it’s also the effect Kubernetes applies automatically when a node is marked NotReady or Unreachable, which is why pods eventually migrate off a node that’s stopped responding.

Tolerations: the pod’s side of the deal

A pod tolerates a taint by declaring a matching toleration in its spec:

tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

A toleration doesn’t force the scheduler to place the pod on the tainted node — it just removes that node from the excluded list. Whether the pod actually lands there still depends on normal scheduling: resource requests, node affinity rules, and how busy other candidate nodes are. If you want to guarantee a workload lands on specific nodes, pair a toleration with node affinity or a node selector — the taint keeps everyone else out, and the affinity rule pulls your pod in.

The operator field also accepts Exists, which tolerates any taint with the given key regardless of its value — useful for a toleration that should match a key even if the value changes over time.

A worked example: dedicated GPU nodes

Say a cluster has a pool of GPU nodes that should only run inference workloads, never routine services that would otherwise happily schedule there and starve the GPU pods of CPU and memory.

  1. Taint every GPU node: kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule.
  2. Give the inference deployment a matching toleration, plus a nodeSelector or affinity rule targeting nodes labeled workload=gpu.
  3. Every other deployment in the cluster, having no toleration, is automatically excluded from the GPU pool — no need to update dozens of unrelated manifests.

This pattern is common for cost control on cloud clusters: taint spot/preemptible node pools so only workloads explicitly built to tolerate interruption (batch jobs, stateless web tiers with enough replicas) get scheduled there, and taint on-demand pools the other direction if you want to reserve them for stateful services.

Built-in taints you’ll see without setting anything

Kubernetes applies several taints automatically, and understanding them explains scheduling behavior that otherwise looks mysterious:

TaintMeaning
node.kubernetes.io/not-readyNode’s kubelet hasn’t reported healthy
node.kubernetes.io/unreachableNode controller can’t reach the node
node.kubernetes.io/memory-pressureNode is low on memory
node.kubernetes.io/disk-pressureNode is low on disk
node.kubernetes.io/network-unavailableNode’s network isn’t configured
node-role.kubernetes.io/control-planeMarks control-plane nodes, keeping ordinary workloads off them

That last one is why a fresh cluster’s control-plane nodes sit idle for regular pods by default — they’re tainted NoSchedule from the moment they join, and only components with an explicit toleration (like core system pods) run there.

Every pod gets a couple of default tolerations for the not-ready and unreachable taints with a short tolerationSeconds, which is why a pod doesn’t get evicted the instant a node blips — it gets a grace period before NoExecute kicks in.

Taints and tolerations vs affinity

It’s worth being precise about which tool does which job, since they’re often confused:

Taints/tolerationsNode affinity
DirectionNode repels podsPod prefers/requires nodes
Default behaviorExclude unless toleratedInclude unless restricted
Typical useReserve nodes for specific workloadsPull a workload toward specific nodes
Combine?Yes — often used togetherYes — often used together

Use taints when the failure mode you’re protecting against is “the wrong pod ends up here.” Use affinity when the failure mode is “the right pod ends up somewhere else.” Most production setups for specialized hardware use both: taint the node so nothing lands there by accident, and add affinity to the workload so it’s not just permitted on those nodes but pulled toward them over ordinary ones.

The takeaway

A taint on a node blocks scheduling by default; a toleration on a pod is a permission slip that lifts that block for pods that declare it, without ever forcing placement. NoSchedule and PreferNoSchedule control future scheduling, while NoExecute also evicts pods already running there — which is the same mechanism Kubernetes uses to migrate workloads off nodes that go unhealthy. For guaranteed placement on specialized nodes, combine a toleration with node affinity: the taint keeps out what doesn’t belong, and the affinity rule pulls in what does.

Chisato Chisato · · 4 min read

Kubernetes Init Containers Explained

Init containers run to completion before a pod's main containers start, making them the standard way to handle setup steps and startup ordering.

#Kubernetes #DevOps #Cloud
Chisato Chisato · · 4 min read

Helm vs Kustomize for Kubernetes Config

Helm templates and packages Kubernetes manifests with a templating language; Kustomize patches plain YAML declaratively, with no templates at all.

#Kubernetes #DevOps #Cloud