Nine tracks

Each track is a question the control plane already answers and almost nobody watches it answer. Prose on one side, a running cluster and live traffic on the other, and every claim an apply you can run against it.

• Desired and observed

  2 lessons

  Who are you actually arguing with?

  The loop, and the two halves of every object: spec is what you asked for, status is what the cluster has. resourceVersion and the conflict you get for writing a stale one, why apply returns before anything has happened, and which controller is going to undo the change you just made by hand.

  • Apply returns, and nothing has happened
  • The controller you are arguing with

• Pods and the kubelet

  2 lessons

  What is the kubelet doing with this right now?

  Pod lifecycle from accepted to running, image pull as a duration rather than an instant, restart backoff and what CrashLoopBackOff is actually counting, init containers and the ordering they buy, and what a pod is before it is a unit of scheduling.

  • What the kubelet is doing with this right now
  • What CrashLoopBackOff is counting

• Probes

  1 lesson

  Which of the three did you mean?

  Readiness, liveness and startup, and the difference between them. The probe that returns 200 before the connection pool is up, the liveness probe whose timeout is under the service's p99 and so restarts healthy pods under load, the cascade that follows, and what happens when you probe a dependency you do not own.

  • Which of the three did you mean?

• Rollouts

  lands in M3

  What will this change cost in failed requests?

  maxSurge and maxUnavailable as arithmetic on small replica counts, where 25% of four is one and the service needed three. Rollout status, rollback, the rollout that is stuck forever with a green-looking cluster, and minReadySeconds as the cheapest fix in the gym.

• Traffic and endpoints

  2 lessons

  When exactly does traffic stop arriving at this pod?

  Service to endpoints to pod, and the race at the end of a pod's life: endpoint removal and SIGTERM are two independent loops and neither waits for the other. preStop and why it is a sleep, graceful shutdown, connection draining, and the long-lived connection nobody rebalances.

  • A terminating pod is still an endpoint
  • The connection nobody rebalanced

• Scheduling

  1 lesson

  Why is this pod still Pending?

  Requests versus limits and which one the scheduler reads, Pending and the reason recorded at the moment the decision was made, taints, tolerations and affinity, the cluster that is half empty and cannot fit anything, and what happens when you set requests by copying the limits.

  • Requests versus limits, and who reads which

• Pressure

  1 lesson

  Is this slow, or is it being held back?

  CPU throttling that looks exactly like a slow dependency, OOMKill and the number that was too small, QoS classes and who the kubelet evicts first, and node pressure with the eviction order it implies.

  • Throttling looks exactly like a slow dependency

• Disruption

  1 lesson

  What happens to this service at two in the morning?

  Drains and what they wait for, the PodDisruptionBudget that blocks a node upgrade indefinitely, preemption by priority, and a horizontal autoscaler acting on a metric that lags the thing it is meant to track.

  • A budget is a veto, and only over one thing

• Reading a generated manifest

  lands in

  What will this rollout cost, before you run it?

  The track the rest exists for. You are handed the Deployment, Service and HPA a model wrote for a stated service, and asked what its rollout will cost in failed requests. Then you apply it and count them.

  /audit