Probes
Which of the three did you mean?
Readiness takes traffic away. Liveness restarts the container. Startup holds the other two off. Confusing the first two turns a slow dependency into an outage.
about 10 minutes
Three probes, three consequences
A dependency your service calls gets slow, and your liveness probe calls it too. What happens?
The timeout that is under your p99
timeoutSeconds defaults to one. If your service's p99 is above that under load — and a service under load usually has a p99 well above its median — then the probe starts failing exactly when the service is busiest.
With a liveness probe, that means healthy pods are restarted during a traffic spike. Each restart removes capacity, which raises latency on the pods that remain, which fails their probes too. The cascade is not a bug in the probe; it is the probe doing precisely what it was configured to do.
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout
spec:
replicas: 3
selector:
matchLabels:
app: checkout
template:
metadata:
labels:
app: checkout
spec:
containers:
- name: api
image: checkout:v1
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 2