Traffic and endpoints
The connection nobody rebalanced
You scaled up and the load did not move. The endpoints are right, the pods are healthy, and every existing caller is still talking to the pod it was talking to an hour ago.
about 10 minutes
Endpoints are consulted once
A proxy picks an endpoint when a connection is opened. If the caller holds that connection — and almost every modern client does, because opening one is expensive — then the pick is made once and never revisited.
So scaling up adds capacity that existing callers never reach. The new pods are healthy, they are in the endpoints, and they get nothing but whatever new connections happen to arrive. The metric that tells you is per-pod request rate, and the shape is unmistakable: the old pods are flat and the new ones are near zero.
The same pinning is why draining hurts more than it should. A caller pinned to a pod that goes away does not get rerouted — nothing is watching on its behalf. It finds out by failing.
This is the one failure in the track that a correct manifest cannot fix. The endpoints are right; the client is the thing that has to reconnect.
apiVersion: v1
kind: Service
metadata:
name: checkout
spec:
selector:
app: checkout
ports:
- port: 80
targetPort: 8080