Skip to content

Checkout is returning 500s during every deploy

Support has had a handful of complaints during each of the last four deploys. Errors spike for about twenty seconds and then stop on their own. Every pod is Running, every pod is Ready, nothing has restarted, and the rollout reports success. The database is fine.

real · about 12 minutes

Pods — in creation order

Every pod in the cluster, in the order it was created — never sorted by phase or by readiness
NamePhaseReadyRestartsNode
checkout-1t48z9-wbbjdReady1/10node-2
checkout-1t48z9-idfrcReady1/10node-2
checkout-1t48z9-8dsjfReady1/10node-1

Requests — simulated, by cause

Requests by the cause recorded when each was emitted, in the order the causes were declared
ok817
refused: pod terminating7
500: not actually ready376

Events — in the order the cluster produced them

  1. 0.3sdeploymentScalingReplicaSetScaled up replica set checkout-56p08a from 0 to 3
  2. 0.5sreplicasetSuccessfulCreateCreated pod: checkout-56p08a-kgft0
  3. 0.6sschedulerScheduledSuccessfully assigned default/checkout-56p08a-kgft0 to node-1
  4. 0.7sreplicasetSuccessfulCreateCreated pod: checkout-56p08a-mbl5g
  5. 0.8skubeletPulledContainer image "checkout:v1" already present on machine
  6. 0.8sschedulerScheduledSuccessfully assigned default/checkout-56p08a-mbl5g to node-2
  7. 0.8sreplicasetSuccessfulCreateCreated pod: checkout-56p08a-cc11j
  8. 0.9skubeletPulledContainer image "checkout:v1" already present on machine
  9. 1.0sschedulerScheduledSuccessfully assigned default/checkout-56p08a-cc11j to node-1
  10. 1.1skubeletCreatedContainer created
  11. 1.1skubeletPulledContainer image "checkout:v1" already present on machine
  12. 1.3skubeletCreatedContainer created
  13. 1.4skubeletStartedContainer started
  14. 1.4skubeletCreatedContainer created
  15. 1.6skubeletStartedContainer started
  16. 1.8skubeletStartedContainer started
  17. 122.3sdeploymentScalingReplicaSetScaled up replica set checkout-1t48z9 from 0 to 0
  18. 122.5sdeploymentScalingReplicaSetScaled up replica set checkout-1t48z9 from 0 to 1
  19. 122.7sreplicasetSuccessfulCreateCreated pod: checkout-1t48z9-wbbjd
  20. 122.8sschedulerScheduledSuccessfully assigned default/checkout-1t48z9-wbbjd to node-2
  21. 123.0skubeletPulledContainer image "checkout:v2" already present on machine
  22. 123.3skubeletCreatedContainer created
  23. 123.5skubeletStartedContainer started
  24. 124.0sdeploymentScalingReplicaSetScaled down replica set checkout-56p08a from 3 to 2
  25. 124.0sreplicasetSuccessfulDeleteDeleted pod: checkout-56p08a-cc11j
  26. 124.2skubeletKillingStopping container api
  27. 124.3sdeploymentScalingReplicaSetScaled up replica set checkout-1t48z9 from 1 to 2
  28. 124.3sreplicasetSuccessfulCreateCreated pod: checkout-1t48z9-idfrc
  29. 124.4sschedulerScheduledSuccessfully assigned default/checkout-1t48z9-idfrc to node-2
  30. 124.5skubeletPulledContainer image "checkout:v2" already present on machine
  31. 124.8skubeletCreatedContainer created
  32. 125.2skubeletStartedContainer started
  33. 125.5sdeploymentScalingReplicaSetScaled down replica set checkout-56p08a from 2 to 1
  34. 125.7sreplicasetSuccessfulDeleteDeleted pod: checkout-56p08a-mbl5g
  35. 125.8skubeletKillingStopping container api
  36. 125.8sdeploymentScalingReplicaSetScaled up replica set checkout-1t48z9 from 2 to 3
  37. 125.8sreplicasetSuccessfulCreateCreated pod: checkout-1t48z9-8dsjf
  38. 126.0sschedulerScheduledSuccessfully assigned default/checkout-1t48z9-8dsjf to node-1
  39. 126.2skubeletPulledContainer image "checkout:v2" already present on machine
  40. 126.5skubeletCreatedContainer created
  41. 126.8skubeletStartedContainer started
  42. 127.3sdeploymentScalingReplicaSetScaled down replica set checkout-56p08a from 1 to 0
  43. 127.3sreplicasetSuccessfulDeleteDeleted pod: checkout-56p08a-kgft0
  44. 127.3skubeletKillingStopping container api

Diagnosis

Which field is responsible? The fix here is always a field — if you find yourself wanting to change the application, you are in the wrong gym and debug.liter8.sh is the right one.