timescale.info
Sign inSign up

Incident 2026-03-14: orders API outage

Three hours of syslog from web-1, web-2, db-1 and backup-1, with the investigation on top: the cause, the outage and what the on-call engineer did. A made-up incident for the logs video.

2,297 moments · 2026-03-14T01:30:00Z CE – 2026-03-14T04:29:57.914461Z CE · by @athan · @athan

Moments

  1. GET /api/orders 200 89ms
  2. GET /api/orders 200 51ms
  3. GET /api/orders 200 68ms
  4. GET /api/orders 200 80ms
  5. GET /api/orders 200 61ms
  6. checkpoint complete
  7. GET /api/orders 200 77ms
  8. GET /api/orders 200 86ms
  9. GET /api/orders 200 94ms
  10. Accepted publickey for deploy
  11. GET /api/orders 200 99ms
  12. GET /api/orders 200 35ms
  13. GET /api/orders 200 62ms
  14. GET /api/orders 200 73ms
  15. GET /api/orders 200 30ms
  16. GET /api/orders 200 66ms
  17. GET /api/orders 200 72ms
  18. GET /api/orders 200 83ms
  19. GET /api/orders 200 65ms
  20. GET /api/orders 200 28ms
  21. Accepted publickey for deploy
  22. GET /api/orders 200 54ms
  23. GET /api/orders 200 28ms
  24. GET /api/orders 200 57ms
  25. GET /api/orders 200 46ms
  26. GET /api/orders 200 50ms
  27. GET /api/orders 200 72ms
  28. GET /api/orders 200 88ms
  29. GET /api/orders 200 35ms
  30. GET /api/orders 200 45ms
  31. GET /api/orders 200 35ms
  32. GET /api/orders 200 22ms
  33. GET /api/orders 200 59ms
  34. GET /api/orders 200 68ms
  35. GET /api/orders 200 27ms
  36. GET /api/orders 200 26ms
  37. GET /api/orders 200 59ms
  38. GET /api/orders 200 42ms
  39. GET /api/orders 200 95ms
  40. checkpoint complete
  41. GET /api/orders 200 71ms
  42. Accepted publickey for deploy
  43. GET /api/orders 200 46ms
  44. GET /api/orders 200 71ms
  45. GET /api/orders 200 44ms
  46. GET /api/orders 200 63ms
  47. GET /api/orders 200 68ms
  48. GET /api/orders 200 91ms
  49. GET /api/orders 200 55ms
  50. GET /api/orders 200 76ms