Skip to content

Writing

A triage order that survives real incidents

My default debugging loop — reproduce, read the real error, localize the layer, check config and diff, fix the root cause — with the war stories behind each step.

All writing
3 min readDebuggingReliabilityDistributed systems

When something breaks in a distributed system, the failure mode I most want to avoid is guessing. Guessing feels like progress and usually isn't. Over hundreds of debugging sessions I've converged on a default order that keeps me honest.

The universal triage order

  1. Reproduce with a real API call; capture the exact status and body.
  2. Read the real error — the bottom of the stack trace, or the actual log line, not the generic message the client sees.
  3. Localize the layer — mesh vs. app vs. downstream vs. DB.
  4. Check config and diff — git diff, and config-in-the-pod vs. local.
  5. Fix the root cause, not the symptom, and add a test so it can't recur.

Every step below is here because skipping it once cost me time.

Reproduce, then read the evidence

The first move is always to reproduce the failure through the API client and capture the exact status and body. A "500" and a "504" send you to completely different places. Once I can reproduce, I pull the correlated logs and only then form a hypothesis.

Localize the layer before asking why

This is the step people skip. For a failing request, ask where it broke before why.

A login through our identity provider's callback once returned a 504 downstream duration timeout, but the application logs showed nothing. Instead of theorizing, I pulled Istio proxy logs and app pod logs from both pods in both clusters and read them side by side:

# Mesh (Istio) access log:
"GET /oauth-callback" 504 DC downstream_duration=30001ms

# App pod log:
(no matching request-received entry)
=> the request never reached the app -> mesh/downstream timeout

That one comparison turns an opaque 504 into an actionable root cause: it's a timeout on a slow hop, not an unhandled exception. A 504 at the mesh with a clean app log points downstream every time.

The error that lies to you

Generic errors are the most dangerous, because they tempt you to fix the wrong layer. A GraphQL userContext query returned data: null with a generic "internal error" and failed only in certain regions. The resolver looked fine. The pod logs told the truth:

WARN o.s.web.cors.DefaultCorsProcessor - Rejected request: Not allowed by CORS

It was a CORS/origin configuration problem, surfaced per-region. The fix was in config, not in the resolver. If I'd trusted the GraphQL error message, I'd have "fixed" perfectly good code.

Config is the recurring tax

If I had to bet on any single root cause, I'd bet on configuration. Missing keys, the wrong active profile, secrets not injected, a truststore path that's wrong locally but fine in the pod. My checklist:

  • Confirm the active profile and that config locations point where you think.
  • Verify every required property (and vault-injected secret) is present — a missing bound value should fail fast, not silently.
  • Diff local config against what the running pod actually has.
Reproduce, read the real error, localize the layer, check config and diff, fix the root cause. In that order. Every time.

Fix the root cause, add a test

The last step is non-negotiable: fix the actual cause, not the symptom, and leave a regression test behind. A bug you can't reproduce with a test is a bug that will come back. This is the same discipline that makes the codebase trustworthy in the first place — tests are part of "done."