A health probe cannot tell
busy from dead.
For weeks, long runs died for no visible reason. The cause was the thing meant to keep them alive.
A timeline: health probe timing out repeatedly during heavy work, with the old behaviour (restart) marked against the new one (wait).
16/9The shape of the bug
The symptom
Long runs — the multi-hour kind this product exists for — would stop partway with no error anyone could point to. It looked like a crash, and every theory about why was a theory about how the engine was dying.
The cause
A watchdog pinged the engine on a timer and restarted it when the ping went unanswered. But an engine that is fully occupied does not answer pings. So the busier a run got, the more likely the watchdog was to decide it was dead and kill it. The supervisor was the killer, and the failure was most likely exactly when the work mattered most.
Why the first two fixes failed
Raising the timeout and adding retries both made it rarer without making it wrong less often. Both treated the probe as a source of truth that needed tuning. It is not one: unanswered says nothing about alive. The third attempt removed the ability to restart rather than guarding it, which is the only version that could not misfire.
Confirmed by a log, not a hope
A three-hour run on a different machine recorded seventy-two health-check timeouts and zero restarts, and finished. Seventy-two moments where the previous build would have killed it. That log is the proof; before it, "fixed" was a claim.
The log excerpt: 72 timeouts, 0 restarts, run completes.
4/3FAQ
So there is no watchdog at all now?
There is no ability to kill a busy engine on the basis of an unanswered probe. That capability was the bug.
What if the engine genuinely hangs?
You can restart it. The difference is that a human deciding has information a timer does not.
Why is this in Research rather than a changelog line?
Because the general lesson is worth more than the fix: a probe measures whether you got an answer, not whether the thing is alive, and any system that conflates the two will eventually kill something healthy.