The 100-second timeout that became 7.8 hours
After a release, devices on our stability stand kept crashing. Instead of reading crash dumps one by one, I diffed the code between the good and the bad release. One constant had been multiplied twice: a 100 s timeout had become 7.8 hours.
After a release, devices on our long-run stability stand started crashing with abnormal restarts. Different modems, different models, no obvious pattern.
When unrelated devices start failing right after a release, the release is the place to look.
Diff, not dumps
I could have gone through the crash dumps one by one. Instead I diffed the record-save, record-send and watchdog code between the last good release and the bad one. Everything matched except one constant.
100 seconds became 7.8 hours
The timeout for the modem’s data-send command was multiplied twice. 100 s became 7.8 h.
With 100 s, a send that gets no answer fails, the task moves on and answers the watchdog. With 7.8 h, the task just waits. The record path stays blocked, watchdog pings go unanswered, and the watchdog restarts the device. A wrong timeout looked like a crash.
Fixing it, and the path around it
The fix itself was one line. Since I was already in that code, I also made the path more forgiving.
The watchdog now warns at 50% and 75%, and retries pings, so a slow answer isn’t treated as a dead task. The record manager slows down at 70% load, with at most 5 s of delay. Logs are rate-limited, and I fixed memory leaks in the SMS and network code. Beacon work waits when load is above 80%.
Conclusion
The stand went quiet. The only restarts left were on one small model under heavy bursts, which is a known limit. The whole branch went into the base firmware, so every build ships with these safeguards.
Comparing three code paths was a much smaller search than a stand full of crash dumps.