alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work one-constant

The 100-second timeout that became 7.8 hours

After a release, devices on our stability stand kept crashing. Instead of reading crash dumps one by one, I diffed the code between the good and the bad release. One constant had been multiplied twice: a 100 s timeout had become 7.8 hours.

7.8 h  to 100 smodem send timeout, bug → fixed

After a release, devices on our long-run stability stand started crashing with abnormal restarts. Different modems, different models, no obvious pattern.

When unrelated devices start failing right after a release, the release is the place to look.

Diff, not dumps

I could have gone through the crash dumps one by one. Instead I diffed the record-save, record-send and watchdog code between the last good release and the bad one. Everything matched except one constant.

100 seconds became 7.8 hours

The timeout for the modem’s data-send command was multiplied twice. 100 s became 7.8 h.

Modem send timeout

With 100 s, a send that gets no answer fails, the task moves on and answers the watchdog. With 7.8 h, the task just waits. The record path stays blocked, watchdog pings go unanswered, and the watchdog restarts the device. A wrong timeout looked like a crash.

Fixing it, and the path around it

The fix itself was one line. Since I was already in that code, I also made the path more forgiving.

The watchdog now warns at 50% and 75%, and retries pings, so a slow answer isn’t treated as a dead task. The record manager slows down at 70% load, with at most 5 s of delay. Logs are rate-limited, and I fixed memory leaks in the SMS and network code. Beacon work waits when load is above 80%.

A send that never fails fast, and the safeguards added after it Part 1 is a flow. The send task issues the modem's data-send command and gets no answer. With the designed 100 second timeout, the command fails, the task gets control back, answers the watchdog and the device stays up. With the 7.8 hour timeout in the release, the task keeps waiting, the record path is blocked, watchdog pings go unanswered, and the watchdog fires an abnormal restart. Part 2 shows the thresholds added with the fix on 0 to 100 percent scales: watchdog warnings at 50 and 75 percent of the watchdog limit with the restart at 100 percent; record manager flow control from 70 percent load with at most 5 seconds of delay; beacon work deferred above 80 percent load. Watchdog pings are retried up to 10 times so a slow answer no longer counts as a dead task. 1 WHY ONE CONSTANT RESTARTED THE DEVICE The send task issues the modem’s data-send command and gets no answer timeout 100 s, as designed timeout 7.8 h, in the release After 100 s the command fails and the task gets control back The task answers the watchdog and the record path keeps moving Device stays up The task keeps waiting, for up to 7.8 h The record path is blocked and watchdog pings go unanswered Watchdog fires: abnormal restart 2 SAFEGUARDS ADDED WITH THE FIX, THRESHOLDS TO SCALE Watchdog timer % of its limit warn warn restart 0 50 75 100 Record manager load, % flow control, max delay 5 s 0 70 100 Beacon work load, % deferred above 80% 0 80 100 Watchdog pings are also retried, up to 10 times, so a slow answer is not a dead task.
A send that never fails fast, and the safeguards added after it

Conclusion

The stand went quiet. The only restarts left were on one small model under heavy bursts, which is a known limit. The whole branch went into the base firmware, so every build ships with these safeguards.

Comparing three code paths was a much smaller search than a stand full of crash dumps.

Tags modem watchdog rtos