One reset, two bugs, one of them in a closed library
Trackers sometimes reset during firmware downgrades and left no crash record. I fixed a watchdog bug, and a weekend test went from 21 resets to 0. QA still saw failures, though. The rest was hiding in a closed vendor library.
Some trackers reset in the middle of a firmware downgrade. It happened on slow 2G links, under load, and not every time. Worse, no crash record survived, so there was nothing to start from.
It turned out to be two bugs. One was ours. The other was inside a Bluetooth library we only had as a binary.
Bug one: the watchdog
A new watchdog re-ping was too strict in three ways. It treated a late answer to a retransmitted ping as a protocol violation. It could post a ping into a full queue, which is fatal on this RTOS. And it reset a task on its first timeout, even when the task was only starved of CPU.
I changed it to accept a late answer as proof of life, skip the ping when the queue is full, give a starved task one more window, and log which task failed. Then I ran the old build and mine side by side over a weekend. The old one reset 21 times: 4 misread late answers and 17 MQTT watchdog timeouts. Mine reset 0 times.
Bug two: 11 empty scans
That looked like the end of it. It wasn’t. QA still saw failures in about 20% of runs.
Every one of those reports had the same thing right before the reset: 11 Bluetooth inquiries in a row that found nothing.
[t+ 0.0 s] bt: inquiry start
[t+ 10.0 s] bt: inquiry timeout, 0 results (1)
[t+ 10.1 s] bt: inquiry start
...
[t+110.2 s] bt: inquiry timeout, 0 results (11)
[t+110.4 s] sys: abnormal reset, crash record lost
That’s too regular to be luck. The odds are about 1 in 1010. So the Bluetooth code was involved, and that code was a closed vendor library.
Inside the closed library
I disassembled it. Inside is a pool of 32 command tokens. A Bluetooth command takes a token and gives it back when it completes. In crowded BLE areas, the vendor’s controller never finishes a classic inquiry, so every scan change queues a command whose token never comes back.
After a few minutes the pool is empty. The library then crashes and takes the crash record down with it. That’s why we never had one.
The fix
Scans now wait until the pool has room and no command is pending. If the radio hangs anyway, a watchdog restarts it and brings back classic scanning, sensor scanning and advertising. Crash breadcrumbs now live in retained RAM, so the next crash leaves something behind. The watchdog code has unit tests.
Conclusion
Each bug got its own proof. For the watchdog, the weekend A/B test went from 21 resets to 0. For the token pool, a 15 h overnight soak had 0 abnormal resets, and 174 of 174 vendor panics were recovered. The vendor got an evidence package and a service ticket, since the real fix belongs in their controller.
A clean test only proves the bug it can trigger. The 20% that didn’t go away was the lead to the second one.