alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work bt-leak

One reset, two bugs, one of them in a closed library

Trackers sometimes reset during firmware downgrades and left no crash record. I fixed a watchdog bug, and a weekend test went from 21 resets to 0. QA still saw failures, though. The rest was hiding in a closed vendor library.

21  to 0resets in a weekend A/B test

Some trackers reset in the middle of a firmware downgrade. It happened on slow 2G links, under load, and not every time. Worse, no crash record survived, so there was nothing to start from.

It turned out to be two bugs. One was ours. The other was inside a Bluetooth library we only had as a binary.

Bug one: the watchdog

A new watchdog re-ping was too strict in three ways. It treated a late answer to a retransmitted ping as a protocol violation. It could post a ping into a full queue, which is fatal on this RTOS. And it reset a task on its first timeout, even when the task was only starved of CPU.

I changed it to accept a late answer as proof of life, skip the ping when the queue is full, give a starved task one more window, and log which task failed. Then I ran the old build and mine side by side over a weekend. The old one reset 21 times: 4 misread late answers and 17 MQTT watchdog timeouts. Mine reset 0 times.

Resets over a weekend, side by side

Bug two: 11 empty scans

That looked like the end of it. It wasn’t. QA still saw failures in about 20% of runs.

Every one of those reports had the same thing right before the reset: 11 Bluetooth inquiries in a row that found nothing.

[t+  0.0 s] bt: inquiry start
[t+ 10.0 s] bt: inquiry timeout, 0 results      (1)
[t+ 10.1 s] bt: inquiry start
          ...
[t+110.2 s] bt: inquiry timeout, 0 results      (11)
[t+110.4 s] sys: abnormal reset, crash record lost
Reconstructed excerpt that shows the pattern. It is not the original log.

That’s too regular to be luck. The odds are about 1 in 1010. So the Bluetooth code was involved, and that code was a closed vendor library.

Inside the closed library

I disassembled it. Inside is a pool of 32 command tokens. A Bluetooth command takes a token and gives it back when it completes. In crowded BLE areas, the vendor’s controller never finishes a classic inquiry, so every scan change queues a command whose token never comes back.

After a few minutes the pool is empty. The library then crashes and takes the crash record down with it. That’s why we never had one.

The fix

Scans now wait until the pool has room and no command is pending. If the radio hangs anyway, a watchdog restarts it and brings back classic scanning, sensor scanning and advertising. Crash breadcrumbs now live in retained RAM, so the next crash leaves something behind. The watchdog code has unit tests.

One reset, two causes. A watchdog misread late answers, and a Bluetooth token pool drained. First cause, weekend A/B test on the stand with both builds side by side: the base build reset 21 times, 4 of them a late watchdog answer counted as a protocol violation and 17 of them MQTT watchdog timeouts. My build with the watchdog fix reset 0 times. QA still saw resets in about 20% of runs with no crash record, so there was a second cause. Second cause: every QA report showed 11 Bluetooth inquiries in a row that found 0 devices, then the reset. The chance of that by luck is about 1 in 10 to the power of 10. In dense BLE areas the vendor controller never completes a classic inquiry, and each scan change queues a Bluetooth command whose token is never returned, from a 32-token pool inside the closed vendor library. The pool goes from 32 free at boot, to 16 free after 16 stranded tokens, to 0 free after 32, within minutes. When it is empty, the closed library crashes and takes the crash record with it, and the device restarts with nothing to read. Fix: a new scan only starts when the pool level is fine and no command is pending, otherwise the scan is held. When the radio hangs, a watchdog restarts it and restores classic scanning, sensor scanning and advertising. Result: a 15 hour overnight soak with 0 abnormal resets, and 174 of 174 vendor panics recovered. FIRST CAUSE FIXED · weekend A/B on the stand, side by side base build 21 resets 4 late answers misread 17 MQTT watchdog timeouts my build 0 resets QA still saw resets in about 20% of runs, with no crash record. There was a second cause. SECOND CAUSE · a 32-token pool inside the closed Bluetooth library The clue: every QA report had 11 empty inquiries right before the reset. Chance by luck: about 1 in 1010 0 0 0 0 0 0 0 0 0 0 0 reset devices found, per inquiry In dense BLE areas the classic inquiry never completes, so each scan change strands one token. boot · 32 free 16 leaked · 16 free 32 leaked · 0 free pool empty, no token for the next command the closed library crashes and takes the crash record with it abnormal reset, nothing to read FIX · gate every scan, restart the radio when it hangs new scan pool level ok, no command pending? yes scan no hold the scan radio hangs watchdog restarts radio restores classic scanning, sensor scanning and advertising 15 h overnight soak: 0 abnormal resets · 174 of 174 vendor panics recovered
One reset, two causes. A watchdog misread late answers, and a Bluetooth token pool drained.

Conclusion

Each bug got its own proof. For the watchdog, the weekend A/B test went from 21 resets to 0. For the token pool, a 15 h overnight soak had 0 abnormal resets, and 174 of 174 vendor panics were recovered. The vendor got an evidence package and a service ticket, since the real fix belongs in their controller.

A clean test only proves the bug it can trigger. The 20% that didn’t go away was the lead to the second one.

Tags ble watchdog reverse-engineering field-bug