alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work offline-loss

1 packet out of 11 survived the outage

After an outage, only 1 of 11 buffered serial packets reached the server, because every packet was saved as sequence 0. I bisected it to a merge from an unrelated feature, fixed it, and wrote the record database’s first unit tests.

1/11  to 11/11buffered serial packets delivered after an offline period

In RS232/RS485 buffered mode, the tracker stores data from a serial sensor and sends it when the server is reachable again. But after an outage, only the last packet was left.

Finding the change

Buffered mode had worked before, so something had broken it. The record database stores records by sequence number in a nested folder tree. I bisected on the device across releases and merges, and landed on a merge from an unrelated feature: a rewrite of the FIFO record database.

The flag nobody read

The multi-save call still took a sequence flag, but the rewritten code never read it. So every packet was written as sequence 0. The database then decided the file was corrupted and erased it, leaving only the newest packet.

Eleven packets buffered offline, one delivered The serial sensor sends 11 packets while the tracker is offline. Before the fix, the multi-save call took a sequence flag that the rewritten FIFO record database never read, so every packet was written to sequence 0. The database treated the file as corrupted and erased it, so only the newest packet, number 11, was left. After reconnecting, 1 of 11 packets was delivered. After the fix, each buffered packet gets its own storage place in the record store, and all 11 of 11 packets are delivered after the offline period. Each square is one packet, so the counts are drawn to scale. BEFORE · the sequence flag was never read packets sent while offline 1 2 3 4 5 6 7 8 9 10 11 record store multi-save, flag ignored all 11 → sequence 0 file seen as corrupted, erased; newest kept delivered 1 of 11 11 1 to 10 lost AFTER · each packet gets its own storage place packets sent while offline 1 2 3 4 5 6 7 8 9 10 11 record store 11 places 1 2 3 4 5 6 7 8 9 10 11 delivered 11 of 11 1 2 3 4 5 6 7 8 9 10 11 Same 11 packets. I bisected the loss on the device to a merge from an unrelated feature. One square is one packet.
Eleven packets buffered offline, one delivered

Two more problems on the same path

Buffered mode always used the TCP path, because only the non-buffered path checked the server type. So a device set up for MQTT never sent its buffered data at all. And in the JSON codec for the cloud, serial data over 18 B was dropped and 0x0A/0x0D bytes were changed. There was also an old edge case in delete-all.

The fixes: each buffered packet gets its own storage slot, which also fixed delete-all. Buffered mode now uses the same server-type check as the rest, so it works over TCP and MQTT, ASCII and binary. The JSON codec sends the full serial data as a hex string.

Conclusion

After an offline period, 11 of 11 packets now arrive, instead of 1 of 11.

Serial packets delivered after an offline period

Nothing caught the original break because the record database had no unit tests. I wrote the first ones, with an in-memory fake file system, a 269-line fake record-manager harness and a 5-step test plan.

The symptom was in one feature and the cause was in another. Bisecting is what connected them.

Tags records testing mqtt field-bug