1 packet out of 11 survived the outage
After an outage, only 1 of 11 buffered serial packets reached the server, because every packet was saved as sequence 0. I bisected it to a merge from an unrelated feature, fixed it, and wrote the record database’s first unit tests.
In RS232/RS485 buffered mode, the tracker stores data from a serial sensor and sends it when the server is reachable again. But after an outage, only the last packet was left.
Finding the change
Buffered mode had worked before, so something had broken it. The record database stores records by sequence number in a nested folder tree. I bisected on the device across releases and merges, and landed on a merge from an unrelated feature: a rewrite of the FIFO record database.
The flag nobody read
The multi-save call still took a sequence flag, but the rewritten code never read it. So every packet was written as sequence 0. The database then decided the file was corrupted and erased it, leaving only the newest packet.
Two more problems on the same path
Buffered mode always used the TCP path, because only the non-buffered path checked the server type. So a device set up for MQTT never sent its buffered data at all. And in the JSON codec for the cloud, serial data over 18 B was dropped and 0x0A/0x0D bytes were changed. There was also an old edge case in delete-all.
The fixes: each buffered packet gets its own storage slot, which also fixed delete-all. Buffered mode now uses the same server-type check as the rest, so it works over TCP and MQTT, ASCII and binary. The JSON codec sends the full serial data as a hex string.
Conclusion
After an offline period, 11 of 11 packets now arrive, instead of 1 of 11.
Nothing caught the original break because the record database had no unit tests. I wrote the first ones, with an in-memory fake file system, a 269-line fake record-manager harness and a 5-step test plan.
The symptom was in one feature and the cause was in another. Bisecting is what connected them.