Hung modems that recover on their own
When a modem hung, trackers lost data, and a failed operator scan could leave them stuck for good. I built recovery in steps at 10, 20 and 30 minutes without data, tested it on five modem types, then fixed what broke along the way.
Trackers in the field lost data whenever their modem stopped working. Nobody is around to power-cycle a tracker in a vehicle, so the firmware has to notice and fix it by itself.
Restarting a modem is the easy part. The hard part is knowing when to do it, how hard to push, and making sure there’s no path where the tracker just gives up.
Escalating step by step
Recovery is driven by how long the tracker has sent no data. At 10 minutes it restarts the modem radio with CFUN. At 20 it resets the modem through its hardware pin. At 30 it resets the whole device. Cheapest step first. I tested it on five modem types.
One watchdog instead of two
I wrote the user story to merge this recovery into the old watchdog and remove the duplicate code: 137 lines added, 183 removed. The watchdog period now grows with the configuration, so a tracker that sends rarely isn’t mistaken for a hung one. Only one recovery can fire per 10 minutes. I documented it with flowcharts and a table for every counter.
What the merge broke
The merge led to about 12 follow-up bugs, and I fixed them. The watchdog counted per server, even though all servers share one modem, so I made it one counter. A CFUN restart was clearing the no-GPRS counter. A record period longer than the watchdog period caused pointless modem restarts. And one modem type has no pin reset, so that step only runs where it’s supported.
Short drops and deep sleep
A short network drop went unnoticed, so the tracker waited out a long timeout and took 1 min 10 s to react. I added a “suspicious” state: after a short drop, the next communication error starts a full recovery right away.
For LTE-M I built a step-by-step sequence: check registration with CEREG?, check and reactivate the data context, then restart the radio. I made these work in deep sleep too, where records had been arriving 6–7 minutes late instead of 2.
A scan that could strand a tracker
The operator scan ignored errors, so an empty result left the tracker stuck for good. On top of that, a dropped data context banned the operator, started no new search, and reset the recovery counter.
Now the scan result is checked, and retries happen at 30, 60 and 120 s instead of every 5 minutes. After 3 failures the modem network restarts, a dropped context starts a search right away, and a ban no longer resets the counter.
Conclusion
The client confirmed the LTE-M sequence works. QA could no longer reproduce the deep-sleep delay. The operator-search fix passed a test with a roaming SIM that had data suspended.
Most of the work came after the first version. Recovery code only runs when things go wrong, so its bugs show up late, one field case at a time.