alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work modem-recovery

Hung modems that recover on their own

When a modem hung, trackers lost data, and a failed operator scan could leave them stuck for good. I built recovery in steps at 10, 20 and 30 minutes without data, tested it on five modem types, then fixed what broke along the way.

10  to 20  to 30 minwithout data before modem restart, pin reset, device reset

Trackers in the field lost data whenever their modem stopped working. Nobody is around to power-cycle a tracker in a vehicle, so the firmware has to notice and fix it by itself.

Restarting a modem is the easy part. The hard part is knowing when to do it, how hard to push, and making sure there’s no path where the tracker just gives up.

Escalating step by step

Recovery is driven by how long the tracker has sent no data. At 10 minutes it restarts the modem radio with CFUN. At 20 it resets the modem through its hardware pin. At 30 it resets the whole device. Cheapest step first. I tested it on five modem types.

One watchdog instead of two

I wrote the user story to merge this recovery into the old watchdog and remove the duplicate code: 137 lines added, 183 removed. The watchdog period now grows with the configuration, so a tracker that sends rarely isn’t mistaken for a hung one. Only one recovery can fire per 10 minutes. I documented it with flowcharts and a table for every counter.

Three recovery steps on a no-data clock, and a fast path for short drops Top, drawn to scale on an axis from 0 to 30 minutes without data sent: at 10 minutes the firmware restarts the modem radio with CFUN, at 20 minutes it resets the modem through its hardware pin on modems that support it, at 30 minutes it resets the whole device. At most one recovery fires per 10 minutes, and each 10 minute period grows by the larger of minimum records times record period and the send period. Tested on five modem types. Bottom, schematic: before, a short network drop went unnoticed and the tracker waited out a long timeout, a latency of 1 minute 10 seconds. After, a short drop puts the tracker in a suspicious state, and the next communication error starts a full recovery at once. The LTE-M recovery goes layer by layer: check registration with CEREG?, check the data context, reactivate the data context, restart the radio with CFUN. The sequences also run in deep sleep, where records had been delayed 6 to 7 minutes instead of 2. ESCALATION · time with no data sent · to scale 0 10 min 20 min 30 min 1 · restart modem radio, CFUN 2 · reset modem by hardware pin, where supported 3 · reset the whole device At most one recovery per 10 min. Tested on five modem types. Each 10 min period grows by max(min records × record period, send period). FAST PATH · after a short network drop schematic before short drop goes unnoticed long timeout recovery 1 min 10 s after short drop "suspicious" next comm. error full recovery at once LTE-M RECOVERY · layer by layer, in order 1 · registration CEREG? 2 · data context check 3 · data context reactivate 4 · radio CFUN restart Works in deep sleep too, where records had been delayed 6 to 7 min instead of 2.
Three recovery steps on a no-data clock, and a fast path for short drops

What the merge broke

The merge led to about 12 follow-up bugs, and I fixed them. The watchdog counted per server, even though all servers share one modem, so I made it one counter. A CFUN restart was clearing the no-GPRS counter. A record period longer than the watchdog period caused pointless modem restarts. And one modem type has no pin reset, so that step only runs where it’s supported.

Short drops and deep sleep

A short network drop went unnoticed, so the tracker waited out a long timeout and took 1 min 10 s to react. I added a “suspicious” state: after a short drop, the next communication error starts a full recovery right away.

For LTE-M I built a step-by-step sequence: check registration with CEREG?, check and reactivate the data context, then restart the radio. I made these work in deep sleep too, where records had been arriving 6–7 minutes late instead of 2.

A scan that could strand a tracker

The operator scan ignored errors, so an empty result left the tracker stuck for good. On top of that, a dropped data context banned the operator, started no new search, and reset the recovery counter.

Now the scan result is checked, and retries happen at 30, 60 and 120 s instead of every 5 minutes. After 3 failures the modem network restarts, a dropped context starts a search right away, and a ban no longer resets the counter.

First retry after a failed operator search

Conclusion

The client confirmed the LTE-M sequence works. QA could no longer reproduce the deep-sleep delay. The operator-search fix passed a test with a roaming SIM that had data suspended.

Most of the work came after the first version. Recovery code only runs when things go wrong, so its bugs show up late, one field case at a time.

Tags modem watchdog network rtos