alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work at-contract

One SMS, 100 replies, and a reboot loop

One SMS command got over 100 replies. One modem event put trackers in a reboot loop. Both times the modem said something the firmware didn’t expect. I capped the resends, taught the parser the new format, and five blocked tickets were cleared.

100+SMS replies to one command, then an endless reboot loop

The firmware talks to the cellular modem through AT commands. It sends a command, and the modem answers or reports events on its own. Every handler in the firmware assumes something about what the modem will say.

Here are two times, a year apart, when the modem said something else.

100 replies to one SMS

In 2022 a user sent one SMS command to a tracker and got more than 100 replies.

Two things together caused it. The firmware resends an SMS reply if sending fails. And this modem returned an error for SMS sends even when the message had gone out. So each reply reached the user, the firmware thought it had failed, and sent it again.

The firmware can’t tell a fake error from a real one, so there’s no clean fix on our side. What I added is a safety net: once 10 replies are queued, the firmware restarts the modem instead of resending forever.

A reboot loop from three words

In 2023, trackers with one of our external modems went into an endless restart loop, with the default configuration. Five other tickets were blocked behind this one.

The modem reports packet data events with +CGEV. The handler for it sits in about 3,000 lines of URC handling in C. This modem also sent three-word events like +CGEV: ME PDN ACT. The handler had no case for that, parsed it wrong, and the device fell into a restart loop.

I added the three-word cases, ME PDN and NW PDN, so they’re parsed in their own format. The fix was merged to the release branch the same week.

The reboot loop: a three-word modem event had no case in the handler and put devices in a restart loop Schematic flow, not to scale. Before: the device boots with the default configuration. The modem sends the packet data event +CGEV: ME PDN ACT, which has three words after the colon. It reaches the +CGEV handler, part of about 3,000 lines of URC handling in C. The handler has no case for three-word events, so it mis-parses the event and the device restarts abnormally, then boots and restarts again in an endless loop. Five other tickets were blocked behind this crash. After: the same event reaches the handler, which now has the three-word cases ME PDN and NW PDN. The event is parsed as its own format and the device keeps running. The fix was merged to the release branch in the same week and all five dependent tickets were unblocked. BEFORE · the handler had no case for three-word events schematic device boots default config +CGEV: ME PDN ACT three words after the colon +CGEV handler ~3,000 lines of URC handling in C no case for three words unhandled format → restart abnormal restart endless restarts dependent tickets 1 2 3 4 5 5 blocked behind this crash AFTER · three-word cases added, merged the same week device boots default config +CGEV: ME PDN ACT same event +CGEV handler same handler, two cases added new cases: ME PDN, NW PDN parsed as its own format keeps running dependent tickets 1 2 3 4 5 all 5 unblocked the same week
The reboot loop: a three-word modem event had no case in the handler and put devices in a restart loop

Back to the same event, later

I came back to +CGEV more than once. In 2024 I used +CGEV: ME DETACH in traces to explain why a link closed right after a server command, and showed the flow was the same on every modem. In 2025 I changed how another modem family handles a network detach: the firmware now asks the modem for the context’s current IP, and keeps the connection open if the modem has already recovered.

Conclusion

The restarts stopped, and all five blocked tickets were cleared. The SMS cap stops the loop, but it doesn’t fix the modem’s false error.

Both bugs come from the same habit: trusting whatever the modem says. A modem handler has to cope with answers that are wrong, and with answers it has never seen.

Tags modem field-bug