alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work gnss-honesty

GNSS off for 5.5 hours, and a cold start that lied

One lost “on” request kept GNSS off for 5.5 hours while the firmware said it was on. Separately, a test claimed 1–3 s cold starts because it read a stale NMEA line. I fixed both. Real cold starts are 30–65 s.

5.5 hGNSS module off after a lost request

A tracker drove out of an underground car park and didn’t get a fix for 5.5 hours. The logs showed the GNSS module had never been powered on, even though the firmware said it was.

Separately, the firmware’s fix-time test was reporting cold starts of 1–3 s. That’s far too good to be true.

Two different bugs, same problem: the firmware was reporting something about the GNSS that wasn’t real.

Reproducing the race

I suspected a race in the restart sequence. I added a test hook that floods the GNSS request path, and that reproduced it on the bench: the firmware says GNSS is on, the module stays off.

How a request got lost

A restart sends “off”, then “on” 2 s later. Requests were de-duplicated against the module’s current state. When “on” arrived, “off” hadn’t been applied yet, so the module still looked on, and “on” was dropped as a duplicate. Then “off” completed, and nothing turned it back on.

The fix

I added a pending-request counter and a FIFO, so requests can’t be lost or reordered while an earlier one is still being applied. As a backup, a check runs every second and resends the request if the wanted and actual states differ for 30 s.

Switching GNSS on or off by command went from 45 s to 3 s, with no reboot. In QA, two trackers got their fix back quickly in driving and long-term tests.

Time to switch GNSS on or off by command

The test that passed too fast

With tick logging, the test “passed” after about 215 ticks. The raw NMEA showed why. A valid $GNGGA line from before the reset arrives after the reset command but before the module’s start message, and the firmware took it as a fresh fix.

I added a 300-tick guard window, about 1.3 s, that ignores GNSS status right after a reset. A fix-holdout feature was also keeping the old status alive, so I cleared it in the two places where the test forces a fix loss. I also tried clearing the assistance data. That made no difference.

A lost "on" request, and a stale NMEA line read as a fix Before: a GNSS restart sends "off" at 0 s and "on" at 2 s. Requests are de-duplicated against the module’s actual state. The off has not been applied yet, so the module still looks on and the "on" request is dropped. Then the off completes, and nothing turns it back on. The firmware reports GNSS on while the module stays off, with no fix for 5.5 hours. After: the "on" request waits in a FIFO with a pending-request counter and runs after the off is applied, so the module powers on. A check runs every 1 s and sends the request again if the wanted and actual states differ for 30 s. Fix-time test: after the reset command, a stale but valid $GNGGA line from before the reset arrives before the module's start message. The old test took it as a new fix at about 215 ticks. A 300-tick guard window, about 1.3 s, now ignores GNSS status after a reset and covers that line. To scale on an axis from 0 to 70 s, the cold start reported by the old test was 1 to 3 s; the fixed test reports 30 to 65 s. BEFORE · restart sends off, then on 2 s later (not to scale) request off · t=0 s on · t=2 s module looks on: dropped module off not applied yet, looks on stays off · no fix for 5.5 h firmware reports GNSS on The firmware state and the real module state disagree, and stay that way. AFTER · the request waits its turn, a 1 s check backs it up request off · t=0 s on · t=2 s FIFO + pending count module off not applied yet, looks on powered on · fix regained check every 1 s: if wanted ≠ actual for 30 s, send again FIX-TIME TEST · what arrived after the reset command 1 · reset command 2 · stale $GNGGA, still valid 3 · module start message Line 2 is from before the reset. The old test took it as a new fix. TICKS AFTER RESET · to scale guard window · 300 ticks ≈ 1.3 s 0 300 ≈215: old test "passed" REPORTED COLD START · to scale old test 1–3 s fixed test 30–65 s 0 30 60 s
A lost "on" request, and a stale NMEA line read as a fix

Conclusion

GNSS now switches in 3 s instead of 45, and requests no longer get dropped. The test reports real numbers: cold start 30–65 s, warm about 3–5 s with assistance data, hot 3–4 s.

A result that looks too good deserves the same suspicion as one that looks bad. The cold-start number was off by more than ten times, and nothing about it looked broken.

Tags gnss rtos testing field-bug