alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work mqtt-latency

Why did locking a car take 10 seconds?

Locking a car remotely had crept past 10 seconds at p90. Commands, records and the protocol engine were all stuck in one MQTT task. After I split it in three, record send p90 dropped from 4.06 s to 0.12 s.

4.06 s  to 0.12 srecord send time, p90

A fleet customer locks and unlocks cars over MQTT. The server sends a command, the tracker does it and replies. Over time, the p90 for a lock crept past 10 seconds. That’s a long time to stand next to a car, waiting for the door.

Measuring first

I wrote a script that pulls the round-trip time of every publish out of the device logs, and added latency probes at each stage. Then I changed one thing at a time, each behind its own flag, and measured again.

One task doing everything

There was no single bug. It was a traffic jam. Commands, record publishing and the protocol engine all ran in one task. That task waited up to 60 s for a socket send, and with no limit at all for an MQTT ACK. One slow write and everything behind it stalled, including a lock command stuck behind a queue of records. On top of that, every record was built twice.

Splitting it up

I split MQTT into three tasks: commands, publishing and the protocol engine. Now a task waiting on the network doesn’t hold up the others.

They still share one TLS session, so writes have to be serialized somewhere. I first tried the shared network layer. It deadlocked. It works in the layer that owns the TLS session.

Both waits now have a cap. The transport answers ok, busy or fatal, so a busy modem no longer kills the TLS session. The duplicate record build is gone. I also fixed a record-storage mode that an earlier update had broken.

One MQTT task with two open waits, split into three tasks Before: lock commands, record publishes and the protocol engine share one MQTT task. Each record is built twice, then the task waits up to 60 seconds for the socket send and then for the MQTT ACK with no limit, so a lock command queued behind records waits for every job ahead of it. After: commands, publishing and the protocol engine run in three separate tasks. They meet in the layer that owns the TLS session, which serializes writes and caps both waits. The modem transport answers ok, busy or fatal, so a busy modem no longer closes the TLS session. To-scale bars: the time one reply held the connection layer went from 495 to 1,144 ms before, to 32 to 240 ms after, on an axis from 0 to 1,200 ms. BEFORE · one task, two open waits queue lock cmd record record one MQTT task for everything record built twice socket send waits ≤ 60 s MQTT ACK no limit The lock command waits for every job ahead of it, and any job can stall on either wait. AFTER · three tasks, one session owner, capped waits lock cmd command task record publish task, 1 build protocol engine task TLS session owner serializes writes send wait capped ACK wait capped ok/busy/fatal modem A busy modem now returns busy, so it no longer tears down the TLS session. HOW LONG ONE REPLY HELD THE CONNECTION LAYER · to scale before 495–1,144 ms after 32–240 ms 0 600 1,200 ms
One MQTT task with two open waits, split into three tasks

Conclusion

Record send p90 went from 4.06 s to 0.12 s, and the median from 1.03 s to 0.05 s. The time one reply held the connection layer went from 495–1,144 ms to 32–240 ms. A calibration command went from 1,664 ms to 4 ms.

Record send time, p90

One caveat: lock replies use the same publish path, so record send time is what I measured. I don’t have the end-to-end lock time after the fix. The delay still visible in the customer’s logs came from sockets dropping while roaming.

Nothing here was slow on its own. The problem was waiting without limits, and it only showed up once each stage was measured.

Tags mqtt rtos network