Why did locking a car take 10 seconds?
Locking a car remotely had crept past 10 seconds at p90. Commands, records and the protocol engine were all stuck in one MQTT task. After I split it in three, record send p90 dropped from 4.06 s to 0.12 s.
A fleet customer locks and unlocks cars over MQTT. The server sends a command, the tracker does it and replies. Over time, the p90 for a lock crept past 10 seconds. That’s a long time to stand next to a car, waiting for the door.
Measuring first
I wrote a script that pulls the round-trip time of every publish out of the device logs, and added latency probes at each stage. Then I changed one thing at a time, each behind its own flag, and measured again.
One task doing everything
There was no single bug. It was a traffic jam. Commands, record publishing and the protocol engine all ran in one task. That task waited up to 60 s for a socket send, and with no limit at all for an MQTT ACK. One slow write and everything behind it stalled, including a lock command stuck behind a queue of records. On top of that, every record was built twice.
Splitting it up
I split MQTT into three tasks: commands, publishing and the protocol engine. Now a task waiting on the network doesn’t hold up the others.
They still share one TLS session, so writes have to be serialized somewhere. I first tried the shared network layer. It deadlocked. It works in the layer that owns the TLS session.
Both waits now have a cap. The transport answers ok, busy or fatal, so a busy modem no longer kills the TLS session. The duplicate record build is gone. I also fixed a record-storage mode that an earlier update had broken.
Conclusion
Record send p90 went from 4.06 s to 0.12 s, and the median from 1.03 s to 0.05 s. The time one reply held the connection layer went from 495–1,144 ms to 32–240 ms. A calibration command went from 1,664 ms to 4 ms.
One caveat: lock replies use the same publish path, so record send time is what I measured. I don’t have the end-to-end lock time after the fix. The delay still visible in the customer’s logs came from sockets dropping while roaming.
Nothing here was slow on its own. The problem was waiting without limits, and it only showed up once each stage was measured.