A weekly crash and one memory dump
A crash showed up about once a week on our stability stand, and never on a desk. One memory dump showed two tasks freeing the same MQTT buffers. I made the race happen on demand, then fixed it.
In 2026, devices on our stability stand crashed about once a week. They were sending records over MQTT with TLS to a cloud broker. The RTOS reported a failed buffer release from the record-send task and reset the device.
Once a week is awkward. You can’t reproduce it on a desk, and you can’t prove a fix by waiting a week and hoping.
Working from one dump
With no way to reproduce it, I had one memory dump to work from. I traced the failed release back to link teardown. Two tasks, the MQTT close path and the record-send close path, had both torn down the same link at the same moment.
The double free
Both paths released the link’s connection buffers. The first release returned them to the pool. The second asked the RTOS to free buffers that were already free. The RTOS treats that as fatal and resets. The two closes rarely line up, so it happened about once a week.
Making it happen on demand
I added a test command that forces both closes to run at the same time. Now I could trigger the race whenever I wanted.
Then the fix: link teardown is now guarded, so the buffers are released exactly once, whichever task gets there first. The second one sees they’re gone and skips.
Conclusion
The fixed firmware ran a full week on the stand with no abnormal restart. To be fair, one clean week is weak proof for a bug that showed up once a week.
What made this solvable was the test command. A rare race doesn’t need a longer soak. It needs a way to force it.