71 tests that run on a real watch
71 tests run on a real smartwatch over BLE, and any new crash fails the test, even if the test’s own checks pass. Together with a memory gate in CI, the setup has found 16 bugs.
A test on a real device can pass while the device crashes in the background. On a watch with little RAM, a change can also pass every test and still eat memory that a later feature needs.
So I built the harness to judge the whole watch, not just the feature under test, and to put a memory cost on every change.
The setup
A pytest harness drives a production smartwatch over BLE and a debug probe. It’s 75 files and 8.1k lines.
A crash guard around every test
Before each test, a guard reads the coredumps, the uptime and the BLE link. After the test, it checks for a new coredump, an unexpected disconnect or an uptime reset. If it finds one, the test fails, even if all its own checks passed.
A memory gate in CI
Every change goes through a RAM budget in bytes, a stack limit worked out from the call graph, and a stack and heap peak recorder that writes to flash. The RAM gate caught a stack-size cut that crashed over-the-air updates.
What it found
The harness turned up 16 bugs, each fixed in its own commit. As a side effect, font loading went from 180 ms to 29 ms.
I also built a binary event journal for the watch. It showed that every ECG abort had been reported with the wrong reason, because of an enum cast. I fixed it with an explicit mapping and a static_assert.
Conclusion
All 71 device tests pass with no new crashes on the current build, and the harness has found 16 bugs.
The crash guard is the part that pays off most. It turns every test into a crash test too.