On the defence side I write C++ for real-time detection and tracking: a visible channel, a thermal channel, an inference stage and a Qt interface, all on embedded NVIDIA hardware. None of it works single-threaded. And nearly every ugly bug I have chased on the tracking system lived in the space between two threads. This post is the short list of what actually bites.
Figure: rendered by me.
Structure threads as a pipeline, not a soup
The design that survived is boring: one thread per stage. Capture per sensor, preprocessing, inference, tracking, display. Stages own their data and talk through queues. No thread reaches into another thread’s state, ever.
The alternative, spawning a thread whenever something feels slow, produces code where you cannot answer the question “who owns this buffer right now”. If you cannot answer that question, you do not have a threading design, you have a race condition with good intentions.
The UI deserves a special rule: the Qt thread only consumes. It receives finished frames and finished track states, renders them, and never blocks on the pipeline. I wrote about the interface side in Building operator interfaces in Qt.
Bounded queues, and the courage to drop frames
Connect two stages with an unbounded queue and you have built a latency bomb. If the consumer is even slightly slower than the producer, the queue grows, quietly, and the operator ends up watching video that is seconds old while the system reports that everything is fine.
Real time does not mean “process every frame fast”. It means “always act on the newest frame”. So the queues between stages are bounded and small, one or two slots, and when a queue is full the oldest frame is dropped. Dropping frames feels wrong the first time you write it. It is the correct call: a tracker with a motion model bridges a missing frame easily, but it cannot un-see a stale one.
The bugs that actually bite
Four patterns cover most of my scars:
Shallow copies. cv::Mat copies share the underlying buffer through a
reference count. Pass one across a thread boundary casually and two threads now
write to the same pixels. Clone at the boundary, or use a proper buffer pool.
Lock ordering. Two mutexes taken in different orders in two places is a deadlock waiting for load. One lock per queue, never nested, and most of the problem disappears.
Condition variables. Wakeups can be spurious and notifies can be missed. Always wait with a predicate. The version without a predicate works on your desk and fails in the field.
False sharing. Two counters updated by two threads, sitting on the same cache line, will crawl. This one does not crash anything, it just silently eats the budget, which on an embedded board is worse.
Threads, OpenMP and CUDA share one budget
Threads give the pipeline its structure. Inside a stage, OpenMP parallelises the hot loops. The heavy math belongs on the GPU. These three compose well, but they draw from the same pool of cores and memory bandwidth: every core you hand to OpenMP is a core the capture and tracking threads do not get. On a Jetson that tension is not theoretical, you feel it the first time preprocessing and the encoder fight over the same cluster. I compared the two CPU-side models in OpenMP vs CUDA.
The rules I now start every pipeline with: every buffer has exactly one owner, data crosses threads through a bounded queue or not at all, and no raw mutex in application code without a comment saying what it protects. Threads are not the hard part. Ownership is.
References
- Wikipedia, Producer-consumer problem: https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
- Wikipedia, Race condition: https://en.wikipedia.org/wiki/Race_condition
- Wikipedia, False sharing: https://en.wikipedia.org/wiki/False_sharing
- OpenCV, cv::Mat reference (memory management and reference counting): https://docs.opencv.org/4.x/d3/d63/classcv_1_1Mat.html
- NVIDIA, CUDA C++ Best Practices Guide (asynchronous transfers and overlap): https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/