On the defence side I work on a real-time detection and tracking pipeline: visible and thermal video, C++, running on an NVIDIA Jetson. The same binary uses OpenMP and CUDA C. People keep asking me which one is better, and the question is wrong. They parallelise different things. The real skill is deciding which stage of the pipeline belongs to which, because both of them spend from the same budget: the milliseconds you have per frame.
Figure: rendered by me on synthetic data.
Two shapes of parallelism
OpenMP is shared-memory parallelism on the CPU. You have a handful of fat
cores. You mark a loop with #pragma omp parallel for and the runtime spreads
the iterations across them. Getting started costs almost nothing. And each core
is good at exactly the things a GPU hates: branches, pointer chasing, code
where every iteration does something different.
CUDA is the opposite shape. Thousands of lightweight threads, all of which want to execute the same instruction on different data. Per-pixel work is the textbook case: a resize, a colour conversion, a filter kernel. But when threads inside a warp take different branches, they serialise, so branchy code wastes the machine.
Both tools come with fine print that only shows up under load. On the OpenMP side the classic trap is false sharing: two threads updating neighbouring elements that live on the same cache line, silently turning your parallel loop into a cache ping-pong match. The other one is oversubscription, where OpenMP grabs every core while the rest of the process still needs a few. On the CUDA side the fine print is launch overhead and occupancy: a kernel launch has a fixed cost, so firing a kernel at a few dozen items is like renting a stadium for a five-person meeting. Both lessons point the same way: the shape of the work picks the tool, not the other way around.
The transfer tax
The first thing CUDA teaches you: the kernel is rarely the expensive part. Moving data is. On a desktop the frame crosses PCIe. On Jetson the CPU and GPU share the same physical memory, which helps, but a careless pipeline still copies frames back and forth for no reason.
My rule: get the frame onto the GPU once, keep it there through preprocessing and inference, and bring back only small things: boxes, scores, track IDs. Never the frame itself. I went deeper into this split in CUDA, CUDA C and the OpenCV CUDA module.
On Jetson there is a second layer to this. Shared DRAM does not mean free transfers: the default OpenCV path still stages a copy when you upload a Mat into a GpuMat, and every synchronisation point stalls the pipeline while CPU and GPU wait for each other. Pinned or managed memory, buffers allocated once and reused every frame, and asynchronous streams take most of that pain away. The habit that stuck with me: treat every cudaMemcpy in the hot path as a bug until it proves it deserves to live.
Where each one wins in a tracking pipeline
In the tracking work the layout settled like this. GPU: undistortion, resize, colour conversion, inference. That is uniform per-pixel arithmetic over millions of pixels, exactly what CUDA exists for. CPU with OpenMP: everything per-object. Association, filter updates, track management, drawing for the operator. A busy scene has dozens of tracks, not millions of pixels, and the logic is full of branches. A GPU has nothing to chew on there. A CPU core eats it.
OpenMP also earns its keep as pipeline glue: parallel sections that overlap capture, processing and display, so the stages hide each other’s latency. That is more a multithreading story than a parallelism story, and I wrote it up separately in C++ multithreading for real-time pipelines.
Three questions that route a stage
After enough iterations the decision stopped being taste and became three questions I ask about every stage of a pipeline.
First: is the work uniform per element? If a stage applies the same arithmetic to millions of pixels, it is GPU shaped. If every iteration branches differently, dozens of tracks each running its own state machine, it is CPU shaped, and OpenMP gets it.
Second: how wide is the parallelism, really? A GPU needs thousands of independent items before it breaks even. Per-pixel work qualifies. Per-track work does not: a busy scene gives you tens of objects, and no GPU can spread that across its hardware. A handful of CPU cores is plenty for it.
Third, and this is the one people skip: where does the data live before and after the stage? A stage that computes fast on the GPU but forces a frame-sized download in the middle of the pipeline is a net loss. I have moved work to the “wrong” processor more than once purely because it kept the frame where it already was. The transfer bill is part of the stage’s cost, and it routinely outweighs the compute.
Case study: routing the defence pipeline, stage by stage
The place where all of this stopped being theory is the defence tracking project: real-time detection and tracking over visible and thermal channels, C++ with CUDA C and OpenMP, running on a Jetson, with a Qt interface in front of the operator. I cannot share numbers, latencies or test data from it. I can share the routing logic, because that is the part that transfers to any pipeline.
Capture and decode stay on the CPU. The sensors hand you frames however they hand you frames; that code is I/O and vendor API calls, nothing to parallelise. From there the frame goes up to the GPU once, and preprocessing happens where it lands: undistortion, resize, format conversion, normalisation for the network. Every one of those is the same arithmetic repeated across every pixel, on both the visible and the thermal channel. The thermal frame is single-channel and needs its own conditioning, but the shape of the work is identical, so it rides the same CUDA path.
Inference runs on the GPU too, straight out of those buffers. Then comes the pivot point of the whole design: what comes back from the GPU is boxes, scores and classes. Kilobytes, not megabytes. Everything downstream of that line is per-object logic: association between detections and existing tracks, filter updates, track birth and death, gating decisions. That is branch-heavy code over tens of items, exactly the wrong shape for a warp and exactly the right shape for OpenMP across CPU cores.
Frame: real output from the project.
The copy question shaped decisions constantly. The pattern looked like this: anything that wanted the full-resolution frame back on the CPU mid-pipeline got treated as a design smell, and I restructured until only the display path touched full frames. The operator overlay, boxes, crosshair, symbols, is drawn onto video that is already headed for the screen, so it adds no extra frame traffic. The Qt interface lives on its own thread with one rule: the UI never blocks the pipeline and the pipeline never blocks the UI. OpenMP parallel sections overlap capture, processing and display so that each stage hides the latency of its neighbours.
The result of that discipline is not a benchmark I can print here. It is that the same routing has survived years of field use without the architecture needing to change: GPU for the pixels, CPU for the objects, and a strict border between them that only small things are allowed to cross.
One clock, so measure it
Amdahl’s law is the boring truth underneath all of this: speed up the parallel part as much as you like, the serial glue sets your ceiling. In practice my bottleneck was almost never the kernel I was proud of. It was a copy I forgot about, or an OpenMP region fighting over cores with the thread that feeds the GPU.
Two habits help. Pin down how many threads OpenMP actually gets, instead of letting it grab everything and starve the rest of the process. And profile before you optimise anything: on Jetson, tegrastats plus a timer around each stage tells you more than any intuition about what “should” be fast.
So the answer is not OpenMP versus CUDA. It is OpenMP and CUDA, each doing the kind of parallel it is shaped for, both billed against the same frame time.