Every detector I trained early on assumed visible light. On the defence side we run detection and tracking on both a visible and a thermal channel, and the first time you point a thermal camera at a scene you realise how much of your intuition was really an assumption about light. This post is the list of things I had to relearn.
Figure: rendered by me on synthetic data.
You are not seeing light anymore
A long-wave infrared camera measures emitted radiation. Everything above absolute zero glows in that band, so the sensor reports temperature contrast, not reflected light. The consequences are immediate. Night is a normal operating condition. Headlights do not blind the sensor. Shadows, in the visual sense, do not exist.
The surprises run the other way too. Glass is a wall: the camera sees the window, not through it. Polished metal has low emissivity, so it behaves like a temperature mirror and shows you the reflected sky instead of itself. You learn to read a thermal frame the way you learn a new language, and the old one keeps interfering.
Contrast is temperature, not texture
In visible imagery a target carries texture, colour, markings, edges. In thermal it carries a temperature difference against its background, and that is nearly all it carries. A person at night is a bright silhouette on cool ground: easy. The same person on sun-heated asphalt at noon can almost vanish, or invert polarity entirely.
There is also crossover: twice a day, around dawn and dusk, target and background temperatures pass through each other and contrast collapses for a while. If your operation can choose its timing, this belongs in the plan, not in the excuses afterwards.
Your RGB training habits break
Colour augmentation is the obvious casualty. HSV jitter on what is effectively a single-channel heat map adds nothing and can hurt. Geometric augmentation still earns its keep; photometric augmentation needs rethinking from scratch.
Two thermal-specific things to handle instead. First, polarity: cameras offer white-hot and black-hot modes, and a model trained on one meets the other in the field. Either normalise the polarity or feed both as augmentation. Second, automatic gain control: the camera remaps raw radiometric data to 8-bit output frame by frame, so the same scene can render with different contrast seconds apart. Whatever representation you train on, deploy on the same one. This is the same lesson dataset work always teaches, the one I wrote about in dataset decisions: the pipeline before the model decides more than the model.
Pretrained RGB backbones still help somewhat, because edges are edges. But budget for more fine-tuning than you expect, and for collecting thermal data of your own, because public thermal datasets are scarce compared to RGB.
Two channels beat one
The channels fail differently, and that is the point. Visible gives you texture and identity but dies at night. Thermal gives you presence in any light but cannot read a marking. In practice thermal often finds the target and visible confirms what it is.
Downstream, nothing changes: the detector still consumes tensors, so NMS and confidence thresholds apply unchanged, and the deployment path through TensorRT on Jetson is the same one we use for the visible channel. The physics changes at the sensor. The engineering after it, thankfully, does not.