Every mapping or measurement pipeline I have built has eventually run into the same complaint: something that is straight in the world is not straight in the image. A runway edge, the side of a building, the horizon. The lens did that, and if you plan to measure anything through that image, you have to model it, because the error is largest exactly where you look least.
I hit this hardest on our SUAS work, where drone frames become a map and map pixels become ground coordinates. A target near the centre of the frame geolocated fine. The same target near the corner landed metres off. The detector was not wrong. The pixels were in the wrong place.
Figure: rendered by me on synthetic data.
What calibration actually estimates
The pinhole model is the starting point: every ray passes through a single point and lands on a flat sensor. Under that model, straight lines in the world stay straight in the image. Real lenses refuse.
Calibration estimates two groups of numbers. The intrinsics: focal lengths fx and fy in pixels, plus the principal point cx, cy, which is where the optical axis actually meets the sensor, almost never the exact centre. And the distortion coefficients, which describe how the lens bends rays away from the pinhole ideal. In OpenCV that is typically k1, k2, k3 for radial terms and p1, p2 for tangential ones.
Two ways lines bend
Radial distortion grows with distance from the image centre. Barrel distortion bows lines outward, and it is the default on the wide lenses we fly, because at 30 metres altitude a wide field of view is worth more than geometric purity. Pincushion bends the other way, common at the tele end. Either way, the centre of the frame is nearly honest and the corners lie.
Tangential distortion comes from the lens not sitting perfectly parallel to the sensor. It is smaller, but asymmetric: it shifts things instead of bowing them. Cheap assembly shows up here.
The checkerboard ritual, done properly
The OpenCV flow is standard: print a checkerboard, shoot it from many angles, findChessboardCorners, calibrateCamera, and you get a reprojection error back. The mechanics are a tutorial. The judgement is not. What I actually enforce:
- Cover the corners of the frame. Distortion is a polynomial fitted to your data, and where you gave it no data, it extrapolates. The corners are exactly where distortion is worst and where lazy capture sessions never put the board.
- Tilt the board. Frontal shots constrain focal length poorly.
- Read the per-view reprojection error, not just the global RMS. One blurred frame can poison the fit while the average still looks respectable.
- Calibrate the unit, not the model. Two cameras of the same model do not share intrinsics. On the defence side we run a visible and a thermal channel side by side, and each gets its own calibration, because a thermal lens has its own geometry and needs a target with thermal contrast instead of a printed board.
And recalibrate after anything mechanical: a refocus, a lens swap, a hard landing.
Where it bites downstream
Undistortion is the boring first step of everything that follows. Stitching frames into a map assumes your homographies map plane to plane; residual distortion becomes misalignment that blending hides and measurement finds. Pixel-to-ground geolocation is a ray cast through the intrinsics, so a wrong principal point moves every target on the map. Thermal detection work inherits the same problem with worse optics.
The pattern I keep relearning: distortion does not fail loudly. It hands you results that are almost right, off by amounts that grow toward the edges. That is exactly the kind of error you ship by accident.