A detector has no memory. Feed it a video and it gives you boxes, frame by frame, with no idea that the player in frame 400 is the same player it saw in frame 399. For a single screenshot that is fine. For anything that happens over time, it is useless.
In our sports project we track players and the ball to detect events like a shot candidate. You cannot define “player X moved toward the goal and the ball left his foot” without knowing which box is player X across two hundred frames. On the defence side, real-time target tracking is the actual product: a detection that cannot be followed is a detection you cannot act on. Identity over time is the whole job.
My own drawing.
Tracking-by-detection in one paragraph
Almost every practical tracker today follows the same loop. For each existing track, predict where it will be in the new frame, usually with a Kalman filter. Then match predictions to the new detections: compute IoU between every predicted box and every detected box, and solve the assignment with the Hungarian algorithm. Matched pairs update their tracks. Unmatched detections may become new tracks. Unmatched tracks survive a few frames on prediction alone, then die.
That is SORT, and it is the skeleton underneath most of what came after. The interesting question is what each tracker does with the detections it does not trust.
What a track actually is
It helps to be concrete about the object we are protecting. A track is a small bundle of state: a Kalman state holding position, box size and velocity, an integer ID, and a lifecycle flag. A new detection does not become a real track immediately. It starts tentative and has to be confirmed by matching again over the next few frames, which is what keeps one-frame false positives from ever getting an ID. A confirmed track that misses its match does not die immediately either. It goes into a lost state where the Kalman filter keeps predicting it forward, blind, until either a detection reclaims it or the track buffer runs out.
Every parameter you will tune later maps onto this lifecycle. How many confirmations before a track counts as real. How many blind frames before it is dead. How far a prediction may drift and still claim a detection. Keep that picture in mind and tracker configs stop looking like a pile of magic numbers.
The one idea in ByteTrack
Classic trackers apply a confidence threshold first and throw the low-score boxes away. ByteTrack’s insight is that this discards exactly the wrong data. A detection score drops when an object is occluded, blurred or half out of frame, which is precisely when a track is about to be lost. The low-score box is often not a false positive. It is your object, having a bad frame.
So ByteTrack associates in two passes. First, high-confidence detections get matched to tracks the normal way. Then, tracks that found no partner get a second chance against the low-confidence boxes. A weak detection is enough to keep an existing track alive, even though it would never be allowed to start a new one.
That asymmetry is the entire trick. No appearance embedding network, no extra model to run. On embedded hardware that matters: the tracker adds almost nothing on top of the detector you were already running.
Where it earns its keep, and where it still breaks
Football is a stress test built for this idea. Players occlude each other constantly, and the ball is small, fast and motion-blurred, which means its detection score is low in exactly the frames where losing it hurts most. The second association pass is what keeps the ball’s identity through a shot, and our shot candidate logic depends on that identity being stable.
What still breaks: long occlusions. The Kalman prediction degrades with every frame it goes unconfirmed, and when a player reappears after a slow ten-frame overlap with a teammate, the IDs can swap. ByteTrack has no notion of appearance, so two similar objects crossing paths is its worst case. If your downstream logic keys on identity, count ID switches in your evaluation, not just tracking accuracy.
Case study: one camera over a five-a-side pitch
The system that taught me most of this watches amateur football through a single fixed wide-angle camera. No pan, no zoom, no second angle. That constraint is a feature: nothing to install per match, no operator, just a camera that sees the whole pitch and a pipeline that turns the raw footage into an annotated match video, per-event clips and a highlight reel. Everything is encoded to H.264 so people can actually play the files.
The pipeline is exactly the loop from above. YOLO detects players and the ball in every frame, ByteTrack carries identities across frames. Once the identities are stable, everything downstream turns into simple bookkeeping.
Team assignment is my favourite example of identity making a hard problem cheap. I do not classify teams with a network. I crop each tracked player, take the jersey pixels and run KMeans on the colours: two clusters, two teams. Any single frame gets some crops wrong, a jersey in shadow, a keeper in a third colour, two players overlapping. But because ByteTrack hands me the same ID across hundreds of frames, I can vote over the whole track and the noise averages out. Without identity I would be classifying every crop in every frame from scratch and living with the flicker.
Frame: real output from the project.
Events work the same way. The shot candidate event comes from the ball’s trajectory: when the ball’s velocity jumps between consecutive tracked positions, something just kicked it. That derivative only exists if the positions belong to the same ball, which is exactly what the second association pass protects. A kicked ball is fast and blurred, its detection score drops, and a classic tracker would cut the trajectory at the most interesting moment. I wrote about why the event is called a shot candidate and not a goal in a separate post.
Frame: real output from the project.
The output side is deliberately boring. Stable IDs plus events give me timestamps, timestamps give me clips, clips concatenate into a highlight reel. The full match also renders as an annotated video with boxes, IDs and team colours burned in, because the deliverable is not a tensor, it is a file a player opens on his phone on the way home. The rest of the system lives on the project page.
Practical notes from running it
The tracker is only as good as the detector under it. Garbage boxes in, garbage tracks out, and no tracker parameter will save you. Spend your effort on the detector first.
Then tune three things. The high/low confidence split decides what counts as “a bad frame” versus noise. The track buffer decides how long a lost track survives, and the right value depends on your occlusion lengths, not on a default. And the match threshold decides how far an object can move between frames, which on fast sports footage is further than you think. Ultralytics ships ByteTrack behind a YAML config, so the experiment loop is short. Use it.
One more note from the five-a-side system: evaluate on your own footage, not on benchmark numbers. When I changed the high/low confidence split, the difference never showed up as a tidy metric first. It showed up as whether the ball kept its ID through a shot. Build a debug view that draws boxes and IDs onto the video before you tune anything. Five minutes of annotated footage tells you more than any table.