Ask the model less
A narrower question, answered better
Referring multi-object tracking asks a model to follow a specific object through a video sequence given a natural language description of which one to follow, "the person in the red jacket," say, or "the truck turning left." The conventional way to build this with a large multimodal model is to have it generate text describing what it sees and match that against the reference. A paper accepted to ECCV 2026, "YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Verification" (arXiv 2609.02318, submitted 2 September 2026), throws that approach out and replaces it with something much narrower: instead of asking the model to describe what it sees, ask it a yes or no question about one candidate at a time, does this box match the reference, yes or no.
What YesTrack changes
The reformulation sounds almost too simple to matter, which is part of the point. A model producing free text has to be right about the content of the description and about its own formatting, and then a separate parsing step has to correctly extract a judgment from whatever prose came out. A model answering yes or no has one job. YesTrack builds on the smallest variant of Qwen3-VL, and the paper's code, released at github.com/ggbondrighthere24/YesTrack under the MIT licence, is evaluated on the Refer-KITTI and Refer-KITTI-V2 benchmarks.
Because yes/no answers form a stream of individual judgments rather than a single description, the authors add two lightweight mechanisms to keep those judgments consistent across time. The first, a Temporal Confidence Prior, carries a running sense of how confident the match has been across recent frames into the current judgment, rather than treating each frame as a fresh question with no memory of the last one. The second, Temporal Reference Propagation, carries the reference match itself forward across frames so that the object being tracked does not have to be re-identified from scratch at every step. The approach also generalises past referring tracking to plain multi-object tracking, released as a separate variant, YesTrack-MOT.
Why constrained beats free text
The reason this matters beyond one benchmark is what a yes or no answer gives you that a paragraph does not: a number you can act on. A binary or near-binary output has a score attached to it that you can threshold, log, and audit later. You can set the threshold higher when false positives are expensive and lower when missed detections are expensive, and you can point at the exact number that produced a given decision when someone asks why an alert fired.
Free text does not give you any of that for free. A generated description has to be parsed before it becomes a decision, and parsing is where the hard part quietly moves to, rather than where it gets solved. A sentence that hedges, or answers a slightly different question than the one asked, or buries the actual judgment in the middle of a paragraph, produces a parsing problem that looks like a formatting bug but is actually the model's underlying uncertainty showing up somewhere you cannot see it clearly. Constraining the output format does not make the underlying model smarter. It does force whatever confidence or uncertainty the model has into a shape you can measure, argue with, and improve on.
What DHI already does for the same reason
This is not a new idea to us, because we made the same bet in our own second-stage verifier. When DHI's optional VLM verifier is turned on as the final gate on an alert, it is not asked to describe the scene. It is asked a constrained question: does the evidence in this frame support the alert that fired, yes or no. The reasoning is the one above. A description of the scene would need a second step to turn into a decision, and that second step is exactly where ambiguity would hide. A yes or no answer, whatever its other limitations, produces something we can log, threshold and inspect directly.
The idea worth stealing: carrying judgment forward
What YesTrack has that our verifier does not is the Temporal Confidence Prior, the mechanism that carries a running judgment forward across frames rather than treating every frame as a cold start. Our verifier, as it stands, evaluates each alert on the evidence in front of it at that moment. It does not currently maintain a running confidence that persists from one frame to the next and informs the next judgment. Every alert gets asked fresh.
That is a real gap, and the paper makes a reasonable case for closing it. A situation rarely changes character from one frame to the next; a forklift that was near a person a second ago is still probably near a person now, and a verifier that has to rediscover that fact from nothing each time is throwing away information it already had. Carrying a confidence estimate forward, the way YesTrack's Temporal Confidence Prior does, is a plausible way to make a sequence of yes or no judgments more stable without asking a bigger, more expensive question at every single frame.
What we have not built
We have not implemented anything like Temporal Confidence Prior or Temporal Reference Propagation in DHI's verifier. This post is not an announcement that we have; it is a note that the paper describes a specific, well motivated mechanism that lines up with a limitation we already knew we had, and that we now have a concrete reference for what closing it might look like.
Asking the model less, one constrained question at a time, was the right instinct, and it is the same instinct behind the yes or no framing YesTrack uses for tracking. Asking it to remember its last answer before it answers the next one is the part we have not done yet. We would rather name that gap directly than describe our verifier as more capable than it currently is.