When your verifier stops looking

A gap between seeing and using what you saw

A vision-language model has two halves: an encoder that turns pixels into a representation, and a language model that turns that representation into words. The assumption behind using one of these systems as a visual verifier is that the second half is actually reading what the first half produced. A paper posted to arXiv on 1 September 2026, "The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence" (arXiv 2609.00868), tests that assumption directly, and the result is uncomfortable for anyone who has wired a VLM into a decision pipeline.

What the paper measured

The method is simple to describe. Take an image, find the region relevant to the question being asked about it, and blur just that region. Everything else in the frame stays sharp. Then ask: does the model's answer change?

Across six vision-language models and three perceptual benchmarks, the answer was no, on a striking share of samples. Blurring the question-relevant region "leaves the next-token distribution nearly unchanged" on 40 to 97 percent of samples, depending on model and benchmark. The model produces essentially the same output whether or not it can actually see the thing the question is about.

The more interesting finding is what happens inside the model while that is going on. The authors trained a linear probe on the vision encoder's own internal representations, and that probe can tell a blurred image from a clean one at 0.72 to 0.79 accuracy. The encoder registers the change. The information that the relevant region has been degraded is present in the network, encoded and available. It just does not reach the output: the language model's answer changes in only 2 to 11 percent of cases where the encoder clearly noticed something different. The authors call this an encoder to LLM disconnect, and they measure it above 0.65 on every single model they tested. The eyes see it. The mouth does not say anything different.

The paper's proposed fix is a Visual Sensitivity Index, a way of scoring how much a given answer actually depended on the image rather than on the question text alone. It performs well, reaching an AUROC of 0.85 to 0.87, on multiple-choice reasoning tasks. On open factuality tasks, where the model has to state a fact rather than pick between offered options, the index is markedly weaker. That split is worth holding onto, and we come back to it below.

Agreement is not verification

Here is why this matters for anyone using a VLM as a check on something else, which is exactly what a second-stage alert verifier is.

The point of putting a language model in front of an alert is to ask it, in effect, does the evidence in this frame actually support the claim being made. If the model's answer is driven mostly by what a plausible answer to a question like this usually looks like, rather than by the specific pixels in front of it, then it is not verifying anything. It is agreeing. A verifier that agrees with whatever framing it is given is worse than no verifier, because it adds the appearance of a check without the substance of one.

This is not a claim that VLMs are useless as verifiers. It is a claim that verification has to be demonstrated, not assumed, because the failure mode the paper documents is invisible from the outside. A model exhibiting the encoder to LLM disconnect still produces confident, well formed, plausible sounding answers. Nothing about its output format tells you whether it looked.

Where DHI's verifier sits, and what we have not measured

DHI runs an optional second-stage VLM verifier as the final gate on alerts. It ships off by default: a customer has to turn it on. It is fail-safe by design, meaning that when the verifier cannot reach a judgment, for example because of a timeout or a malformed response, the alert is kept rather than dropped. The system is built to err towards showing you something rather than silently swallowing it.

We do not currently have a published measurement of our own verifier's visual sensitivity. We have not run the blur and compare test the paper describes against our own pipeline, and until we do, we do not know where our verifier sits on the 40 to 97 percent range the paper reports across other models. Saying otherwise before we have measured it would be exactly the kind of unearned confidence this paper is warning against.

Why the multiple-choice framing is mild reassurance, not evidence

There is one structural reason for cautious optimism, and we want to be precise about how much weight it can carry. The verifier's actual job is closer to a constrained, effectively multiple-choice question: is this really a forklift near a person, yes or no, given the frame and the alert that triggered it. That is a narrower task than the open factuality questions where the paper found the Visual Sensitivity Index weakest. The paper's own results show its detection method working better on multiple-choice-shaped reasoning than on open-ended factual claims.

That is a reason to expect our situation is not the worst case in the paper. It is not a reason to believe it is fine. A yes or no question can still be answered from priors: "alerts like this are usually correct" or "forklifts near people are common on this camera" are both language-prior shortcuts that would produce a confident answer without the model actually re-examining the frame. The paper measured this exact behaviour in constrained-choice settings, including some that were multiple-choice, and still found meaningful insensitivity to the image. Being in a friendlier category of task lowers the odds of the failure mode. It does not retire the question.

What we would need to actually know

The honest position is this: we have a verifier we believe is a useful check, built to fail safe when it is uncertain, and we do not yet have the measurement that would tell us how much of its judgment is coming from the frame in front of it versus from what a plausible answer usually sounds like. That measurement is now a known, published method, applicable to our own pipeline. Until we run it, any claim about how well our verifier is actually verifying is a hope, not a result, and we would rather say that than borrow confidence from a paper about other people's models.

For a fail-safe design, the practical consequence of this failure mode is skewed in one direction. If a verifier is agreeing rather than checking, it will tend to keep alerts it should have cleared, because keeping is the default when judgment is uncertain or absent. That produces more false alerts, not more missed ones. That is the less dangerous direction to be wrong in, and it is still wrong, and it still costs someone time reviewing footage that did not need reviewing. Fail-safe is a reasonable default. It is not a reason to skip measuring what the safety net is actually catching.