Our thermal model's evaluation was wrong twice before it told us the truth
A model that failed, then failed differently, then turned out fine
We are building a self-supervised thermal perception backbone (a masked-autoencoder ViT trained on infrared frames, then probed with a frozen linear classifier to check whether it learned anything useful). Twice, our own evaluation told us the training had gone wrong. Both times it was lying. The model was fine. The test was not.
This is a story about why that happened, and a case for treating "does this evaluation actually work" as its own engineering problem, not an afterthought you bolt on once the model exists.
The first probe barely had room to say anything
Public thermal datasets are small relative to what a foundation-scale vision backbone wants. Our own pretraining recipe sets a floor of 200,000 frames before we trust a model's learned features at all. The two public real-thermal datasets available to us without a login wall, a public thermal-and-visible pedestrian dataset and a public thermal driving-scene dataset, do not clear that floor even combined.
So instead of a downstream benchmark, we built a frozen linear probe: freeze the trained encoder, run a closed-form ridge regression on top of its features, and see whether it can predict something simple, in this case whether an infrared image from the pedestrian dataset shows a crowd (two or more people) or not. Compare that probe's accuracy against two baselines: always guessing the majority class, and running the identical probe on the identical images through an untrained, randomly initialized encoder of the same architecture.
On the pedestrian dataset's test set, the majority-class baseline was 0.6526. The random-init baseline was 0.6359. That left 1.67 points of headroom between "guess the common answer every time" and "an untrained network's random projection." A probe with that little room to move cannot reliably tell you whether pretraining taught the model anything real. Almost any reasonably-behaved feature extractor, trained well or trained badly, will land somewhere in that narrow band. We built this probe anyway, because it was the annotation we had. It went on to give us two confidently wrong answers.
Two wrong verdicts, in opposite directions
Verdict one. We trained a ViT-B backbone on a larger, multi-source corpus (102,254 frames pulled from several public thermal sources) and ran the pedestrian-dataset crowding probe against it. The trained backbone scored 0.4791. Random init scored 0.652. The trained model was 17.3 points worse than a network that had never seen a single training step. Read literally, that says pretraining actively destroyed useful structure in the representation.
Verdict two. We found the reason for verdict one (below), rebuilt the corpus, and retrained. Same probe, same pedestrian-dataset test set, new backbone. Trained accuracy: 0.6139. Random init: 0.6177. A gap of 0.4 points, statistically indistinguishable from noise. Read literally, that says the retrained model learned nothing at all.
Two different backbones, evaluated the same way, and the same test told us "training hurts" and then "training does nothing." Neither was true. Once we replaced the probe (see below), both backbones turned out to clearly beat their own random-init control. The problem was never fully in the model. It was in an evaluation with 1.67 points of real signal trying to detect effects of that size or smaller.
What was actually broken: a corpus that was not what we thought it was
Verdict one had a second, separate cause worth being honest about: the 102,254-frame "multi-source thermal" corpus was contaminated. One of its contributing sources was built with an unrestricted file glob, and a large share of that source's images turned out to be ordinary RGB or false-color photographs, not single-channel thermal captures. The pipeline never checked; it just resized, cast to the expected dtype, and fed everything into masked-autoencoder pretraining as if it were thermal data.
This is the part that should worry anyone who trusts a loss curve. The training loss on that contaminated run descended in a completely normal, healthy-looking shape, start to finish. Nothing about the optimization looked wrong. A model can learn a perfectly stable, well-converged representation of the wrong data, and nothing in the training log will tell you that happened.
The bug surfaced only because we ran the trained-versus-random-init comparison and got an answer that made no sense on its face: a trained network scoring worse than a network that had never trained at all. That is not a result you would expect from a model that simply learned something unhelpful. It is closer to a symptom of the model learning to solve a task that does not match the one it is being graded on, because the images it trained on were not the images the test assumes.
The fix was a channel-divergence filter: a true single-channel thermal JPEG has near-identical red, green, and blue channels once decoded, since it started life as one grayscale value. A colorized or genuinely RGB frame does not. Applying that filter to the contaminated source rejected 6,877 files as non-thermal, and the corpus was rebuilt around what was left, 47,706 frames. That is the backbone behind "verdict two" above, and it is also the multi-source backbone in the results table further down.
Replacing the probe instead of just rerunning it
Finding the contamination bug fixed the data. It did not fix the evaluation, which still had only 1.67 points of headroom and had already produced two wrong answers. So we built a different probe on a different dataset: a 4-way person-count bucket (0, 1, 2 to 3, 4 or more people) on thermal images from the driving-scene dataset, using real bounding-box person annotations rather than a filename heuristic. The majority-class baseline on that task is 0.3237, well under 50%, because there is real class variety to get wrong. That is a probe with actual room to detect something.
Before trusting it, we validated the new probe two ways, not one:
- Against known-informative features. Does a trained encoder actually separate from its own random-init control on this task, and by a margin that looks like a real effect rather than noise? Yes, on both backbones we had, and the margins point the same direction.
- Against a random-init control run every time, not as a one-off sanity check. Every result below is reported next to the identical probe run on an untrained encoder of the same architecture, on the same images, so a reader never has to take the trained number on faith.
The results, reported next to their own baselines
- Majority-class baseline (always guess the most common bucket): 0.3237 accuracy, 0.25 balanced accuracy, 0.1223 macro F1. This is the floor any real backbone has to clear.
- Pedestrian-dataset-only ViT-B, 15,488 frames: 0.6136 accuracy, 0.586 balanced accuracy, 0.5834 macro F1. Its own random-init control scored 0.5379, so the trained encoder beats it by 0.0758.
- Multi-source ViT-B, 47,706 frames, trained after the contamination fix: 0.5158 accuracy, 0.4887 balanced accuracy, 0.4881 macro F1. Its own random-init control scored 0.4311, a gap of 0.0847.
Both backbones clearly beat their own random-init control and the majority baseline. Both are learning real thermal structure. Neither corpus clears the 200,000-frame floor we set for trusting a backbone as production-ready, so neither is, and the production gate on this work stays closed.
The honest part: we cannot yet rank the two backbones
It would be easy to read that table as "the pedestrian-dataset-only backbone (0.6136) beats the multi-source one (0.5158), so smaller and cleaner wins." We are not making that claim, because the numbers do not support it cleanly.
Look at the random-init controls: 0.5379 for one backbone's evaluation pass and 0.4311 for the other, a gap of 0.107 between two runs of the same untrained architecture. That is not a property of either backbone. It is the kind of spread you get from separate random draws and separate feature-extraction passes. And once you look at each backbone's improvement over its own control, the two numbers are nearly identical: +0.0758 and +0.0847. The raw accuracy gap between the backbones (0.6136 versus 0.5158, about 0.098) is close to the same size as the gap between their untrained controls (0.107). We cannot separate "one backbone is meaningfully better" from "one backbone happened to draw a lower random-init baseline in this evaluation pass." Both readings are consistent with the data we have. So we are stating plainly: the ranking between these two backbones is not resolved, and anyone who tells you otherwise from these numbers is overreading them.
What we think a trustworthy probe actually requires
Generalizing past this one case, we think an evaluation is not ready to certify a model until it has three things, checked in this order:
- 01Real headroom. The gap between a majority-class baseline and a random-init baseline has to be large enough that the effect you are trying to measure could plausibly show up inside it. 1.67 points is not that. 28 points (0.3237 majority versus roughly 0.43 to 0.54 random-init on the new probe) is closer to enough.
- 02Validation against known-informative features, before the probe is trusted on the model you actually care about. If a probe cannot separate a trained encoder from a random one on a task with real signal, it will not reliably tell you anything about a model where the signal is smaller or the training is subtler.
- 03A same-config random-init control, reported alongside every number, every time. Not as a one-time sanity check that gets dropped from later runs. It is the only thing that caught both the degenerate-probe problem and the corpus contamination in this story, and a loss curve alone caught neither.
What this means if you are building on public thermal data
Public thermal datasets like the pedestrian and driving-scene sets used here are useful. They got us a self-supervised backbone that measurably beats random initialization and a majority-class guess on a real person-counting task, using only data anyone can download. That is a genuine, if modest, result, and we are publishing the honest ceiling on it rather than rounding it up: neither backbone clears our own pretraining floor, and we cannot yet say which of the two is better.
The larger lesson generalizes beyond thermal: production-grade perception for a fixed deployed sensor needs footage from that sensor, at that site, not a public research dataset collected on different hardware, under different licensing, for a different purpose. Public data is where you prove the training and evaluation machinery works end to end, and where you catch bugs like this one before they reach a customer's camera. It is not where the final model comes from.