Randomize Everything. Just Don't Mix the Domains.
Domain randomization is supposed to stop a model from memorizing its surroundings. Our thermal-imaging experiment showed us exactly where that idea stops working.
Synthetic data comes with a familiar rule: if you don't want the model to memorize something, randomize it. Change the lighting. Change the camera angle. Change the texture. Swap the background, again and again. Do it enough, and the model should stop caring about the scenery and start learning the object itself.
That's the theory. Then we tried it with thermal imaging — and the background still mattered.
We thought more variety would win
We were building an object detector for thermal imagery, where real data is expensive, limited, and hard to collect at scale. Synthetic data was the obvious way to expand the training set — which left one question: what should go behind the synthetic objects?
We tested two options. The first was real thermal backgrounds — actual infrared scenes from environments like the one the model would operate in. The second was grayscale RGB backgrounds — ordinary photographs converted to grayscale, easy to source, visually diverse, and close enough to thermal at first glance.
If domain randomization worked exactly the way we hoped, the second option had everything going for it: more scenes, more variation, more chances for the model to stop leaning on the background. The result went the other way. Real thermal backgrounds beat grayscale RGB — the bigger, more varied library lost. That was the interesting part.
Grayscale isn't thermal
A grayscale photo can look surprisingly like a thermal image — to a human. To the sensor, they come from completely different worlds. RGB cameras measure reflected visible light; thermal cameras measure infrared radiation tied to temperature and material properties such as emissivity. The resulting edges, textures, intensity relationships, and noise are fundamentally different — which is exactly why synthesizing convincing thermal imagery is a hard, physics-constrained problem in its own right ¹. Converting RGB to grayscale removes color. It does not turn reflected light into emitted heat.
That matters because the model never looks at a background and decides that's irrelevant, I'll ignore it. It sees pixels, and pixels have statistics. Image classifiers have been shown to pull real predictive signal from backgrounds alone — the background isn't empty space around the object, it's part of the distribution the model learns ². So our grayscale backgrounds weren't neutral. They were teaching the model something — just not the world it would eventually see.
More variation can't fix the wrong distribution
This changed how we think about domain randomization. The usual advice is randomize what you don't want the model to rely on. We'd add one line: randomize it inside the world where the model will actually operate.
There's plenty we should vary in thermal imagery — environments, temperatures, times of day, seasons, viewpoints, other objects in the scene — and that variation genuinely makes the model more robust. But swapping thermal backgrounds for grayscale RGB doesn't just add variation; it changes the domain. Once that happens, more data isn't necessarily helping — you're only showing the model more examples of a world its sensor will never meet. Structured domain-randomization research has reached a similar conclusion: realistic scene context can outperform unconstrained randomization, because the context itself carries information ³. The goal isn't maximum randomness. It's useful randomness.
A simple test before you randomize
Before building a giant background library, there's a much easier question to ask: could this scene actually appear in front of the sensor?
Different streets, different factories, different weather, terrain, lighting, and camera positions — yes, randomize all of it. But if the honest answer is no, adding more examples may just widen the synthetic-to-real gap. For thermal specifically, that gap is still an active research problem; recent work focuses precisely on making synthetic thermal imagery behave more like real sensor data ⁴. Visual similarity isn't the same thing as domain similarity.
The background isn't background
That was the bigger lesson. We started from a familiar assumption — randomize the background until the model learns to ignore it. What we found was slightly different: the model never really ignores the background. It learns whatever world you put around the object. So that world has to be the right one.
Synthetic data gives us enormous freedom to create variation — one of its biggest advantages. But freedom still needs boundaries. Randomize everything you want. Just make sure it still belongs to the world your model will actually see.
References
[1] Upadhyaya, Sharma, Mukherjee, Singhal & Lall (2024). A Comprehensive Survey on Synthetic Infrared Image Synthesis. arXiv:2408.06868.
[2] Xiao, Engstrom, Ilyas & Madry (2021). Noise or Signal: The Role of Image Backgrounds in Object Recognition. ICLR. arXiv:2006.09994.
[3] Prakash et al. (2019). Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data. ICRA. arXiv:1810.10093.
[4] Mayr et al. (2024). Narrowing the Synthetic-to-Real Gap for Thermal Infrared Semantic Image Segmentation Using Diffusion-Based Conditional Image Synthesis. CVPR Workshops (CVPRW).
