1. [Publications](/publications)
2. Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image
 
 # Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Image

  ![](/sites/default/files/styles/wide/public/publications/WACV2026.png?itok=IqRWafnD)

 Text-to-image diffusion models can generate stunning visuals, yet they often fail at tasks children find trivial - like placing a dog to the right of a teddy bear rather than to the left. When combinations get more unusual - a giraffe above an airplane—these failures become even more pronounced. Existing methods attempt to fix these spatial reasoning failures through model fine-tuning or test-time optimization with handcrafted losses that are suboptimal. Rather than imposing our assumptions about spatial encoding, we propose learning these objectives directly from the model’s internal representations. We introduce Learn-to-Steer, a novel framework that learns data-driven objectives for test-time optimization rather than handcrafting them. Our key insight is to train a lightweight classifier that decodes spatial relationships from the diffusion model’s cross-attention maps, then deploy this classifier as a learned loss function during inference. Training such classifiers poses a surprising challenge: they can take shortcuts by detecting linguistic traces
rather than learning true spatial patterns. We solve this with a dual-inversion strategy that enforces geometric understanding. Our method dramatically improves spatial accuracy: from 0.20 to 0.61 on FLUX.1-dev and from 0.07 to 0.54 on SD2.1 across standard benchmarks. Moreover, our approach generalizes to multiple relations and significantly improves accuracy.



 ## Authors



Sapir Yflah (Bar-Ilan University)

[Yuval Atzmon](/person/yuval-atzmon)

[Gal Chechik](/person/gal-chechik)

 

 

 ## Publication Date



Sunday, November 9, 2025

 

 ## Published in



[WACV 2026](https://arxiv.org/pdf/2509.02295)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence)

[Computer Graphics](/research-area/computer-graphics)

 

 

 ## External Links



[Project Page](https://learn-to-steer-paper.github.io/)