1. [Publications](/publications) 2. GRS: Generating robotic simulation tasks from real-world images # GRS: Generating robotic simulation tasks from real-world images ![](/sites/default/files/styles/wide/public/publications/flappy_designer3.jpg?itok=qWo4K1uk) Game design hinges on understanding how static rules and content translate into dynamic player behavior---something modern generative systems that inspect only a game's code or assets struggle to capture. We present an automated design iteration framework that closes this gap by pairing a reinforcement learning (RL) agent, which playtests the game, with a large multimodal model (LMM), which revises the game based on what the agent does. In each loop the RL player completes several episodes, producing (i) numerical play metrics and/or (ii) a compact image strip summarising recent video frames. The LMM designer receives a gameplay goal and the current game configuration, analyses the play traces, and edits the configuration to steer future behaviour toward the goal. We demonstrate results that LMMs can reason over behavioral traces supplied by RL agents to iteratively refine game mechanics, pointing toward practical, scalable tools for AI-assisted game design. ## Authors [Alex Zook](/person/alex-zook) [Josef Spjut](/person/josef-spjut) [Jonathan Tremblay](/person/jonathan-tremblay) ## Publication Date Wednesday, June 11, 2025 ## Published in [CVPR 2025](https://cvpr.thecvf.com/Conferences/2025) ## Research Area [Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence) [Computer Vision](/research-area/computer-vision) [Physical AI](/research-area/physical-ai) ## Uploaded Files [Fly\_\_Fail\_\_Fix.pdf](https://d1qx31qr3h6wln.cloudfront.net/publications/Fly__Fail__Fix.pdf "Open file in new window")4.91 MB ## Copyright This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to .