GameGAN

All videos and GIFs are taken from people playing with GameGAN.

Abstract

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.

Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba,
Sanja Fidler
Learning to Simulate Dynamic Environments with GameGAN
CVPR, 2020.
[Preprint] [Code] [Bibtex] [Video]

Overview

We are interested in training a game simulator that can model both deterministic and stochastic nature of the environment. GameGAN is composed of three modules. 1) The dynamics engine maintains an internal state variable which is recurrently updated. 2) For environments that require long-term consistency, an external memory module is used to remember what the model has generated so far 3) Finally, the rendering engine is used to decode the output image at each time instance. All modules are neural networks and trained end-to-end.

GameGAN trained on the official version of
Pac-Man from Bandai-Namco entertainment

GameGAN learns to follow the user command - how Pacman moves - as well as other rules of the game. It turns ghosts to purple when a capsule is consumed, and also learns their policy. You can see that ghosts chase Pacman in the normal state, but they start to run away when a capsule is consumed.

Credits:
PAC-MAN™&©BANDAI NAMCO Entertainment Inc.

GameGAN trained on a custom version of Pac-Man and VizDoom

Credits:
1. We modified the version of Pacman from http://ai.berkeley.edu/project_overview.html to create random mazes for training data.
2. We used https://github.com/hardmaru/WorldModelsExperiments to extract training data for VizDoom

Left: Final output (Static + Dynamic), Middle: Static component, Right: Dynamic component

GameGAN trained with
memory module & disentangling rendering engine

GameGAN learns to disentangle static components such as background and blue wall from dynamic components such as the firball-shooting enemies. In the Pac-man video, observe how it remembers the layouts (blue walls) that it has generated. It learns to recover the generated layout when Pac-man comes back to the same location.

Swapping foreground / background

With disentangled components, we can swap out the static/dynamic components with anything we like! We can play Pac-man with mario or play VizDoom in your favorite scenes rather than staying in the depressing room from VizDoom.