Google DeepMind unveils SIMA, an AI agent that plays games
Google DeepMind has introduced SIMA (Scalable Instructable Multiworld Agent), a new AI model designed to act as a collaborative partner in 3D video game environments. Unlike traditional gaming AI, which is often hard-coded to compete against humans or perform specific, limited tasks, SIMA is built to interpret verbal instructions and behave more like a human player.
The agent does not interact with a game’s internal code or rulesets. Instead, it perceives the game world solely through the visual stream of pixels on the screen, mirroring how a human player processes information. By analyzing thousands of hours of recorded human gameplay, the model learns to associate visual patterns with specific actions, such as navigating toward a door or manipulating objects.
Tim Harley, a lead researcher on the project, emphasized the shift in design philosophy: “Rather than having a superhuman agent you play against, you can have SIMA players beside you that are cooperative, that you can give instructions to.”
To achieve this, the team focused on several key training methods:
- Imitation Learning: The model was trained on video data of human players, combined with annotations from data labelers that link visual actions to text-based commands.
- Instruction Following: Researchers recorded interactions where players instructed one another, allowing the AI to map natural language to in-game primitives.
- Generalization: By training across a variety of titles, including Valheim and Goat Simulator 3, developers tested if the agent could perform in games it had never encountered before.
The research team noted that traditional reinforcement learning—where an agent receives a “reward” signal like a score or a win/loss outcome—is often impractical for open-ended commercial games. Because these games lack a unified reward structure for every possible player goal, the DeepMind team opted for imitation learning. This approach allows the agent to pursue abstract goals rather than being strictly limited to maximizing a numerical score.
While the AI shows promise in generalizing its skills to new environments, the researchers acknowledge that unique game mechanics can still pose challenges. The current version of SIMA relies on a library of “primitives”—basic actions like moving, interacting, or building—that it can combine to execute more complex tasks requested by the user.