Super Mario Bros. Becomes The Latest AI Performance Benchmark

Researchers at the University of California San Diego’s Hao AI Lab have introduced a demanding new stress test for artificial intelligence: the fast-paced world of Super Mario Bros. While gaming has long served as a sandbox for machine learning, this experiment highlights a growing divide between how models handle static logic and real-time execution.

The study utilized a custom-built framework dubbed GamingAgent, which integrates with an emulator to grant AI models control over the iconic platformer. By feeding the systems basic tactical instructions alongside in-game screenshots, the researchers tasked the models with generating Python code to navigate Mario through obstacles and enemies.

Performance Discrepancies in Real-Time

The results showcased a surprising hierarchy among current industry-leading models. Anthropic’s Claude 3.7 emerged as the top performer, followed by its predecessor, Claude 3.5. Conversely, models often cited as state-of-the-art—specifically OpenAI’s GPT-4o and Google’s Gemini 1.5 Pro—struggled to maintain consistent gameplay.

Perhaps the most counterintuitive finding involved “reasoning” models like OpenAI’s o1. Despite their reputation for superior step-by-step problem-solving, these models performed worse than their non-reasoning counterparts. The researchers point to a fundamental issue: latency.

• The Timing Factor: In Super Mario Bros., a delay of even one second can result in a failed jump or a character’s death.
• Processing Speed: Reasoning models typically require several seconds to deliberate on actions, making them ill-suited for the immediate, twitch-reflex demands of the game.

The Industry’s Growing Evaluation Crisis

The use of video games as a metric for AI intelligence remains a subject of debate. Critics argue that games provide an abstract, simplified environment that does not accurately reflect the complexities of the real world. This trend arrives amid what Andrej Karpathy, a founding member of OpenAI, recently characterized as an “evaluation crisis.”

In a post on X, Karpathy expressed uncertainty regarding current benchmarking standards, stating, “I don’t really know how good these models are right now.” As researchers continue to push models into new domains like gaming, the industry faces an ongoing struggle to define which metrics truly correlate with meaningful technological advancement.