Microsoft on Wednesday introduced Muse, its first generative AI model for video game creation. Developed by the Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams with Xbox Games Studios’ Ninja Theory, Muse introduces the World and Human Action Model (WHAM).
Muse – Generative AI Model for Gameplay
This AI, detailed in a Nature paper by Katja Hofmann, Senior Principal Research Manager and lead of the Game Intelligence team, can generate game visuals and controller actions based on a 3D understanding of game worlds.
Muse, using WHAM-1.6B with 1.6 billion parameters, was trained on over 1 billion images and controller actions—equivalent to more than 7 years of continuous human gameplay—from Ninja Theory’s Bleeding Edge.
Collaboration and Open-Source Access
Muse emerged from a partnership between Ninja Theory and Microsoft Research, both based in Cambridge, UK. Trained on Bleeding Edge, a 4v4 online game from 2020, the model uses ethically collected gameplay data (visuals and controller actions at 300×180 px resolution) with player consent via the End User License Agreement.
Katja Hofmann noted, “I’m incredibly proud of our teams and the milestone we have achieved,” highlighting Muse’s ability to learn rich game structures. To support further research, Microsoft is open-sourcing Muse’s weights, sample data, and the WHAM Demonstrator—a visual interface for interacting with WHAM models—available on Azure AI Foundry.
How Muse Works and What It Can Do
Muse excels in generating complex gameplay sequences lasting up to 2 minutes. In demos, it’s prompted with 10 frames (1 second) of human gameplay and controller actions, then predicts game evolution in “world model mode.”
The more its output matches Bleeding Edge dynamics, the better it captures the game. Examples show behavioral diversity (e.g., camera shifts, path choices) and visual diversity (e.g., different hoverboards).
Evaluated for consistency, diversity, and persistency, Muse ensures gameplay respects physics, offers variety, and adapts user changes—like adding a character into a scene, which it then integrates naturally.
The Research Journey Behind Muse
The project began in December 2022, sparked by ChatGPT’s release, which inspired Hofmann to explore transformer-based models for gaming. With years of Bleeding Edge data from Ninja Theory, the team scaled training from a V100 cluster (100 GPUs) to H100s.
Early demos, like one by researcher Tim Pearce, showed Muse improving over 10k, 100k, and 1M training updates—progressing from basic recognition to mastering mechanics like flying after 1M updates.
Gavin Costello, Ninja Theory’s technical director, said, “It’s been eye-opening to see the potential this technology has,” from AI agents to new gameplay sequences.
Muse’s Potential Impact on Gaming
Muse pushes AI-driven game design forward. Fatima Kardar, Corporate Vice President of Gaming AI, said, “We are already using Muse to develop a real-time playable AI model trained on other first-party games.”
It could revive classic games tied to old hardware, optimizing them for any device. Kardar added, “This could radically change how we preserve and experience classic games.” Developers might also use Muse to prototype ideas or expand existing titles, with short AI game demos soon available on Copilot Labs.
Shaping Muse with Creators in Mind
Microsoft interviewed 27 game creators globally—from indies to AAA studios—to align Muse with real needs. Linda Wen, a design researcher, emphasized including underrepresented voices to ensure broad benefits.
During a hackathon, teams built the WHAM Demonstrator, letting users prompt Muse with visuals (e.g., a Bleeding Edge promo image) and tweak outcomes using controllers.
Cecily Morrison, Senior Principal Research Manager, noted, “It was a great opportunity to shape model capabilities to suit the needs of creatives right from the start.”
Xbox’s AI Vision and Future Steps
Xbox has long used AI, and Kardar said, “We’re now seeing an acceleration of generative AI research.” Muse reflects a creator-first approach, with usage guided by each team’s vision.
Looking ahead, Xbox aims to reduce barriers in gaming via AI. Kardar explained, “We’ll share our AI product innovation earlier on,” inviting players and creators to co-build features.
Guided by principles like fairness and transparency, Xbox plans more AI updates in 2025 to enhance tools and experiences across devices.
Availability and Access
Muse is available now for researchers and developers. Its model weights, sample data, and WHAM Demonstrator can be accessed on Azure AI Foundry. Short interactive AI game demos will soon be testable on Copilot Labs, with broader updates expected in 2025.