JengaBench

Evaluates visual-spatial reasoning and strategic decision-making of LLMs in a deterministic 3D Jenga environment.

Comment

Inspiration

We were inspired by how current AI systems often appear intelligent in language tasks while still struggling with intuitive understanding of the physical world. One example that stood out to us was a video where an AI was asked to predict the outcome of a simple balancing trick involving a pen and two hands. The AI relied purely on abstract physics assumptions and failed to recognize that a person could simply grip the pen with the remaining hand. That disconnect between theoretical reasoning and real-world intuition made us wonder how well modern models actually understand physical interaction and spatial structure.

That led us to JengaBench. Jenga felt like a natural fit because it requires observation, planning, stability analysis, and careful interaction with an unstable structure. It also creates situations where small physical differences completely change the outcome, which makes it a good benchmark for testing whether models can reason about cause and effect in a grounded environment.

We also chose Jenga because it is immediately understandable and visually interesting. Even without knowing the technical details, anyone can watch a tower collapse and understand whether the AI made a good decision.

What it does

JengaBench is a deterministic 3D physics benchmark built for the Mesocosm platform. The project evaluates how well AI models can reason about physical stability, spatial relationships, and long-term planning inside a simulated Jenga environment.

The AI interacts with the tower entirely through visual observations and constrained actions. It can reposition the camera to inspect the structure, apply directional pushes to blocks, and attempt to remove and replace blocks without collapsing the tower. The benchmark intentionally avoids giving the model hidden simulator state so that it must infer stability from what it can actually observe.

The environment exports replay traces for every episode, which allows games to be reconstructed in a custom 3D viewer. This makes it possible to inspect the model’s reasoning, camera movement, actions, and the resulting tower behavior step by step.

Alongside the benchmark environment, we also built a fully playable version of the game so people can directly interact with the same mechanics the AI uses.

How we built it

We built the backend in Python and the frontend using JavaScript, HTML, and CSS. The physics simulation runs in PyBullet, which handles rigid body dynamics, collision resolution, force application, and tower stability.

A major design goal was keeping the interaction space simple and interpretable for AI agents. Instead of allowing arbitrary physics controls, we constrained the action space into structured operations such as camera movement and directional pushes. This let us focus the benchmark on reasoning and decision making rather than low-level control complexity.

The backend handles:

deterministic tower generation, physics stepping, collapse detection, reward calculation, replay export, and Mesocosm environment integration.

The frontend was built as a replay-driven 3D viewer that reconstructs full episodes from exported traces. We used it to visualize tower states, AI actions, camera movement, and collapses over time.

A large amount of the work went into tuning the simulation itself. Small changes in friction, settling behavior, force ramps, and collision thresholds could completely change how the tower behaved. We iterated heavily to make the environment feel realistic while still remaining deterministic and reproducible.

We also spent a significant amount of time integrating with Mesocosm. That included designing the observation format, adapting the environment to the required endpoint contract, validating replay data, and ensuring multimodal observations worked correctly with the platform’s inference pipeline.

Challenges we ran into

The biggest technical challenge was getting the physics to behave consistently while still feeling believable. Jenga is extremely sensitive to tiny force differences, friction values, and contact resolution. Early versions either felt unrealistic or became unstable too easily. We spent a lot of time tuning pushes, settling logic, and collapse detection so the game remained playable while still punishing bad decisions.

Another major challenge was Mesocosm integration. We had to repeatedly adjust endpoints, observation formats, replay serialization, and action schemas to make sure the environment worked correctly with the platform’s benchmarking pipeline and multimodal model support.

Hosting was also more difficult than expected. Physics simulations consume a significant amount of memory and compute resources, especially when running deterministic replay exports and rendering observations at scale.

Surprisingly, the hardest overall part ended up being the UI and UX. We went through many different interface designs before landing on something that felt intuitive and easy to inspect. In particular, we were not fully satisfied with how the replay mode and benchmark tab turned out. They worked functionally, but we wished we had more time to polish the experience and make the interface feel more cohesive and refined.

Another challenge we ran into was managing time effectively during such a fast-paced event. As the deadline approached, we realized one of our major required features had barely been started, which forced us to rapidly implement and integrate it near the end of the hackathon. That experience taught us a lot about prioritization, MVP planning, and how important it is to identify core deliverables early when working under extreme time constraints.

Accomplishments that we're proud of

One thing we are especially proud of is that the project became more than just a benchmark. Building both the AI benchmark environment and a fully playable version made the project feel complete and accessible to people outside of the evaluation space.

We are also proud of the physics system itself. Getting a deterministic Jenga simulation to feel believable while still remaining stable and replayable required a large amount of iteration and tuning.

Another accomplishment was creating a replay-driven 3D viewer that lets people inspect how the AI thinks and interacts with the tower over time. Watching the model reposition the camera, probe blocks, and eventually succeed or collapse the tower makes the benchmark much easier to understand.

What we learned

The biggest thing we learned was how to collaborate effectively as a team. Most of us had primarily worked on solo projects before this, so coordinating development across GitHub, sharing responsibilities, and designing systems together was a completely different experience.

We also learned a lot about deterministic simulation design, multimodal AI evaluation, and how difficult it is to create environments that are both technically reliable and genuinely interesting for models to interact with.

Another important lesson was that designing good benchmarks is very different from simply building a game. Small implementation details can dramatically affect whether a benchmark measures meaningful reasoning or can be exploited through shortcuts.

We also learned how critical prioritization and MVP scoping are during time-restricted events. Near the end of the project, we found ourselves rushing to complete a major required feature that had not been fully implemented yet. That forced us to make fast development decisions under pressure and showed us the importance of identifying the most essential systems earlier in the process.

What's next for JengaBench

There are several directions we would like to explore next.

One major extension would be adapting the benchmark toward robotics and embodied interaction. The current benchmark focuses on reasoning through constrained actions, but the same environment could eventually support robotic manipulation tasks and more advanced physical planning.

We are also interested in using the environment not just for evaluation, but for spatial reasoning training and curriculum generation for future multimodal models.

Beyond benchmarking, the physics and replay systems could support entirely new puzzle environments and physics-based games built on the same infrastructure.

We would also like to expand the customization system so users can fully modify the playable environment, including tower generation settings, camera behavior, physics presets, and replay visualization options.

Built With

Share this project:

Updates