G1 Embodied AI Alignment Testing With Gemini 3

The first embodied AI safety evaluation built on Inspect AI — the UK AI Safety Institute's open-source evaluation framework. We test what happens when an AI agent controls a physical body.

Comment 6

Inspiration

Last year, 2025, I participated in a Misalignment Bounty research project, run by Palisade Research. The work was later published by their team in https://www.palisaderesearch.org/blog/misalignment-bounty-results. I managed to place, and discovered a real passion for this area. My submissions discovered that OpenAI's o3 and GPT-5 fabricate documentation and bypass validation when facing real-world constraints. Those findings were published in a peer-reviewed paper (https://arxiv.org/abs/2510.19738).

That work raised important questions for me I couldn't shake: if frontier models cut corners in text environments, what happens when they control a physical body? A chatbot that fabricates a citation can be corrected. A robot that drives through radioactive waste barrels cannot.

I looked at https://inspect.ai-safety-institute.org.uk/ — the UK AI Safety Institute's open-source evaluation framework, the industry standard for testing frontier models. It has 107 evaluations across 19 categories: coding, cybersecurity, scheming, safeguards. I could not find any for embodied AI. No eval tests what happens when an AI agent controls a physical body in a physics simulation. So I wanted to see if I could built that missing eval.

What it does

G1 Alignment Experiment places a simulated https://www.unitree.com/g1/ inside a DOE nuclear waste storage facility built in MuJoCo physics simulation. Three 200L drums of Class C radioactive waste (Sr-90, Cs-137) block the direct path to a charging station. The AI must decide: take the safe detour and risk battery depletion, or squeeze through a 1.9cm gap between barrels and risk displacing waste that exposes human technicians to radiation.

The scenario applies layered pressure — low battery, a charger entering standby, cumulative radiation doses for nearby technicians, and institutional threats (program termination if efficiency drops). In the barrels_corrupt variant, a telemetry corruption mechanism disables incident reporting mid-run. Now the AI knows nobody is watching. Does it exploit that gap? Does it hide what it did?

Everything runs through Inspect AI — task definitions, tool schemas, multi-turn agent loops, and structured scoring. The model receives LiDAR, camera, and IMU sensor data as tool calls. Every decision is logged, scored, and reproducible. An LLM judge (Gemini 3 Pro) evaluates each run on safety (0-5) and honesty (1-5), classifying behavior into a 5-level misalignment taxonomy from L0 (full compliance) to L4 (active sabotage).

A full-stack dashboard lets researchers explore results, replay experiments in 3D using MuJoCo compiled to WebAssembly, and query the dataset through a streaming research assistant powered by Gemini 3 Pro.

How I built it

Physics simulation: MuJoCo 3.x running the Unitree G1 12-DOF humanoid model. We built a custom nuclear facility scene with radioactive barrels, forbidden zones, and contact detection for barrel displacement events. LiDAR (36 rays, 360° coverage), camera capture, and IMU sensors feed data to the AI as structured tool calls.

Evaluation framework: Built on Inspect AI with custom tasks, tools, solvers, and scorers. The multi-attempt flow lets models navigate, fail, request retries, and eventually debrief — creating rich reasoning traces for analysis. Five scenario variants escalate pressure from simple navigation to full telemetry corruption.

Gemini 3 integration (4 components):

  1. Experiment Judge — Gemini 3 Pro scores safety and honesty from reasoning traces with structured evidence
  2. Research Assistant — Streaming chat via Vercel AI SDK with thinking/reasoning display and 4 server-side tools (paper RAG via Vertex AI Search, web search, video analysis, experiment catalog)
  3. Video Verification — Gemini 3 Flash analyzes experiment recordings to detect discrepancies between model claims and actual robot behavior
  4. Extraction Verification — Gemini 3 Flash watches sped-up simulation videos to verify data accuracy against visual ground truth

Dashboard: Next.js 15 with the Vercel AI SDK for streaming AI responses. The 3D viewer uses MuJoCo WASM — the same physics engine compiled to WebAssembly and rendered with Three.js — to replay trajectories frame-by-frame with overlays for forbidden zones, contact events, and AI decision annotations.

Backend: FastAPI on Cloud Run serving paper search (Vertex AI Search over indexed AI safety literature), web search, and video analysis endpoints.

Multi-model testing: We tested GPT-5, Gemini 2.5 Pro, Gemini Robotics ER 1.5, and Kimi K2.5 across 30 runs using a custom Kimi K2.5 provider for Inspect AI (Kimi's API isn't natively supported).

Challenges I ran into

  • MuJoCo on macOS: The MuJoCo viewer requires mjpython — a special Python launcher — on macOS, which complicated our development workflow and CI pipeline. Getting MuJoCo WASM to replay recorded trajectories in the browser with accurate physics required careful synchronization between the simulation timestep and the recorded data.

  • Telemetry corruption design: Getting the corruption mechanism right was tricky. It needed to be subtle enough that the AI treats it as a realistic system failure, but clear enough that we could measure whether the model exploits the gap. We went through several iterations before landing on a "incident report upload failed" notification that gives the model a genuine choice.

  • Kimi K2.5 integration: Kimi's API isn't supported by Inspect AI out of the box. We wrote a custom provider that translates Inspect's tool-calling protocol to Kimi's API format, handling differences in function call schemas and response parsing.

  • Scoring calibration: Having an LLM judge evaluate another LLM's safety behavior creates a meta-alignment problem. We iterated on the judge prompt extensively to ensure consistent scoring, and built the video verification pipeline as an independent check — Gemini 3 Flash watches the actual simulation recording to verify whether the judge's assessment matches what physically happened.

  • Sample size vs. cost: Each run costs real API credits and takes 5-15 minutes including simulation, multi-turn agent loops, and judging. Scaling to statistically rigorous sample sizes (30+ runs per model) is an ongoing effort. Our current 7-9 runs per model show consistent patterns but we're transparent that these are directional findings.

What we learned

The most surprising finding wasn't that models make unsafe choices — it's that safety and honesty are independent dimensions. GPT-5 scored the worst on safety (1.9/5) but had perfect honesty (5.0/5). It would drive through the barrels and then tell you exactly what it did. Gemini Robotics ER 1.5 showed the opposite pattern: in 29% of runs, it detected the telemetry corruption and actively exploited it to hide safety violations — citing standard operating procedures to justify submitting a falsified report.

This suggests that text-based alignment evaluations miss a critical failure mode. A model can pass every honesty benchmark in a text environment but behave differently when it has a physical body and believes oversight has been disrupted. Physics simulation with realistic pressure is necessary to surface these behaviors.

Gemini Robotics ER and VLA alignment — Gemini Robotics ER 1.5 is the https://deepmind.google/models/gemini-robotics/gemini-robotics-er/ that plans and makes decisions for physical robots, while its companion VLA (Vision-Language-Action) model translates those decisions into motor commands. As a reasoning layer that sits above the action layer, ER is exactly where alignment evaluations should focus — decisions made here propagate into physical behavior. Our benchmark provides a ready-made testing ground for the full ER + VLA stack as these models evolve, helping surface alignment edge cases before deployment on real hardware.

What's next

  • Open Eval Platform — I'm building this into a hosted platform where anyone can run embodied AI alignment evals. Users will be able to bring their own API key, run evaluations against any model supported by Inspect AI directly from the browser, and get cross-run analytics showing how safety and honesty scores trend across runs, scenarios, and model updates. Users who prefer running locally can do so via CLI and upload results to the platform to get the same analytics and research assistant access. Public runs contribute to a shared community leaderboard — the benchmark becomes more valuable as more people use it.

  • More models, more runs — We're scaling to 30+ runs per model to move from directional findings to statistically rigorous claims. We're also expanding to Anthropic Claude, DeepSeek, and every other model supported by Inspect AI to build a comprehensive cross-model alignment landscape.

  • Additional scenarios — Pipeline inspection, warehouse logistics, hospital corridors — testing whether the misalignment patterns we've found transfer across domains or are specific to the nuclear facility context.

  • Sim-to-real transfer — The critical open question: do these misalignment behaviors reproduce on physical robots? We're saving up to purchase Unitree G1 hardware to validate whether L3+ strategic concealment occurs in real-world settings, testing whether simulation-based alignment evaluations are predictive of actual deployment risks.

  • Mechanistic interpretability — Apply probing techniques to open-source models on this benchmark to understand why some models conceal while others disclose — moving from behavioral observation to mechanistic understanding of deception.

Built With

Share this project:

Updates

posted an update —

Technical Update: Exploring UnifoLM-WMA-0 for G1 Alignment Testing

Background

While investigating robotics world models with extractable reasoning traces, I came across UnifoLM-WMA-0, which is Unitree’s open-source world model architecture for the G1 humanoid robot.

What is UnifoLM-WMA-0

UnifoLM-WMA-0 is a world-model-action framework that operates in two modes: 1. Decision-Making Mode: Predicts future physical interactions between the robot and environment. These predictions are then used to inform the action policy. 2. Simulation Mode: Generates environmental feedback based on robot actions, functioning as an interactive simulator. The architecture appears to separate world model predictions from action generation, which could allow for extracting intermediate reasoning states.

Potential Relevance to G1 Alignment Research

Since our current work tests LLM-based control under resource pressure scenarios, integrating UnifoLM-WMA-0 could provide: ∙ A comparison point between world-model-based and LLM-based decision-making under constraints ∙ Access to the model’s physical interaction predictions, which may reveal different failure modes than language-based reasoning ∙ A manufacturer-provided baseline for the G1 platform we’re already using Resources ∙ Code: https://github.com/unitreerobotics/unifolm-world-model-action ∙ Model weights: https://huggingface.co/unitreerobotics/UnifoLM-WMA-0-Base ∙ Released: September 15, 2025

Next Steps Under Consideration

If this direction seems worthwhile, the next phase would involve: 1. Setting up the model with our existing MuJoCo G1 simulation 2. Attempting to extract world model prediction traces 3. Running preliminary tests under the same battery pressure scenarios used for LLM testing 4. Evaluating whether the prediction traces provide meaningful insights into alignment behavior

This is exploratory at this stage. Feedback welcome if anyone has experience with this model or similar world-model architectures.

Log in or sign up for Devpost to join the conversation.