ProtocolRun-VR

An autonomous AI agent that runs VR studies while researchers step away—guiding participants, diagnosing failures from live telemetry, recovering sessions, and producing evidence-based reports.

Comment

Inspiration

A technical failure during a VR study can silently invalidate task timing, participant responses, and interaction records. However, allowing a generative model to modify a Unity scene directly would create another risk: the model could select the wrong object, rely on stale evidence, change protected components, or report success without a real physical retest.

ProtocolRun-VR was inspired by this tension. The goal was not to build an unrestricted “AI fixes Unity” system, but a bounded agentic workflow in which Gemini can reason about a verified failure while deterministic software retains execution authority.

What it does

ProtocolRun-VR detects a specific hand-interaction failure during a Meta Quest study and coordinates a controlled recovery workflow.

In the recorded demonstration:

  1. Unity registers immutable healthy baselines for three study objects.
  2. CUBE_A is grabbed and released as the normal practice object.
  3. A protocol-controlled demo fault disables CUBE_B’s captured direct hand-grab paths.
  4. Three distinct near-target pinch attempts produce matching failure evidence.
  5. A Google ADK agent sends a compact, filtered evidence summary to Gemini 3.5 Flash.
  6. Gemini must call one bounded propose_recovery tool.
  7. A deterministic server firewall independently validates the target, evidence, baseline, protocol phase, expiry, and permitted action.
  8. The server issues pause, restoration, and retest commands.
  9. Unity validates and acknowledges each command before restoring only the captured CUBE_B baseline.
  10. Recovery is accepted only after a new SDK-observed CUBE_B hand grab.

Gemini cannot directly select the target, edit transforms, modify participant data, change protected CUBE_C, or declare recovery successful. If credentials are unavailable, evidence is insufficient, a command expires, or any guard fails, the workflow stops or moves to manual review instead of generating a simulated success.

How we built it

The verified demonstration path uses:

  • Meta Quest 3 hand tracking
  • Unity 6000.3.16f1
  • Meta XR All-in-One SDK 205.0.0
  • OpenXR 1.17.0
  • FastAPI
  • SQLite for local protocol state and audit records
  • Google Agent Development Kit
  • Gemini 3.5 Flash through a private Google AI Studio API key

Unity records protocol events and communicates with the FastAPI service using session-scoped authentication. The backend validates event ordering, maintains the recovery state machine, minimizes the fields sent to Gemini, applies the deterministic firewall, and issues expiring commands. Unity then applies its own local command guard before changing any component.

The Gemini prompt excludes raw hand positions, hand rotations, head pose, participant survey text, arbitrary logs, and credentials. Only decision-relevant identifiers and bounded status fields are included.

The repository also contains a Google Cloud deployment path using Cloud Run, Firestore Native, Secret Manager, Vertex AI configuration, Cloud Build, and scoped service accounts. For this submission, Cloud Run container deployment and startup were verified, and a Firestore Native database was provisioned in us-central1.

The recorded Quest recovery itself used the local FastAPI and SQLite path with a real Gemini API call. A public Cloud Run request, deployed Firestore application transaction, Vertex AI call from Cloud Run, and remote Quest-to-Cloud Run recovery cycle are not claimed as verified.

Challenges we ran into

The main challenge was separating model reasoning from execution authority. A model proposal alone is not sufficient evidence that a physical interaction failed or that a repair succeeded. We therefore implemented independent checks for object identity, immutable baselines, evidence freshness, distinct attempt IDs, protocol phase, command expiry, acknowledgements, and a new same-object physical retest.

Another challenge was preserving research integrity. Failure events and interrupted timing segments must remain auditable rather than being deleted after recovery. ProtocolRun-VR retains raw events, audit decisions, command acknowledgements, and verification results.

The Unity integration also required careful handling of captured Meta interaction components. The actuator restores only previously captured enabled states and refuses recovery when the component layout, Unity run identity, protocol step, target state, or tracking conditions do not match expectations.

During final repository verification, we also corrected an OpenXR version mismatch in the static verifier, restored required shell executable modes, and replaced a missing local build-plugin import with the pinned official @openai/sites-vite-plugin package.

Accomplishments that we are proud of

  • Demonstrated a real Quest 3 hand-interaction failure and recovery cycle
  • Used a real Google ADK tool call with Gemini 3.5 Flash
  • Prevented Gemini output from directly modifying Unity
  • Implemented fail-closed behavior without fake AI fallbacks
  • Required three distinct failure attempts before proposing recovery
  • Protected CUBE_C from restoration
  • Required a fresh CUBE_B physical grab before declaring success
  • Preserved raw events and audit records
  • Deployed and started the backend container on Google Cloud Run
  • Provisioned Firestore Native on Google Cloud
  • Completed a green CI repair-and-verification workflow covering Python tests, Unity static verification, TypeScript, the dashboard production build, dashboard tests, and the backend Docker image build

What we learned

We learned that trustworthy physical agents need more than a good model prompt. Model reasoning, policy validation, command execution, acknowledgements, and physical verification should be separate stages with independently enforceable contracts.

We also learned that data minimization improves both privacy and reliability. Sending Gemini a compact evidence whitelist makes the decision boundary easier to inspect and reduces the chance that irrelevant telemetry or untrusted text affects the proposal.

Finally, cloud deployment evidence and end-to-end cloud execution are different claims. We documented them separately so that the submission clearly distinguishes what was demonstrated locally, what was provisioned on Google Cloud, and what remains unverified.

What’s next for ProtocolRun-VR

Future work includes validating a complete Quest-to-Cloud Run recovery cycle, recording a successful Firestore transaction from the deployed service, validating a Vertex AI call from Cloud Run, and testing the workflow with broader hardware and participant conditions.

The current project is a bounded technical demonstration, not a validated unattended human-subject research system or an unrestricted autonomous Unity repair tool.

Built With

Share this project:

Updates

Submission history