Inspiration
Software teams have CI/CD. A bad change is built, tested, compared with a baseline, and blocked before release. Physical AI still has a much more fragmented loop. A vehicle policy fails in a simulator or a recorded scene, someone watches the video, guesses at the cause, creates new data by hand, starts another training run, and then tries to remember whether the new checkpoint broke something else.
I wanted an agent that did more than discuss a failure. It needed to gather evidence, run experiments, change a model, test the result, and leave behind a record that another person could audit.
What it does
Servo turns a physical-AI failure into a complete improvement workflow. By giving Servo a policy, a world, and a capability goal. Servo runs the policy and records synchronized evidence such as frames, detections, controls, trajectory, and collision events. Gemini 3.7 Flash or 3.6 Flash examines that evidence and proposes competing explanations. Google ADK then routes bounded tools that run counterfactual experiments instead of accepting the first plausible answer.
Once the evidence establishes a capability gap, Servo creates a targeted curriculum, reserves hidden test seeds before training begins, trains a new PyTorch checkpoint, and evaluates it against both the hidden exam and protected regression suites. Deterministic code, not the language model, makes the final promote or reject decision.
The desktop also turns recorded media into explorable Gaussian worlds. Those worlds provide visual context and coverage diagnostics. Servo keeps reconstructed appearance separate from collision truth and clearly marks inferred or unknown geometry.
The result is a Taskmaster workflow that can move through without asking a person to manually guide every transition.
How we built it
Servo is a Windows desktop application that I built with Qt, QML, C++, Vulkan, and QRhi. The world pipeline uses FFmpeg, COLMAP, PyTorch CUDA, gsplat, and a native Gaussian renderer. The policy and RealityCI services are written in Python with real PyTorch checkpoint training and content-addressed artifacts.
The agent workflow uses Google ADK 2.7.1. A SequentialAgent graph wraps the campaign states, and a runner executes the graph with explicit session state. Google Gen AI SDK 2.19.0 calls Gemini 3.7 Flash for structured diagnosis, bounded tool selection, and evidence summaries. Gemini can propose an action, but schemas, allowlists, causal gates, hidden exams, and promotion rules remain deterministic.
The Google Cloud control plane is deployed in project servo-1f808:
- A Firebase-authenticated API runs on Cloud Run.
- A Cloud Run Job is configured for asynchronous campaigns.
- Vertex AI provides the Gemini execution path for cloud campaigns.
- Firestore stores bounded campaign metadata, hashes, status, and
gs://pointers. - Cloud Storage stores large evidence bundles, Gaussian worlds, videos, and checkpoints.
- Cloud Build and Artifact Registry produce and store the deployment images.
- Cloud Logging captures service and job logs.
The public Cloud Run endpoint rejects requests without a valid Firebase bearer token. The desktop exposes authenticated cloud readiness instead of inferring success from configuration.
Challenges we ran into
The hardest problem was World reconstruction. A forward-facing video can produce a convincing view along the recorded path while falling apart when the camera moves into unobserved space. I spent many iterations improving reconstruction and learned to stop treating visual quality as geometric truth. Servo now exposes appearance, inferred depth, structure, coverage, and provenance separately, and it never calls Gaussian opacity collision geometry.
Accomplishments that we're proud of
Servo is not a chatbot wrapped around a dashboard. The agent can execute an ordered campaign, inspect failures, select bounded interventions, train a checkpoint whose weights actually change, run a hidden exam, protect existing capabilities, and produce a deterministic promotion decision.
One live Ask Servo run used Gemini 3.7 Flash and Google ADK to inspect a campaign, choose the allowed run_to_completion action, and verify 21 ordered events and 80 artifacts. The candidate was rejected by the promotion gate. I are proud of that rejection because it proves the system does not turn every agent run into a success story.
The repository also includes a reproducible native application, a Vulkan Gaussian renderer, Firebase authentication, a deployed Cloud Run API and Job, Firestore and Cloud Storage integrations, a four-minute demo plan, an architecture diagram, and automated tests for the agent graph, Gemini boundary, cloud dispatch, training, hidden evaluation, and fail-closed decisions.
What we learned
The most useful agent is not always the agent with the most freedom. Servo became more autonomous when each action had a clear contract, evidence requirement, and recovery path.
I also learned that appearance, physics, semantics, and evidence should be separate layers. A visually realistic world is useful, but it should not silently become a collision map. A simulator event is useful, but it should not be presented as an observation from the original camera. Clear provenance makes the product more trustworthy and easier to debug.
Our Reality Debt score captures the remaining verified capability gaps as
$$ D = \sum_i w_i(1-c_i), $$
where (w_i) is the importance of a capability and (c_i) is its evidence-backed confidence. The exact score is computed by code, not by Gemini. This gives the autonomous loop a measurable reason to choose the next weakness.
What's next for Servo
After the hackathon, I want to add more policy adapters, metric sensor-backed road geometry, richer multi-camera capture, per-object dynamic representations, and longer-running cloud campaigns. Google Sign-In can also be added as an optional operator login once its native OAuth redirect flow is configured and tested. The core principle will remain unchanged: every improvement must come with evidence strong enough to reject it when it is wrong.
Log in or sign up for Devpost to join the conversation.