Inspiration
Every barrel of oil comes up with salt water. That water fills tanks at each facility and gets pumped down pipelines to a disposal site. When a pump fails at 2 AM, its tank starts filling. If it reaches the shutdown line, every well feeding that tank is shut in, and the field loses oil every hour until the repair.
The night-shift operator has three questions: what's failing, how long do I have, and what's the cheapest way to keep running? Most consoles answer with a wall of numbers. We wanted an agent that answers those three questions in plain words, shows the field in 3D, and lets the operator act before anything goes dark.
What it does
gaia loads any EOG dataset folder (network.json, ops.json, telemetry.csv) and works from the raw readings alone.
- Finds what's failing. It detects pump failures, pipeline leaks and worn pumps. Each comes with when it started, whether it's still going, a confidence, and the evidence in plain English.
- Says how long you have. For every failed pump, it gives the tank's level, the hours until shutdown, and the exact trip time.
- Finds the cheapest fix. It plans trucks (one shared fleet per system, with response and round-trip times), shutting in wells early, or accepting the trip, and picks the cheapest combination. Failures in the same system are planned together, so a truck sent to one tank can't also help another.
- Shows the price of waiting. Last Call says when the cheapest plan stops working and how the bill climbs after that.
- Never moves equipment on its own. A fix is a proposal, and a person holds to confirm it.
The experience
- Your field, generated in 3D. gaia turns the network into a 3D field with our Blender pipeline, in seconds per system. Every well, tank, pump, pipeline and disposal site from the data is in it.
- An agent you talk to. Hold space and say "walk me through it." gaia flies the camera to the failed pump, traces its pipes, jumps the timeline to the shutdown, draws the tank's level and opens the plans, one visible step at a time. Point at anything and ask "what's wrong with this?" It draws charts on request ("plot S1 tank C level").
- Hand control. Point to select, pinch to orbit, open your palm to stop. It runs in the browser, and the camera feed never leaves the laptop.
- Alerts you can't miss. A big alert card and a red glow around the whole screen. One sentence gives what failed, the countdown and the cheapest fix with its cost. The same alert can go to the operator's phone as a text and a voice call through Twilio, or to the phone pager, where they can hold to confirm.
- Time travel. Drag the time slider to see the next six hours, or replay any moment of the week as "now". "Two nights" plays the same failure twice, side by side: nobody acts, versus the plan.
- Everything EOG asked for: a map and 3D of every system, trends for any sensor, alerts with actions, a plain-English shift handoff, a what-if replay, a fleet-size what-if, and an
events.jsonexport in EOG's format.
How we built it
- Backend: Python and FastAPI. A dataset loader, a detector and an exact planner, plus a physics replay of each tank with trucks, shut-ins and repair times. One service that every screen reads from, so the status line, the alerts, the plans and gaia never disagree.
- gaia's AI is layered:
- Three agents (watch, forecast and planner) are deterministic templates over that data.
- gaia answers through browser intents first, then backend rules, then Claude (the Anthropic API) with tools.
- Claude can read alerts, forecasts and plans, act on the 3D view in steps, and propose a fix. It can never send one.
- Every number it says comes from a tool result, with its source and age.
- Web: three.js and Vite.
- the gaia console, a chat with a 3D canvas;
- the /dataset page, the judge flow and a guided tour;
- MediaPipe hand tracking and browser speech recognition.
- 3D: a Blender generator that turns a blueprint or a sponsor network into a named, clickable glTF field, with a fast fallback kit when Blender isn't installed.
- SpacetimeDB Maincloud: the backend mirrors live state, alerts and proposals to Maincloud, and our asset library lives there.
- Checks:
- more than 1,100 backend tests;
- a score command that grades gaia against EOG's answer keys;
- headless browser checks for every screen.
Accomplishments that we're proud of
- It matches EOG's answer keys. Across the full week and the three snapshots, gaia found 72 of 72 events with zero false alarms. For every pump failure still in progress, its hours to shutdown are within 0.01 h of the key, and its plan costs exactly what the reference plan costs ($905, $1,298, $1,810, $1,215, $975). The only status difference is a leak that was repaired while its pump was off, which no reading can show.
- It works on networks it has never seen. We renamed every ID in the data to a new network (S6 to S10). gaia found the same events with no code change, so EOG's unseen
test/folder just works. - EOG's own opening scene, found on its own. The tour replays the real week and stops just before a pump fails. gaia picks that failure by itself and arrives at the README's example: one truck, $905, instead of about $5,000 if nobody acts.
Challenges we ran into
- One source of truth. Early on, two parts of the app computed plans separately and disagreed by a few dollars. We cut it down to one planner, and every screen reads from it.
- Shared trucks. Three pumps failing in one system compete for the same two trucks, so planning each failure alone gives wrong answers. The planner solves them together.
- Readings that can't tell you. A leak that starts or ends while its pump is off is invisible in the data. gaia says it can't vouch rather than calling anything safe.
- Keeping the AI honest. gaia never invents a number, never calls a stale reading safe, and never moves equipment. Getting an agent to be genuinely helpful inside those rules took real design.
What we learned
Simple physics you can explain beats a black box at 2 AM. And an agent is only as trustworthy as the numbers you let it say.
What's next for gaia
- Live field connectors: OPC UA, Sparkplug B, Modbus, MQTT and DNP3 are partly built.
- A halt switch, and a forecast check before every confirm.
- Tamper-evident audit logs.
- Running it on a real lease.
Built with
python, fastapi, three.js, vite, blender, claude, anthropic, mediapipe, spacetimedb, twilio, cloudflare, playwright, pytest, javascript, webgl
Team
- Sriyan Bodla: lead, 3D generation, ingest, integration.
- Mounish Mavuduru: backend, dataset mode, detection and planning.
- Ronak Kasaram: web console, /dataset, Last Call.
- Vinny: scenario, gaia's tools and connectors, demo, Devpost.
Built With
- anthropic
- blender
- claude
- cloudflare
- fastapi
- javascript
- mediapipe
- playwright
- pytest
- python
- spacetimedb
- three.js
- twilio
- vite
- webgl

