Field Assist

Field Assist lets Codex and a human expert turn a live phone camera into verified, object-tracked guidance—pointing users to the correct physical control.

Comment

Inspiration

Earlier this week, before I discovered The WebMCP Challenge, I published experimental WebMCP support for GoFastr. When I saw the hackathon, my honest reaction was basically: “heck yeah—this is exactly what that package is for.”

I wanted to test the browser-agent bridge against something much harder than a catalog or form demo. Remote support repeatedly fails at the last few inches: the operator is holding a phone, the expert says “press the button underneath,” and everyone is forced to translate vague language into physical action. Field Assist asks what happens when Codex, a human support representative, and the person holding the camera can share one live, structured workspace.

Field Assist itself was built for this hackathon on top of that newly published GoFastr capability.

What it does

Field Assist creates a two-device support session. A support representative opens the desktop console and shares a one-time QR code. The operator scans it, grants rear-camera access, and selects a starter scenario or describes any problem in free form. The camera travels directly to the support browser over peer-to-peer WebRTC; the application server never records or relays the video.

Codex joins through the authenticated support page’s WebMCP tools. It loads the application’s operating contract, reads the operator’s request, inspects the visible scene, and combines vision with product knowledge to form a hypothesis. When identity is uncertain, Codex asks the operator a bounded question on the phone. Once the device and its actionable control are separately verified, Codex can send a large phone-visible instruction and place an arrow at the control.

The support representative can inspect, correct, compose, or remove the same shared guidance. OpenCV, Depth Anything V2, Canvas tracking, and short-lived phone orientation data help keep the arrow associated with the physical target as the camera moves. When geometry becomes unsafe, Field Assist suppresses the marker and asks for reacquisition instead of pretending it still knows.

Why WebMCP

Why is this a strong fit for WebMCP? Field Assist depends on live, authenticated browser state: a private camera session, changing scene geometry, operator answers, tracking confidence, and delivery receipts. WebMCP lets Codex use those application concepts directly instead of guessing at UI controls or receiving a giant screenshot-and-prompt dump.

How does it create a better user experience? The operator keeps one hand on the phone and receives large, timely instructions plus spatial guidance where they are already looking. Codex and human support share the same backend-authoritative scene, conversation, question, and banner state, so the operator does not repeat context or translate ambiguous chat into physical action.

What can people and agents do together that was difficult before? The person supplies intent, judgment, and physical confirmation. Codex contributes vision, product knowledge, structured questions, and rapid guidance. Together they can identify an unfamiliar device, confirm the correct object, locate a small control, guide the operator toward it, and withdraw the annotation if tracking becomes unsafe—all inside one live support session.

How did I implement WebMCP? The authenticated support page exposes twenty-five tools generated by GoFastr through the standard document.modelContext.registerTool API. Read tools return versioned session and scene state. Mutating tools call the same authenticated Go handlers as the human controls and must include the current contextVersion. Landing and operator pages expose no support capabilities.

How we built it

Field Assist is one Go 1.27 service built on the released GoFastr v0.75.0 module. GoFastr provides the server-rendered application host, typed handlers, security middleware, WebSocket hub, lifecycle endpoints, structured logs, static plugin assets, and experimental WebMCP bridge.

The authenticated support page declares twenty-five semantic tools for application context, scene inspection, object registration, room context, conversation, operator questions, banners, snapshots, and reversible spatial guidance. The mandatory get_app_info tool copies a versioned operating contract and the user’s current request into Codex context. Every state-changing tool must return that context version, which prevents an agent from acting against stale instructions.

Browser-native WebRTC sends video directly from the phone to the support computer. GoFastr WebSockets carry only signaling and compact shared state. The support computer runs OpenCV ORB/RANSAC homography and PnP tracking alongside Depth Anything V2 Small through ONNX Runtime Web. A lightweight phone tracker and bounded DeviceOrientation samples bridge short visual gaps over a peer-to-peer data channel. Camera pixels, descriptors, depth maps, orientation samples, SDP, and ICE candidates are never stored by the application server.

The interfaces are responsive and deliberately asymmetric: the phone prioritizes the live camera, large instructions, questions, answer acknowledgment, and chat; the desktop prioritizes the shared scene, support conversation, guidance controls, delivery state, and progressive diagnostics.

Challenges we ran into

The hardest challenge was not drawing an arrow—it was deciding when an arrow was trustworthy. Early versions anchored annotations to screen coordinates, confused a broad device with a tiny physical control, or kept displaying stale guidance after tracking loss. Reflective televisions and dark hardware exposed every weakness in monocular depth and feature matching.

Another challenge was keeping three participants synchronized: the operator, human support, and Codex. Local optimistic UI caused banners and answers to disagree between screens, so the backend became authoritative for conversation, questions, banners, scene versions, and delivery receipts.

Testing a two-device, hands-busy experience was also physically awkward. I repeatedly had to hold the phone in front of real hardware while Codex operated the support console. Those sessions forced the application contract to become explicit: inspect first, confirm uncertain identity, register the containing device before its control, send movement instructions to the phone banner, verify rendered delivery, and fail closed when tracking confidence drops.

Finally, WebMCP is experimental. The application needed to make its own operating expectations discoverable without asking judges to install a separate skill. get_app_info became the handshake that teaches any compatible Codex session how to use Field Assist safely.

Accomplishments that we're proud of

  • A real iPhone-to-desktop, video-only WebRTC support session with no account required.
  • Twenty-five authenticated WebMCP tools available only on the support page.
  • Codex as the only runtime agent; Field Assist embeds no separate model API.
  • Shared two-way conversation, phone-visible questions, answer acknowledgment, and backend-authoritative banners.
  • Separate semantic objects for a device and its actionable physical control.
  • Reversible object-relative arrows with explicit tracking-loss behavior.
  • Support-side OpenCV and Depth Anything inference without uploading camera frames.
  • Ten-minute, single-use QR links exchanged for role-scoped HTTP-only cookies.
  • Deterministic Codex evaluations, Go tests, and two-browser Playwright coverage.
  • A public live deployment, public MIT-licensed repository, and a complete narrated demo under three minutes.

What we learned

A useful agent integration needs more than tool schemas. The agent needs the application’s safety model, ordering constraints, versioned context, and definitions of success. Separating identity, localization, tracking, delivery, and physical confirmation prevented plausible-looking UI from being mistaken for proof.

Depth estimation does not magically create a persistent 3D world model. It becomes useful when combined with visual features, stable object identity, bounded camera-pose estimation, and honest loss behavior. Reflective or texture-poor objects remain difficult, and the product should say so.

Realtime UX also needs one source of truth. Once questions, messages, banners, observations, and guidance receipts came from backend state, the operator, support representative, and Codex stopped telling different stories.

Most importantly, WebMCP works best when it exposes application meaning rather than mirroring buttons. Codex should ask for “the current scene” or “send this operator instruction,” not hunt for coordinates in a support dashboard.

What's next for Field Assist

The next reliability step is TURN-backed WebRTC for restrictive networks, followed by persistent multi-instance session state. Spatial guidance can improve with stronger visual-inertial tracking and optional WebXR or LiDAR signals when the device supports them, while retaining the current browser-only fallback.

The same collaboration model can extend beyond consumer electronics to appliance setup, accessibility support, field service, lab procedures, industrial maintenance, guided inspections, telehealth equipment setup, education, and emergency troubleshooting. The essential pattern stays the same: a person controls the physical world, Codex supplies fast contextual reasoning, and the application keeps their shared guidance explicit, synchronized, and reversible.

Built With

Share this project:

Updates

Submission history