Inspiration

A downed streetlight cable, a burst water main and an unprotected lane on a busy road all demand that a field crew make the right decisions in a strict order. In South African municipalities those procedures are often learned from manuals, toolbox talks and supervision on live sites, where a mistake can cost a life. Reading a lockout procedure is not the same as seating your own padlock, proving a tester and turning a valve key with your own hands. We built CivicOps Field Training so that a trainee can make the mistake in a headset, seated, and have the drill stop them, long before they reach a live street.

What it does

CivicOps Field Training is a seated, hands-first WebXR training app that places a trainee in full-size South African street scenes for four municipal jobs:

  • Electrical safe isolation: identify the correct circuit, notify control, switch off, apply a personal padlock and danger tag, then prove the tester, test dead and re-prove before a deliberate held "Safe to work" confirmation.
  • Water burst-main isolation: protect the area, plan the valve route on a network schematic, notify control, close two valves with real wrist turns, open the washout, confirm isolation and check the trench before holding the site at "Ready for repair".
  • Roads work-zone protection: in rain, with minibus taxis on the road, the trainee works from a kerb refuge to place the beacon, advance warning, cone taper, keep-left board and barriers, briefs and dispatches STOP/GO marshals, then runs a traffic demonstration that reacts to the actual layout.
  • Fire and EMS scene safety: position the engine, place cones and flares, identify hazards, exclude a downed cable, approach safely, triage four simulated casualties and hand over.

Every job begins with six pre-shift checks covering supervisor authority, site hazards and PPE or crew readiness. Unsafe choices stop the attempt; other mistakes must be corrected and remain in the score. The debrief ranks safety first, then accuracy, then pace, and records a local Bronze, Silver and Gold practice ladder with deterministic daily drills and refresher dates. Guided practice coaches each step; Unguided assessment removes every cue that names the next action. An "Explain why" instructor gives the safety reason behind any step.

Accessibility is built in: captions in three sizes, high contrast, reduced motion, left-hand and one-handed layouts, gaze dwell that presses a button after 1.2 seconds, and a seated reach tray that brings the controls within reach of a seated shoulder. On-site Assist adds an immersive AR preview with plane placement, persistent anchors, human STOP gates and a structured offline log.

How we built it

The app is TypeScript on Meta's Immersive Web SDK (IWSDK 1.0.1) with Vite and three.js, and pmndrs/uikit panels authored in UIKitML. Every procedure is a pure reducer: an action goes in, and the next state, score and feedback come out. The same reducers drive the headset scenes, more than 850 unit tests and the saved attempt records. Hand input uses IWSDK's poke, ray-pinch and grab systems; Roads checks evaluate real placed geometry (taper length and angle, sign facing, enclosure coverage, refuge and sight lines) rather than fixed slots. Scenes are original Blender work plus licensed CC0 and CC BY assets, compressed as glTF with meshopt and KTX2 textures, and all sound is synthesised in WebAudio. Captions and browser speech provide the instructor in every build, and in the judge build "Explain why" asks Meta's Muse Spark model live through the Meta AI API. The call goes through a rate-limited server-side Azure Function, so the key never reaches the browser, and the scripted answer is the fallback. Every module was run end to end with emulated tracked hands in IWER, the Quest 3 emulator, with no state injection. The judge build bundles IWER so the app can also be tested in a desktop browser, and it is hosted on Azure Static Web Apps.

Challenges we ran into

  • Measured hands are not exact. Twelve emulated wrist quarter-turns summed to 2.999875 turns: the panel read 3.00, yet the valve did not count as closed. We defined closure with an explicit tolerance and added tests for both that case and a genuinely short turn.
  • Seated reach. Panels designed for a standing trainee were out of reach when seated. We built a tilted reach tray that follows the working shoulder and measured the distance to every control.
  • Honest assessment. Coaching that names the next action makes an assessment meaningless, so Unguided mode strips those cues from captions, titles and the instructor while keeping safety stops.
  • A full street on a mobile-class GPU. Rain, traffic and crews required instancing, shared atlases and per-scene draw and texture budgets.

Accomplishments that we're proud of

  • Four complete, hands-only procedures in which unsafe decisions genuinely stop the drill.
  • A roads exercise scored on the actual geometry the trainee builds, with traffic that responds to it.
  • Accessibility designed in from the first build: seated tray, gaze dwell, captions, contrast and one-handed use.
  • Streets that feel local, with minibus taxis, STOP/GO marshals and municipal crews, built only from original and properly licensed assets.

What we learned

Order matters more than detail. Modelling each job as a strict sequence with hard safety stops gave trainees clearer feedback than adding more props, and it made every step testable without a headset.

What's next for CivicOps Field Training

Validate on Meta Quest hardware with municipal crews and supervisors, and tune comfort and frame rate on device. Reduce the download size. Add the steps we deliberately left out: Water clamp repair and recommissioning, Roads patching, and Electrical earthing and permits. Complete room-aware mixed reality for On-site Assist, add further departments, and give supervisors a view of practice results.

Disclosures

  • The procedures are illustrative training based on generic municipal practice, not employer SOPs. Practice certificates are not authorisation for live work or patient care. Triage is simulated.
  • All scenarios and data are synthetic. No real people are depicted; crews and vehicles are generic and unbranded.
  • Third-party assets are licensed (CC0, CC BY 4.0 with attribution, and Adobe Mixamo motion for in-app use) and credited in the in-app credits page and ASSETS.md.
  • Demo footage captured in IWER is labelled as emulator footage.
  • No association with or endorsement by Meta is implied.

Assets removed from the public repository: the hosted build and the demo video use licensed Adobe Mixamo character animation (the street crew and the Roads marshal). Mixamo's terms allow that animation to be used and shown inside the app, but not redistributed as standalone files, so the animated models were removed from the public GitHub repository, which keeps the same characters without their animation clips. The judge link runs the complete build.

Built With

Share this project:

Updates

Submission history