NEXUS

Safety-first robot mission control: turn plain-language objectives into memory-backed jobs, live digital-twin evidence, and human-gated deployment, with Boston Dynamics Spot camera support.

Comment

Robots can enter hazardous buildings before responders, but today's mission software is fragmented: objectives live in chat, maps live in another tool, robot telemetry lives in vendor software, and safety decisions are difficult to audit. The dangerous shortcut is to let a powerful model directly command hardware without showing what it remembered, why it chose a route, or where human authority begins.

We built NEXUS to explore a better pattern: useful autonomy with visible evidence and explicit authority boundaries.

What it does

NEXUS is a safety-first robot mission control plane that turns a plain-language field objective into a reviewable mission:

  1. An operator describes a job in text or selects a destination on the map.
  2. NEXUS recalls persistent location, procedure, and previous-experience memory.
  3. It deterministically assigns the best robot and explains why.
  4. It calculates a route through a metric three-floor digital twin, accounting for walls, door openings, robot clearance, stairs, and lifts.
  5. A six-stage pipeline makes intent, memory, safety validation, navigation, inspection, evidence, and learning visible.
  6. The job runs in simulation and stores experience memory for future work.
  7. Any physical request remains human-gated and records hardware_command_sent: false.

The live demo includes three continuously moving virtual robots streamed over a sequenced WebSocket at four updates per second. Judges can stop and restart the fleet to prove the scene is live. The same spatial model is rendered as an interactive north-up 2D plan and a navigable 3D building.

NEXUS also includes a changing Demo Spot camera feed for reliable judging and an optional server-side Boston Dynamics Spot adapter for real identity, battery, camera discovery, and current images. The adapter is deliberately read-only: it does not acquire a lease, power motors, or send movement commands.

An optional OpenAI Responses API feature uses GPT-5.6 with Structured Outputs to produce a bounded engineering review from current project evidence. The reviewer receives no tools and cannot approve or execute a physical action. The complete deterministic demo works without an API key.

How we built it

The control plane is a FastAPI service backed by PostgreSQL, SQLAlchemy, and Pydantic. It owns objectives, requirements, robot memory, jobs, approvals, telemetry, WebSocket events, and a SHA-256 hash-linked audit chain.

The operator console is built with React, TypeScript, Vite, and Three.js. It renders the shared metric building model, live robot paths, mission assurance, job explanations, camera state, approvals, and audit evidence.

The navigation foundation includes a dependency-free C++ occupancy-grid core. Docker Compose starts PostgreSQL, the API, and the production dashboard on the fixed demo URL http://localhost:3006. A seeded project, three robots, building geometry, hazards, memory, and Demo Spot feed make the reviewer path repeatable without external services.

How we used Codex and GPT-5.6

Codex was the primary development environment for the hackathon build. We used it to inspect and migrate the original Windows-oriented repository to macOS; implement the FastAPI, PostgreSQL, React, Docker, virtual-fleet, memory-backed job, and read-only Spot changes; diagnose integration issues; write regression tests and operating documentation; rebuild the full stack; and exercise the final UI in a real browser.

GPT-5.6 was used through Codex for repository reasoning, implementation, debugging, test generation, documentation, and acceptance-test planning. The human developer supplied the product goals, chose the safety boundary, reviewed the visible behavior, and remains responsible for submission and all physical-robot decisions.

AI-generated changes were treated as proposed engineering work, not as safety evidence. We verified them with deterministic tests, strict typing, production builds, Docker health checks, route-geometry audits, WebSocket sampling, camera frame checks, and browser acceptance tests.

Challenges we ran into

  • Migrating a Windows-first project to a repeatable macOS and Docker workflow.
  • Making three simulated robots move smoothly while keeping the server as the authoritative source of truth.
  • Preventing visual routes from cutting through walls while still allowing valid door and vertical-connector transitions.
  • Turning natural-language intent into explainable, memory-backed jobs without hiding assignment or safety decisions inside a black box.
  • Supporting a convincing hardware story without falsely claiming that virtual telemetry came from a physical robot.
  • Preserving a hard safety boundary: camera and telemetry access must never imply motor or lease authority.

Accomplishments that we're proud of

  • A complete text-or-map-to-job pipeline with visible memory recall, assignment, verified route, six execution stages, evidence, and learning.
  • Three live virtual robots with smooth four-Hz WebSocket movement and explicit stop/restart controls.
  • A reliable changing Demo Spot feed plus an optional real Spot read-only camera adapter.
  • A derived Mission Assurance view instead of a fabricated readiness score.
  • Human approval gates, emergency-stop semantics, and a verified hash-linked decision history.
  • A fresh full-stack verification with 43 backend tests, 78 frontend tests, and the C++ navigation test passing, plus clean lint, strict typing, production builds, healthy Docker services, and zero browser console errors.

What we learned

The most useful unit of autonomy is not a generated command; it is an inspectable evidence chain. Memory, planning, route geometry, simulation, approval, and audit data must agree before a physical decision is even considered. Models are most valuable when they help engineers reason over that evidence—not when they bypass the control plane.

What's next for NEXUS

The next milestone is a Gazebo/ROS 2 simulation profile with signed scenario artifacts and repeatable failure fixtures. A physical pilot would follow only after authentication, durable jobs and events, expiring capability grants, robot-edge allowlists, watchdogs, an independent emergency stop, signed command receipts, and formal safety-case review are implemented and tested.

NEXUS is a simulation-first engineering foundation, not a certified robot controller—but it demonstrates how AI-assisted mission planning can remain useful, explainable, and human-controlled.

Built With

Share this project:

Updates