JAMES

A touch-free, multi-agent maintenance copilot for macOS. Point at the machine, ask out loud, pinch to confirm each step. 1st prize, AgentX-2026.

Comment

Inspiration

A technician with gloves on and both hands busy cannot scroll a PDF manual. What they need lives in manuals, sensor logs, past repairs and other people's heads. JAMES started from one question: what should a computer look like when your hands are never free?

What it does

JAMES watches the camera, listens, finds the right procedure in the manual, checks the machine's sensor log and past repairs, and walks through the fix one step at a time.

  • Hands-free by design: point to pick a target, V sign to talk, pinch and hold to confirm
  • Every answer has a source: steps cite manual pages, readings come from the supplied log, and weak evidence is called out instead of hidden
  • A safety gate with no AI in it: WARDEN, a plain-Python rule engine, refuses unsafe steps. Removing a coupling guard stays blocked until lockout and zero energy are confirmed
  • Learns from approved jobs: once a senior approves a repair, the next similar request recalls it first. Unapproved jobs are never recalled
  • ATLAS: draws the targeted machine as an exploded schematic and lights the part the current step touches, amber when allowed and red when blocked. It also runs as its own app: a library of openly licensed models (NASA's Perseverance rover, Ingenuity, the James Webb Space Telescope and more) you take apart with your hands
  • CANTUS: a rhythm game in the launcher. Swing through blocks on the beat, left hand red, right hand blue
  • Local first: local model, local search, local memory

How we built it

A team of small agents, each with one job:

  • LENS sees hands and asset tags (MediaPipe Hand Landmarker, OpenCV ArUco)
  • ECHO hears the request (faster-whisper, Qwen3-VL 4B via Ollama, keyword rules when offline)
  • PAGE finds the procedure (PyMuPDF, BM25, nomic-embed-text in Qdrant) and cites pages
  • PULSE reads the sensor log with pandas and never invents a reading
  • LEDGER remembers approved jobs in an append-only SQLite store
  • FOREMAN plans with a LangGraph fan-out to PAGE, PULSE and LEDGER
  • WARDEN blocks unsafe steps with the industry pack's rules

Machine knowledge (rules, faults, sensors, manuals, asset tags) lives in industry packs, not in the code, and the app refuses to start if a pack file changed after its manifest. Every logged event is hash-chained, so an edited or removed event can be named later.

Challenges we ran into

The pinch has to be a pinch. A pinch that starts as a closing fist, or a hand moving toward the camera while held (reaching for a tool), must never confirm a step. A tracking-trust layer flags low light, dark gloves and dropped hands with an amber marker instead of guessing.

Accomplishments that we're proud of

  • 1st prize, AgentX-2026, a national-level hackathon at Teegala Krishna Reddy Engineering College, Hyderabad (25-26 Sep 2026), Rs 20,000. Entered under Multi-Agent Systems and Collaborative AI and pitched live to the chief guest and jury. Covered in Sakshi (27 Sep 2026).
  • Seeded safety suite: 8 of 8 violations blocked, 0 of 4 false blocks
  • 607 tests passing, public under MIT with a product site

What we learned

Safety and trust are features you have to show on screen. JAMES's status bar always says which backend is live (model or rules, BM25 or BM25 plus Qdrant), so nothing is overstated.

What's next

  • Signed industry packs and a second pack for an air-handling unit
  • OEM models in ATLAS, pinned to the real machine using the asset tag's pose
  • A simulated Modbus / MQTT sensor adapter
  • Windows, Linux and edge devices
  • Telugu and Hindi voice

Demo build with sample data and demo safety rules, not certified for real safety use.

Built by Victor Isaac Chintha. Pitched at AgentX-2026 as Team Ace Azael with Sunil Dehru and P. Bharyasri.

Built With

Share this project:

Updates