Lumen

An AI agent who listen with its eyes.

Comment

Lumen: Project Story

Inspiration

Most AI assistants and medical facilities are designed around voice. But what happens when someone communicates through sign language?

I wanted to make sign language a first-class input modality for AI agents, giving people who communicate through signing a way to interact with systems that can actually take action. In a clinical setting, that could mean communicating a need to a doctor, recording intake information, or asking for help.

I also wanted to address what happens when someone's in a vulnerable state and cannot respond to a routine check-in. That led us to tackle the KAZE challenge with Lumen's safety system, designed to detect missed check-ins and escalate alerts to trusted contacts.

What it does

Lumen turns a set of recognized signs into agent actions. A Meta Quest 3S uses WebXR hand tracking, and a compact model classifies isolated ASL signs in the headset browser. The current default model recognizes HELLO, PLEASE, THANK_YOU, YES, NO, and HELP, plus a background class. It is a prototype recognizer, not a sign-language interpreter. It does not understand full ASL grammar, facial expression, or body language. The current demo model and recordings come from one signer, so recognition has not been shown to generalize to other signers.

In live sign mode, the headset sends recognized glosses and confidence values to the server, not video or raw hand-joint streams. In the explicit recording mode, hand-joint clips are uploaded for training and evaluation.

Agent and clinic workflow

The server turns confirmed glosses into a plain-English sentence, applies a code-defined safety policy, and gives a planner access to MCP tools. Those tools can:

  • Speak a phrase for the clinician.
  • Save a clinic-intake entry or flag an allergy.
  • Ask the signer to confirm a read-back.
  • Request an emergency-contact notification.

Clear phrases can be spoken after a cancel countdown. A phrase the policy considers risky is held for read-back approval. Contact notifications require separate approval enforced by the tool. The normalizer and planner can use hosted models when configured, with deterministic rule and offline paths available.

The clinician dashboard accepts text and microphone input on a phone or laptop to communicate with the patient where it would transcribe into text bubbles for the patient to see.

Lumen safety check-ins

An operator can configure a check-in interval and trusted contacts. On the Quest launch screen, the selected interval controls how long the watch waits before prompting.

  1. Lumen prompts the person when the interval expires.
  2. The person gets a 60-second response window.
  3. If that window expires, a further configured grace period runs before escalation begins.
  4. Contacts can acknowledge the alert, and the service can continue through its configured escalation.

How we built it

Sign recognition on Meta Quest 3S

  • Read WebXR hand tracking and map the joints into the project's 21-joint-per-hand representation.
  • Segment the stream into clips and extract features with matching JavaScript and Python implementations.
  • Train a compact PyTorch model with a convolutional stem, Transformer blocks, and attention pooling.
  • Export the model to ONNX and run it in the Quest browser with ONNX Runtime Web and WebAssembly.
  • Keep live recognition on the headset; upload joint clips only in the deliberate recording workflow.

AI agent and clinic workflows

  • Use a Python FastAPI server for the web app, dashboard, HTTP APIs, and WebSocket connections.
  • Expose application actions through the MCP Python SDK. The web app connects to the MCP server in-process; standalone MCP clients can use its stdio transport.
  • Normalize confirmed glosses into a sentence, using Gemini when configured and a rule-based fallback otherwise.
  • Let Gemini to choose among the available tools.
  • Keep repeat, read-back, cancel, and emergency-contact approval rules in code and in guarded tools.
  • Optionally transcribe browser-streamed microphone audio with faster-whisper for clinician captions.

Lumen's safety check-in implementation

  • Run the check-in timer on the server, with configurable intervals, a response window, and an alert grace period.
  • Provide a laptop setup console, device client, Quest prompt, and contact acknowledgement page.
  • Use SQLite for local/default check-in runs. The current Vultr deployment uses Tiger Data managed PostgreSQL; its live health endpoint reported a working database connection and a TimescaleDB hypertable.
  • Keep Lumen's separate append-only event log in SQLite. Quest recordings, model files, and prepared datasets remain files rather than database records.
  • Support Twilio, Telegram, ntfy, and generic webhooks. If no backend is configured or configured delivery fails, Lumen reports a dry run or failure instead of claiming an alert was delivered.

Challenges we ran into

Collecting Quest-matched data

Public sign datasets do not directly match the Quest hand-tracking format and capture conditions. We built a recording path, but the available demo data is small and comes from one signer, so the accuracy across different signers may be significantly lower than from one specific signer [Lack generalization]. Therefore, collecting data from more signers and evaluating on held-out signers would improve Lumen's model significantly.

Making agent actions bounded

Recognition alone is not enough. The system must handle uncertain words, let the signer correct or cancel a phrase, and prevent an LLM from approving its own sensitive actions. I separated normalization, policy, planning, and tool execution.

  • If a planner fails before acting, Lumen can retry or use its offline path.
  • If it fails after a tool call, Lumen stops rather than replaying calls that could duplicate actions.

Handling a missed check-in

A timer needs to allow for a delayed response while still progressing if the person does not check in. Lumen uses a server-side schedule, response window, alert grace period, and contact acknowledgement/escalation logic. Connectivity failures, accidental alerts, and real contact delivery still need broader field evaluation.

Accomplishments we're proud of

  • An end-to-end prototype connecting Quest hand tracking, on-device sign recognition, and MCP tool calls.
  • A Quest recording workflow and a small first-party dataset that tests the training/export pipeline.
  • Explicit code-level gates for uncertain phrases, read-back approval, cancellation, and contact notifications.
  • Lumen's configurable, server-authoritative check-in feature, with laptop and device interfaces, a Quest prompt, contact acknowledgement, and a live Tiger Data/TimescaleDB deployment for the KAZE challenge.

What we learned

  • Data matters as much as model architecture. A small single-signer dataset cannot establish how recognition will work for other people.
  • Separate interpretation from action. A planner can suggest tool calls, but explicit rules should constrain what it may do, especially when speech or contacting someone is involved.
  • A working prototype is a starting point. Recognition, usability, privacy expectations, network failure behavior, and alert delivery all need evaluation with the people who may use Lumen.

What's next for Lumen

  • Collect consented Quest recordings from more signers and environments; evaluate using held-out signers.
  • Create a database to store chat history that the user can enable.
  • Tune rejection and confirmation thresholds using real deployment-condition data.
  • Work with fluent ASL signers to verify the sign guide, vocabulary, and interaction design.
  • Conduct privacy, accessibility, and safety reviews before considering real-world use.

Built With

Share this project:

Updates

Submission history