Calyx

Calyx turns EMS call data into instant reports. It rebuilds timelines from video, audio, and logs, reducing PTSD, cutting errors, and delivering actionable insights fast.

Comment

Inspiration

EMTs operate in high-stress, life-or-death environments. After every call, they are required to reconstruct the entire incident using memory, body cam footage, dispatch audio, and CAD logs.

This process is slow, cognitively demanding, and forces responders to repeatedly relive traumatic moments. Over time, this contributes to burnout and PTSD.

We asked a simple question:

Why are humans manually reconstructing events when the data already exists across multiple sources?

What it does

Calyx is a multimodal intelligence system that transforms fragmented EMS data into a complete, structured after-action report.

It ingests:

  • Patient Care Reports (PCR)
  • Body cam video
  • Dispatch / radio audio
  • CAD (Computer-Aided Dispatch) logs

Then it:

  • Reconstructs a unified, timestamped timeline
  • Aligns conflicting sources into a single ground truth
  • Detects discrepancies, missing documentation, and protocol deviations
  • Generates a reviewer-ready QI (Quality Improvement) report

On top of that, it includes a 3D ambulance simulation layer that visually replays the call:

  • Real route playback on a 3D map
  • Event markers placed along the route
  • Discrepancy highlights at exact locations
  • Interactive callouts with evidence context

This turns abstract reports into something spatial, intuitive, and reviewable.

How we built it

Calyx is built by harnessing multimodal data, agentic pipelines, and deterministic orchestration into a single system.

At a high level, we fuse four different perspectives of the same incident:

$$ \text{Ground Truth} \approx \text{Fusion}(\text{PCR}, \text{Video}, \text{Audio}, \text{CAD}) $$

Each modality is processed independently by specialized agents:

  • PCR agent extracts structured clinical actions
  • Video agent extracts visible interventions
  • Audio agent extracts dispatch and radio events
  • CAD parser provides timestamp anchors and metadata

All outputs are normalized into a shared event representation, allowing cross-source reasoning.

Multi-agent orchestration layer

Instead of using one large model, we decompose reasoning into coordinated agents:

Event Extraction (parallel)
All modalities are processed simultaneously to produce structured event streams.

Event Clustering
Events that refer to the same real-world action are grouped together across sources.

Parallel Cluster Review
Each cluster is evaluated independently to:

  • Measure disagreement
  • Resolve timestamps
  • Identify inconsistencies

Conditional Escalation (Critic Layer)
Only high-uncertainty clusters are escalated to stronger models.
This creates a hybrid system balancing cost and accuracy.

$$ \text{Cost Efficiency} = \frac{\text{Cheap Agents for Easy Cases}}{\text{Expensive Agents only for Hard Cases}} $$

Canonical Timeline Assembly
All validated clusters are merged into a single ordered timeline:

  • Evidence-backed
  • Fully traceable
  • Discrepancy-aware

Findings + Report Generation
Specialized agents generate:

  • Clinical assessments
  • Documentation scoring
  • Recommendations
  • Final determination

Critical outputs are constrained with deterministic rules for reliability.

3D Ambulance Simulation (GPS + event replay)

We extend the system beyond static reports by mapping events into physical space.

The ambulance route is modeled as a sequence of geographic coordinates over time:

$$ \text{Route} = {(lat_1, lon_1, t_1), (lat_2, lon_2, t_2), \dots, (lat_n, lon_n, t_n)} $$

The ambulance position is interpolated continuously:

$$ \text{Position}(t) = (1 - \alpha)(lat_i, lon_i) + \alpha(lat_{i+1}, lon_{i+1}) $$

Each clinical event is projected onto the route:

$$ \text{Event Position} = \text{argmin}{(lat, lon) \in \text{Route}} |t{\text{event}} - t| $$

This allows us to:

  • Place discrepancies at exact spatial points
  • Trigger event callouts based on proximity
  • Replay the incident as a continuous 3D simulation

We use:

  • Mapbox for 3D maps and building extrusions
  • Deck.gl for rendering routes and event layers
  • ScenegraphLayer for a 3D ambulance model
  • Animated markers to represent severity and discrepancies

The system effectively creates a spatial-temporal replay of the incident, turning raw data into an interactive investigation tool.

System philosophy

Calyx is built on three core principles:

1. Harnessing over guessing
We combine multiple sources instead of trusting any single one.

2. Agentic decomposition
Each model does one job well instead of one model doing everything poorly.

3. Deterministic control
Critical outputs are governed by rules, not just model outputs.

Challenges we ran into

  • Aligning inconsistent timestamps across modalities
  • Handling incomplete and noisy real-world EMS data
  • Balancing latency, cost, and model complexity
  • Preventing hallucinations while preserving reasoning power
  • Designing a system that degrades gracefully under failure
  • Mapping abstract events into physical GPS space

Accomplishments that we're proud of

  • Built a full multimodal reasoning pipeline
  • Created a canonical timeline from conflicting sources
  • Designed a cost-efficient multi-agent system
  • Integrated a 3D spatial replay engine with event-level precision
  • Delivered structured, auditable outputs instead of black-box AI

What we learned

  • Multi-agent systems outperform monolithic prompts for complex reasoning
  • The hardest problem is data alignment, not model capability
  • Deterministic logic is essential for high-stakes systems
  • Spatial visualization dramatically improves understanding of events
  • Real-world systems must handle failure as a first-class concern

What's next for EMT tool

  • Full protocol-checking engine for clinical validation
  • Real-time EMS system integration
  • Dynamic GPS routes per case using live CAD data
  • Automated extraction of real video/audio clips
  • Training feedback loops for EMT teams
  • Organization-level analytics for systemic improvements

Long-term, Calyx becomes the intelligence and visualization layer for emergency response, transforming raw multimodal data into actionable insight and continuous improvement.

Built With

Share this project:

Updates