• 

  Aegis Preview

Aegis: An Optimal MedTech Robotics Paradigm

Project Story

Inspiration

The inspiration for Aegis stems from the impending obsolescence of legacy hospital information systems, specifically platforms like SAP IS-H, which are scheduled for discontinuation by 2030. We recognised an urgent need for a replacement architecture capable of automatically generating deeply structured, semantic data points per patient treatment to fuel clinical research. Furthermore, we saw that as medical AI evolves, traditional software-as-a-service billing models fail when applied to autonomous agents. We wanted to build a system where the AI is not just a clinical assistant, but an independent economic actor capable of managing its own financial transactions.

What it does

Aegis is an autonomous AI surgical monitoring and clinical workflow agent. Operating as a digital assistant in the operating room, it leverages imaging and real-time robotic telemetry to ensure maximum patient safety. It listens to the surgeon via hands-free, hallucination-free medical speech-to-text, actively formatting clinical notes into strict Fast Healthcare Interoperability Resources (FHIR) standards.

Beyond its clinical duties, Aegis operates autonomously in the machine economy. It dynamically leases its own computational resources and pays for multi-vendor API costs in real-time, executing outcome-based billing to the hospital seamlessly.

How we built it

We used a Miro digital workspace as our immutable single source of truth, populating it with FHIR database schemas and sequence diagrams. Using the Miro Model Context Protocol (MCP) Server and the Lovable.dev AI coding agent, we instantly translated these visual diagrams into a functional React frontend and a Supabase backend.

Our core medical intelligence is powered by:

• Google HAI-DEF (MedSigLIP): We laid the architectural groundwork and prototyped the integration for MedSigLIP to perform zero-shot classification and semantic image retrieval for anatomical landmarks in DICOM files.
• Ollama & Mistral 7B: To ensure strict data privacy and rapid execution, we deployed a locally hosted Mistral 7B model via Ollama to handle the core clinical reasoning and semantic text processing.
• Webots & ElevenLabs: A Webots physics simulator generates real-time kinematic telemetry of a 6-axis surgical arm, while ElevenLabs Scribe v2 handles extreme-accuracy speech-to-text for clinical observations.

For the autonomous economic layer:

• Stripe: We utilised Stripe and Shared Payment Tokens so the agent could securely lease computational power.
• Solana: We leveraged the Solana blockchain for sub-cent, high-frequency API micropayments using Token-2022 Confidential Transfers.
• Paid.ai: We wrapped core functions in telemetry trace calls to track multi-vendor API costs and implement outcome-based hospital billing.

Challenges we ran into

• Balancing Latency and Privacy in Robotics: Relying entirely on remote cloud inference engines often introduces network latency and data sovereignty concerns, which are unacceptable in high-stakes robotic surgery. We solved this by pivoting to a local-first reasoning architecture utilizing Ollama and Mistral 7B, allowing us to process clinical logic instantly while keeping sensitive telemetry entirely on-device.
• Strict Healthcare Compliance: The EU AI Act and DORA regulations classify medical AI as high-risk systems. We orchestrated our pipelines to guarantee immutable model lifecycle management. Furthermore, we integrated incident.io adaptive agents (like the Incident Commander) to automatically triage and mitigate any robotic kinematic anomalies.
• Financial Privacy: Healthcare B2B transactions require absolute privacy. We overcame this by implementing Solana's Confidential Transfers, which utilise zero-knowledge proofs and homomorphic encryption to mathematically obscure transaction amounts while keeping accounts public for auditor verification.

What we learned

We learned that the true bottleneck in modern MedTech innovation isn't just the AI's reasoning capacity, but operationalising that AI securely. By combining high-speed visual coding with localized inference infrastructures and programmatic financial primitives (Stripe & Solana), we proved that the next generation of healthcare robotics will be fully autonomous economic actors.

Built With

• elevenlabs-scribe-v2
• fhir
• hai-def
• incident.io
• lovable
• miro
• ollamamistal7b
• paid.ai
• postgresql
• pythoncontrollers
• react
• solana
• stripe
• supabase
• synthea
• webots