AURA: Intelligent Health Network for Oculomics

AURA Architecture: A scalable, Multi-Agent Orchestration Layer Integrating SOTA Predictive Oculomics (IRIS) with specialized AI Agents

Inspiration

Modern healthcare is fundamentally broken in two critical ways: it is reactive rather than proactive, and it is deeply fragmented. Patients often wait until symptoms are severe to seek help. When they finally receive a diagnosis, they are left alone to navigate a confusing maze of medical jargon, dietary changes, mental health struggles, and complex insurance plans.

We were deeply inspired by the emerging field of Oculomics—the science of using the eye as a non-invasive window into the body's systemic health. The microvascular structures in the retina are the only blood vessels in the human body that can be observed directly and non-invasively.

We asked ourselves a bold question: What if we could turn a standard smartphone camera into a holistic, full-body diagnostic tool, and then instantly surround the patient with a personalized, AI-driven care team to guide them through the "what next?"

This vision birthed AURA: an intelligent health network that bridges the gap between instant, non-invasive diagnostics and continuous, proactive health maintenance.

What it does

AURA is a comprehensive, multimodal AI-driven healthcare platform designed for modern clinical workflows and personal health management. It operates in two main phases:

1. Instant, Non-Invasive Diagnostics (IRIS) Users capture a retinal scan using their mobile device. Our flagship Smart Oculomics engine, IRIS, analyzes this single image to predict 10 distinct health dimensions:

Demographics: Age & Gender correlation.
Systemic Health Risks: Diabetes, Hypertension, and overall Cardiovascular Risk.
Complications: Acute Myocardial Infarction (AMI), Neuropathy, and Nephropathy.
Ocular Diseases: Diabetic Retinopathy and Macular Edema.

2. The Intelligent Care Umbrella (Multi-Agent Ecosystem) A diagnosis without a plan is a dead end. Once IRIS identifies a risk (e.g., high cardiovascular risk), the AURA Orchestrator (our master AI router) seamlessly delegates the user to a network of specialized AI agents:

APOLLO (Virtual Doctor): Provides immediate symptom consultation, medical triage, and actionable first-aid guidance.
NORA (Dietary Expert): Instantly generates personalized nutritional advice, caloric tracking, and diet plans tailored to the new diagnosis.
SAGE (Wellbeing Counsellor): Offers emotional support and mental health counseling using Cognitive Behavioral Therapy (CBT) protocols to help users cope with health anxiety.
PRISM (Diagnostic Assistant): Synthesizes uploaded lab reports and clinical documentation to provide a unified health view.
COMPASS (Insurance Guide): An Agentic RAG system that actively retrieves live US health insurance plans, ACA Marketplace data, and geo-localized premiums to help users afford the care they need.

How we built it

AURA is a highly modular, production-ready platform built on a robust modern tech stack.

Frontend: We built a seamless, app-like user experience using React 18, TypeScript, Vite, and a custom MNC-grade Vanilla CSS architecture.
Backend: The core logic is powered by Python 3.11 and FastAPI, providing high-performance asynchronous API endpoints.
Database Layer: We used SQLite for relational data (user profiles, chat threads) and ChromaDB for local, persistent vector memory per user, enabling personalized Retrieval-Augmented Generation (RAG).
AI Orchestration: We utilized LangGraph to build the complex state machines required for multi-agent routing. The AURA Orchestrator acts as the central brain, analyzing user intent and passing context (state) to the specialized sub-agents.
LLM Engine: We leveraged Google Gemini 1.5 Pro for complex reasoning, vision tasks (IRIS), and empathetic counseling (SAGE), while using Gemini 1.5 Flash for low-latency tasks like routing, dietary calculations (NORA), and query formulation.
Oculomics (IRIS): Instead of traditional CNNs, we utilized state-of-the-art Vision Transformers (ViTs). By dividing the retinal image into patches and applying self-attention mechanisms, the model learns complex global dependencies between microvascular abnormalities and systemic biomarkers, trained against datasets like mBRSET and MIMIC-CXR.
External Integration: COMPASS proxies our custom, externally deployed Hugging Face API node to fetch live insurance data, demonstrating robust external service integration within the LangGraph flow.

Challenges we ran into

Building a clinical-grade, multi-agent system came with significant hurdles:

Oculomics Data Imbalance: Training models to correlate retinal pixels with systemic diseases like AMI is notoriously difficult due to class imbalances in medical datasets. We had to implement custom weighted loss functions to ensure the model didn't simply predict the majority class.
Multi-Agent State Management (The "Amnesia" Problem): Initially, our agents would lose the context of the user's medical history when switching tasks (e.g., moving from the Doctor to the Dietitian). Implementing LangGraph was crucial. It allowed us to define strict state schemas (TypedDict) and conditional edges, ensuring the Orchestrator could gracefully pass memory and context between agents.
Latency vs. Reasoning: Running complex medical reasoning takes time. We had to carefully architect our routing so that simple tasks utilized the blazing-fast Gemini 1.5 Flash, while deep diagnostic synthesis was routed to the more powerful Gemini 1.5 Pro, optimizing the user experience.
RAG Hallucinations in Healthcare: For COMPASS (Insurance) and PRISM (Diagnostics), hallucinated information is dangerous. We had to build rigorous "Retrieve & Grade" nodes within our LangGraph flow to ensure the agents only answered based on retrieved, factual documents, falling back to web search (Tavily/DuckDuckGo) only when necessary.

Accomplishments that we're proud of

The IRIS ViT Implementation: Successfully proving that a single retinal scan can predict 10 distinct systemic and demographic factors using Vision Transformers is a massive technical leap. We turned a smartphone into a holistic diagnostic tool.
Seamless Multi-Agent Orchestration: We are incredibly proud of the AURA Orchestrator. Watching it seamlessly parse a complex user query ("I just found out I have high blood pressure, what should I eat and how do I pay for meds?") and route it simultaneously to NORA (Diet) and COMPASS (Insurance) feels like magic.
Production-Ready Architecture: We didn't just build a prototype script; we built a scalable, modular architecture with FastAPI, persistent vector databases, and a beautiful React frontend that is ready for real-world deployment.

What we learned

The Power of Agentic Workflows: We learned that LLMs are exponentially more powerful when constrained to specific "personas" and given distinct tools, rather than acting as a single omniscient chatbot. LangGraph was a game-changer for this.
Multimodal AI is the Future of MedTech: Combining text (clinical notes), vision (retinal scans), and structured data (insurance APIs) into a single cohesive pipeline proved that AI can replicate the multidisciplinary approach of a real hospital.
Explainability is Non-Negotiable: In healthcare, a "black box" prediction isn't enough. We learned how to structure our prompts and model outputs so that every agent, from IRIS to APOLLO, provides the reasoning behind its guidance, building crucial user trust.

What's next for AURA: Intelligent Health Network for Oculomics

We view AURA as the foundational operating system for personal health. Our next steps include:

Mobile Native App: Porting the React frontend to React Native to utilize native smartphone macro-lenses for even higher fidelity retinal captures.
Wearable Integration: Connecting AURA to Apple Health and Google Fit APIs so agents like NORA and APOLLO have real-time continuous biometric data (heart rate, sleep, continuous glucose monitors) to supplement the IRIS scans.
Clinical Validation: Partnering with local clinics to run shadow trials, comparing IRIS's systemic predictions against traditional blood panel results in real-world patient populations.
Expanding the Agent Network: Adding specialized agents for specific chronic conditions (e.g., an Oncology Navigator or a specialized Diabetes Manager) to further personalize the care umbrella.