Medica

Inspiration

An emergency call is only the beginning.

When someone suddenly collapses, gets seriously injured, has trouble breathing, or experiences another medical emergency, the person nearby can call emergency services. The call goes through. Help is on the way.

But then there is the waiting.

The person standing beside the patient may not be a doctor, paramedic, or trained medical professional. They may be a family member, bystander, teacher, security guard, police officer, first-aid responder, or simply someone who happened to be there.

They may have a phone in their hand, but they may not know what information matters, what they should be observing, or what guidance is relevant to the situation.

That gap between calling for professional help and professional help arriving is where we saw an opportunity for AI.

We started with a simple question:

What if the phone could provide useful medical decision support during those minutes?

But there was another problem.

What happens when there is no internet?

An emergency can happen anywhere. Connectivity can be weak, unavailable, overloaded, or completely absent.

So we didn't want Medica to be an online AI assistant with an "offline mode" added later.

We wanted to build something that could actually carry its own medical intelligence.

That led to the idea of Medica: an AI-assisted medical decision-support system combining multimodal AI, on-device intelligence, local retrieval, and a vast Medical Knowledge Vault.

Medica isn't designed to replace emergency services, doctors, paramedics, or other professionals.

It is designed to help the person who is already there.

---

What it does

Medica allows a responder to build a picture of an emergency using multiple forms of information.

They can provide:

• Photos
• Video
• Audio
• Text
• Observations
• Additional context about the situation

Medica then uses that information to provide structured medical decision support.

The system can identify relevant observations, surface possible explanations to consider, retrieve supporting medical resources, communicate uncertainty, highlight important warnings, and provide context-aware guidance for the situation.

The goal isn't to give a definitive diagnosis.

The goal is to help the person at the scene make better-informed decisions while professional help is being arranged or while they wait for it to arrive.

Online

When internet connectivity is available, Medica can use cloud-based AI and online capabilities.

Responder
    ↓
Photo / Video / Audio / Context
    ↓
Cloud AI
    ↓
Medical reasoning + retrieval
    ↓
Structured result + evidence

Offline

When there is no internet, the entire intelligence pipeline can move onto the phone.

Responder
    ↓
Photo / Video / Audio / Context
    ↓
On-device AI
    ↓
Local Retrieval
    ↓
Medical Knowledge Vault
    ↓
Relevant evidence
    ↓
Structured result

This is where the Medical Knowledge Vault becomes central to Medica.

The Medical Knowledge Vault

We don't want the offline system to depend solely on whatever information happens to be stored inside the model.

Instead, Medica is designed around a large, structured and continuously expandable local medical resource library.

The Vault can contain:

• Medical protocols
• Clinical reference documents
• Medical photographs
• Anatomical images
• Diagrams
• Emergency flowcharts
• Decision trees
• Training videos
• Audio resources
• Structured medical information
• Other curated reference material

And these aren't simply dumped into a folder.

Every resource can contain metadata such as:

• Source
• Version
• Publication date
• Review date
• Category
• Tags
• Description
• Relationships to other resources
• Local location
• Reference information

This makes the Vault searchable and retrievable.

A video, for example, can be represented through its transcript, timestamps, keyframes and searchable segments rather than requiring the entire video to be processed every time.

A flowchart can exist both as its original visual representation and as structured data containing nodes, questions, branches and actions.

Images, documents, protocols and other resources can also be connected to related material.

The result is intended to be a multimodal medical knowledge layer, not simply an offline PDF database.

---

How we built it

Medica is designed as a modular system where the AI model is only one part of the architecture.

At the top is the multimodal input layer, which collects the information available at the scene.

That information goes through the AI and retrieval layers.

The architecture is designed to support both cloud and local execution:

                         MEDICA
                            │
                    Multimodal Input
                            │
             ┌──────────────┴──────────────┐
             │                             │
          ONLINE                         OFFLINE
             │                             │
         Cloud AI                    On-device AI
             │                             │
             │                       Local Retrieval
             │                             │
             │                  Medical Knowledge Vault
             │                             │
             └──────────────┬──────────────┘
                            ↓
                     Relevant Evidence
                            ↓
                    Structured Analysis
                            ↓
                    Human Confirmation

The Knowledge Vault itself is multimodal:

Medical Knowledge Vault
│
├── Protocols
├── Reference Documents
├── Images
├── Diagrams
├── Flowcharts
├── Decision Trees
├── Videos
├── Audio
└── Structured Medical Data

The retrieval system can use factors such as semantic similarity, category, tags, protocol relevance, source, version, review date and relationships between resources.

This means Medica doesn't need to give the AI the entire Vault.

It can retrieve the most relevant evidence for the current case.

We also designed the system around an abstraction between Medica and the underlying AI provider.

That means the application can eventually work with different cloud models and different on-device models depending on the device's capabilities.

The long-term goal is for the core offline experience to require no server for its essential functionality.

The phone should be capable of carrying the AI, the retrieval system, the knowledge base and the case data itself.

---

Challenges we ran into

The biggest challenge was trying to bring something that normally depends on cloud infrastructure onto a mobile device.

A cloud application can rely on large amounts of compute and storage.

An offline mobile system cannot.

The phone has limited:

• RAM
• Storage
• Processing power
• Battery
• Model capacity

And we weren't only trying to fit an AI model onto it.

We also wanted to carry a potentially huge multimodal medical resource library.

That meant we had to think about how information should be stored, indexed, compressed, retrieved and represented.

Another major challenge was multimodality.

A medical image isn't the same as a protocol.

A video isn't the same as a flowchart.

An audio recording isn't the same as a text description.

We needed an architecture where all of these different forms of information could contribute to the same case without treating them as identical data.

The Knowledge Vault introduced another challenge: scale.

If the Vault eventually contains thousands of documents, images, videos, diagrams and protocols, simply searching everything isn't practical.

That made retrieval quality extremely important.

The system needs to find the right information rather than just find information.

We also had to think about evidence and uncertainty.

In a high-stakes domain, an AI shouldn't simply produce a confident-sounding answer.

Medica needs to communicate what it knows, what evidence it used, and where uncertainty remains.

Finally, we had to maintain a clear boundary around the role of the system.

Medica is decision support.

It isn't intended to replace professional medical care or independently make medical decisions.

---

Accomplishments that we're proud of

We're proud that Medica became much more than an AI chatbot for emergencies.

The core idea is an entire medical intelligence stack that can exist on a phone.

We designed both an online and offline architecture rather than treating offline functionality as a fallback.

We're especially proud of the Medical Knowledge Vault.

The idea isn't simply to give an offline model a few medical PDFs.

We designed a foundation for a large, structured and multimodal library containing protocols, documents, images, diagrams, flowcharts, decision trees, videos, audio and other medical resources.

The Vault is designed to be:

Multimodal. Searchable. Versioned. Traceable. Expandable. Available offline.

We're also proud of the way resources can be represented.

A long video can become searchable through its transcript, timestamps and keyframes.

A visual flowchart can become machine-readable.

A medical image can be connected to relevant reference material.

A protocol can be linked to related resources.

This gives the AI something much more useful than a pile of files: a structured knowledge environment.

And perhaps most importantly, we designed Medica around the person at the scene.

The emergency service is still responsible for professional emergency response.

Medica exists in the space between the call and the arrival of that help.

---

What we learned

We learned that building useful AI isn't just about picking the biggest model.

The model is only one component.

A useful system needs:

Model + Context + Retrieval + Knowledge + Evidence + Device + Human

We also learned that offline AI completely changes how you think about software.

When everything runs in the cloud, compute, storage and connectivity can be abstracted away.

When everything needs to work on a phone, every part of the architecture matters.

We learned that a large knowledge base is only useful if you can retrieve the right information from it.

Having thousands of resources doesn't automatically make an AI smarter.

The retrieval system has to understand what is relevant.

We also learned that multimodal knowledge needs multimodal representation.

A video shouldn't just be stored as a video.

A flowchart shouldn't just be stored as an image.

A document shouldn't just be stored as text.

The system needs to understand the structure behind each resource.

And finally, we learned that uncertainty matters.

In a high-stakes application, pretending to know something is worse than acknowledging that the available information isn't enough.

---

What's next for Medica

The next step is turning the architecture into a fully functional on-device system.

We're working toward:

• Efficient multimodal models that can run directly on phones
• A larger curated Medical Knowledge Vault
• Better local semantic retrieval
• Image and diagram retrieval
• Video understanding using transcripts and keyframes
• Structured flowchart reasoning
• Better evidence tracking and provenance
• Automatic resource versioning
• Device capability detection
• Efficient local storage
• Stronger uncertainty handling
• Better offline performance
• Continuous Knowledge Vault updates when connectivity becomes available
• Extensive testing and validation with qualified medical professionals

Longer term, we want Medica's Knowledge Vault to become a living medical knowledge layer.

When connected, the device could receive updated and reviewed resources.

When disconnected, it could continue using the knowledge already stored locally.

The underlying AI model could also evolve independently from the knowledge layer.

A better model can be added.

The retrieval system can improve.

The Vault can grow.

The evidence can be updated.

The architecture remains.

Because the goal of Medica isn't simply to build another medical AI.

It's to explore what happens when AI, a vast multimodal medical knowledge base, and on-device computing come together at the exact moment someone needs useful information.

When an emergency happens, the ambulance is already being called.

The question isn't whether AI can replace it.

It can't.

The question is:

What can technology do for the person standing beside the patient during the minutes before professional help arrives?

That's what we're building with Medica.

( Note: ^This is an android app, and the APK is provided in the GitHub Repository's Readme, with all the guides. ^This app is also highly based on the smartphone's hardware. )

Built With

• android-aicore
• android-room-2.7.0
• cryptographic-sha-256-verification
• gemini-nano
• gemma(google)
• hybrid-dense
• jetpack-compose
• junit-4.13.2
• kotlin-2.2.10
• lexical
• litert
• llama-(meta-ai)
• mistral
• mobilenet-v4
• mvvm
• onnx-mobile
• robolectric-4.16.1
• sqlite-fts4
• whispermobile