Consilium

Inspiration

Every year, many cancer patients receive treatment in community, rural, or under-resourced hospitals where access to a full multidisciplinary board is limited. At major academic centers, complex oncology cases are often reviewed collaboratively by radiologists, surgeons, medical oncologists, radiation oncologists, clinical trial specialists, and care coordinators. That kind of coordinated expertise can be difficult to access elsewhere because of staffing constraints, cost, and time pressures. We built Consilium to help close that gap and make high-level specialist support more widely available.

What it does

Consilium is a multi-agent AI clinical support platform built for complex oncology cases. A clinician can upload a radiology report PDF or MRI scan, and Gemini Vision extracts key findings, tumor characteristics, and relevant clinical details to automatically structure the case. Once the case is prepared, six specialist AI agents review it simultaneously. A radiology agent analyzes the imaging findings, a medical oncology agent proposes likely diagnoses and treatment pathways, a neurosurgery agent evaluates surgical feasibility, a radiation oncology agent outlines planning considerations, a clinical trials agent searches for relevant studies, and a care coordination agent synthesizes everything into one cohesive recommendation. The platform then produces a consensus-style report in under 90 seconds, along with a plain-language patient summary and an exportable clinical PDF.

How we built it

The backend runs on FastAPI and uses six specialist AI agents powered by ASI1-Mini from Fetch.ai. Each agent was designed with domain-specific clinical prompting so that every specialty approaches the case from a different perspective. The agents are registered on Agentverse using the Chat Protocol, which makes them discoverable through ASI:One. We used Gemini 2.0 Flash from Google for MRI image analysis, PDF parsing, and the patient Q&A chatbot. Its multimodal capabilities allowed us to work directly from imaging and reports rather than relying only on typed inputs. The frontend was built in React, TypeScript, and Tailwind CSS, with local JSON persistence for storing case data.

Challenges we ran into

One of the biggest technical blockers was getting uAgents to run locally on Python 3.13, where dependency conflicts caused Chat Protocol verification failures. We solved that by moving execution to Python 3.12 and registering the agents directly on Agentverse. Another challenge was ensuring consistent structured output across six specialist domains. Each agent needed to reason differently while still returning usable JSON, so we spent a significant amount of time refining prompts, building fallback parsing, and validating outputs. We also had to solve frontend issues caused by models occasionally returning nested objects instead of simple strings, which initially created rendering crashes in React.

Accomplishments that we're proud of

We’re proud that we successfully deployed six fully registered Fetch.ai agents that can be discovered through ASI:One and work together as a coordinated panel. We built a functioning Gemini Vision pipeline that extracts clinical data from radiology reports and MRI scans with minimal manual input. We also created a polished interface that feels like a real healthcare product rather than a prototype, along with a PDF export system that generates professional multi-page reports. Another highlight was the patient Q&A chatbot, which can answer questions using the full context of the tumor board discussion.

What we learned

This project taught us that true multi-agent orchestration is far more difficult than it first appears. Getting multiple agents to collaborate, preserve shared context, and generate reliable structured outputs required much more engineering and iteration than expected. We also came away believing that some of the most meaningful uses of AI in healthcare are not about replacing doctors, but about helping more clinicians access specialist-level support regardless of geography or institutional resources.

What's next for Consilium

Our next step is to expand beyond neuro-oncology into additional cancer types and broader multidisciplinary workflows. We also want to integrate with electronic health record systems so recommendations can fit naturally into clinical practice. Over time, we hope to add longitudinal case tracking so clinicians can compare recommendations across visits and monitor how plans evolve. Most importantly, we want to pursue real clinical validation studies to understand how closely Consilium aligns with expert tumor board decisions and where it can provide the most value.

Built With

• agentverse
• fastapi
• fetch.ai
• gemini-api
• python
• react
• tailwind-css
• typescript