HEALTHCLICK

Inspiration

Living in a community where healthcare facilities are often congested and geographically spread out, I frequently noticed a dangerous trend: when people fall ill, their immediate instinct is to look up their symptoms online. This generic approach leads to unnecessary panic or, conversely, false reassurance. Furthermore, text searches completely fail when a symptom is highly visual—like an unusual skin rash, an allergic reaction, or an eye infection—which a user cannot easily describe in words. I was inspired to bridge this gap by creating HealthClick, an autonomous, multimodal AI Autopilot Agent designed to give individuals instant, localized clarity and automated routing to professional care when they need it most.

How we built it

I designed HealthClick to operate as an intelligent digital pipeline that transforms raw human inputs into structured, actionable medical navigation. I built the prototype using an agile, high-performance tech stack:

Frontend & Interface: Developed using Streamlit (Python) to spin up a clean, conversational chat interface rapidly.

Multimodal AI Engine: Integrated the Google Gemini 1.5 Flash API. By utilizing its native multi-input processing, the agent ingests text chat, voice audio, and smartphone images simultaneously into a single context window.

Structured Triage JSON: I engineered strict system instructions forcing the underlying LLM to act as a structured data generator, converting messy user descriptions into standardized, parsed JSON arrays.

Autonomous Tool Execution: If the agent flags an urgent condition, it automatically triggers a Python-based geolocation module to cross-reference coordinates with a local medical directory database.

Challenges we ran into

$$P(\text{Emergency} \mid S_{\text{text}}, S_{\text{vision}}) > \lambda$$

Accomplishments that we're proud of

True Multimodal Integration: Successfully building a working prototype that seamlessly interprets text, voice, and imagery simultaneously within a massive hackathon time crunch.

0 to 1 Execution: Taking a conceptual vision and turning it into a tangible, live tool that connects digital AI analysis with real-world physical infrastructure (local hospitals).

Autonomous Agent Architecture: Successfully moving past the simple "chatbot" mold to build a functioning Autopilot Agent that uses conditional tools autonomously to solve a high-stakes problem.

What we learned

Building HealthClick taught me the immense power of the Autopilot Agent design paradigm. I realized that the true value of modern AI isn't just in generating text, but in its ability to orchestrate workflows autonomously. By elegantly connecting computer vision, natural language processing, and location services into a unified user workflow, a single developer can build a high-impact solution capable of steering people out of confusion and driving them directly toward professional medical care.

What's next for HEALTHCLICK

Transition to an Edge Agent: I plan to optimize and compress smaller open-source models (like Llama-3 or Phi) to run directly on low-end smartphone hardware. This will allow HealthClick to operate fully offline in remote, rural areas with zero internet connectivity.

Localized Audio Overlays: Integrating local speech-to-text models to allow users to express their symptoms in regional languages and dialects, breaking down literacy and language barriers completely.

USSD Deployment: Building a lightweight text-based fallback version over USSD channels so that users without smartphones can still utilize the basic text triage and hospital locator services.

Built With

• geopy
• pil
• pillow
• python
• streamlit