Voice-Native Vitals for Elfie

Inspiration

Digital therapeutics platforms like Elfie do an incredible job of gamifying health management. However, a significant barrier remains: manual data entry. For elderly patients or those with lower tech literacy, typing numbers into an app daily can feel tedious or confusing. We were inspired to build a bridge between the sophisticated Elfie ecosystem and the most natural human interface available: the human voice.

What it does

Aura acts as an intelligent voice layer over the Elfie app. Instead of navigating menus to log daily health metrics, patients simply tap a button and speak naturally.

For example, a user can say:

"I just checked my blood pressure, it's 120 over 80, and my blood sugar is 105. I also took my morning pills."

Aura instantly:

1. Transcribes the spoken audio into text via our ASR module.
2. Analyzes the text to extract specific medical entities.
3. Logs these metrics automatically into the user's Elfie profile to earn their daily gamified rewards.

How we built it

We utilized Alibaba Cloud's Model Studio to host our backend and leverage the Qwen-Max Large Language Model. Qwen handles the heavy lifting of zero-shot entity extraction.

Here is a simplified example of how we format the prompt for Qwen to ensure accurate data extraction:

`python def extract_patient_vitals(user_transcript): # System prompt instructing Qwen to act as a structured data extractor prompt = f""" You are a medical data extraction assistant. Analyze the following patient transcript and extract the vitals. Transcript: "{user_transcript}"

Return the data strictly in JSON format with the following keys:
- blood_pressure
- blood_sugar
- medication_taken (boolean)
"""

response = alibaba_cloud.qwen.generate(
    model="qwen-max", 
    prompt=prompt,
    temperature=0.1 # Low temperature for strict factual extraction
)

return response.json()

`

Challenges we ran into

Our biggest hurdle was ensuring data accuracy. When dealing with healthcare data, an AI hallucination (making up false information) is unacceptable. We spent a significant portion of the build window refining our system prompts to ensure Qwen strictly formatted the data and would trigger a clarifying question to the user if it didn't clearly hear a specific number, rather than guessing.

Accomplishments that we're proud of

We are incredibly proud of achieving Zero-Shot Entity Extraction with near-perfect accuracy using Qwen. This means the AI was able to correctly pull the complex medical data out of highly varied, unstructured voice sentences without needing to be pre-trained on thousands of specific examples beforehand.

What we learned

We learned that the true power of AI in healthcare isn't just in deep data analysis, but in accessibility. By spending time adjusting the "persona" of our Qwen model, we learned how to make an enterprise-grade AI sound like an empathetic, encouraging healthcare companion.

What's next for Aura

• Multilingual Support: Utilizing Qwen's advanced language capabilities to allow users to log vitals in regional Vietnamese dialects.
• Proactive Voice Alerts: Allowing the app to proactively call out to the user to remind them to take their medication if they haven't logged it by a certain time of day.

Built With

• python