Lora - AI Health Assistant

Inspiration

We all have iPhones and we wanted to be able to track and calculate our own wellness or state of health or fitness better, but when opening the Health app, we found it hard to navigate the metrics and find the right ones. Furthermore, raw data like number of steps might not be insightful and people don't know how to interpret them and make meaningful goals out of them. Moreover, we know a lot of people have an Apple Watch, WHOOP, or Oura Ring. These devices provide a large variety of metrics, but in many cases require subscriptions to platforms to really get insights from them.

What it does

Lora is an assistant that works via an agent loop. It's presented as a chat interface and has access to functions that give it access to the user's health data, only the metrics that they've given permission to of course. Lora analyzes your data and provides key insights, answers, and charts via conversation. Lora also has an interactive voice-to-voice mode, where Lora can explain what's going on with your fitness hands-free. Lora also has an interactive dashboard with some top-level insights. Lastly, and possibly most importantly, Lora supports both the Gemini 3 Flash model and local models such as Qwen 3 and Gemma 3. This is key for privacy, as the users' health data never leaves their phone, especially when interacting via text-only.

How we built it

We knew we had to move fast, so we used Expo and React Native for the mobile app. This let us test on our real devices and simulators quickly. For the API, we already wanted to use Gemini for fast yet thoughtful responses, and it worked out great that this was a sponsor track. We used the generation API for Gemini from AI Studio. We also used ElevenLabs' Scribe and Text-to-Speech APIs for a Speech-to-Speech interface when talking to Lora.

Challenges we ran into

We were building our mobile app with React Native and Expo for fast prototyping and since we were comfortable with TypeScript. But when we wanted to integrate local models into the mix, this was much more well supported in Swift via mlx-swift. Thus we needed to find a way to incorporate swift package manager into our expo build pipeline and create a bridge to utilize the services.

We also ran into an issue with making the voice interaction interactive and seamless. It required an async loop to create a VAD sensing algorithm.

Accomplishments that we're proud of

We're proud of being able to incorporate local models effectively. We're also really happy with how the voice mode turned out, and how immersive it is.

What we learned

We learned a lot about how to orchestrate agents. We received a lot of raw data from HealthKit, and there were many variations in the health data present from person to person. We had to dive into the data and extrapolate the key metrics while filtering out duplicates and more. But this experience led to learning the importance of data, and how much better the assistant got as we did this.

What's next for Lora - AI Health Assistant

Support for Google Fit and Android is a big one, since we went with Expo, the mobile app can largely be ported over with additional support needed for the specific fitness platform for the data.

Integration with Healthcare providers and services. This would enable users to interact with their actual clinical data while maintaining privacy just like currently.