Grinny

• 

  Bilingual conversations with audio and summary

• 

  Architecture

Inspiration

I recently learned that medical and dental offices are required to use human interpreters to assist patients who are not proficient in English. Considering the recent advances in AI, text to speech, and speech to text technology, this requirement for a human translator seems a bit outdated.

What it does

Grinny is a mobile application and RAG API designed to directly converse with patients who do not speak English. It is designed to collect basic, patient information and chief complaint and then summarize that information at the end for the dentist.

How we built it

The front end is built with Flutter, Gemini, ML Kit, Deepgram, and ElevenLabs. Patients can speak directly into the app where their voice is transcribed into text by a Deepgram model. This is sent to Gemini 2.0 flash to generate a response and responses then converted into speech by a multilingual ElevenLabs model.

At any point, you can choose to summarize the conversation which will use Gemini to create a summary of the chief complaint and then reach out to the RAG system to collect extra context. The RAG system was created with LlamaIndex and an open source dentistry textbook. It uses FastAPI to serve endpoints and can be tested locally using uvicorn.

Challenges we ran into

Deepgram's text-to-speech models only support English at the moment, so I needed to do some quick research to find additional models that are capable of this. Deepgram can also not detect the language when using the streaming API so most of the interactions in the app have to be performed using file reads and writes.

Python is also a miserable language to work with 😅

Accomplishments that we're proud of

This project ties together 5+ different AI models and services and I was able to accomplish it in less than 24 hours.

What we learned

What's next for Grinny

Since Grinny is built on Flutter an obvious next step would be to make it available on iOS and the web.

My RAG system is also built on single dental textbook and is only referenced once to summarize a conversation with a patient. one way to improve this would be to add additional textbooks and sources to make the database more robust.

I plan on integrating the rag service into the conversation more frequently to guide the questions that Grinny asks.

The app only supports English to Spanish interactions but all of the AI services I'm using many more than that. Adding additional languages would make this much more useful.

Built With

• dart
• deepgram
• elevenlabs
• fastapi
• flutter
• gemini
• huggingface
• ml-kit
• ollama
• python