Thescribe

Problem

In many communities, conversations naturally code-switch between English and local languages.

In busy clinics, this forces doctors to type notes or rely on memory, increasing the chance of errors. Especially in East Asia, current patient-tracking tools cannot keep up accurately when conversations switch between languages like Mandarin, Taiwanese, and English.

What it does

We present TheScribe, a secure, real-time medical scribe that captures multi-speaker, multi-language patient–doctor conversations and displays a live, speaker-tagged transcript in a web dashboard.

It captures real-time audio in the browser, gracefully handling code-switching to prevent mangled translations.

TheScribe can also generate SOAP summaries to improve clinicians' efficiency when meeting again with patients.

How we built it

We created a concise and intuitive frontend using Streamlit, a Python framework for interactive web apps.

TheScribe utilizes Gladia for low-latency speaker diarization and multilingual transcription.

TheScribe also takes advantage of Fetch.AI's model orchestration for translation and summary using Gemini. Using Fetch.AI allows us to scale with having fine-tuned translation models for niche dialects. Gemini's state-of-the-art translation gives clinicians the choice to summarize the conversation and save it.

Challenges we ran into

As this was our first hackathon submission, finding the correct tech stack that appropriate for our needs led to a lot of wasted time.
- There were many technologies that at first seemed to be useful to us, but then we later found had limitations that led to incomplete prototypes of our idea.
Dependency support for development
- OpenAI's open-source speech recognition model, Whisper, has only been tested on Mac and Linux. This led to problems between teammates during production who used different operating systems, and eventually had to scrap using Whisper in our project.
API Requests
- Although this isn't the first time we have encountered API's, it is the first time we've used API's in an enterprise setting. Thus, being mindful of our API requests, keys, and rates led to some hiccups when prototyping.

Accomplishments that we're proud of

Integrating live transcriptions onto the web
- It turns out transcribing a prerecorded file on the web is a lot easier than real-time transcribing. However, we were eventually able to implement a live translation through Gladia successfully.
Being able to integrate Fetch.ai into our project
- This allows us to add other models in the future (such as finer-tuned custom models) and experiment with and gain experience with new technology.
Getting this far on our first hackathon
- It may not be what we envisioned at first, but to make this much progress with a lot of setbacks gives us confidence for future endeavors.

What we learned

How to read API documentation
Integrating LLMs inside projects
Exposure to the intersection between linguistics and technology.

What's next for TheScribe

Automated SOAP summaries and EHR exports at the end of each session.
Mobile-first apps so rural clinics without desktops can use TheScribe on tablets or phones.
An on-device Offline Mode for areas with poor connectivity.