There are 1 million working medical professionals across the US today, performing a variety of disciplines across the industry. However, as of last year, there are only ~300 deaf physicians. If that weren't already a fraction, the slice further shrinks when you compare that to the 11 million deaf adults in the US. We believe that there is an inherent, systemic issue in today's medical education when it comes to providing resources for the deaf. This is especially potent in surgery, during which the surgeon's hands are occupied and can only speak while operating.

Take today's standard surgical residency class. The surgeon is performing with immense focus, dictating all that he / she is doing to the residents in the room furiously taking notes. The deaf resident in this situation is almost entirely excluded from the learnings. Sign language from the surgeon is out of the question, but to make matters worse, everyone's face is covered in a surgical mask so reading lips isn't an option either.

This needs to change. And to make that change, we built DashboardMD.

What it does

DashboardMD is a robust web-app that relies on speech-to-text technologies to make surgeon transcripts immediately available to deaf residents post-surgery. It fluidly presents all surgeries conducted by the surgeon in a dashboard view - each surgery leads to the transcript as well as an array of biometric information collected throughout the session. This has an immense three-fold benefit:

1) Most importantly, this gives deaf residents the resources they wouldn't otherwise have in today's surgical education setting. They can get the visual experience and follow up right after with a breakdown of what the surgeon narrated by the timestamp.

2) This tool disrupts how surgeons organize their previous surgeries. This dashboard would be a great tool to see how they performed in the past, what approaches they took, and how they could iterate on these techniques moving forward. They now have exactly what they said and did in the surgery room, something previously incomparable to the accuracy and ease we can now provide.

3) At a high level, if surgeons around the world used this dashboard, we could create an immense knowledge graph of various surgeons' approaches to their surgeries. This could be a foundational reference tool for surgeons exploring how their peers tackle similar problems and learn from others' techniques. The best part is, because their explanations are all anonymized - it's about the process, not the patient - this should all be easily available without legal concern.

How we built it

We have three components to this app: an iOS client, a backend server, and a React web-app. The iOS client helps record the audio and sends it over to's speech-to-text processing API. When the resulting transcript returns, we pipe that over to the backend server we host using Microsoft Azure. This serves as both our data store and transfer layer to the frontend web-app. This is the front-facing dashboard described earlier, and cleanly displays all surgeries conducted and the respective patient information.

Challenges we ran into | What we learned

On the iOS side, working with audio uploads, file format, and more complicated POST API requests was quite challenging to overcome. It required a lot of researching into the HTTP request protocols and teamwork to resolve but we finally got it! Setting up the server was also quite a challenge as we had learn how to connect to the right ports and use Flask. On the webapp side, connecting the server data to React was an unfamiliar flow and we all worked several hours to pull from our data source with axios (HTTP request framework) and dynamically populate the app.

Accomplishments that we're proud of

We started out with the healthcare space but weren't sure how to progress. After ideating together for two hours, we finally found the problem of accessibility in surgical education and went full force on how to solve this. This was our team's first time building a React app, Flask backend, and iOS + web dual clients! We're super proud of having a functional product at the end of the day and can't wait to share it with all of you!

What's next for DashboardMD

We think DashboardMD has immense potential, especially given the three benefits described earlier. We fully intend on connecting with medical professionals in the industry and seeing how we can go from MVP to real-world users. Accessibility is often an afterthought in most industries, tech included, and we want to reverse that trend. Especially in a sphere like medicine where there are never enough surgeons to match the waiting lists, DashboardMD might just be the tool we need to do just that.

Share this project: