Patient Agent

Agentic web extension to help seniors and non-English speakers navigate patient portals.

Comment

Inspiration

The inspiration behind the project was to build something convenient for non-English speakers as well as the elderly. First, we wanted to build something simple and clean that was easy for users to use and understand. A lot of applications tend to underestimate how complex it can be to navigate, so we went with the approach of building an extension.

What it does

Our agentic web extension provides the user with translation of the entire website, explanations about complicated topics like specific drugs and lab results, and can talk to the user in their native language. We use a shrinkable sidebar that contains the UI for our web extension, which also opens once the user logs into their portal. The user can then change the language of the website, click on question bubbles to get simplified explanations of medical terminology, and communicate with our agent about any of their medical needs.

How we built it

Patient Guide was built using JavaScript, HTML, and CSS for the entire extension upon Manifest V3 extension platform. We use three main scripts: a content script that reads the page and injects our sidebar UI, a service worker background script that handles the AI calls, and a pop-up for settings. For handling the plain-language explanations, we use our scripts to scrape relevant text from the website and the Anthropic Claude API to send the information and get back jargon-free explanations. Our web extension’s speech capabilities are powered by the browser’s built-in Web Speech API, where we use SpeechRecognition for talking to the assistant and speechSynthesis for reading answers back from our Claude outputs. Our one-click translations are handled through the MyMemory Translation API, the world’s largest translation memory

Challenges we ran into

Challenges we ran into while building our product were thinking of what would be the best way for our application to live in, and the second challenge was building it as an extension, because for the most part, it was all of our first time really trying to understand how extensions work and what the best approach could be to architecting our application. Third was trying to understand how to get some of our functionalities to work, like our Web Speech API to converse with the chatbot, and getting translations using the MyMemory API when switching between languages.

Accomplishments that we're proud of

Getting Web Speech API and Claude to actually hold a conversation together was a genuine good moment for us. Here's a real difference between a chatbot that responds and one that responds fast enough to feel natural, and we got there. We're also proud of the sidebar itself; it took a lot of trial and error to get a content script injecting into someone else's webpage without breaking it, but we ended up with something that feels like it belongs on the page instead of bolted on top of it. Most of us had never built a browser extension before, so just getting the content script, service worker, and popup to all talk to each other was amazing in itself

What we learned

We learned a lot about what it means to develop tools that support the elderly or those who have not adjusted to modern tech, and how to design UI/UX surrounding that market. By putting ourselves in their shoes and approaching the problem with their mindset, we were able to make the necessary changes to our hack to make it more user-friendly for disadvantaged users. Finally, we learned a lot about HIPAA regulations and how to handle user data in order to prevent any forms of data leakage.

What's next for Patient Agent

Our end goal for Patient Agent is to be the OpenClaw for patient portals. We plan on including a Drug-Drug Interaction tool for our agents to allow users to check whether they can take drugs along with their prescribed medicine based on certified medical databases. Additionally, we are working to understand billing documents and to produce plain-language explanations for users. Finally, we plan to add features that allow our agent to directly interact with the website, handling actions such as booking appointments or requesting medication refills based on the patient’s requests.

Built With

Share this project:

Updates