Mashwara

Drug Relation Map
Unidentified Speech Ambiguity
Summarized account with transcript

Inspiration

According to the FIP, only around 5 percent of all pharmacies in Pakistan are staffed with a pharmacist, resulting in dangerous drug combinations can arise involuntarily through lack of education and information accessibility. Someone walks in with back pain and buys Panadol. They're already taking Tylenol at home. Both are paracetamol. Neither box is wrong on its own, and nobody at the counter is trained to notice that together they can reach a dose that damages the liver. That's the kind of harm that comes from nobody asking and not from someone being careless.

What it does

Uses artifical intelligence to decipher patient information from non-english speakers and deciphers informal language into patient data, then sends it to a pharmacist page to approve or deny dangerous medication. This helps community volunteers connect Urdu-speaking patients with pharmacist review, with accessibility and human judgment at the center.

How we built it

Implemented ElevenLabs's speech to text to decipher patient conversations in Urdu, and translate it into english. Then used an Google Gemini and a drug combination list to process confirmed dangerous drug combinations and used D3.js to visualize key drug combinations with citations. xAI Speech to text was also used to provide natural follow-up questions to patients in Urdu. Google Gemini also inferred and highlighted possible undisclosed drugs from vague speech that may need to be disclosed, and proposes follow up questions to patients.

Challenges we ran into

Data visualization was a real challenge, as the intention of this was to make pharmacist decision making quicker and clearer. The formatting and design of this graph took a lot of trial and error, and planning. Determining drug identification scope, as we had to balance between practicality inferring too much from vague language.

Accomplishments that we're proud of

We are very proud to use artifical intelligence to identify ambiguous statements from patients, and highlight that to volunteers to clarify possible dangers. Used natural language processing and LLMs to process patient statements in Urdu.

What we learned

The importance of UI/UX in actually creating an app supports philanthropic behavior. Too much friction, and the app becomes too confusing and tedious for volunteers and pharmacists. How to use natural language processing to infer dangerous drug combinations. And the fact that being trusted matters more than just being smart. It's easy to get a language model to sound confident about a drug interaction. Getting it to say "I don't know, ask her this" is harder, and it's the only version a pharmacist would actually use.

What's next for Mashwara

Implement a local AI model to run on machines such that internet connectivity is not required, and patient data is secured. Additionally, the memory is built but not yet switched on since we don't collect a phone number at intake, which is what links a returning patient to her history. Expanding the interaction table is next: only a small share of rows currently carry a citation, and by design an uncited row never produces a flag.

Sources: (https://gpo.fip.org/wp-content/uploads/2023/01/FIP-Global-Report-Pakistan-2020.pdf)