Inspiration

10% of people in the world deal with the struggles of dyslexia on a daily basis. This can make simple tasks from reading a grocery list to reading a medical consent form incredibly difficult, so we sought out to alleviate this problem. We were inspired by extensions like Dyslexia Reader, but we noticed that it wasn't extensible to physical documents. They also lacked integrated TTS features to deliver a seamless experience to dyslexic users. This project brings all those features together along with an AI summary for the best experience.

What it does

Overall, Medlexia helps dyslexic readers who struggle with interpreting medical documents by appealing to various types of dyslexia (readers who prefer certain fonts and people who prefer listening instead of reading). Start by uploading a PDF with some kind of medical document, and Medilexia not only scrapes the text on the PDF to give text more digestible for people with dyslexia based on professional research, but it allows text to speech options as well as AI generated summaries to allow dyslexic reader to navigate the medical world of documents with ease. Once the original text is generated, the user can customize the options to better fit their needs, catering to the various types of dyslexia a user could have.

How we built it

We used Vite.js with React and TypeScript for our frontend to create a fast, responsive user interface. The backend is powered by Flask with SQLAlchemy for database management and Flask-CORS for cross-origin requests. We implemented a REST API with full CRUD operations for recipe management, storing step-by-step instructions as JSON arrays in a SQLite database.

We used Vite.js with React for our front end to create a fast, responsive user interface. The backend is powered by Flask and AWS S3 and Lambda to allow for the implementation of AI for our AI summary feature. For the image scraping we used Pytesseract and we used a library called react-text-to-speech for the text to speech feature.

Challenges we ran into

We had a lot of difficulty linking the backend to an S3 bucket due to permission problems in IAM. We also ran into trouble when trying to fetch the response from our Lambda function back to the Flask backend and then to the frontend.

Accomplishments that we're proud of

Originally, we set out to focus on developing the core components of the app (uploading a file, scraping text, allowing changed format) and only to leave extra features to if we had more time. Because of our collaborative workflow allowing use to divide the workload efficiently, we were not only able to create an app with the core features, but we were able to add additional features like text to speech and AI implementation.

What we learned

Our application is a full-stack web-app and it gave us the opportunity to improve both our frontend and backend skills. We got a better sense of creating the architecture of a React application when it comes to what should be made into a component and how to use props and state to make our app more interactive and dynamic. We also learned how to use Amazon Web Services to increase the robustness of our app through the AI feature.

What's next for Medilexa

After demonstrating our web prototype, Medilexa’s goal is to be ported into mobile apps for accessibility. This can quickly be done with technologies like Ionic Capacitor or React Native. As a mobile app, users would be easily able to scan documents with their phone cameras.

Our already implemented AI summary can be extended into a much more versatile multi-turn conversation feature, using the model’s vision capabilities and understanding to inform users how to interact with forms and allow them to ask more questions.