Inspiration According to the World Health Organization, an estimated 1.3 billion people experience significant disability globally. In high-stress medical environments, communication barriers can be a matter of life and death. Deaf or non-verbal patients struggle to communicate symptoms, hard-of-hearing individuals face immense difficulty when doctors wear surgical masks, and marginalized communities often lack immediate access to dermatological triage. We wanted to design a single, unified "health gateway" to open doors for these individuals—but we know that streaming live video to cloud AI models causes massive lag. We were inspired to design AegisHub: a platform aiming to solve these problems instantly without ever transmitting heavy video files over the internet.
What it will do AegisHub is designed to be a multi-modal accessibility dashboard featuring three core tools:
Sign Language Interpreter: Users will sign their symptoms (e.g., "Pain", "Help") into their camera, and the app will translate this into actionable text for healthcare providers in real time.
Visual Lip-Reader / Captioner: A tool to watch a speaker's lips and listen to their voice, mapping phonetic audio to visual lip-mesh structural changes to generate highly accurate, context-aware subtitles.
Derma-Scan Triage: Users will upload a standard JPEG of a skin anomaly. The AI will analyze it and return a plain-language diagnostic summary to help them decide if they need urgent clinical care.
Our Architectural Plan (How we are building it) To avoid latency and network crashing during this hackathon, we are engineering a Split-Processing Architecture.
The Frontend: Praise is building a custom, highly accessible interactive hub using React, Next.js, and Tailwind CSS. We are bypassing heavy video streaming by utilizing browser-native APIs so the client's device extracts only lightweight spatial data (X, Y, Z coordinates and facial meshes) locally.
The Backend: Neche and Timothy are engineering API bridges using Python, FastAPI, and Pydantic validation.
The AI Brain: Scroll is developing our core AI algorithms, including an LSTM (Long Short-Term Memory) network to interpret time-series hand coordinate arrays, and a transfer-learning image classification model for the Derma-Scan.
By keeping the frontend and backend strictly separated, our goal is for the React client to send only tiny arrays of numbers to our Python backend, allowing it to process the math and return text in milliseconds.
Anticipated Challenges We expect our biggest hurdle to be synchronizing continuous sequence data. If a user drops a frame while signing, an LSTM model can easily misclassify the gesture. To combat this, Timothy is actively implementing strict Pydantic validation models on our FastAPI endpoints to clean, validate, and normalize the coordinate matrices before they ever hit Scroll's AI models.
Our Core Objectives Our primary goal for this hackathon is to successfully execute our split-processing architecture. By shifting the computational load of video tracking to the client's browser and only sending coordinate data to the cloud, we aim to reduce our API payload sizes by over 95%. We are also focused on engineering a fully custom, high-contrast user interface from scratch.
What we hope to learn We are using this 23-day sprint to master the strict separation of concerns. By defining the exact JSON data structures on Day 1, our frontend and backend teams aim to work completely independently. Neche and Timothy will be testing everything locally using Postman to ensure our final integration between the Next.js frontend and FastAPI backend is seamless.
What's next for AegisHub Beyond this hackathon, our long-term vision is to expand the Sign Language LSTM model to include regional dialects and a wider vocabulary of complex medical symptoms. We also plan to integrate a Large Language Model (LLM) endpoint to automatically simplify complex medical documents and prescriptions into 5th-grade reading levels for patients with cognitive disabilities.
Built With
- computer-vision
- fastapi
- lstm
- machine-learning
- momen
- numpy
- opencv
- pydantic
- python
- pytorch