Inspiration
Getting an autism assessment as an adult can be difficult, expensive, and hard to access. Long term, I want ASD Insight Companion to become a tool that helps people decide whether pursuing a professional autism evaluation might be worth considering.
Camera-based methods already appear in autism research and clinical workflows, but tools that use behavioral or camera information shouldn't overpromise, act like they can diagnose someone, or handle sensitive video footage carelessly.
This project is a proof of concept for that foundation. It uses clear consent, optional on-device camera processing, privacy-preserving feature handling, and honest results. The current build doesn't diagnose autism or give a probability. Instead, it shows how a future clinically validated preliminary screening tool could be built with privacy, accessibility, and transparency from the start.
What it does
ASD Insight Companion is a proof of concept for a future autism screening tool for adults.
The current build guides someone through consent, a short placeholder self-report questionnaire, an optional camera activity, and a summary of what data was available or limited during the session. It doesn't diagnose autism, provide a score, or tell someone whether they're autistic.
Consent is fail-closed, meaning the required statements must be acknowledged before continuing. The camera portion remains separate and optional. The questionnaire shows one question at a time, tracks progress, and is clearly labeled as not clinically validated.
During the optional camera activity, the user completes a quality check and short calibration, then watches a licensed silent social-interaction clip. MediaPipe Face Landmarker runs locally in the browser while the user watches it. This part uses video only and never requests microphone input.
The results page reports session completeness and limitations. If face tracking was unavailable or low quality, the site says so instead of inventing data. Raw video, audio, images, frames, and facial landmarks aren't uploaded or retained by the server. Temporary tracking information is processed locally and cleared after being summarized. The server receives only approved aggregate numeric information, such as tracking duration, tracking ratio, and quality indicators.
How I built it
I built ASD Insight Companion as a local-first web application.
The frontend uses React, TypeScript, and Vite. MediaPipe Face Landmarker runs directly in the browser through self-hosted WebAssembly and model files, so camera frames are processed on the participant's device instead of being sent to a third-party AI service.
Temporary tracking information stays in browser memory and is converted into a small numeric summary before being cleared. The camera stream is stopped when the user finishes, skips, goes back, refreshes, closes the tab, or leaves the activity.
The backend uses FastAPI, Pydantic, and SQLite. Pydantic rejects unexpected fields, which helps prevent accidental raw-media uploads. Sessions are anonymous and don't require accounts.
The frontend and backend share configuration for the stimulus and quality thresholds. The repository also uses gated pull requests, automated tests, CodeQL, and a CI privacy guard that checks for raw-media upload paths.
The demo uses licensed silent Pexels footage by Artem Podrez, with the source, license, non-endorsement statement, and descriptive transcript documented. The project is open source under the MIT License.
Challenges I ran into
The hardest part was building a camera-based experience without treating someone’s video footage like normal application data. Face tracking needed to happen locally in the browser, temporary tracking information needed to stay in memory, and the camera needed to shut down correctly after completion, skips, errors, refreshes, tab closes, and navigation.
Another challenge was staying honest about what the prototype can do. It would've been easy to show a score or autism-likelihood estimate, but the current project hasn't been clinically validated. That's why it reports what data was available and what was limited instead.
I also had to solve the stimulus problem without filming custom footage, since this is currently a proof of concept. I used licensed stock video, documented its rights, removed audio, and created a transcript that accurately describes the scene.
Finally, scope control mattered. The release freeze, amendments, tests, and verification matrix helped keep the project focused on a reliable and defensible prototype that can be built on later.
Accomplishments that I’m proud of
I built a privacy model that can be demonstrated, not just claimed. A judge can open the browser Network panel during the camera activity and see that the server receives numeric data only. No raw video, audio, images, frames, or facial landmarks are uploaded.
I built an on-device AI workflow instead of sending camera footage to a cloud model. MediaPipe runs locally in the browser, with GPU support when available and a CPU fallback when it isn't.
I made skipping a normal part of the experience. Users can decline camera access, skip calibration, or skip the video portion entirely. The app returns an honest partial session instead of pretending unavailable data was measured.
I also built accessibility into the project through keyboard-operable controls, visible focus states, text-based quality feedback, reduced-motion support, descriptive transcripts, and light/dark themes with contrast checks.
Finally, I built in engineering guardrails. There are more than 190 automated tests in the project, including a test confirming that the video portion never requests microphone access. The release matrix also connects verified behaviors to the tests that prove them.
What I learned
Privacy needs to be part of the architecture, not just a policy statement. It has to show up in the API contract, interface, tests, and development process.
I also learned that honest limitations don't necessarily indicate low ambition, as it's more important to be transparent about the current state of the project because that makes the long-term goal more credible.
I also learned that webcam-based tracking has real limitations, as results can vary based on lighting, fidelity, posture, and demographic factors such as skin tone. So, any future clinical versions would need to measure, publish, and address those differences before making any claims.
What's next for ASD Insight Companion
The long-term goal is to build a clinically validated, privacy-conscious autism screening aid for adults to use.
The project would first need to be developed in collaboration with clinicians, researchers, and autistic individuals to create a properly designed and validated stimulus set that's actually effective. The current 10-second licensed clip is only a technical demo, and a practical version would need longer tasks that are proven to cause a response that can indicate someone has autism.
Next, the data pipeline would need clinical validation, which means studying whether self-report responses, task behavior, and camera-derived features can provide useful information without creating misleading conclusions as a result. Any future screening outputs would have to be evaluated against real clinical outcomes across diverse populations and devices too.
A hosted version may eventually be needed for more advanced analysis. If and when that happens, it would need clear consent, secure processing, defined retention limits, deletion procedures, user withdrawal options, and a real compliance program for biometric and health-related data to protect privacy.
Only after these changes will I consider this project truly ready to be used for the goal that I have worked to develop it towards.
Log in or sign up for Devpost to join the conversation.