SleepSense AI — Devpost Submission

Inspiration

Sleep is the foundation of health, yet most people have no idea what happens during those 7–8 hours. Fitness trackers give you a number — "72% sleep quality" — with no explanation. EEG labs give you raw data nobody can read. We wanted something in between: a tool that speaks plain English about your sleep, connects your voice and dreams to your biology, and watches over you automatically.

What it does

SleepSense AI is an end-to-end sleep intelligence platform:

• Sleep Stage Classification — upload an EEG recording (.edf) or wearable export (Apple Health XML, Fitbit JSON, Garmin FIT, or CSV) and get a full hypnogram with AASM-compliant metrics: sleep efficiency, REM latency, stage percentages, awakenings, and a quality score
• AI Sleep Report — Claude generates a personalized 2-paragraph interpretation of your results and answers follow-up questions via a live chat assistant
• Voice Health Check — record 20–30 seconds of your voice each morning; we extract 8 acoustic biomarkers (F0, jitter, shimmer, HNR, spectral centroid, RMS energy, ZCR, speaking rate) and score fatigue, stress, and cognitive load — with live transcription as you speak and a sentiment score
• Dream Journal — describe your dream using text or voice; Claude correlates the content with your actual REM timing, period count, and morning voice biomarkers to give you a complete overnight picture
• Autonomous Health Agent — a Fetch AI uAgent runs daily, compares tonight's sleep to your 7-night baseline via z-score, and sends an alert if something's significantly off

How we built it

• Backend: FastAPI with two custom ML models — TinySleepNet (ResNet CNN + BiLSTM, 91.12% MF1 on Sleep-EDF) for EEG and WristSleepNet (1D-CNN + BiLSTM, 89% MF1 on DREAMT) for wearables, both trained from scratch
• AI layer: Claude (claude-sonnet-4-6) for sleep reports, chat, voice interpretation, dream correlation, and sentiment scoring via claude-haiku for speed
• Voice pipeline: scipy + librosa for acoustic feature extraction; Deepgram nova-2 for transcription; Web Speech API for real-time live transcript display
• Agent: Fetch AI uAgents registered on Agentverse with ASI:One discovery tags, running anomaly detection on Redis-persisted session history
• Frontend: Vanilla JS dark-mode SPA with Chart.js — no framework, one HTML file, deploys instantly
• Infrastructure: Railway with Railpack builder, PostgreSQL + Redis add-ons, Sentry for error monitoring, Arize for ML observability

Challenges we ran into

• Railway 512 MB RAM limit — TensorFlow alone is ~450 MB. We had to exclude it from the deploy and serve TF inference via demo mode, routing around the memory ceiling entirely
• pip backtracking — unpinned arize>=7.0.0 triggered 20+ minute dependency resolution loops. Pinning to arize==7.54.0 cut build time to under 2 minutes
• Agentverse registration — the register_chat_agent API expects an HTTP endpoint URL as its second argument, not an agent address. The error message doesn't say this — took deep diving into the SDK source
• Two-branch chaos — GitHub had main (README only) and master (all code) because of an unrelated repo initialization. Force-pushed master:main to reunify before Railway would pick up the real code
• WebM audio decoding — browsers emit Opus-encoded WebM; soundfile can't read it. Added a PyAV fallback decoder for the voice pipeline without breaking WAV/FLAC for non-browser paths

Accomplishments that we're proud of

• Trained two production sleep ML models from scratch on real clinical datasets (Sleep-EDF and DREAMT v2.2.0) in the same hackathon window
• Full-stack acoustic biomarker pipeline — from raw WebM audio to fatigue/stress/cognitive load scores — with zero third-party ML API dependency
• Live transcription in the browser via Web Speech API with no latency and no API cost, with a Deepgram fallback for unsupported browsers
• A real autonomous agent that runs on a 24-hour interval, loads sessions from Redis, runs statistical anomaly detection, and integrates with Agentverse for ASI:One discovery
• Every sponsor integration degrades gracefully — the app runs fully without any external key set

What we learned

• Railway's Railpack builder requires requirements.txt at the repo root — not inside a subdirectory. A one-line fix that cost two hours to diagnose
• Sleep stage classification on wearables is genuinely hard: 14-subject training data caps accuracy at ~34% MF1. More data matters more than better architecture at this scale
• Acoustic biomarkers are surprisingly informative — jitter and shimmer alone meaningfully separate fatigued speech from rested speech without any trained model
• Claude Haiku is fast enough for synchronous sentiment scoring inside an API route without the user feeling latency

What's next for SleepSense AI

• Multi-Night Trend Dashboard — Chart.js trend lines across sessions, Claude weekly narrative, sleep debt tracking
• Sleep Disorder Screener — AASM-based rule engine flagging OSA, insomnia, and circadian rhythm patterns
• Apple Watch / Garmin live sync — real-time streaming from wearable SDK instead of file upload
• Provider dashboard — clinician-facing view with multi-patient session history and risk flags

Built With

• deepgram
• fetch
• sentry