Inspiration
Dermatology wait-lists can exceed 6 – 12 weeks in the UK and US, while anxiety over an unexplained rash sets in within hours.
Our teammate’s younger sister lives with chronic eczema; every flare-up means juggling GP visits, photos sent over WhatsApp, and trial-and-error creams.
When ElevenLabs & Bolt.new challenged us to build something conversational and impactful, we knew we could turn our existing Cognitive Health skin-diagnosis model into a self-serve, privacy-first platform that speaks back to users and routes tricky cases straight to dermatologists.
What it does
- Snap / Upload – Users drop a photo or ≤ 30 s video of their skin concern.
- Instant AI Triage – Our vision model detects 12 common conditions (psoriasis, acne, vitiligo …) and returns a confidence score + first-line care advice in < 5 s.
- Research-Grounded Chatbot – Patients ask follow-up questions; an OpenAI Assistant answers using a curated dermatology knowledge base.
- Daily Symptom Tracker – Push/email reminders collect itch, pain & trigger data; trend charts help spot patterns.
- Doctor Portal – Verified dermatologists review low-confidence cases, see the full media timeline, and launch secure video consults.
- Voice Coaching (ElevenLabs) – Key advice is read aloud for accessibility and reassurance.
How we built it
- Bolt.new Super-Prompt → 5 min scaffold of a monorepo (Next.js 14 + TypeScript + Tailwind).
- AI Inference – Existing Cloud Run endpoint (FastAPI + Torch) wrapped in
/api/diagnose.
Chat Agent – OpenAI Assistants v2 with a Pinecone vector store of peer-reviewed papers.
Realtime & Auth – tRPC + Clerk for role-based access; Firestore for data; Firebase Cloud Messaging for reminders.
Tele-Derm – Twilio Programmable Video embedded in the doctor portal.
Voice – ElevenLabs TTS proxy in/api/voice, streamed back to the PWA.
CI/CD – GitHub Actions → Vercel (front-ends) + Fly.io (tRPC services). All infra spun up in **< 30 hours.
Challenges we ran into
Privacy vs. Speed – Keeping uploads encrypted while hitting sub-5 s latency took careful caching & signed URLs.
Video Frames – Handling 30 fps clips on the edge and picking the “best” frame for inference.
Token Budget – Slimming medical answers to stay within OpenAI context limits without losing nuance.
ElevenLabs Streaming – Syncing audio playback with UI state when users rapidly re-query.
Accomplishments that we’re proud of
- End-to-end demo diagnosing live webcam images accurately 83 % on our test set.
- First time any of us used Bolt.new—generated > 8,000 LOC we could iterate on instead of write from scratch.
- Fully responsive PWA + dark mode + WCAG-AA contrast in one weekend.
- Deployed HIPAA-mode toggle that strips analytics in one env variable.
What we learned
- Generative dev tools can compress weeks of boilerplate into hours, letting us focus on domain logic.
- Voice feedback dramatically improves user trust—testers preferred spoken guidance 3 × over text only.
- Skin images vary wildly; lighting normalization & confidence thresholds are critical for safe triage.
- Small UX touches (progress rings, emoji reactions) keep health tech from feeling clinical and cold.
What’s next for Skin AI
Regulatory Path – Begin ISO 13485 documentation and seek UK CA / FDA SaMD clearance.
Model Expansion – Add melanoma risk-scoring with dermoscopy support.
Stripe Connect – Pay-per-consult to compensate partner dermatologists.
EHR Integration – HL7 / FHIR push so GPs can import AI reports with one click.
Offline Mobile Kit – On-device diagnosis using MediaPipe + GGML for low-bandwidth regions.
Built With
- clerk
- docker
- elevenlabs-text-to-speech-api
- fastapi
- firebase-cloud-messaging
- firestore
- fly.io
- github-actions
- google-cloud
- google-cloud-run
- next.js-14
- openai-assistants-api
- pinecone-vector-db
- prisma
- python
- pytorch
- react
- shadcn/ui
- tailwind-css
- trpc
- twilio-programmable-video
- typescript
- vercel
Log in or sign up for Devpost to join the conversation.