E.D.I.T.H.

E.D.I.T.H. sees, hears, remembers, and talks back, reimagining interactive learning through sound and visuals. Built on Vercel, powered by Backboard.io, blueprinted by Omi.

Comment

About the Project Inspiration I grew up watching Iron Man and always wanted my own J.A.R.V.I.S. — an AI that sees me, hears me, talks to me, and actually remembers everything. When I saw the Interactive Learning track at Milpitas Hacks 3, it clicked: kids learn best through sound and visuals, not reading. What if an AI could watch a child play, hear their questions, recognize their family members, and respond with a warm, natural voice — no typing required? That was the spark for E.D.I.T.H. What I Learned This project taught me that integration is harder than isolation. Building one AI feature is easy. Making 7 sponsor technologies work together in a single browser app — that's the real challenge. I learned:

  • Vercel's deployment pipeline — from git push to global CDN in 23 seconds, with environment variable management for API keys
  • Backboard.io's API architecture — threading, persistent memory, and model routing abstract away the complexity of raw LLM calls
  • ElevenLabs' TTS limitations — CORS is allowed but Chrome's autoplay policy requires DOM audio elements, not new Audio()
  • face-api.js + TensorFlow — running 4 ML models simultaneously in-browser is possible, but version conflicts between TF 1.7 and TF 4.22 will crash everything silently
  • Google Apps Script — it's a free, serverless backend but requires careful deployment: "Anyone" access AND new version publishing for code changes to take effect How I Built It The stack is entirely client-side — no backend, no database, no servers. React 19 + Vite 7 + Tailwind v4 form the foundation. Every API call goes directly from the browser: Camera → face-api.js (4 models, ~3.6MB) Mic → Web Speech API → wake word "EDITH" → speech-to-text AI → Backboard.io → OpenRouter → Gemini 2.5 Flash (multimodal) Voice → ElevenLabs "Bella" (eleven_turbo_v2_5) Search → DuckDuckGo + LangSearch (free tier) Email/Calendar → Google Apps Script (2 web apps) Storage → localStorage (offline-first, ~5MB) Hosting → Vercel (auto-deploy from GitHub) The hardest part was the speech recognition lifecycle. Chrome's Web Speech API requires a user gesture to start, fires onend asynchronously, and fails with cryptic "not-allowed" errors if permission is denied. I solved this by using continuous: false with a 150ms restart delay in onend, identity-checked refs to prevent stale closures, and a wake-word state machine that handles both "EDITH what time is it" and "EDITH" pause "what time is it" patterns. Challenges Faced
  • TensorFlow version hell: @tensorflow/tfjs-core@4.22 (from COCO-SSD) conflicted with face-api.js which requires tfjs-core@1.7. The app crashed with "t3 is not a function". Solution: removed COCO-SSD entirely, kept face-api.js.
  • ElevenLabs key permissions: The provided API key lacked voices_read permission, so listing available voices returned 401. Solved by testing voice IDs directly via POST /v1/text-to-speech/{id}.
  • Google Apps Script POST redirect: When deployed with "Anyone" access, GET requests work but POST requests return 302 redirects unless deployed as a new version after adding doPost. The email sender worked perfectly, but the calendar reader silently failed until I realized the deployment version was stale.
  • Chrome autoplay blocking: new Audio(url).play() silently fails without a user gesture. Fixed by using a persistent DOM element created during the first interaction.
  • SpeechRecognition race conditions: rec.onend fires asynchronously after rec.stop(), causing rapid mic toggle flickering. Fixed by nullifying recRef.current BEFORE calling stop(), and using identity checks in all handlers.
  • localStorage rename aftermath: When I renamed the app from EDITH to JARVIS and back, all localStorage keys changed, wiping saved faces, memories, and API keys. Lesson: separate data keys from display names. Built With Vercel, Backboard.io, Omi, ElevenLabs, OpenRouter, Google Apps Script, TensorFlow.js, React, Vite, Tailwind CSS

Built With

  • backboard
  • css
  • elevenlabs
  • face-api.js
  • googleappscript
  • html
  • javascript
  • omi
  • openrouter
  • tensorflow.js
  • vercel
Share this project:

Updates