NutriVoice AI

Autonomous, ambient voice nutrition tracker using Amazon Bedrock Converse API and Model Context Protocol (MCP) for zero-friction macro logging.

Comment

Inspiration

Over 60% of people abandon dietary tracking within the first two weeks because manually searching food databases, measuring portions, and tapping through screens creates excessive friction[cite: 2]. Traditional voice skills often make this worse with rigid invocation syntax and fragile one-shot commands[cite: 2].

We built NutriVoice AI to make nutritional awareness truly ambient and hands-free[cite: 2]. By combining the Amazon Bedrock Converse API with the Model Context Protocol (MCP), NutriVoice AI turns natural speech into verified, clinically sound macronutrient tracking without touching a screen[cite: 1, 2].

What it does

NutriVoice AI functions as an autonomous, ambient nutrition companion[cite: 2]:

  • Ambient Wake-Word Ingestion: Operates a continuous background listener that detects "Alexa" and processes natural food inputs without manual button clicks[cite: 2].
  • Deterministic MCP Tool Routing: Translates natural language into strict Pydantic tool calls (log_meal, delete_last_meal, recommend_recipe)[cite: 1].
  • Conversational Retraction (Undo): Allows users to rectify mistakes naturally (e.g., "Alexa, wait, remove that last entry") with instantaneous database rollbacks[cite: 1, 2].
  • Multi-Turn Ambiguity Resolution: Clarifies incomplete food items (crust type, portion size) before committing dummy macros[cite: 2].
  • Proactive Macro Budgeting: Recommends meals based on remaining daily allowances ("Alexa, what can I eat for dinner with 400 calories left?")[cite: 1, 2].
  • Real-Time Visual Telemetry: Companion dashboard streaming audio waveforms, LLM agent thought breakdowns (Intent → Entity → Tool → Database), token counts, and sub-second execution latency[cite: 2].

How we built it

  • Agentic Reasoning & Tool Dispatch: Implemented using the Amazon Bedrock Converse API (bedrock_runtime.converse) powered by Anthropic Claude 3.5 Sonnet and Claude 3 Haiku[cite: 1, 2].
  • Model Context Protocol (MCP): Configured strict tool execution contracts in mcp_server.py, ensuring deterministic data validation before writing to SQLite/DynamoDB[cite: 1].
  • Backend API Gateway: Built with Python and FastAPI, serving endpoints for voice ingestion, meal retrieval, and static asset streaming[cite: 1, 2].
  • Ambient Voice State Machine: Engineered an armed state-machine using the browser's Web Speech API with clean audio-bus unhooking and TTS synchronization to eliminate silent disconnects[cite: 1].
  • Telemetry Dashboard: Built with React, Vite, and Tailwind CSS to surface real-time execution metrics and responsive daily macro progress rings[cite: 1, 2].

Challenges we ran into

  • Continuous Ambient Audio in Web Browsers: Chrome's SpeechRecognition automatically suspends upon silence or TTS playback, often throwing InvalidStateError on rapid restart. We engineered an armed two-stage state machine with a 300ms debounce loop to sustain uninterrupted listening.
  • Sub-Second Voice Latency: Multi-modal analysis with larger LLMs can exceed voice tolerance thresholds. We architected a hybrid routing strategy: routing fast ambient logging turns through Claude 3 Haiku (~300ms) while reserving Claude 3.5 Sonnet for complex multi-ingredient parsing and ambiguity resolution[cite: 1, 2].
  • MCP Schema Binding with Bedrock: Synchronizing Bedrock Converse tool specifications with local MCP servers required strict validation handlers to prevent silent schema drift[cite: 1].

Accomplishments that we're proud of

  • Achieving sub-second end-to-end voice processing latency while keeping tool execution 100% deterministic[cite: 2].
  • Building a truly responsive conversational undo system that eliminates database drift without needing complex form navigation[cite: 1, 2].
  • Creating a real-time developer observability dashboard showcasing agent thought steps, token usage, and execution costs per turn[cite: 2].

What we learned

  • How to leverage the Model Context Protocol (MCP) to decouple LLM agent logic from underlying database schemas and external APIs[cite: 1, 2].
  • Techniques for balancing model latency and context window management across Amazon Bedrock foundation models[cite: 1, 2].
  • Best practices for designing conversational safety guardrails against extreme caloric deficits[cite: 1].

What's next for NutriVoice AI

  • Echo Show Native Widget: Packaging the companion dashboard as a native APL (Alexa Presentation Language) multimodal display experience[cite: 2].
  • Multi-Modal Plate Scanner: Deploying live camera capture to estimate portion weights alongside voice commands via Claude 3.5 Sonnet Vision.
  • Wearable Biometric Sync: Integrating with Apple Health and Google Health Connect to adjust daily macro rings dynamically based on active calorie burn.

Built With

  • amazon-bedrock
  • claude-3-5-sonnet
  • claude-3-haiku
  • fastapi
  • mcp
  • model-context-protocol
  • python
  • react
  • tailwind-css
  • web-speech-api
Share this project:

Updates

Submission history