Team Cue

An MCP Powered Semantic Memory Layer For Personalized Assistance, Connected Workflows, And Sensor Informed Elderly Care.

Comment

Inspiration

Alexa+ starts every conversation knowing nothing about you, and its add-ons are consumer services — rides, restaurants, travel. Two everyday needs fall straight through that gap. "Tell the team the demo moved" fails because Alexa doesn't know who "the team" is. "How is Mum doing?" fails because Alexa can't see the home of someone who lives alone. Neither is a model problem; both are a missing integration — and the Model Context Protocol is exactly how an add-on supplies one.

What it does

Cue is an MCP add-on for Alexa+ with three jobs.

  • It remembers. People, groups, projects, preferences, habits and open tasks, per user, with citations. "The team" resolves to #cue-team without being told twice.
  • It acts — with consent. One sentence — "tell the team the demo moved to Thursday and block my evening" — compiles into a multi-app plan (Slack, Google Calendar, Notion), is read back, and runs only after you say yes. Then Cue learns what made it work. Standing watches ("remind me if Arjun hasn't sent the deck by Wednesday") fire once, in the background.
  • It watches over a home. A Unity 6 elder-care simulator — a furnished house, an elderly resident, five virtual motion sensors, room-occupancy tracking, a local care-rule engine and seven scenarios (morning routine, prolonged bathroom occupancy, unexpected inactivity, medication reminder, night-time wandering, sensor failure, caregiver acknowledgment) — streams events to Cue. Ask "How is Margaret doing?" and Alexa answers from them. A quiet house is reported as exactly that — never as a fall — and medication status is only what was entered, never inferred from movement.

Twelve MCP tools, each returning a speech string under ~40 words written to be heard, and a card for an Echo Show screen.

How we built it

  • MCP server: Python, FastAPI and the official MCP SDK over Streamable HTTP; PostgreSQL 16 with pgvector; bearer tokens stored as SHA-256 hashes select the tenant. Plans are persisted before execution, so execute_plan runs exactly what was read aloud.
  • Alexa+ simulator: Next.js on Vercel — a genuine MCP client whose LLM brain reads tools/list and decides what to call, with push-to-talk, spoken replies, Echo-Show-style cards and a live JSON-RPC panel. An access-code gate protects it.
  • Elder-care simulator: Unity 6 (URP). An editor tool generates the whole house from primitives; sensors use cooldowns and movement thresholds so standing still never re-triggers; occupancy is tracked separately from motion; a pure-C# rule engine records auditable evaluations (inputs, thresholds, result, explanation). It runs on the desktop or in the browser (WebGL), offline or connected, and posts through a bounded, retrying queue with stable event ids.
  • AWS: one CDK stack — ECS Fargate behind an ALB, RDS PostgreSQL, API Gateway for HTTPS, Secrets Manager, CloudWatch Logs; Bedrock, Transcribe and Polly behind flags.
  • Quality: 70 server tests, 15 client tests, 78 Unity edit-mode and 10 play-mode tests, and a 40-check smoke test run against the live deployment.

The tool contract, in one line — every reply must fit what a person can comfortably hear:

$$ \text{speech} \le 40 \text{ words}, \qquad \text{latency}_{\text{tool}} \ll 1\,\text{s} $$

Challenges we ran into

  • 421 on every MCP call behind the load balancer: the ALB lower-cases Host, CloudFormation reports it mixed-case, and the SDK compares exactly.
  • A redirect that stole our token: /mcp → /mcp/ behind API Gateway pointed at the origin host, so clients dropped Authorization.
  • An account where Bedrock said "Operation not allowed" to every model and CloudFront required verification — so the deployed stack runs its brain on Groq and gets HTTPS from API Gateway, with Bedrock still the default path in code.
  • Models that announce instead of act — "I'm checking…" with no tool call — fixed with a nudge in the brain and by surfacing home status from the tool the model always calls first.
  • Being honest in a care context: designing rules and speech so that inactivity is an observation, missing data is never mistaken for stillness, and no output ever claims a fall.

Accomplishments that we're proud of

  • The whole chain is live and verified end to end: Unity → AWS → MCP → Alexa on Vercel, over HTTPS, authenticated at every hop.
  • A 40-check smoke test that asserts every tool's output, passing against the deployment.
  • A Unity simulator with real tests: play-mode tests prove the house is walkable and every scenario completes, with zero autopilot obstructions.
  • Security as a feature: hashed tokens, an ingest-only simulator key, an access-code gate that fails closed, production hardening, and secrets that never touch a template, an image or a client build.
  • 23 detailed friction-log entries, each with a reproduction and a suggestion.

What we learned

Tool descriptions and annotations are the real interface: the host's model reads them, not your documentation, so they deserve as much care as the code. Deploy-time configuration (an account restriction, a host-name case, a redirect) causes more outages than logic does. And in a care setting, the right default for what an assistant may say is only what the data supports.

What's next for Team Cue

A real sensor feed behind the same ingestion contract; caregiver notifications through an actual channel; care-plan and medication schedules from an authorised source; CloudFront in front of the simulator and Bedrock back as the default once our account is verified; and — given a sandbox — testing against Alexa+ itself.

AWS Builder Mini Challenge — Which AWS services did you incorporate and how?

Everything below is defined in one AWS CDK (Python) stack (infra/cue_stack.py) and was deployed and tested live in us-east-1.

  • Amazon ECS on AWS Fargate — runs the Cue MCP server container (FastAPI + MCP SDK), one task with a deployment circuit breaker that rolls back a bad image automatically. The same image runs locally in Docker Compose.
  • Elastic Load Balancing (Application Load Balancer) — fronts the Fargate service, health-checks /health, and also serves the Unity WebGL simulator at /simulator/ from the same task.
  • Amazon API Gateway (HTTP API) — the HTTPS front door: TLS on an execute-api domain proxying to the ALB, so the Alexa client (Vercel) and the desktop Unity simulator never send a token in clear text. We added it when our account could not yet create CloudFront.
  • Amazon RDS for PostgreSQL 16 — with the pgvector extension: semantic memory, plans, watches, the audit log and the sensor_events table the simulator writes to. Private subnets, encrypted storage.
  • AWS Secrets Manager — generates and stores the add-on bearer token, the simulator's ingest-only key, the internal watcher secret and third-party keys; injected into the task as ECS secrets, never written into the template or image.
  • Amazon Bedrock — the default LLM (Claude via the Converse API) for plan compilation, fact extraction and diagram generation, and Titan Text Embeddings v2 for semantic memory (1024 dimensions). Our account currently returns "Operation not allowed" for Bedrock, so the live stack is deployed with a CDK flag that switches the LLM to Groq and embeddings to a local provider without changing the schema; the Bedrock path remains the default in code.
  • Amazon Transcribe Streaming — optional speech-to-text for the Alexa simulator: the server signs a short-lived presigned WebSocket URL so the browser streams audio to AWS directly (Vercel can't hold WebSockets).
  • Amazon Polly — optional neural text-to-speech for spoken replies.
  • Amazon VPC — two AZs, one NAT gateway, private subnets for the database and tasks.
  • Amazon ECR — the container image, built and pushed by CDK assets.
  • Amazon CloudWatch Logs — structured JSON logs for the service, one-week retention; we used them to diagnose the Bedrock restriction live.
  • AWS IAM — a least-privilege task role (Bedrock invoke, Transcribe, Polly) and execution role with secret access.
  • AWS CloudFormation — the deployment engine behind CDK; cdk diff before every deploy.
  • Amazon EventBridge (API destinations) — wired to call /internal/run-watches every five minutes when a certificate is supplied; without one, the in-process watcher is used.
Share this project:

Updates

Submission history