Orby for Alexa+

An Alexa+ add-on that remembers, then follows up once

Comment

Inspiration

Most of what we forget isn't a task. It's a thing someone mentioned: a birthday next week, a desk you might buy, a landlord who hasn't written back, slides a coworker owes you. Voice assistants are great at the moment you speak and bad at the day after. Orby is a planner we already had as a phone app and a text-message thread. It remembers what you tell it and brings it back at the right time. For this hackathon we asked what Orby would look like as an Alexa+ add-on on a home display: a memory and a set of follow-ups for one person, wherever they are in the house.

What it does

You talk to it like a friend who happens to be organized. Real life is small things: a bill, an errand, an email you're waiting on.

  • "Remember my sister Maya's birthday is Saturday." Orby keeps it and says it will bring it up Friday, so you have time to get something.
  • "Remind me to call the bank Thursday at 10." A reminder, read back in plain words.
  • "Rent is due on the 1st. Remind me the day before." Orby sets it for Saturday, October 31 at 9 AM.
  • "I emailed the landlord about the leaky faucet. If he hasn't replied by Wednesday, remind me to follow up." On Wednesday at 9 AM Orby opens with it, on its own.
  • "I saw a standing desk, I might buy it." That isn't a task yet, so Orby keeps it as an open loop.

The next morning, in a brand-new session, Orby speaks first: "Morning, Jordan. You said you might get that standing desk. Still on your radar, or should I drop it?" Say "drop it" (or tap Drop it on the card) and it leaves the follow-ups but stays in memory if you ask.

"What do I have this week?" gets the whole week in one breath, with a "Your week" card on screen. "What about next week?" always checks the plan; Orby never guesses a schedule. Change your mind out loud ("actually, make it 11", "not Tuesday, make that Wednesday") and Orby changes the reminder you already have and keeps everything else. Say "Thanks!" and you get a short warm answer, not a reading of your day. The cards fit the device's safe areas and screen height.

It follows up once, never twice. It holds nudges during quiet hours, keeps to a small daily budget, and never brings up something you muted. Reminders you asked for always fire. It never saves its own words as a memory, and never saves what you told it not to ("don't write that down").

On a screen, answers come with cards: your day or week, a reminder, a memory, a check-in. The cards are an MCP App served by Orby's own MCP server, and their buttons call Orby's tools.

How we built it

  • The add-on is a self-hosted MCP server. Nine tools over Streamable HTTP: remember, recall, forget, add_reminder, complete_reminder, whats_on_my_day, check_nudges, answer_nudge, mute_nudges. It speaks protocol 2025-11-25 and also accepts 2025-03-26, the version in Amazon's own client sample. State lives in SQLite. No model runs inside a tool, so tools answer in a few milliseconds, far inside the 500 ms budget Alexa+ asks for.
  • Account linking the way Alexa+ expects it. An OAuth 2.1 authorization server with PKCE (S256 only): client credentials for service calls like tools/list, and authorization code for user tools. Protected-resource and authorization-server metadata are published, and tokens are stored only as hashes.
  • The proactive nudge engine decides what to bring up and when: open loops later, events the day before, reminders on time, follow-ups that wait on a reply. "Never twice" is a unique key in the database.
  • The assistant brain runs on Amazon Bedrock. Amazon Nova Lite, through the Strands Agents SDK, picks and chains Orby's tools over a real MCP client connection, the same way Alexa+ would call an add-on. Nova Micro is the cheaper fallback. In run 20 of our live eval, the median turn took 2.2 seconds end to end, tool calls included, and 9 in 10 finished within 3.4 seconds.
  • Guardrails we'd want in production. What Orby says comes from the tools' own replies, so dates and facts can't drift. For answers to a check-in, a small rule-based router knows the right tool and arguments and the model can't swap them. Data-changing tools only run when the person's words ask for them. A Nova-only model registry with a pinned default makes a silent model change fail the tests, and a spend breaker checks the budget before every model call. The whole build, with every model call metered, cost $1.73 of the $2 cap on Amazon Bedrock.
  • A simulated Alexa+ experience. The Alexa+ MCP toolkit is limited to select partners, so we built a smart-display simulation in the browser: voice in and out (Web Speech API), a notification light when Orby has something for you, cards, and a "How it worked" drawer that shows every MCP tool call and the Nova model behind it. A demo clock shows days passing in a three-minute video. This is a hackathon build, not a published add-on, and we have not connected it to a real Alexa+ device.
  • Tested against real Nova, not just mocks. A 51-phrase live eval runs everyday wording through the whole stack on Nova Lite, 21 runs in all. The first run passed 33 of 49 judged cases and showed Nova changing data nobody asked it to change; after the fixes, twenty more runs passed 94 to 100% of the turns we could judge, and the final build passed 51 of 51. Then we wrote four sets of 30 new phrasings that we never tuned on and ran each once: 20 of 30, 26 of 30, and, on the second night, 17 of 29 judged and 18 of 29 judged. Those last two are the honest number for wording we have never seen: about 60%. Later runs of the same sets (27 to 29 of 30) are labeled "after fixes" and are not held out. We publish every run in the repo.
  • A security review of the server, OAuth and the cards, with every finding fixed: the display refuses requests from other sites, the card widget can't reach the network, and replayed OAuth codes revoke what they issued. A second review pass on the correction, week and bill features found and fixed edge cases (daylight saving time, stale alerts, recall with your own name).
  • Checked against Amazon's Alexa+ documentation. We went through the Alexa+ MCP documentation line by line: 110 requirements, with a test for each one we could check locally. 71 are met, 13 partly, 1 not met, and 25 need the real Alexa+ host, which we say plainly instead of faking.
  • 2,544 automated tests (2,541 pass, 3 mark open Alexa+ gaps as to-do) across the nudge rules, time parsing in four time zones, the MCP server, OAuth, the cards, the guards and the full demo flow over HTTP.

Challenges we ran into

  • Bedrock's Converse API needs the conversation to start with the user. A session where the assistant speaks first (the whole point of a proactive add-on) broke every later call until we anchored the history.
  • Nova Lite sometimes reached for the wrong tool with a clear phrase: it called forget for "drop it" and once made up an id. Our live eval found more: "yes please" marked a reminder done, and questions saved new memories. We now let clear wording pick the tool for answers to a check-in, and data-changing tools only run when the words ask for them.
  • Tool calling on Nova Lite varies from run to run. On the second night we saw it pass the whole question as the search words, put the user's own name inside a search, call recall with nothing, read the day on small talk, answer "keep going" or "what about next week?" from memory instead of calling the tool (once with an invented schedule), save its own confirmation as a memory, and save a gate code right after the person said "don't write that down". Each one is now caught by a guard in our code, not by prompting alone.
  • Our own time parsing quietly used the computer's time zone instead of the user's. It looked right on our machine because both were in New York; on a laptop in another zone, "Friday at 6:30pm" landed hours off. The test suite now runs in four time zones.
  • Nova sometimes writes its private reasoning (a thinking block) into the answer. On a voice device that would be read aloud, so replies go through a sanitizer.
  • The MCP spec and Amazon's Alexa+ checklist disagree about one header: the spec wants a 401 to carry WWW-Authenticate, the checklist says to leave it out. We kept both: the default follows the spec, and an Alexa+ profile (ORBY_HOST_PROFILE=alexa) leaves it out.
  • We wanted Nova Sonic for speech, but its streaming API doesn't accept Bedrock API keys, so the demo uses the browser's voice.
  • Getting "follow up once" right is harder than it sounds: quiet hours, budgets and mutes all interact, and each needed its own tests.
  • One known gap, stated plainly: when no check-in is waiting, Nova can still save a vague note from its own narration. The next step is a check that a saved fact uses the person's own words.
  • Our full friction log, with steps, expected versus actual, and suggestions, is in the repo as docs/friction-log.md.

Accomplishments that we're proud of

  • Memory that survives a new session and a new day, shown in the demo video, not just described.
  • A nudge engine with tests for every rule: never twice, quiet hours, budget, mutes.
  • One MCP server serves the tools, the cards and account linking, ready for a real Alexa+ connection if we get access.
  • We kept test sets we never tuned on, and we report their lower scores next to the flattering one.

What we learned

  • Voice output needs different rules from chat: one to three sentences, 12-hour times, no lists read as lists, never "check your screen".
  • Keep models out of tools. Tools stay fast and predictable; the agent decides which ones to call.
  • Test the model with real wording, not the demo script. The demo passed every time; 51 everyday phrasings didn't, until we fixed what they found.
  • Keep a test set you never tune on, and write a new one when the old one has been used. Our final main-set build passed 51 of 51; a fresh set of 30 phrasings scored 17 of 29 judged. Even one of our own fixes backfired: asking Nova again with the tool forced, when it skipped a clearly requested one, forced a reminder the person had said not to set. Only a test on new wording caught it.
  • Only offer what a tool can do next. An early version offered "Want me to nudge you at 9:45?" with no tool behind it, and a "yes" went nowhere.

What's next for Orby for Alexa+

  • Connect to a real Alexa+ account once the toolkit is open to us, using the OAuth server we already built.
  • Link the add-on to a user's existing Orby memory, with their permission.
  • Recurring routines and a morning brief on the display.

What was new during the submission period

Orby existed before as a phone app and a text-message planner. This entry is a new repository whose first commit is dated October 18, 2026, inside the submission period (August 31 to October 23, 2026). The MCP add-on, the nudge engine, the Bedrock agent brain, the cards and the simulated display are all new work. Product feedback and our friction log are in their own form fields and in the repo.

Built With

  • alexa+
  • amazon-bedrock
  • amazon-nova
  • amazon-nova-lite
  • amazon-nova-micro
  • mcp-apps
  • model-context-protocol
  • node.js
  • oauth-2.1
  • sqlite
  • strands-agents-sdk
  • streamable-http
  • typescript
  • web-speech-api
Share this project:

Updates

Submission history