Inspiration

"Did the electrician come back like he said?"

A household runs on small spoken promises: "I'll be back Wednesday at 10 to finish the wiring, there's $250 still due." Then he calls: "Can we do Thursday instead, same time?" By Saturday nobody remembers which day was agreed, or whether he showed up.

We already own the devices that know the answer, but each knows only part of it. Ring knows who came to the door and when. Bee heard what was promised. Alexa is how we'd want to ask. None of them can answer, because the answer needs what was promised, what happened, and the fact that the promise changed in between. It's a join across devices and across time. Kept is the memory layer that does that join.

What it does

Kept listens through Bee for promises, tracks them as they get renegotiated, checks them against what the Ring doorbell saw in the promised window, and answers through Alexa+ with receipts: the sentence that was said and the frame that was captured.

Monday 6:40 PM: a promise. Kept creates Electrician · visit · Wed 10–11 AM · $250 due, citing that one sentence.
Tuesday 1:15 PM: it changes. "Thursday instead, same time?" Kept records an update, not a new memory. Wednesday is struck through, and the Tuesday sentence is attached as the reason.
Wednesday passes. Nothing is expected any more, so nothing is flagged.
Thursday 10:12 AM: the world answers. Ring sees motion. Kept fetches the frame, checks only whether a person is present, and marks the promise seen at the door, not yet confirmed who.
Thursday evening: you confirm. One tap: yes, it was him. Now it's kept.
Saturday: you ask. Alexa+ answers: "The electrician was due Thursday at 10 AM, moved from Wednesday in Tuesday's conversation. Someone was at your door from 10:12 to 11:40 on Thursday, and you confirmed it was him. $250 is still due." An Echo Show-style card shows both receipts.

If nobody comes, Kept says not seen, never "broken", because cameras miss visits. A cancelled promise is closed with the cancelling sentence attached. Vague hedges like "I might swing by sometime" are shown to you but not tracked.

How we built it

Memory operations, not summaries. For each batch of new speech, a model sees the currently open promises and must answer through one forced tool call with operations: ADD, UPDATE, CANCEL or CANDIDATE. A reschedule becomes an update to the existing promise, which lets Kept judge against the current time. Our live run used GitHub Models (openai/gpt-4o-mini) through Kept's OpenAI-compatible adapter. Amazon Bedrock Nova, Groq and local Ollama models plug into the same interface.

Code owns the facts. The model only copies the time words ("Thursday instead, same time"). Deterministic code resolves them to exact times in the household's timezone. Promise versions are linked by SUPERSEDES and carry two clocks: when something was said, and when Kept learned it. History is never deleted.

The join is a rule, not a prompt. A door event links to a promise version when its time falls inside that version's window, and only the latest version counts. A small state machine moves each promise through pending → due → evidence found → kept / not seen, plus cancelled and renegotiated.

The model proposes, the rules decide, the human confirms. Nothing that says a person did something becomes true without your tap. Every verdict must link (CITES) to a sentence or a frame, and a verdict without one is rejected.

Bee (primary). A collector runs next to the official Bee CLI on the wearer's machine. It backfills with bee conversations transcript --since, then follows bee stream --json utterance by utterance, because a Bee conversation can stay open for hours. It batches on 4 utterances, 45 seconds of silence or a conversation change, and sends the server only the operations and the one cited sentence. The Bee token and raw transcripts never leave the machine. Bee doesn't label who is speaking, so the promisor is inferred with a confidence score. Below 0.7, the review page asks "who said this?", and the correction feeds back into extraction.

Ring. Webhooks are verified with HMAC-SHA256 over the raw body and deduplicated on request_id. Kept answers within Ring's 5-second limit and does the work afterwards. For the Developer Playground, an event-history poller feeds the same path. Snapshots come from the image-download endpoint. The watermark band is cropped only from the copy sent to the presence check, and the stored, cited frame is the original. There is no face recognition and no license-plate reading: presence in a window plus your confirmation is the whole inference.

Alexa+. Kept has a self-hosted MCP server (official Python SDK, Streamable HTTP) with five tools: list_open_promises, check_promise, get_receipts, confirm_outcome and remind_me. confirm_outcome refuses without an explicit confirm=true, so the assistant can't decide who came. check_promise returns an MCP App card (ui://kept/promise-card). Since the Alexa+ add-on toolkit isn't open to participants, our web simulator plays Alexa+. It's a real MCP client and MCP Apps host, with voice in and out, and it shows the protocol exchange live. Fail-open recall: each tool has an 800 ms budget, so a slow lookup returns a truthful partial answer instead of an error.

Runs anywhere. Every external dependency (Bee, Ring, model, storage, clock) has a live and a replay implementation. make demo runs the whole week with no hardware, accounts or network, and checks the result.

Stack: Python, FastAPI, MCP Python SDK, SQLite (Neo4j optional), OpenCV, GitHub Models / Bedrock, Bee CLI, Ring Partner API.

Challenges we ran into

Bee doesn't label the wearer. Every utterance arrives with an unknown speaker, so "I'll return your drill" (your promise) and "I'll come Thursday" (theirs) look alike. We infer the promisor with a confidence score and ask you when unsure.
Bee conversations stay open for hours, so waiting for a finished conversation wasn't an option. We process speech utterance by utterance, keep a checkpoint, and deduplicate so a restart never doubles anything.
Ring's Partner API serves US devices only, and Playground tokens last about 30 minutes. From India we built on the Playground's event history and recorded every payload, so sessions replay exactly.
Recorded Ring timestamps never fall inside a simulated window. Replay rewrites them onto the demo timeline; without that, the final step silently fails.
No public Alexa+ client to test against. We built a simulator that speaks real MCP and hosts the real MCP App card.
Webhook auth and event-type names differed between Ring's docs and the sample app, and watermark bounds aren't published. Each item is in our friction log.

Accomplishments that we're proud of

Renegotiation done right. On our 27-thread synthetic eval, Kept's rules reach 100% final-verdict accuracy with 0% false "kept". The same rules with versioning turned off drop to 57% accuracy and a 25% false-"kept" rate: they say a promise was kept when it wasn't.
Every answer has receipts. Each spoken sentence traces back to a quoted utterance or a doorbell frame.
Privacy by construction. No identity inference from video, no transcripts stored, and Kept never contacts the other person. Deleting a promise deletes its sentences and frames from disk, and frames expire after 30 days.
Honest failure. Absence of evidence is "not seen", never "broken". Confidence is shown, not hidden.
Fully reproducible. One command runs the whole demo week offline, and CI tests it on every commit.

What we learned

Language models are good at turning messy speech into structure, but bad at deciding facts. Splitting the work between them ("the model proposes, the rules decide, the human confirms") made the system both smarter and safer.
Storing operations against open memories, instead of summaries, is what makes a changed plan easy to handle.
Designing every integration with a replay mode from day one let us build and test without hardware, and made the demo work for anyone.
For a household product, honesty matters more than cleverness. "Not seen" is the truthful answer when a camera misses something.

What's next for KEPT

A real Alexa+ add-on with OAuth account linking, once the toolkit opens.
Wearer voice labelling from Bee, to attribute promises precisely.
More promise types: deliveries, returns ("I'll bring your drill back"), and payments.
Caretaking: adult children checking that a parent's help showed up, and small landlords tracking contractor visits.