ChronoGrid: git for the physical world

Git for the physical world: ask Alexa+ where you left anything, diff your workbench, and lint your home for hazards.

Comment

ChronoGrid

Inspiration

I did my apprenticeship in computer repair shops, fixing printers, motherboards, chargers and power supplies. Several of us shared the same benches, so the screwdriver or multimeter you'd just put down was never where you left it, half-finished jobs got moved, and a hot soldering iron could sit on a bench with nobody watching it. "Who had the…?" was the most-asked question in the shop, and nothing could answer it.

Homes and workshops are full of cameras that record everything and remember nothing. Developers solved "what changed and who changed it" for code decades ago with git. We wanted the same for physical space, with an assistant you can just ask.

What it does

ChronoGrid turns an on-device camera into an append-only ledger of what was put where and when, and exposes it to Alexa+ through a self-hosted MCP server as git-style verbs:

  • git log / blame: "Alexa, where did I leave my mug?" Alexa answers by voice while the screen shows a rewind MCP App looping the 5-second moment it was set down, with a box around it. The D-pad steps through the object's history.
  • git commit / diff: "Commit drawer 2 as clean." Later: "What changed?" A red and green diff shows what went missing, where it went, and what's new.
  • Real-world linter: "Warn me if the soldering iron is on and nobody's at the bench for two minutes." Amazon Bedrock compiles that into a typed rule, shown as a pull request you approve. A deterministic daemon enforces it and slides a hazard card onto the screen while Alexa speaks the warning.
  • On the TV: ChronoGrid TV, a native Fire TV app built with React Native for Vega, follows the conversation. Ask where something is and the TV jumps to the rewind and loops the real clip; approve a rule PR or snooze a hazard with the remote's OK button.

Ambient screen

How we built it

  • Edge (Python): YOLO-World open-vocabulary detection on a laptop CPU and a placement tracker that turns noisy per-frame detections into placed, moved and removed events plus person presence, with stability counting and occlusion checks. Objects are named from a plain word list, so a new item means a new word, not retraining. 5-second H.264 clips with faces blurred are saved locally first and mirrored to a private Amazon S3 bucket (presigned URLs, 7-day expiry).
  • MCP server: the official MCP Python SDK over Streamable HTTP, negotiating spec 2025-11-25. Ten git-verb tools, each returning a deterministic say sentence, plus an MCP App (ui://chronogrid/rewind.html).
  • ChronoLint: a Pydantic rule language whose JSON schema is the propose_lint_rule tool input, so the LLM writes rules but never judges them.
  • Alexa+ brain on AWS: a Strands Agents SDK agent on Amazon Nova 2 Lite, deployed to Amazon Bedrock AgentCore Runtime. It reaches the household's MCP server as a normal MCP client through a tunnel that exposes only /mcp behind a bearer token. AgentCore Memory remembers the household across sessions, and Amazon Polly is the voice. Warm turns take 3–4 s end to end.
  • Ambient screen: a React 10-foot UI with D-pad navigation, push-to-talk, a live MCP tool trace and a minimal MCP Apps host.
  • Fire TV: ChronoGrid TV in React Native for Vega, with D-pad focus, W3C Media clip playback and a live WebSocket link to the server, developed on Windows by running the Vega Virtual Device under WSL2 with KVM.

The seeded day of history exists only so judges can try ChronoGrid without a camera; real use builds the git log from camera events. In the demo video the camera feed is free stock footage (Pexels) replayed through the same edge pipeline; every label, clip and S3 upload you see was produced live by ChronoGrid.

Challenges we ran into

  • Naming hand tools on a live webcam. We first ran YOLO-World on live footage of our own props, and every model size (small, medium, large) gave poor results: only the phone was reliable, pliers and a hammer were called "scissors", and the hammer was never recognised under any wording. Wordier prompts ("red pliers"), the medium model and one label per object helped; the full benchmark is in our friction log (#12).

Webcam benchmark

  • Fire TV on Windows: the Vega Virtual Device needed KVM enabled inside WSL2 by hand, can't load WebView apps (so the TV app was rewritten natively), only repaints under WSLg while the mouse moves, and splits large WebSocket messages.
  • A cloud agent calling into a home safely: the tunnel guard rejects everything except /mcp with the right token.
  • Bedrock access changed mid-build; Nova 2 Lite turned out faster and handled the nested rule schema well.

Accomplishments that we're proud of

  • Voice answers grounded in evidence: every answer cites a ledger row and has a clip behind it.
  • Safety rules written in plain English but enforced deterministically.
  • One MCP server that works with any MCP host, including MCP Apps, and drives both a web screen and a native Fire TV app.
  • A production-shaped Alexa+ brain: serverless on AgentCore with managed long-term memory, answering in 3–4 s.

How this demo video was made

  • Camera feed: free Pexels stock clips replayed through ChronoGrid's unchanged edge pipeline (edge.run --source clip.mp4). Our laptop webcam couldn't reliably name hand tools (benchmark in friction log #12). Every label, 5-second clip, ledger event and S3 upload shown was produced live by ChronoGrid.
  • Alexa+: simulated. Real Alexa+ can't connect to a self-hosted MCP server yet, so a web screen stands in for an Alexa+ ambient screen. The brain is our Strands agent on Amazon Bedrock AgentCore (Nova 2 Lite); the voice is Amazon Polly. Voice input uses the browser's speech recognition; one rule was typed after it heard "soldering iron" as "sovereign union", which the rule pull-request card let us catch before approving.
  • Fire TV: ChronoGrid TV running on the Vega Virtual Device (WSL2), driven by the keyboard and on-screen remote.
  • Editing: takes were recorded separately and joined; loading pauses and mouse wiggles were trimmed. The hazard fired ~2 s after the iron came on because the bench had already been empty for over 20 s.

What we learned

MCP fits ambient assistants well: long-lived state lives in the server and any assistant can borrow it. Keeping the LLM on the language side of a deterministic boundary is what makes a safety feature believable. And open-vocabulary vision is impressive on everyday objects but still needs light, distance and wording care for tools.

What's next for ChronoGrid

Ring camera ingestion, testing ChronoGrid TV on a real Fire TV Stick, multi-camera homes, a caregiver mode with family-approved rules, and better tool recognition (a short fine-tune on the household's own objects, or a Nova second opinion on the object crop when a placement is detected).

Built With

Share this project:

Updates

Submission history