Maestro

House analysis with a focus on elder safety
4 of 10 subagents
visualization of post suggested infrastructural changes of the house

Inspiration

Home is supposed to be the safest place in our lives, but homes remain static while the people inside them keep changing. A baby starts crawling, a parent begins using a walker, or a room suddenly needs to serve an entirely different purpose.

Families are usually left with generic checklists and online advice that cannot see their actual space. We built Maestro to help them understand what needs to change and act on it.

Making recommendations was only half the challenge. Eleven agents needed to interpret the home, exchange findings, find products, verify payments, and recover from failures. A dropped message could leave a family waiting. A repeated message could execute the same action twice.

We needed to understand the entire journey. Sentry became central to how we built, debugged, and improved Maestro.

What it does

Maestro turns a short video or photos of a home into an interactive 3D model that specialized AI agents can analyze.

Users choose elder safety, babyproofing, space transformation, or insurance inventory. Maestro examines the actual layout, including rooms, doors, furniture, obstacles, and spatial relationships. Its agents identify hazards, assess clearances, plan safer walking paths, suggest layouts, and recommend products that fit the available space and budget.

Every recommendation connects to an object and location in the 3D home. Users can see the problem, inspect the evidence, preview a change, and review a product plan through the conversation in ASI:One.

We chose 3D because these are spatial problems. A recommendation becomes more useful when someone can see where a hazard is, whether a proposed change fits, and how it affects the rest of the room.

Behind that experience, Sentry connects the work across the agent network. When a step fails, Maestro's Recovery agent handles the failure and returns a recovery card linked to the relevant Sentry trace.

How we built it

Maestro uses 11 Fetch.ai agents: one Coordinator and ten supporting agents. The Coordinator manages the ASI:One conversation through the Agent Chat Protocol and Interactive Cards. Supporting agents handle space building, four specialist assessments, product search, cart construction, payment verification, insurance document generation, and recovery.

Agents exchange typed messages through Agentverse mailboxes. A FastAPI Space Service uses Anthropic's Claude API to convert uploaded media into structured rooms, doors, objects, and estimated measurements. Base44 provides the dashboard, while our React and Three.js viewer renders the home and is hosted on Vercel. Paper defines the shared design system.

The Product Finder verifies product pages, the Cart Builder applies the user's budget, and Stripe handles test checkout. The Payment Verifier independently checks payment status before the workflow proceeds. The Insurance Claim Filer generates an inventory spreadsheet and cover letter.

Connecting this system required custom Sentry instrumentation. uAgent messages are not HTTP requests, so automatic HTTP tracing alone could not connect the handoffs.

We carried a TraceCtx containing sentry-trace and baggage on every agent message. Each receiving agent continues the originating trace, letting us follow one request through the Coordinator, Space Builder, specialist, Product Finder, Cart, and Payment Verifier.

We instrumented agent invocations, handoffs, tool executions, and LLM calls. Sentry's AI Agents insights expose token usage and cost, while distributed traces distinguish model latency, tool execution, and communication overhead.

Sentry also covers the surrounding application. Session Replay captures viewer interactions, continuous profiling covers the Space Service, and logs and User Feedback connect reported problems to their traces. Cron monitors check every agent's liveness against Agentverse each minute. Metric monitors alert on excessive build duration and token usage, and every deployment is tracked as a Release.

Challenges we ran into

Our hardest problems happened between agents. Sentry made those failures visible and gave us evidence for what to change.

ASI's planner sometimes rewrote requests, repeated old answers, dropped video links, or combined assessments. Slow downstream agents could also push the Coordinator past its response timeout. User Feedback saying "your last reply came back empty" was linked to the corresponding traces. We identified the stalled handoffs and added timeout handling, bounded retries, and controlled fallbacks. Failed replies to ASI:One fell from 33 to zero in our measured test sets.

Space reconstruction needed a different timing strategy from ordinary chat. Traces showed real builds taking nearly 170 seconds, exceeding our original 150-second timeout. We raised the limit to 180 seconds, moved reconstruction into the background, and cached repeated videos to avoid unnecessary Claude calls.

Tracing also exposed duplicate Action agent executions when users re-sent messages. We added idempotency keys and tied payment confirmation to the current order, with each payment confirmable only once. Duplicate executions fell from 18 to zero, and we observed zero duplicate paid actions across 100 retry and replay tests.

Session Replay caught another failure that backend success alone would have missed: hazards with incomplete coordinates crashed the 3D viewer. We added schema validation before those results reached the visualization. Crashes in the affected test cases fell from 12 to zero.

Finally, distributed traces exposed sequential calls and Agentverse overhead. After optimizing those handoffs, median agent-to-agent latency fell from 6.2 seconds to 1.3 seconds. Median end-to-end latency fell from 18.6 seconds to 9.2 seconds, a 51 percent reduction across comparable requests. Full 3D reconstruction has a separate, longer runtime.

Accomplishments that we're proud of

We built a connected experience from home capture to assessment, visualization, product planning, and verified test payment.

We are equally proud that we can explain what happens when that experience breaks. Every agent reports failed steps to Recovery, which attempts bounded recovery or a controlled fallback and returns a user-facing notice linked to its Sentry trace.

Across 250 end-to-end test runs, the workflow achieved a 99.2 percent success rate. Recovery automatically handled 87 percent of injected failures.

Our Sentry project contains 1,284 live traces, including 216 spanning at least three agents, associated with release last-mile-home@8f3a21c. We can inspect how requests cross agent boundaries and connect failures to the code that produced them.

The same action agents support every specialist mode. Elder safety, babyproofing, space transformation, and insurance inventory share a consistent path from findings to the appropriate next action.

Latency figures are medians over comparable requests. Payments use Stripe test mode, and synthetic data tagged environment=test is kept separate from production data.

What we learned

Structured AI output makes a multi-agent system possible. Explicit contracts, state management, and observability make it dependable.

A structured room containing objects, measurements, evidence, and confidence scores can power several agents and interfaces. But a valid message can still arrive late, execute twice, or contain coordinates that crash a viewer.

Sentry helped us connect technical failures to their effect on the user. An empty response became a traceable timeout. A duplicate action became a reproducible state-management bug. A viewer crash became a missing validation rule.

We learned that observability needs to be designed into agent communication from the beginning. Carrying trace context across every message gave us the visibility to improve the complete workflow. Most importantly, we learned to begin with the person. The purpose of the 3D model and agent network is to help a family adapt its home during a moment of change. Reliability is part of delivering that help.

What's next for Maestro

We want to improve reconstruction accuracy through depth estimation, device sensors, and user-confirmed measurements. We also want families to update the same digital home over time instead of rebuilding it for every situation.

We plan to add specialist agents for accessibility, temporary injuries, pets, emergency preparation, and other household changes. Recommendations could eventually connect users with verified contractors, occupational therapists, installers, and local service providers.

As the agent network grows, we will expand our Sentry instrumentation and evaluation coverage alongside it. We want to track recovery outcomes by failure type, compare performance across releases, and catch regressions before they interrupt a family's workflow.