What it does
Home Orchestrator is a local household AI digital-twin prototype for a safe evening arrival. A user can say “I’m heading home,” and the system reads fictional ETA, temperature, device, and inventory state; plans a routine; executes low-risk simulator actions; creates a human approval card for a grocery suggestion; and refuses high-risk door and gas actions.
The problem it solves
Home automation is often framed as a convenience problem, but a useful household AI must coordinate actions with unequal risk. Pre-cooling a room, switching on lights, suggesting food, opening a door, operating gas, and making a purchase should not share the same autonomy level.
How we built it
How it works
The web console talks to a FastAPI backend and a fictional local digital twin. In Live mode, one Home Orchestrator Agent uses GPT-5.6 Terra through the OpenAI Agents SDK and narrow function tools. The app owns local state, tool implementations, deterministic permission checks, SQLite audit records, and Server-Sent Event updates. In Demo mode, the same orchestrator runs deterministically without an API call. In Offline Safe Return, only low-risk local rules are available.
How GPT-5.6 is used
GPT-5.6 Terra is the Live-mode intent and plan layer. The single agent reads local state through function tools, produces a structured Pydantic plan, and invokes the safe arrival playbook. It uses a low reasoning effort for a latency-conscious routine and a stable, privacy-preserving safety identifier. The model never has direct state-write authority.
Permission and safety design
- AUTO: fictional climate, lights, read-only fridge inspection, and no-heat robot preparation.
- CONFIRM: fictional grocery replenishment and other purchase/high-heat/long-term schedule actions.
- DENY: door unlocks, gas, audit deletion, high-value automatic orders, and safety bypasses.
The decisive check is executable application code immediately before each effect. The audit timeline visibly records plans, execution, waiting approvals, refusals, and Offline fallback.
Architecture
The project has one agent, a FastAPI API, simulator adapters, a deterministic permission service, SQLite state and audit tables, SSE events, a native HTML/CSS/JavaScript console, tests, local evals, and Playwright E2E automation. It is deliberately a digital twin, not a claim of real car, robot, payment, or smart-home access.
Challenges
The hard part was not making a chat response sound helpful. It was keeping the plan, tool calls, permission decisions, UI changes, approval state, and audit log consistent—especially when Live mode is unavailable. The prototype therefore keeps a bounded single-agent architecture and a deterministic Offline path.
Accomplishments
- A runnable evening-arrival execution loop with visible backend-driven state changes.
- Hard approval and denial gates, not prompt-only guidance.
- Fictional food-expiry inspection and a non-payment approval flow.
- Deterministic Demo and Offline scenarios, tests, evals, E2E automation, diagrams, and video assets.
What I learned
Useful home AI needs a narrow definition of autonomy. Models are valuable for intent understanding and planning, while product code must own permission boundaries, idempotency, auditability, and safe degradation.
What's next
The next steps are authenticated local profiles, durable multi-process event delivery, per-device safety certification, user-controlled consent for a real integration, and eval-driven improvement of the Live planning path. No real integration should be enabled merely because a model can describe one.
How Codex was used
Codex was used throughout local development: translating the product brief into the FastAPI application, reviewing current OpenAI documentation, implementing the Agents SDK route and deterministic fallback, writing tests/evals/E2E automation, generating diagrams, and preparing these submission materials. Key product and engineering decisions were made by the user.
Log in or sign up for Devpost to join the conversation.