FlowPilot

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of## Inspiration

Knowledge workers rarely survive a workday with their 9:00 AM schedule intact. Unexpected manager 1-on-1s, urgent production issues, and shifting priorities derail static calendars. At the same time, users do not trust autonomous AI agents that silently overwrite calendar events, drop tasks, or hallucinate time blocks. We built FlowPilot for the Alexa+ Track (Simulated Experience) to show how an ambient, voice-first agentic assistant can proactively orchestrate a person's day while keeping the human strictly in control.

What it does

FlowPilot is a multi-modal, stateful Simulated Alexa+ Personal Day & Task Agent that combines conversational voice/text interaction with visual approval cards:

• Ambient Day Planning: Fits pending tasks (respecting full durations, priority ranks, and hard deadlines) around protected, immutable commitments like Team Standups and Lunch within a 09:00–17:00 workday.
• Dynamic Interruption Replanning: When an unexpected meeting arrives mid-day (e.g., "My manager scheduled a 1-on-1 at 2 PM for 30 minutes — replan around it"), FlowPilot preserves completed/elapsed blocks and immutable commitments while shifting movable work into valid open intervals.
• Voice + Multi-Turn Task Drafts: Users can speak naturally in English or Hindi (Web Speech API + spoken TTS replies). If a request is missing a task duration or target, FlowPilot asks a concise clarifying question, remembers the conversation history, and generates a reviewable Task Change Draft (add, update, remove).
• Strict Human-in-the-Loop Approval & Explainable Diffs: The AI agent can only execute READ and PROPOSE tools—never COMMIT. Every schedule or task change renders a 6-way visual diff (kept, moved, shortened, lengthened, dropped, added) that commits atomically only when the user clicks Confirm.
• Personal Workspaces & Weekday Routines: Supports multi-day navigation and recurring weekday routines that materialize idempotently without invalidating pending reviews.

How we built it

• Backend (FastAPI + SQLite in WAL mode): Engineered a deterministic interval-fitting scheduler (server/engine/scheduler.py), an invariant validator (validator.py), and a persistent block-identity differ (differ.py).
• Bounded Agentic Tool Architecture: Built a 3-tier tool registry (READ, PROPOSE, COMMIT). GeminiProvider (google-genai SDK) is structurally restricted to READ and PROPOSE declarations with JSON Schema validation, bounded single-turn argument self-repair, and HTML-escaped prompt-injection sandboxing.
• Resilient Circuit-Breaker Failover: FallbackProvider wraps the live LLM in a SQLite SAVEPOINT with a 2-strike, 60-second cooldown circuit breaker that seamlessly degrades to DeterministicProvider so the user experience never breaks offline or under API timeouts.
• Frontend (React 19 + TypeScript + Vite): Contract-first SPA using openapi-typescript generated schemas, featuring an occupancy block/timeline visualizer (DayLoad), PlanPreview diff cards, TaskDraftPreview, and VoiceInput push-to-talk controls.

Challenges we ran into

• Preventing AI Over-Privilege: Ensuring voice transcripts could never trick the model into self-approving a proposal required strict architectural separation: commit_schedule and task-draft confirmation are excluded from the model's tool registry and only reachable via version-checked (state_version) HTTP endpoints.
• Optimistic Concurrency & Partial Failures: If a user edits a task while a schedule proposal is pending, committing that proposal would corrupt the calendar. We solved this with monotonic state_version checks (409 STALE_PLAN / REVIEW_PENDING) and SQLite SAVEPOINT rollbacks.

Accomplishments that we're proud of

• Zero-Trust Approval Boundary: 200+ automated backend and integration tests verifying all 7 scheduling invariants, prompt/SQL injection resistance, idempotency on double-confirmation, and atomic rollback on mid-commit faults.
• End-to-End Contract Integrity: Automated OpenAPI-to-TypeScript contract verification (npm run contract:check) ensuring zero drift between FastAPI models and React components.

What we learned

Designing for Alexa+-style ambient experiences requires pairing conversational speed with visual transparency. Users want voice to capture intent effortlessly, but they rely on structured visual diffs before committing changes to their calendar.

What's next for FlowPilot

• Exposing FlowPilot's READ and PROPOSE tool registry as a Streamable HTTP MCP server (spec 2025-11-25+) for native Alexa+ multi-device orchestration.
• Cross-day overflow scheduling when high-priority interruptions exceed single-day capacity.

What we learned

What's next for FlowPilot

Built With

• css3
• fastapi
• google-gemini
• html5
• openapi
• pydantic
• pytest
• python
• react
• sqlite
• typescript
• vite
• web-speech-api