ByteCoder

Thinking steps ahead: Autonomous multi-action task orchestration for Alexa+

Comment

Inspiration

Modern personal productivity tools and voice assistants suffer from a fundamental design flaw: they treat users like micro-managers. Current voice platforms excel at logging atomic actions—"Alexa, add buy milk to my list" or "Alexa, what's on my calendar at 3 PM?"—but they abandon the user when it comes to the most taxing cognitive task: deciding what to do next. When faced with dozens of overlapping deadlines, fluctuating personal energy levels, and rigid calendar commitments, users experience acute decision fatigue. We built ByteCoder (powered by ActionPilot) to elevate Alexa+ from a passive command-executor into an autonomous situational decision partner. We wanted to create an ambient assistant that doesn't just store tasks, but actively evaluates context, models schedule changes before committing them, and guides the user from vague daily goals to concrete focus blocks.

What it does

ByteCoder acts as an intelligent cognitive orchestration layer for Alexa+, introducing three core capabilities: Next-Best-Action (NBA) Decision Engine: Rather than dumping an overwhelming list of backlog items, ByteCoder evaluates active task vectors and presents a single, explainable recommendation accompanied by a confidence score. Consequence-Aware "What-If" Rescheduling: Rescheduling an item is rarely isolated. ByteCoder runs a shadow simulation on proposed calendar changes to detect fixed-meeting collisions and cascading deadline breaches before confirming the move. Contextual Memory & Learned Habits: By integrating with Amazon Bedrock AgentCore Memory, ByteCoder remembers individual focus habits (such as preferred 90-minute morning deep-work blocks and mandatory 15-minute meeting buffers) to tailor schedules to human cognitive patterns.

How we built it

Large Language Models (LLMs) excel at natural language understanding (NLU), but they frequently hallucinate interval arithmetic, timezone conversions, and non-linear constraint validation. To eliminate scheduling errors, we built ByteCoder using a Two-Tier Cognitive Architecture:

  1. Deterministic Math & Ranking EngineAll interval calculations, dependency trees, and ranking models are processed in pure TypeScript: Multi-Factor Priority Scoring: Priority P(t) for any task t is computed using a composite non-linear weight model: P(t) = w_u.U(t) + w_i.I(t) + w_e.E(t, r) - D(t) Where: U(t) is the non-linear time-to-deadline urgency step function: U(t) = 95 if T_due <= 12h, 85 if T_due <= 24h, 65 if T_due <= 48h, 40 otherwise I(t) represents intrinsic task importance (1<= I <= 10) E(t,r) measures the alignment between the required cognitive energy of task t and the user's current energy curve at time r. D(t) applies a strict dependency penalty (D(t) = 60) if prerequisite tasks remain incomplete. Interval Arithmetic & Buffering: Continuous focus windows are computed by merging calendar events and automatically padding 10-minute buffers b around fixed meetings: FreeSlot = [t_start, t_end] \U{k} [t_k,start -b, t_k,end + b]

  2. Generative NLU & Explainable Synthesis Tier Amazon Bedrock (Claude 3.5 Sonnet): Translates conversational voice utterances into structured tool requests and converts mathematical decision traces into concise, explainable Alexa speech. Model Context Protocol (MCP 2025-11-25): Exposes 13 tools via Streamable HTTP (GET /sse and POST /mcp), allowing Alexa+ to trigger tools autonomously over persistent Server-Sent Events.

  3. AWS Infrastructure & Frontend ObserverAWS CDK v2: Deploys an Application Load Balanced ECS Fargate Service, Amazon DynamoDB tables (equipped with Global Secondary Indexes for status and urgency queries), and CloudWatch metrics.React Web Dashboard: Built with Vite and Tailwind CSS to display real-time execution waterfalls, next-action hero cards, and live JSON-RPC 2.0 messages.

Challenges we ran into

Eliminating Math Hallucinations in Voice AI: Standard LLMs frequently miscalculate time offsets (e.g., mistaking a 45-minute task starting at 2:45 PM for ending at 3:15 PM). We solved this by creating strict boundary separation: Bedrock handles natural language parsing, while our deterministic engine executes all time arithmetic. Maintaining SSE Stream Stability on AWS ALBs: AWS Application Load Balancers drop idle streaming connections after 60 seconds with an HTTP 504 Gateway Timeout. We engineered an automatic 15-second heartbeat ping (event: ping) inside streamableHttp.ts alongside an unauthenticated health check endpoint to maintain stream stability indefinitely. Cascading Ripple-Effect Traversal: Modeling the downstream impact of moving a task required traversing directed acyclic graphs (DAGs) representing task dependencies. We implemented atomic dry-run evaluations to detect if shifting Task A pushes dependent Task B beyond its deadline T_due.

Accomplishments that we're proud of

37 Passing Automated Tests: Engineered a fully tested codebase across 5 test suites covering algorithms, tool JSON schemas, protocol compliance, and multi-turn agent workflows. 100% Protocol Compliance: Implemented full compliance with the latest Model Context Protocol (MCP) Spec 2025-11-25 over Streamable HTTP. Zero-Config Resilient Offline Fallback: Created an architecture that allows developers and hackathon judges to clone, test, and run the complete experience offline using in-memory fallbacks without immediate cloud provisioning. Production-Ready Infrastructure: Synthesized a 650+ line production AWS CloudFormation template via AWS CDK v2 in a single command.

What we learned

Remote MCP is a Game-Changer for Ambient Assistants: Decoupling the tool runtime from the voice platform using the Model Context Protocol (MCP 2025-11-25) allowed us to build, unit-test, and deploy a complex orchestration engine completely independent of proprietary voice hardware. Hybrid Cognitive Architectures are Essential: Combining deterministic TypeScript constraint solving with generative LLM reasoning yields systems that are both naturally conversational and mathematically precise. Observability Matters in Agentic Loops: Building a real-time web observer with a live JSON-RPC inspector proved invaluable for debugging multi-step agent flows.

What's next for ByteCoder

Third-Party Calendar Connectors: Extending our MCP server to proxy tools directly into Google Calendar, Microsoft 365, Asana, and Jira.Biometric & Wearable Integration: Ingesting sleep and recovery metrics (via Apple Health or Whoop) to dynamically adjust energy curves $E(t, \tau)$ based on real-time fatigue.Multi-Agent Meeting Coordination: Enabling autonomous ByteCoder instances to negotiate meeting times directly between users without manual scheduling back-and-forth.

Built With

Share this project:

Updates

Submission history