Milo

An autonomous AI coding agent that reasons, plans, and builds.

Comment

Inspiration

Modern software development requires constant context switching: translating vague specifications into phased architectural plans, inspecting codebases, writing boilerplate, executing builds in terminal shells, diagnosing runtime stack traces, and searching documentation. While modern generative AI tools assist with autocomplete and single-file generation, they remain largely passive:

They cannot inspect file hierarchies autonomously. They cannot execute terminal commands or verify test suites. They frequently produce code that breaks runtime dependencies. They require developers to manually copy, paste, debug, and coordinate changes.

We were inspired to build Milo: a fully autonomous, full-stack AI coding partner that operates like a senior engineer. When assigned a project, Milo does not merely output a code snippet—it decomposes requirements into a structured, phased implementation plan, explores the workspace, surgically modifies code, runs commands in an isolated sandbox, searches the live web for up-to-date documentation, and autonomously fixes errors until the job is complete.

What it does

Milo pairs an intelligent reasoning core with a dark-mode IDE interface:

Autonomous ReAct Engine: Milo executes a continuous Reasoning + Acting (ReAct) loop powered by NVIDIA Nemotron open models via the Nebius Token Factory. Dual-Persona System: Seamlessly switches between a read-only Architect/Planner (focused on system design and phased milestones) and an active Engineer/Coder (with full read/write/execution access). 9 Native Tools: File manipulation (read_file, write_file, edit_file), workspace discovery (list_dir, grep_search, find_files), terminal execution (run_command), web intelligence (web_search via Tavily), and state management (task_plan). Real-Time Streaming IDE: Token-by-token generation, collapsible chain-of-thought blocks, live tool-execution feedback, and an interactive task tracker over WebSockets and SSE. ACID Session Persistence: Full workspace and conversation survival backed by SQLite in WAL mode.

How we built it

┌────────────────────────────────────────────────────────────────────────┐ │ React 18 + Vite Frontend │ │ ┌──────────────┐ ┌───────────────────────┐ ┌──────────────────┐ │ │ │ File Explorer│ │ Real-time Chat Stream │ │ Phased Task Plan │ │ │ │ & Sessions │ │ & Collapsible CoT │ │ & Progress Visual│ │ │ └──────────────┘ └───────────────────────┘ └──────────────────┘ │ └───────────────────────────────────┬────────────────────────────────────┘ │ Bidirectional WebSocket / SSE ┌───────────────────────────────────▼────────────────────────────────────┐ │ FastAPI Backend Core │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ │ ReAct Agent Orchestrator │ │ │ │ • Context Compactor • Dynamic System Prompt Builder │ │ │ │ • Dual Personas • Hybrid Provider Demarcation │ │ │ └──────────────────┬─────────────────────────────┬─────────────────┘ │ │ │ │ │ │ ┌──────────────────▼────────┐ ┌────────────────▼─────────────────┐ │ │ │ Nebius Token Factory │ │ Sandboxed Tool Registry │ │ │ │ NVIDIA Nemotron 70B │ │ read, write, edit, grep, find, │ │ │ │ OpenAI-Compatible SDK │ │ run_command, web_search, plan │ │ │ └───────────────────────────┘ └──────────────────────────────────┘ │ └────────────────────────────────────────────────────────────────────────┘

Challenges we ran into

Context Window Compaction & Tool-Call Closure

As the agent generates hundreds of lines of code and inspects large directory trees, history length exceeds token budget L_max: Sum of tau(m) for m in H > L_max.

Naïve truncation often split the conversation between an assistant's tool_calls message and its corresponding role="tool" response message. When serialized to the Nebius/OpenAI endpoint, this triggered validation exceptions.

Solution: We designed an Atomic Tool-Call Closure Algorithm. A split index k is valid if and only if all tool call IDs generated before k are resolved before k, and all tool calls after k are resolved after k: C(H_1:k) = R(H_1:k) AND C(H_k+1:N) = R(H_k+1:N), where C(.) is the set of declared tool_call_ids and R(.) is the set of tool observation responses. The compactor summarizes only closed historical transactions.

Mid-Stream Provider Failures and Token Demarcation

When streaming tokens from LLMs, network drops or upstream provider timeouts can occur after emitting m tokens. If a fallback model begins streaming from the beginning without coordination, client text buffers experience token duplication: T_corrupted = (t_1, t_2, ..., t_m) || (t_1, t_2, ..., t_n).

Solution: We introduced a first-class is_reset flag in our StreamChunk interface. When mid-stream failure triggers fallback, the engine emits a reset demarcation chunk: StreamChunk(is_reset=True). Both downstream WebSocket accumulators and frontend reducers immediately clear their volatile buffers before consuming the clean fallback stream: T_buffer = empty set if chunk.is_reset=True, otherwise T_buffer = T_buffer UNION {chunk.token}.

Cross-Platform Directory Traversal & Shell Sandboxing

Allowing an autonomous agent to execute bash/PowerShell commands and edit files risks catastrophic directory traversal (../..), symbolic link escapes, or destructive operations (rm -rf /, rmdir /s /q).

Solution: Canonical Path Sandboxing: Every path is canonically resolved and checked against the active workspace: Canonical(p) = realpath(p), IsPermitted(p, W) IFF Canonical(p) is a subset of Canonical(W).

Command Blocklists & Confirmation Hooks: Destructive patterns (disk formats, fork bombs, system directory removals) are intercepted before shell invocation, accompanied by an async user confirmation hook.

Accomplishments that we're proud of

100% Pass Rate Across 70+ Adversarial & Stress Tests: We subjected Milo to a rigorous battery of adversarial tests (test_critical_fixes.py, test_browser.py, and test_delegate.py), testing boundary conditions, symlink escapes, simulated process kills, and corrupted frames. Milo achieved a flawless 70/70 passing test record with zero regressions across legacy browser and delegation modules

Mathematical Invariant Guarantees for Context Compaction: Long-running agent conversations frequently crash when sliding-window compaction blindly truncates messages between an LLM's tool call and the environment's observation. We formulated and implemented the Atomic Tool-Call Closure Invariant

A True End-to-End Autonomous Developer Experience: We moved beyond basic chat completion to a cohesive, full-stack IDE experience. Watching Milo dynamically break a single prompt into a multi-phase engineering plan, write complete modular architectures, execute test commands, inspect failures, and repair its own code in real time has been the proudest highlight of our project.

What we learned

Agent Reliability is a State Machine Problem: The raw intelligence of an LLM is only as effective as the boundaries around it. Production-grade coding agents require strict mathematical invariants around message history, schema serialization, and idempotency. Streaming Requires Protocol Discipline: Streaming tool-calls and multi-step reasoning over WebSockets demands clear demarcation events. Separating thought tokens from response tokens in real-time creates an intuitive, transparent developer experience. NVIDIA Nemotron on Nebius Token Factory Delivers High-Precision Tool Calling: Open-weights models like Nemotron-70B exhibit exceptional adherence to JSON tool schemas and complex reasoning when provided with structured persona framing.

What's next for Milo

Multi-Agent Swarm Collaboration: Extending the persona engine into autonomous swarms (e.g., a dedicated Test Runner agent critiquing code written by the Coder agent). Language Server Protocol (LSP) Integration: Incorporating real-time LSP diagnostics directly into Milo’s observation loop for instant linting and type-checking feedback. Interactive In-Browser Diff Approvals: Visual side-by-side git diff review before committing changes to the primary repository branch

Built With

Share this project:

Updates

Submission history