Autopilot

🖥️ Autopilot — The AI Automated Software Engineering Platform

"Set the destination. It flies the plane."

🎯 The One-Line Describes It

"Give it a task in plain English. It picks the right agent, runs real code in a sandbox, browses the web with vision, generates artifacts, and records every step into a replayable film strip."

🔥 The Problem

💬 Models reason — but they cannot act.

Chatbots stop at text. No files are written. No code compiles. No pages are visited. No PDFs ship. There is a wall between "the model knows how" and "the computer did it" — and that wall is where every brilliant idea quietly dies.

🧩 The engineering loop is shattered across a dozen tools.

Search engine. Terminal. Headless browser. LaTeX compiler. Chat app. Issue tracker. Each tool lives on its own island, and the context evaporates every time you cross the bridge. Nobody sees the whole loop at once.

🕶️ Black-box agents can't be trusted.

Without a visible record of what an AI actually did — the commands it ran, the pages it visited, the files it changed — there's no way to audit, verify, or replay the work. And a judge can't score what they can't see.

What people say 🙊	What actually happens 🙈
"The AI can code."	It prints code. You copy, paste, and pray.
"The AI can research."	One smooth paragraph. Zero sources. Zero confidence.
"The AI can browse."	It scrapes one page and breaks on the next selector.
"The AI finished the task."	A chat bubble. No proof. No artifact.
"The AI is safe to run."	`rm -rf /` with a smiley face. 😱
"The AI is transparent."	A black box wrapped in a hype video.

💡 The Solution

Autopilot — AI for Automation 🤖

Autopilot is an AI automation platform. Give it a task in plain English, and it performs the entire task without human intervention: runs real code in a sandbox ⚡, browses the web with vision 🌐, controls the full desktop 🖱️, generates artifacts 📄, and records every step into a replayable film strip 🎞️.

It automates software engineering — writing and executing real Node, Python, and Shell code in an isolated sandbox, no copy-paste, no human in the loop 💻. It automates research — planning, searching, deep-diving sources, and producing cited reports unattended 🔬. It automates browser tasks — opening the browser, reading the screen, and clicking through UI flows using vision 🌐. It automates desktop tasks — full computer use: opening apps, navigating OS settings, and controlling the desktop like a human. In the demo, it changed the Firefox default search engine from Google to DuckDuckGo, end to end 🖱️. And it automates documents — compiling raw LaTeX into a finished PDF 📄.

AI is the engine 🧠. A multi-agent orchestrator routes each task to a specialist agent — CodeAct for execution ⚡, DeepResearch for research 🔬. A Vercel AI SDK service streams tool-augmented generations, and a vision-capable MCP browser layer drives real clicks and screenshots. Every tool call is recorded into a replayable timeline 🎞️.

Most AI tools return text. Autopilot performs the task and leaves an auditable trail — every action is real, executed, and replayable 🚀.

🚀 Capability	What it actually does
🧠 Agent Orchestration	Routes every task to a specialist agent (CodeAct, DeepResearch) via `select_agent`, `switch_agent`, and dependency-graph `coordinate_agents`. One brain, three specialists.
⚡ CodeAct Sandbox	Executes real Node.js, Python & Shell in isolated per-language workspaces — with auto `npm`/`pip` installs, dangerous-command blocking, SIGTERM→SIGKILL timeouts, and persistent memory.
🔬 DeepResearch	Plan-and-execute research across DuckDuckGo/Google/Bing: domain filters, source tracking, insight extraction, and cited Markdown / HTML / JSON reports.
🌐 MCP Browser	Vision-capable browser control via Model Context Protocol — click, type, drag, hover, wait, extract, screenshot — with exponential-backoff reconnects.
📄 LaTeX PDF	Compiles raw LaTeX into a downloadable, inline-rendered PDF via `pdflatex`. Zero extra tooling.
🎞️ Replay Engine	Every tool call becomes a film-strip frame. Scrub it. Replay it. Export it as a self-contained HTML file for judges and demos.
🗄️ SQLite Persistence	`bun:sqlite` — sessions, messages, and tool calls survive restarts with cascade deletes and four indexed queries.
📦 One-Binary Deploy	Ships as a single `bun build --compile` executable, a Docker WebTop desktop, or CI-built release binaries.

🥊 What Makes It Different

Dimension	😴 Typical AI tool	🚀 Autopilot
Execution	Returns text about code	Actually runs it and streams stdout/stderr, exit codes, durations
Sandboxing	No sandbox (or "you build one")	Per-language isolation, path-traversal guards, command blacklist, auto deps
Research	One hallucinated paragraph	Structured plan → multi-engine search → deep-dive → cited report
Browser	Fragile selector-scraping	Vision-based MCP browser with backoff reconnect + screenshots
Observability	Opaque black box	Film-strip timeline of every tool call + standalone HTML replay
Deployment	A Node app you fight to keep alive	One compiled binary, Docker desktop, 4-binary auto-built by CI

🏗️ How It Works — Architecture

🧬 The data flow, end to end

The React + Vite + Tailwind frontend talks to an Express + Socket.IO server over REST, SSE, and websockets. AIService wraps the Vercel AI SDK and streams tool-augmented generations. AgentTARS decides which agent handles the work and how subtasks coordinate; the tool registry exposes every capability as a Zod-validated tool the model can call in a loop of up to 100 steps. Every event is persisted to SQLite and mirrored into an in-memory replay timeline that can be exported to HTML.

┌──────────────────────────────────────────────────────────────────────────┐
│                        🖥️  React SPA (Vite + Tailwind)                  │
│        Dashboard · Chat · Film-strip Workspace · Sidebar · Themes        │
└──────────────────────────────┬───────────────────────────────────────────┘
                               │  Socket.IO / SSE / REST
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                        ⚙️  Express + Socket.IO server                    │
│        Helmet CSP · CORS · 10MB JSON · SPA routing · graceful shutdown   │
└──────────────────────────────┬───────────────────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                   🧠  AIService (Vercel AI SDK)                          │
│      Anthropic · OpenAI · Gemini · Ollama · OpenRouter                   │
│      generateText · streamText · maxSteps:100 · attachments (MIME)       │
└──────────────────────────────┬───────────────────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│               🎛️  AgentTARS Orchestration                               │
│     select_agent · list_agents · switch_agent · coordinate_agents        │
│     keyword routing + dependency-graph topological execution order       │
└──────────────────────────────┬───────────────────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                        🧰  Tool Registry (Zod-validated)                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │ ⚡ CodeAct   │  │ 🔬 Deep      │  │ 📄 LaTeX PDF │  │ 📁 FileSystem │  │
│  │ Node/Python  │  │ Research     │  │ pdflatex →   │  │ workspace/    │  │
│  │ + memory     │  │ plan→dive→   │  │ /api/pdf/:n  │  │ path-safe     │  │
│  │              │  │ report       │  │              │  │               │  │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘  │
│  ┌──────────────────────────────────────────────────────────────────┐    │
│  │ 🌐 MCP Browser — Manager: stdio transport, backoff reconnect    │    │
│  │   retry caps, timeouts, health checks, kill-switch              │    │
│  │   → @agent-infra/mcp-server-browser --vision  (Puppeteer/Chrom) │    │
│  └──────────────────────────────────────────────────────────────────┘    │
└──────────────────────────────┬───────────────────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  🗄️  SQLite (sessions · messages · tool_calls)                          │
│  └──►  🎞️  Replay engine (event stream) ──►  self-contained HTML export │
└──────────────────────────────────────────────────────────────────────────┘

🧠 The agent brain: `AgentTARS`

Three archetypes, one coordinator:

Agent	Tools	Best for
AI Assistant (multi_agent)	Everything, dynamically	General tasks, mixed workflows
CodeAct Agent	`node_code`, `python_code`, `shell_code`, `codeact_memory`	Execute / debug / automate
DeepResearch Agent	`search`, `deep_dive`, `research_plan`, `report_generator`, `session_manager`	Research + cited reports

switch_agent preserves conversational context; coordinate_agents builds a dependency graph and executes subtasks in sequential, parallel, or dependency order — even resolving cycles.

⚡ The CodeAct sandbox

CODEACT_WORKSPACE/.codeact/{node,python,shell,output} — per-language isolation, auto-installed dependencies, a dangerous-command blacklist (rm -rf /, dd if=, mkfs, fork bombs, sudo…), timeout escalation (SIGTERM → 5s → SIGKILL), and a JSON key/value persistent memory file the agent reads and writes across turns.

🔬 DeepResearch loop

research_plan (create/update/complete steps) → search (DuckDuckGo with site:/-site: filters, relevance scoring) → deep_dive (visit top sources, extract focus-area insights, dedupe) → report_generator (Markdown/HTML/JSON with full metadata) → session_manager (list/inspect/delete with live stats). Sessions persist to data/research-sessions.json.

🌐 MCP lifecycle manager

MCPManager (singleton) owns the entire connection lifecycle: stdio transport via the AI SDK, exponential-backoff reconnects (capped at 30s), retry attempts, per-server tool-execution timeouts (Promise.race), health checks that flag stale clients, and a config kill-switch. Five management tools (mcp_list_servers, mcp_list_tools, mcp_execute_tool, mcp_health_check, mcp_configure) let the agent introspect and even reconfigure its own tooling.

🎞️ The replay & streaming engine

SocketService records a timestamped event stream per session: user_message, assistant_thinking, message_chunk, tool_call, tool_result, assistant_message. That stream powers the film-strip Workspace panel (21+ typed renderers), the JSON replay API, and the self-contained HTML export — a downloadable timeline with embedded React + Tailwind that opens offline.

🛡️ Security stack

Path-validation guards on every file op → dangerous-command blacklists → Helmet CSP headers → CORS allow-listing → sanitized PDF filenames (403 on escape) → 10MB JSON cap → 10MB file-read cap → start-server.sh feeds API keys via stdin so secrets never appear in ps.

🧗 The Eight Hard Problems

🏗️ Orchestration — One brain, three specialists; keyword routing plus a real dependency-graph coordinator with context preservation.
🛡️ Sandboxing — Real execution without the danger: isolated per-language workspaces, traversal guards, command blacklist, timeout escalation.
🔬 Research — Replaced single-shot answers with a monitored plan: search → deep-dive → report, with sources tracked and deduped.
🔌 MCP lifecycle — stdio transport, exponential-backoff reconnects, retry caps, per-server timeouts, health checks, kill-switch.
🎞️ Replay — Every user message, thinking frame, tool call, and result recorded; exportable as a standalone HTML timeline.
⚡ Streaming — Dual path: Socket.IO for the app, SSE for the API; tokens, tool results, and thinking state arrive live.
🗄️ Persistence — bun:sqlite with three tables, cascade deletes, four indexes; sessions survive restarts.
📦 Deployment — bun build --compile per target, Docker WebTop desktop with Chromium, GitHub Actions 4-binary release matrix on every push.

🎬 See It In Action

Judge-ready prompts — paste one in and watch the film strip fill up:

#	Prompt	What you'll see
1	"Write a Python script that plots a sine wave and saves it as a PNG."	⚡ CodeAct runs real Python, `pip` deps installed on the fly, output streamed
2	"Run this Node.js snippet, print the output, and save the result to memory."	⚡ CodeAct Node + persistent memory across turns
3	"Research the latest MCP spec changes and write a cited report in Markdown."	🔬 Plan → search → deep-dive → cited report, step by step
4	"Open Wikipedia, search for 'Model Context Protocol', extract the summary."	🌐 Vision MCP browser: screenshots, navigation, content extraction
5	"Write a LaTeX CV for a software engineer and compile it to a PDF."	📄 `pdflatex` compiles; PDF renders inline and downloads
6	"Research solid-state batteries, write a Python script computing theoretical energy density, and compile it all into a PDF."	🧠 Full orchestration: research → code → artifact, one request
7	After any session, hit Export Replay.	🎞️ Download a self-contained HTML timeline and open it offline

🛠️ Built With

Frontend: React 18, Vite, Tailwind CSS, Framer Motion, Monaco Editor, Socket.IO client
Backend: Bun, Express, Socket.IO, Helmet, Vercel AI SDK, Zod
Execution: child_process sandboxing, Node.js, Python, Shell
Data: SQLite (bun:sqlite)
Research: DuckDuckGo API, Mozilla Readability, Turndown, JSDOM
Browser: Model Context Protocol, @agent-infra/mcp-server-browser, Puppeteer
Artifacts: pdflatex (LaTeX → PDF)
Deploy: Docker (WebTop), Bun --compile binaries, GitHub Actions
AI Providers: Anthropic, OpenAI, Google Gemini, Ollama, OpenRouter

🤝 AI Tools Disclosure

This project was built with the assistance of Claude Code (Opus 4.7, Claude Max tier) and OpenHands, plus standard AI-assisted development tooling. Runtime AI providers supported: Anthropic, OpenAI, Google Gemini, Ollama, OpenRouter.

🗺️ Roadmap

🛰️ SSE transport for remote MCP servers — URL-based connections with auth headers
🧠 Per-agent memory pipelines shared across CodeAct and DeepResearch sessions
🖱️ Vision-first GUI automation agent
🔐 Multi-user auth with sessions scoped per user
📚 RAG / knowledge-graph backend
🚢 One-click "Ship" flow: agent → PR/artifact → deployed demo
🎥 Full browser-session recording & playback

❤️ Why This Matters

Software engineering is not a chat conversation — it is a chain of effects: files written, processes run, pages visited, artifacts shipped. Autopilot treats the agent like an engineer on your team, not a chatbot. It works in a real workspace, uses real tools, leaves an auditable trail, and produces something you can run, inspect, and reuse. It closes the gap between "the model knows how" and "the computer did it" — and it makes AI work something a judge can trust, because they can watch it happen and replay it afterward.

🧠 Agents that answer are a commodity. Agents that build are the future.

Autopilot — where agents don't just answer, they build.

🔗 Links

Repository: github.com/digiflowhelp-stack/Autopilot
License: MIT
🖥️ Live demo: Local-first by design. Sandboxed code execution, a Chromium-backed MCP browser, and SQLite persistence are intentionally not hosted — they need real processes, a real browser, and real disk. Instead, everything ships in a single Bun-compiled binary or a Docker desktop image that runs entirely on your machine. bun run dev:watch, open http://localhost:9005, and fly. 🛫

Built With

ai-agent
anthropic
autonomous-agents
bun
docker
express.js
latex
model-context-protocol
openai
puppeteer
react
self-hosted
socket.io
sqlite
tailwind-css
typescript
vercel-ai-sdk
vite
zod

Autopilot

🖥️ Autopilot — The AI Automated Software Engineering Platform

🎯 The One-Line Describes It

🔥 The Problem

💬 Models reason — but they cannot act.

🧩 The engineering loop is shattered across a dozen tools.

🕶️ Black-box agents can't be trusted.

💡 The Solution

🥊 What Makes It Different

🏗️ How It Works — Architecture

🧬 The data flow, end to end

🧠 The agent brain: `AgentTARS`

⚡ The CodeAct sandbox

🔬 DeepResearch loop

🌐 MCP lifecycle manager

🎞️ The replay & streaming engine

🛡️ Security stack

🧗 The Eight Hard Problems

🎬 See It In Action

🛠️ Built With

🤝 AI Tools Disclosure

🗺️ Roadmap

❤️ Why This Matters

🔗 Links

Built With

Updates

Submission history

🖥️ Autopilot — The AI Automated Software Engineering Platform

🎯 The One-Line Describes It

🔥 The Problem

💬 Models reason — but they cannot act.

🧩 The engineering loop is shattered across a dozen tools.

🕶️ Black-box agents can't be trusted.

💡 The Solution

🥊 What Makes It Different

🏗️ How It Works — Architecture

🧬 The data flow, end to end

🧠 The agent brain: AgentTARS

⚡ The CodeAct sandbox

🔬 DeepResearch loop

🌐 MCP lifecycle manager

🎞️ The replay & streaming engine

🛡️ Security stack

🧗 The Eight Hard Problems

🎬 See It In Action

🛠️ Built With

🤝 AI Tools Disclosure

🗺️ Roadmap

❤️ Why This Matters

🔗 Links

Built With

Updates

Submission history

🧠 The agent brain: `AgentTARS`