AGI WORKFORCE

The last AI app you'll ever need

Comment

What it does

AGI Workforce is a full AI application suite spanning six first-class surfaces:

Surface Role
Web Account, projects, synced chats, artifacts, billing
Mobile On-device Local LLM chat, continuity, approvals
Desktop Local-private compute host, rich app shell, MCP/connectors
Chrome Extension Browser context, page capture, native messaging
VS Code Extension IDE-native developer surface over CLI runtime
CLI Developer agent engine, terminal-native workflows

Three Trust Modes

Every conversation has an immutable execution mode:

  • Local Mode — 100% offline. Models run on-device via Ollama/llama.cpp. Your data never leaves your machine.
  • BYOK (Bring Your Own Key) — Use your own OpenAI, Anthropic, Google, or other provider keys directly. AGI never sees them.
  • Managed Cloud — AGI-managed provider access (public alpha, open by default). One subscription, every frontier model.

Transitioning from Local → BYOK is an explicit fork with context selection, secret scanning, payload preview, and user consent — never a silent mode flip.

Swarm Orchestration

The core is a multi-agent swarm architecture in Rust:

$$ \text{Manager Agent} \xrightarrow{\text{spawn}} {W_1, W_2, \ldots, W_n} \xrightarrow{\text{aggregate}} \text{Result} $$

  • A Manager Agent decomposes goals into subtasks
  • Worker Agents are dynamically spawned with resource limits (CPU, RAM, network) and semaphore-controlled concurrency
  • Workers are frozen during execution — they don't learn mid-task, ensuring deterministic behavior
  • A Circuit Breaker pattern handles failures: \( \text{Healthy} \to \text{Degraded} \to \text{CircuitOpen} \to \text{Recovering} \)
  • Communication flows through Tokio mpsc channels (tasks) and oneshot channels (results)

Deep OS Integration

AGI doesn't just chat — it acts:

  • 🖱️ Mouse & keyboard control via enigo
  • 🖥️ Screen capture via xcap
  • 💻 Terminal emulation via portable-pty
  • 🎤 Offline speech-to-text via whisper-rs
  • 👁️ OCR via Tesseract
  • 🔐 OS-level credential storage via keyring

How we built it

Architecture

The system is a monorepo with shared Rust crates and TypeScript packages:

web/# Next.js - account, projects, billing mobile/# React Native - local LLM + cloud chat desktop/# Tauri 2.x - local-first compute host extension/# Chrome MV3 - browser context & capture extension-vscode/# VS Code - IDE integration cli/# Rust - developer agent engine packages/ # Shared TS: contracts, providers, UI, tools crates/# Shared Rust: protocol, registry, sandbox services/ # API gateway, signaling, managed compute

Backend — Rust / Tauri

The backend is a high-performance Rust core:

  • Tauri 2.x for the desktop shell, plugin system, and JS↔Rust bridge
  • Tokio async runtime for agent orchestration and automation
  • Serde for serialization across JSON, Bincode, and YAML
  • rusqlite for local agent state and persistent memory
  • reqwest + tungstenite for HTTP and WebSocket communication

Frontend — React / TypeScript

  • React 19 with Vite for the desktop SPA
  • Radix UI for accessible component primitives
  • Framer Motion for animations
  • Monaco Editor for integrated code editing
  • Xterm.js for integrated terminal
  • Zustand for state management
  • KaTeX + Mermaid for math and diagram rendering

AI Stack

  • Ollama API for local LLM management
  • llama-cpp-2 for local inference
  • whisper-rs for offline STT
  • webrtc-vad for voice activity detection
  • Multi-provider routing: OpenAI, Anthropic, Google, Mistral, and more — all from a single model registry

Infrastructure

  • clerk for auth and cloud sync
  • Neon (PostgreSQL) for Web data
  • Vitest + Playwright for testing

Challenges we ran into

1. Trust Boundary Enforcement

The hardest architectural problem was ensuring that Local data never silently leaks to cloud providers. Every IPC message, every API call, every model selection had to be audited against the conversation's immutable execution_mode. We built Rust-level provider admission gates ttside the conversation's trust boundary — and added secret scanning to the Local→BYOK fork flow.

2. Multi-Surface Consistency

Maintaining a consistent UX across six surfaces (Web, Mobile, Desktop, Chrome, VS Code, CLI) with three different trust modes required a shared contracts layer (packages/contracts) that defines PrivacyMode, ProviderMode, and ChatExecutionMode — enforced at compile time in Typeust.

3. Agent Determinism in Swarms

When you spawn \( n \) concurrent agents, non-determinism explodes. Frozen sub-agents (ource locking (file locks, UI element locks), and semaphore-controlled concurrency wereessential to prevent agent conflicts without sacrificing parallelism.

4. Offline-First AI

Running Whisper, Tesseract, and llama.cpp locally means managing native C/C++ bindings across macOS, Windows, and Linux — each with different FFI quirks, library paths, and GPU acceleration stacks. We wrapped everything in safe Rust abstractions with graceful fallbacks.

5. Circuit Breaker Tuning

Getting the circuit breaker thresholds right was iterative. Too aggressive and healthy agnt and cascading failures brought down entire swarms. We settled on a five-state model(Healthy → Degraded → CircuitOpen → Recovering → Terminated) with configurable thresholds per task type.

Accomplishments we're proud of

  • True model freedom: One app, every major AI provider, plus fully offline local models — no vendor lock-in
  • Privacy that's provable: Local Mode conversations physically cannot reach the net
  • Active AI autonomy: Agents that control your OS, terminal, and browser — not just generate text
  • Six production surfaces from one monorepo with shared contracts
  • $7/month entry with a Free tier.

What we learned

  • Rust is the right choice for trust-critical AI infrastructure. Ownership semantics catch entire categories of privacy bugs at compile time.
  • Privacy is an architecture, not a feature toggle. You can't bolt on "local mode" tohas to be the foundation.
  • Agent swarms need governance, not just orchestration. Spawning agents is easy. Preventing them from conflicting, leaking data, or cascading failures is the real engineering.
  • Model registries beat hardcoded IDs. AI models change weekly. A hand-maintained cattions lets us ship updates without code changes.

What's next

  • Visual artifact workspace — canvas, artboards, prototyping, and design export
  • Desktop AGI Code — full developer mode with inline code execution
  • Enterprise SSO & audit — SAML, SCIM, and compliance logging
  • Plugin marketplace — community-built MCP connectors, skills, and agent templates
  • Video generation — available on Max 15x and Enterprise plans

Built With

Share this project:

Updates