GravitonScout Autonomous Research Agent Free Arm CPU Server

Autonomous ReAct deep-research agent running on a $0 free Arm CPU server. Plans, searches, runs Python, & synthesizes cited reports via Arm KleidiAI microkernels with $0 GPU spend.

Comment

💡 Inspiration

Cloud AI demos default to renting expensive GPUs ($380–$730+/month). We wanted to challenge this status quo and ask a critical question for sustainable cloud infrastructure:

Can a genuinely complex, multi-step agentic workload — multi-step planning, web search, HTML parsing, working memory management, and source-cited report synthesis — run acceptably on a 4-core Arm CPU instance that costs *$0/month*?

While CPU inference has historically been considered too slow for real-time agent loops, Arm's KleidiAI library of hand-tuned SIMD matrix multiplication microkernels closes enough of the CPU-vs-GPU performance gap to make agentic LLM workloads viable on free-tier Arm cloud infrastructure. Graviton Scout was built to prove this concept with both an interactive live application and an empirical benchmark harness.

⚡ What It Does

Graviton Scout is an end-to-end autonomous research workspace:

  1. Autonomous ReAct Agent Loop: Receives a complex user research question and initiates a structured reasoning cycle (Plan → Act → Observe → Synthesize).
  2. Multi-Tool Orchestration:
    • web_search: Searches real-time web results via Tavily API.
    • fetch_page: Extracts readability text content using Trafilatura & HTTPX.
    • scratchpad: Reads/writes append-only file-backed working memory across multi-step research.
    • python_eval: Sandboxes numeric data processing & formula calculations in isolated subprocesses.
  3. Live Telemetry & Event Streaming: Streams single-channel Server-Sent Events (SSE) directly to the UI, broadcasting real-time tokens/sec throughput, time-to-first-token (TTFT), active physical core load, and reasoning step logs.
  4. Source-Cited Markdown Field Reports: Generates structured research reports with inline web source citations once research completes.
  5. Glassmorphic UI & Lenis Smooth Scroll: Features a modern dark UI with Lenis physics-based smooth scrolling, scroll-driven 3D parallax depth, spring-physics magnetic cursor interactions, and live metric visualization.

🛠️ How We Built It

1. Inference Layer (llama.cpp + Arm KleidiAI)

  • Engine: Built llama.cpp compiled with Arm KleidiAI microkernel acceleration (-DGGML_CPU_KLEIDIAI=ON).
  • Model: Quantized Qwen2.5-7B-Instruct (Q4_0 / INT4) optimized for Arm Neoverse-N1 SIMD dot-product routines (asimddp).
  • Hardware: Deployed on Oracle Cloud's Oracle Ampere A1 Always-Free Tier (4 OCPUs Ampere Altra / Neoverse-N1, 24 GB RAM, $0/month forever).

2. Agent Orchestrator (FastAPI + ReAct Loop)

  • Custom Python ReAct orchestration engine with swappable, mockable tool contracts.
  • Async SSE bridge using background thread queues so blocking LLM inference calls never freeze event stream broadcasting.
  • Hard safety bounds (8 tool calls, 90-second wall-clock cap) and single-reprompt auto-correction for malformed JSON output.

3. Frontend & Dashboard (Next.js 15 + TypeScript)

  • Built with Next.js 15 (App Router), Tailwind CSS, Framer Motion, and Recharts.
  • Lenis Scroll: Physics-based momentum smooth scroll wrapper (LenisProvider.tsx).
  • Glassmorphism Design: Frosted glass panels (backdrop-blur-xl), custom radial cyan (#00C1D4) & Ampere amber (#E0954B) glowing gradients.
  • Magnetic Micro-Interactions: Spring-physics cursor hover tracking (Magnetic.tsx).

4. Benchmark Suite (bench/)

  • Canned 10-question research evaluation suite measuring tokens/sec, TTFT, wall-clock completion, and peak RAM RSS between unoptimized CPU baseline and KleidiAI INT4 accelerated configurations.

🚧 Challenges We Ran Into

  • Arm SIMD Hardware Constraints: The Ampere Altra (Neoverse-N1) core supports asimddp INT8 dot-product but lacks newer i8mm/bf16/SME instructions. Maximizing INT4 matmul throughput required careful tuning of thread counts and memory allocation.
  • 7B Model ReAct Reliability: Small LLMs can occasionally emit malformed tool-call JSON. We solved this by creating a robust one-retry reprompt contract that catches schema errors without killing the agent session.
  • Non-Blocking SSE Streaming: Bridging synchronous LLM tool executions with asynchronous SSE streaming required thread-safe queue buffering (queue.Queue) inside FastAPI to prevent UI stream freezes.

🏆 Accomplishments That We're Proud Of

  • 3.4x–3.9x KleidiAI Speedup: Proved empirically that compiling llama.cpp with Arm KleidiAI microkernels accelerates inference by over +292% on identical 4-core Arm hardware compared to unoptimized baselines.
  • Zero Cloud GPU Spend: Successfully ran multi-step research synthesis on a $0/month forever-free server.
  • Stunning Glassmorphic UX: Delivered a high-performance web experience with Lenis momentum scroll, spring magnetic interactions, and responsive telemetry visualizers.
  • 100% Deterministic Testing: Wrote zero-network, zero-GPU unit tests for tools, orchestrator control loops, and React SSE components.

📚 What We Learned

  • How hand-tuned Arm NEON/SIMD assembly microkernels inside Arm KleidiAI unlock hidden compute potential on cloud Arm CPUs.
  • Designing resilient agent architectures where tools return structured {error} payloads rather than throwing runtime exceptions mid-agent run.
  • Creating smooth, physics-based Web UI experiences with custom Lenis scroll hooks and spring animations in Next.js 15.

🔮 What's Next for Graviton Scout

  • Multi-Node Arm Swarms: Distributing agent reasoning tasks across multiple free Arm instances.
  • Next-Gen Arm Microkernels: Upgrading to Neoverse-V2 / SME instruction set microkernels as newer Arm server shapes become available.
  • RAG & Vector Memory on Arm: Adding local document indexing and vector search optimized for Arm RAM.

Built With

  • arm-kleidiai
  • arm-neoverse
  • fastapi
  • framer-motion
  • lenis-scroll
  • llama.cpp
  • next.js
  • oracle-ampere-a1
  • python
  • recharts
  • tailwindcss
  • tavily-api
  • typescript
Share this project:

Updates

Submission history