Research Swarm

A distributed swarm of Gemini agents that researches for hours unattended — idempotent, self-healing, and cost-bounded by design. Built on Cloud Run + Pub/Sub.

Comment

Inspiration

Most "AI agent" demos are really just chatbots with extra steps — you ask, it thinks for a few seconds, it answers. We wanted to build something that actually earns the word agentic: a system you hand a hard, open-ended question to, then walk away from, because it keeps working, coordinating, and improving its own plan long after you've closed the tab.

The hackathon prompt — "agents that run in the background, handle the heavy lifting of massive datasets, and automate complex workflows asynchronously" — was the brief we took literally. So we built Research Swarm: a coordinator agent that decomposes a broad research question, deploys a fleet of worker agents to investigate it in parallel, and continuously re-plans and re-synthesizes the answer as new findings come in — for as long as the question needs.

What it does

You create a Workspace (e.g. "AI Regulation," "Food Research") to keep related research organized, then submit a broad question inside it — for example, "How is the EU AI Act going to affect small AI startups?" You get a job back immediately. No waiting, no spinner.

Behind the scenes:

  1. A Coordinator Agent (Gemini) decomposes the question into parallel sub-questions, each with a search strategy hint.
  2. A fleet of Worker Agents picks these up concurrently via Pub/Sub, each running Gemini with real Google Search grounding to investigate its sub-question and extract a structured, source-cited finding.
  3. A Re-planner continuously reviews incoming findings and can dynamically spawn new follow-up sub-questions if a worker surfaces an unexpected angle — the workflow isn't a fixed DAG, it grows based on what the swarm actually discovers.
  4. A Synthesizer Agent rebuilds a living, versioned Markdown report as findings arrive — clustering results by theme, with citations verified against real grounding metadata (not model-guessed URLs) — so the report evolves visibly from v1 to v6 as the swarm works.
  5. A three-panel live dashboard — sidebar of workspaces and files, a real-time activity and follow-up chat panel, and the rendered report — lets you watch the swarm work, see exactly which agent is doing what right now, and even send a follow-up message mid-run to redirect or deepen the research, just like steering a conversation.

How we built it

Architecture: a monorepo with clearly separated agent roles — coordinator/, worker/, synthesizer/, sharing common types and infra through lib/, fronted by an Express server/ and a Next.js frontend/.

Stack: Gemini 2.5 Flash (function calling + Search Grounding + JSON-schema structured output) for every agent; Cloud Run for horizontally scalable, independently deployable Coordinator/Worker/Synthesizer services; Pub/Sub to decouple fan-out and fan-in between agent roles; Firestore as the live job/task/workspace/report store, powering real-time listeners across the whole UI with no polling.

We treated this as a distributed system, not a script:

  • Idempotency: workers check for an existing finding before processing, so Pub/Sub's at-least-once delivery can never create duplicate work.
  • Backpressure: the Re-planner is bounded by maxTasks and maxDurationMinutes per job, so dynamic fan-out is powerful but never unbounded or runaway-expensive.
  • Circuit breaking: a shared breaker across the worker fleet trips after repeated upstream failures, so the swarm fails fast and recovers instead of hammering a struggling dependency.
  • Grounded citations: the Synthesizer can only cite URLs present in a worker's actual Gemini grounding metadata — never a model-guessed source — with a "Grounding Audit Pass" check surfaced directly in the UI.
  • Observability: every log line carries a traceId (the job ID), and every task records queued/started/completed timestamps, visualized live in the activity feed.

Challenges we ran into

  • Making the async nature visible, not just real. Early on, our progress indicators and worker counts read from different, out-of-sync data sources — one part of the UI would say "3/6 done" while another correctly said "6/6." We consolidated everything onto a single live Firestore listener per job so every UI element — the progress bar, the worker list, the live status indicator — always agrees.
  • Making sure the swarm answers the actual question asked. We caught and fixed a serious bug where a job's report could reflect content from a different prior job. Tracing it forced us to audit context-passing through the whole Coordinator → Worker → Synthesizer pipeline and add explicit per-job logging so this class of bug is now immediately visible if it ever recurs.
  • Designing a report that improves rather than just grows. We rebuilt the Synthesizer to re-cluster all findings by theme on every update, so the report reads like a coherent document at any point in its lifecycle, not a running log.
  • Adding conversational steering without breaking the architecture. The follow-up chat had to reuse the exact same Worker → Finding → Synthesizer pipeline and the same budget guards as the original run — no special-cased shortcut — so a mid-run user message is treated with the same rigor as the initial question.

Accomplishments that we're proud of

  • A genuinely asynchronous, multi-hour-capable agent system — not a simulated delay, an actual background job architecture built on Cloud Run + Pub/Sub.
  • Real multi-agent coordination with visible message-passing between distinct Coordinator, Worker, and Synthesizer roles.
  • Grounded, verifiable citations — the report can only cite sources the swarm actually retrieved, with an audit pass shown in the UI.
  • Production-grade correctness and safety: idempotent processing, bounded fan-out, circuit breaking, and full execution tracing.
  • A live dashboard that makes the swarm's async, parallel nature genuinely visible — workspaces to organize research, a real-time activity feed, and the ability to redirect a running swarm mid-investigation with a simple follow-up message.

What we learned

Agentic systems live or die on the boring distributed-systems fundamentals, not the prompting. The hardest and most valuable work here wasn't getting Gemini to decompose a question well — it was making sure duplicate messages don't double-process, that a re-planning loop can't run away unbounded, that every UI element agrees with the same source of truth, and that a report can't silently answer the wrong question. Treating each agent as an independently scalable microservice, rather than one big script, is what let the system stay understandable and safe even as its behavior became more dynamic and conversational.

What's next for Research Swarm

  • Source-level verification agents that cross-check claims across multiple sources before they enter the report, surfacing disagreement explicitly rather than picking one answer.
  • Persistent, queryable research memory across a workspace, so related future questions can build on prior swarms' findings instead of starting cold.
  • Cost/latency-aware scheduling — dynamically trading off worker parallelism against Gemini API budget for a given job's priority.
  • Shared workspaces for teams to collaborate on the same live research threads.

Built With

Share this project:

Updates

Submission history