Rabbithole

What if you could ask a question and watch an AI agent recursively explore it 10 layers deep, discovering connections you'd never find manually? Thats Rabbithole

Comment

Rabbit Hole - Devpost Submission

Inspiration

You start researching a topic, find an interesting paper, which cites another paper, which mentions a company... and suddenly you have 47 browser tabs open.

Traditional research tools give surface-level answers. Search engines return links. AI chatbots give single responses. But real understanding comes from going deep - following threads and discovering non-obvious connections.

I built an AI agent that researches like curious humans do - recursively exploring questions and building mental models. But unlike humans, it goes 10+ layers deep without getting tired, tracks hundreds of entities simultaneously, and visualizes the entire knowledge graph in real-time.

What it does

Rabbit Hole takes any question and explores it 10+ layers deep, building a live knowledge graph as it discovers entities, relationships, and insights.

Key features:

  • Recursive Exploration: Agent uses Claude to plan research, searches multiple sources (Tavily, Semantic Scholar, news APIs, arXiv, patents), extracts entities/relationships, and recursively follows promising leads
  • Live Knowledge Graph: Real-time D3.js visualization of nodes (people, papers, companies, concepts) and edges (authored, funded_by, contradicts, cites)
  • Insight Generation: Finds contradictions, surprises, and research gaps - not just data collection
  • Comprehensive Reports: Synthesized findings with citations after 10 layers
  • Real-time Streaming: WebSocket updates as the agent thinks

Example: "What are emerging alternatives to lithium-ion batteries?" → Discovers solid-state/sodium-ion concepts → Finds researchers and companies → Explores papers and patents → Uncovers funding relationships → Recursively explores manufacturing, market dynamics, regulations, etc.

How I built it

Tech Stack:

  • Kiro (AWS): AI-assisted development
  • Claude (Anthropic): Agent reasoning - planning, extraction, reflection
  • Aerospike: Real-time graph database with sub-millisecond latency
  • Airbyte: Data pipeline orchestration
  • Auth0: Authentication
  • Overmind: API cost optimization (3-4x reduction)
  • Next.js 14 + D3.js: Frontend with force-directed graph visualization
  • FastAPI: Async Python backend with WebSocket support

Agent Loop:

async def research_loop(question, depth, max_depth=10):
    plan = await agent.plan(question, current_graph_state)
    results = await search_all_sources(plan.queries)
    entities = await agent.extract_entities(results)
    await store.save_graph(entities)
    await websocket.broadcast_update(entities)
    insights = await agent.reflect(current_graph_state)

    if depth < max_depth:
        for lead in plan.top_leads[:3]:
            await research_loop(lead.question, depth + 1, max_depth)

Data Sources: Tavily, Semantic Scholar, News API, arXiv, USPTO/Google Patents, Wikipedia

Challenges I ran into

API Costs: Initial implementation made 100+ Claude calls per session. Integrated Overmind for intelligent batching/caching, reducing costs 3-4x.

Graph Performance: D3.js re-rendering entire graph on updates caused lag. Implemented incremental updates with natural force simulation stabilization.

Circular Research: Agent would revisit topics. Added embedding-based deduplication and reflection step to detect stalled progress.

Entity Extraction: Inconsistent formats from Claude. Used Pydantic models with structured outputs and few-shot examples.

Large Graph Visualization: 200+ nodes became cluttered. Added clustering by entity type, zoom/pan, and filtering.

Accomplishments that we're proud of

  • Real-time knowledge graph visualization that makes the agent's thinking visible
  • 10+ layers of exploration (most tools stop at 1-2)
  • Insight generation that finds genuinely interesting contradictions and surprises
  • Production-ready architecture with Aerospike, Auth0, Airbyte
  • 3-4x cost reduction with Overmind optimization
  • Multi-source integration (academic papers, news, patents, web)
  • Polished UX despite tight timeline

What I learned

Technical: Aerospike is incredibly fast for graph workloads. WebSocket streaming beats polling for UX. D3.js needs careful performance tuning. Structured outputs make LLMs reliable. Async Python perfect for agent workloads. Kiro saved 10+ hours of coding time.

Agent Design: Reflection is crucial - agents need to think about what they've learned. Parallel searches are much faster. Planning before searching produces better results. Embeddings essential for deduplication.

Product: Visualization makes intelligence tangible. Users want to explore, not just read reports. Insights (contradictions, surprises) more valuable than raw data. Real-time feedback keeps users engaged.

What's next for Rabbit Hole

Short-term (next 2-4 weeks):

  • Multi-modal support: Analyze images, charts, videos in research sources
  • Collaborative research: Multiple users exploring the same question, sharing insights
  • Export capabilities: Generate research papers, presentations, or reports from the knowledge graph
  • Custom data sources: Let users connect their own APIs, databases, or document repositories
  • Advanced filtering: Filter graph by confidence, depth, entity type, or time period

Medium-term (2-3 months):

  • Research templates: Pre-configured agent behaviors for specific domains (scientific literature review, market research, competitive analysis, due diligence)
  • Agent memory: Remember previous research sessions and build on them
  • Human-in-the-loop: Let users guide the agent mid-research ("explore this branch more", "skip this topic")
  • Comparison mode: Research multiple questions simultaneously and compare findings
  • API access: Let developers integrate Rabbit Hole into their own applications

Long-term (6+ months):

  • Multi-agent collaboration: Multiple specialized agents (one for academic papers, one for news, one for patents) working together
  • Continuous research: Agents that monitor topics over time and alert you to new developments
  • Research marketplace: Share and discover research sessions from other users
  • Enterprise features: Team workspaces, access controls, audit logs, custom LLM deployments
  • Mobile app: Research on the go with optimized mobile graph visualization

Potential Applications:

  • Academic literature reviews
  • Market research and competitive intelligence
  • Due diligence for investments
  • Investigative journalism
  • Patent landscape analysis
  • Technology trend forecasting
  • Policy research
  • Medical research synthesis

Built With

Share this project:

Updates