ProofPilot

When a user drops an LLM assertion into ProofPilot, it doesn’t just run a basic keyword search. It parses claim, pulls source evidence and delivers audit: Verified, Hallucinated, or Unverifiable.

Comment

✈️ The ProofPilot Story: Charting the Flight Path of Truth

Behind every line of code, there is a story of inspiration, experimentation, late-night debugging, and the pursuit of a solution to a real-world problem. This is the story of how ProofPilot came to be, what we learned along the way, how we built it, and the turbulence we navigated.

💡 The Inspiration

We live in an era of information overload. Large Language Models (LLMs) have made it trivial to generate content, but they have also made it trivial to generate hallucinations. Rumors, misleading statistics, and half-truths spread across the internet at the speed of a click.

During our research, we noticed a critical gap:

  • Most fact-checking tools are slow, academic, or manually operated.
  • Existing AI tools often give a simple, binary "True" or "False" verdict, failing to capture the nuance of "gray zone" claims—statements that are partially true but contain incorrect details (like dates or numbers) or subjective opinions that are inherently unverifiable.

We wanted to build something different. We envisioned ProofPilot—an automated, real-time, non-blocking claim verification engine that operates like a co-pilot for information integrity. It doesn't just judge; it decomposes, investigates, reasons, and presents its verdict with crystal-clear explanations and citations.

🛠️ How We Built It

ProofPilot is designed from the ground up to be lightweight, high-performance, and visually stunning. We opted for a decoupled architecture:

graph TD
    User([User Client]) -->|1. Enters Claim| FE[React SPA]
    FE -->|2. POST /check| BE[FastAPI Gateway]

    subgraph Backend Pipeline
        BE -->|3. Evaluate Mode| Mode{Live / Mock?}
        Mode -->|Live Mode| Tavily[AsyncTavilyClient]
        Tavily -->|4. Search Queries| Web[Tavily Search API]
        Web -->|5. Context (Basic Depth)| Tavily
        Tavily --> BE

        BE -->|6. Call Llama 3.3| Groq[AsyncGroq Client]
        Groq -->|7. Strict JSON Output| BE

        Mode -->|Mock Mode| Mock[Mock Engine]
        Mock -->|Auto-Generated Verdict| BE
    end

    BE -->|8. Structured Response| FE
    FE -->|9. Render Cards| User
    FE -->|10. SDK Event Track| Novus[Novus.ai Telemetry]

1. The Backend (FastAPI + Groq + Tavily)

  • FastAPI: Chosen for its native asynchronous support, type safety via Pydantic, and high throughput.
  • Tavily Search API: Used AsyncTavilyClient to retrieve credible, real-time web search indices globally without blocking the event loop.
  • Groq Cloud: Leveraged Groq's high-speed completion API to query llama-3.1-8b-instant (and llama-3.3-70b-versatile during testing) with a carefully designed system prompt to evaluate search context and return structured JSON verdicts.
  • Smart Fallbacks: Implemented an automated mock engine fallback if API keys are missing or if any network failure/timeout occurs.

2. The Frontend (React + Vite + Tailwind CSS)

  • Vite & React: Provides an ultra-fast developer loop and optimal production builds.
  • Glassmorphism UI: Built with custom Tailwind CSS themes, using dynamic HSL gradients, subtle micro-animations, and glassmorphism elements to provide a modern, premium dark-mode dashboard.
  • Step-by-Step Loading States: To maintain user engagement during the 2-5 second analysis pipeline, we implemented animated text stages showing the active task (e.g., Decomposing claim, Fetching sources, Computing final consensus).
  • Novus.ai Integration: Integrated telemetry tracking (claim_check_started, claim_checked, claim_check_failed) to capture usage events and performance metrics in real time.

⚠️ Challenges We Faced & How We Overcame Them

Challenge 1: The "Gray Zone" Claim Dilemma

The Problem: LLMs often struggle to categorize claims that are almost true. For example: "OpenAI was founded in 2015 by Sam Altman and Steve Jobs." (Steve Jobs is incorrect). A basic LLM might mark this as "Verified" because the primary subject matches.

The Solution: We updated our SYSTEM_PROMPT to mandate claim decomposition. The LLM must break down the sentence into individual assertions (OpenAI founded in 2015? True. Founded by Sam Altman? True. Founded by Steve Jobs? False.). Under our rules, if any component is inaccurate, the claim is marked as LIKELY_HALLUCINATED.

Challenge 2: Network Latency and API Timeouts

The Problem: Querying two external APIs sequentially (Tavily search -> Groq inference) can take upwards of 10-15 seconds if search results are large, which degrades user experience.

The Solution:

  1. We configured Tavily to search with basic depth and a maximum of 2 highly relevant results.
  2. We set strict connection timeouts (8s for Tavily, 12s for Groq) and switched to llama-3.1-8b-instant for ultra-fast response times.
  3. We built a background simulation/fallback mock engine that immediately takes over if endpoints time out or if API keys are unconfigured, ensuring the application remains 100% functional.

Challenge 3: Ensuring Deterministic JSON Responses

The Problem: LLMs occasionally wrap JSON in markdown code blocks (json ...) or append conversational preamble, which crashes traditional JSON parsers.

The Solution: We activated Groq's native response_format={"type": "json_object"} and added defensive python parsing: .strip().removeprefix("json").removesuffix("").strip() to ensure parsing never fails.

🎓 What We Learned

  1. Asynchronous Architecture is Crucial: Running concurrent, non-blocking requests in FastAPI allows us to scale efficiently and handle multiple requests without blocking server throughput.
  2. UX Fills the Latency Gap: When a task takes several seconds, transparent loading stages (revealing exactly what the backend is doing) make the wait feel instantaneous and build user trust.
  3. Telemetry Drives Insight: Incorporating Novus.ai telemetry from day one helped us immediately identify drop-offs and track which claims users searched for most, guiding future feature priorities.

ProofPilot is more than just a fact-checker—it is our contribution to a more transparent, verifiable internet.

Share this project:

Updates