True Source

A GitLab Duo agent that fact-checks every claim in your issues and merge requests against the actual repository, then fixes what it can autonomously.

Comment

Inspiration

I run a news verification platform called TrueSource, and one thing I kept noticing is that misinformation isn't just a social media problem. I'd see developers drop claims into issues like "this fixes the CVE affecting all versions" or "our API handles 10k RPS" and nobody would question it. These claims end up in docs, get referenced in MRs, and eventually shape decisions.

When I saw the GitLab AI Hackathon, I thought: why not bring the same verification engine I built for news directly into GitLab, right where the claims are being made?

What it does

Mention TrueSource Verify on any issue or MR and it breaks down the text into individual factual claims, then checks each one. You get a structured report with verdicts (True, False, Partially True, Unverifiable), confidence scores, and flags for things like outdated statistics or absolutist language.

The part I'm most proud of is the semantic precision analysis. It catches stuff like "SQL injection is the #1 OWASP vulnerability," which was true in 2017 but not in 2021. Technically misleading claims are harder to catch than outright false ones.

How I built it

The core idea is dialectical verification. Instead of just asking an LLM "is this true?", the system looks for evidence on both sides before making a call. This was something I learned the hard way building TrueSource: single-pass fact-checking has a confirmation bias problem.

The standalone pipeline is TypeScript and pulls from multiple sources: Brave Search API for real-time results, Google News RSS as a fallback, Wikipedia for reference claims, and the Google Fact Check Tools API for existing professional fact-checks. Each source gets weighted by reliability scores from Media Bias/Fact Check, which covers over 5,000 outlets.

On the GitLab side, it runs as a Duo agent with access to GitLab's native tools for reading issues, searching the codebase, and posting results back as comments.

Challenges

The Duo Agent Platform is brand new (GA in 18.8) and the docs are still catching up. I spent more time than I'd like to admit figuring out the right configuration for agents, flows, and triggers. Lots of trial and error.

The other hard part was semantic precision. Getting an LLM to understand that "always" and "never" are red flags, or that a statistic from 2019 might not hold in 2026, required careful prompt engineering and multiple iterations.

What I learned

Honestly, the biggest takeaway is that the best place to catch bad information is where it's written, not after it's spread. Putting fact-checking inside the developer workflow feels obvious in hindsight. I also gained a much deeper understanding of the GitLab Duo Agent Platform, which I think has huge potential for domain-specific agents beyond what GitLab ships by default

Built With

  • ad-fontes-media
  • allsides
  • anthropic-sdk
  • brave-search-api
  • claude-api
  • gitlab-ai-catalog
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • google-fact-check-tools-api
  • google-news-rss
  • jsdom
  • media-bias/fact-check
  • mozilla
  • node.js
  • typescript
  • wikipedia-rest-api
Share this project:

Updates