news-divergence

Inspiration

In our International Relations class, we studied how UN workers coordinated humanitarian relief during the fall of Kabul by manually cross-referencing news segments from Taliban statements, local Afghan press, international wire services, and NGO reports just to piece together basic ground facts. However, we realized no single source was reliable. Official data was unavailable, suppressed, or contested. The only path to understanding what actually happened was slow, manual triangulation across dozens of sources, often in multiple languages. This problem isn't unique to Afghanistan. Organizations like ACLED, ICRC, and UNAMA face it constantly: serious events go under-analyzed not because no one cares, but because no one has the capacity to read every source. We built this agent to take that first pass off the analyst's plate.

What it does

A news divergence agent that takes a natural language query about a crisis event, fetches real multilingual coverage from GDELT, and uses Gemini to produce a structured comparison of what each source reported — highlighting where sources agree, where they contradict, and what only one source is claiming.

How we built it

On each query, Gemini first extracts structured search parameters from the natural language input. The GDELT Doc 2.0 API returns real news articles matching those parameters, trafilatura fetches full text, and a SHA256 dedup layer prevents re-indexing articles already in the corpus. The agent then runs as a LangGraph ReAct loop, calling the Elastic MCP server to retrieve semantically relevant articles before composing the structured analysis. Elastic's semantic_text mapping was essential. Without semantic indexing, keyword search alone couldn't reliably filter coverage to the same underlying event across sources with different framing and vocabulary. The whole pipeline runs in a Flask server deployed on Google Cloud Run.

Challenges we ran into

GDELT enforces a global rate limit of roughly one request per five seconds, and roughly 40% of article URLs are behind paywalls, so we built a caching layer early to avoid re-fetching on repeated queries. GCP Agent Builder Studio's MCP integration didn't yet support API key authentication at the time we built this, which pushed us toward a code-based agent using the Vertex AI Python SDK directly. Gemini 2.5 also returns response content as typed blocks rather than a plain string — we had to extract only type: text blocks to prevent internal reasoning traces from leaking into the output.

Accomplishments that we're proud of

Elastic's semantic search does the real work here. It's what makes it possible to retrieve articles about the same event even when sources use different vocabulary, languages, or framing. We also learned that the humanitarian framing sharpens the product constraint: the tool should never claim to know what really happened, only what each source said. Holding that line made the output more honest and, we think, more useful to the analysts it's meant to serve.

Built With

• docker
• elastic-agent-builder-mcp-server
• elasticsearch-(elastic-cloud-serverless)
• flask
• gdelt-doc-2.0-api
• gemini-2.5-flash
• google-artifact-registry
• google-cloud-build
• google-cloud-run
• html/css/javascript
• langchain
• langchain-mcp-adapters
• langgraph
• multilingual-e5-small-elasticsearch-(elastic-in-cluster-embedding-model)
• pandas
• python
• trafilatura
• vertex-ai