Spotlight

Spotlight Landing Page
ArcGIS Map with Events
AI Insights Webscraper and Bot
Analytics Page
Workflows (AI Agents)
Workflows (Inputs)
Workflows (Output)
Architecture

Inspiration

Conflict, Peace, and Digital Technologies

Humanitarian organizations face significant challenges in establishing efficient workflows and minimizing inefficiencies to maximize their impact in assisting the world's most vulnerable populations. The rapid flow of global events often overwhelms existing systems, making it difficult to process and act on real-time data effectively. To address this, I developed Spotlight, a cutting-edge data visualization tool powered by Agentic AI, designed to provide deep, contextual insights into global events as they unfold. By aggregating and analyzing real-time data, Spotlight empowers users—from researchers to humanitarian agencies—to make informed decisions, understand crisis dynamics, and act swiftly in response to emerging situations.

What it does

Spotlight is an interactive intelligence platform that enables users to visualize, explore, and analyze real-time data on global events. It collects vast amounts of structured and unstructured data from multiple sources, including news reports, government bulletins, and crisis response organizations. By leveraging AI-driven analytics and graph database technology, Spotlight creates a dynamic, ever-evolving map of world events, enabling users to:

Monitor unfolding conflicts and crises in near real-time.
Track historical trends and patterns to understand the progression of global events.
Filter and customize views based on geographic regions, event types, and severity levels.
Leverage AI-powered insights to contextualize data, helping users grasp complex relationships between events and actors.
Enhance decision-making for humanitarian response efforts by providing actionable intelligence.

With an intuitive interface, Spotlight bridges the gap between complex datasets and user-friendly insights, ensuring that individuals and organizations can quickly assess global events and respond effectively.

Dataset

Spotlight relies on a rich collection of data sources to provide comprehensive coverage of global events. The primary dataset used is GDELT (Global Database of Events, Language, and Tone), one of the most extensive open-source datasets tracking news media and event occurrences worldwide. It continuously updates every 15 minutes, ensuring that Spotlight maintains an up-to-the-minute pulse on global developments.

Key data sources include:

GDELT Datasets Overview: A detailed breakdown of the different datasets within the GDELT ecosystem.
GDELT 2.0: Our Global World in Real-Time: The latest iteration of GDELT, providing improved analytics and enhanced real-time capabilities.
Real-Time Data Updates (15-minute intervals): The most up-to-date global event data available, enabling immediate analysis and visualization.

By integrating these datasets into a powerful AI-driven system, Spotlight ensures a high degree of accuracy and responsiveness in capturing and analyzing global events.

How I used ArangoDB

Spotlight’s backend is built on ArangoDB, a versatile multi-model database optimized for graph-based data processing and complex queries. The system architecture is designed to efficiently handle and analyze real-time global event data, providing a seamless experience for users.

Core components of ArangoDB in Spotlight:

Data Ingestion: Raw event data, sourced from GDELT and other real-time datasets, is transformed into a graph-based structure within ArangoDB. This structured representation allows for deep relational analysis and efficient querying.
Dual Query Paths: ArangoDB supports two distinct querying methodologies:
- Prisma for workflow-based data display: Prisma acts as the ORM (Object-Relational Mapper), ensuring efficient data retrieval for visualization.
- LangChain for AI-driven querying: LangChain enables Spotlight to process natural language queries, converting them into ArangoDB graph queries to deliver relevant insights.
Visualization Layer: Data retrieved from ArangoDB is processed through Prisma and formatted into interactive visual elements, allowing users to see complex event relationships in an intuitive manner.
AI-Powered Queries: Using natural language processing (NLP), LangChain enables users to ask questions about global events in plain language. These queries are converted into structured ArangoDB searches, returning relevant and actionable insights.
Frontend Implementation: The frontend is built using Next.js, a powerful React-based framework that provides a responsive and interactive user experience. The UI integrates seamlessly with both Prisma and LangChain, delivering real-time analytics and interactive visualizations.
User Management: Security and authentication are handled by Clerk, ensuring role-based access control, session management, and data protection. This ensures that humanitarian agencies and researchers can use Spotlight securely and efficiently.

By leveraging this sophisticated tech stack, Spotlight offers an unparalleled real-time data experience that empowers users with actionable intelligence on global events.

Key Files:

components/ArangoDB/CSV_to_ArangoDB.py converts CSV file into a Graph in Arango components/ArangoDB/langchain.py (Instead of a Jupyter Notebook) uses ArangoGraphQAChain to query ArangoDB for Agentic AI

What's next for Spotlight

Spotlight is just the beginning of a broader mission to revolutionize real-time global event monitoring and analysis. As the platform evolves, future enhancements will focus on:

Enhancing AI-powered insights to provide deeper contextual understanding of global events, trends, and interconnections.
Expanding data sources beyond traditional news media to include social media signals, satellite imagery, and IoT data, creating a more comprehensive view of unfolding crises.
Integrating predictive analytics to anticipate potential crises, enabling proactive response strategies for humanitarian organizations.
Building collaboration features that allow humanitarian agencies, researchers, and journalists to share insights, annotate data, and coordinate responses in real time.
Developing a mobile-friendly version of Spotlight to ensure accessibility for users in remote or crisis-affected areas.

Spotlight represents a commitment to innovation, transparency, and global awareness. By equipping individuals and organizations with cutting-edge AI-driven tools, it aims to make real-time global intelligence more accessible and actionable.

🚀 Want to try it out? Read the README on GitHub to set up and explore Spotlight yourself!