-
-
Monitoring AI agents for mission drift, emerging risks, and long-term societal impact through real-time observability and governance.
LegacyLens: The AI Risk Observatory
Inspiration
As AI systems become increasingly autonomous, organizations face a growing challenge: how do you know when an AI system begins drifting away from its original purpose?
Most AI monitoring tools focus on uptime, latency, and infrastructure health. Very few monitor whether an AI agent remains aligned with its intended mission, ethical constraints, and long-term societal goals.
We built LegacyLens to answer a critical question:
"Who watches the AI agents?"
LegacyLens acts as an AI Risk Observatory, continuously evaluating AI-generated decisions for mission drift, harmful optimization, misalignment, and long-term risk. Instead of only monitoring system performance, it monitors AI behavior.
What it does
LegacyLens analyzes outputs from AI agents and evaluates:
- Risk Score
- Drift Score
- Alignment Score
- Risk Level Classification
- Long-Term Impact Forecasting
- Recommended Safety Interventions
The platform detects when an AI system begins optimizing for short-term gains at the expense of its original mission.
For example, an agricultural AI originally designed to promote sustainable farming may gradually begin recommending monoculture farming practices that maximize short-term yield while damaging long-term soil health.
LegacyLens identifies this behavioral drift before it becomes a large-scale problem.
How we built it
The platform consists of four major components:
1. AI Risk Analysis Engine
Built with FastAPI and OpenAI.
Each AI-generated output is analyzed against its original mission and evaluated for:
- Alignment
- Drift
- Harm potential
- Long-term societal impact
2. Zero-Trust AI Gateway
Before an AI output is allowed downstream, LegacyLens evaluates its safety profile.
Outputs exceeding risk thresholds can be flagged or blocked automatically.
3. Splunk Observability Layer
Splunk acts as the operational intelligence backbone.
Every AI event is streamed into Splunk using HTTP Event Collector (HEC), enabling:
- Real-time monitoring
- Risk analytics
- Historical investigation
- Operational alerting
4. Executive Risk Dashboard
Using Splunk dashboards, organizations can visualize:
- Risk score trends over time
- Drift score distribution
- High-risk AI events
- Gateway decisions
- Risk activity by domain
Challenges we ran into
One of our biggest challenges was designing a framework that measures AI mission drift rather than traditional system metrics.
We also faced integration challenges involving:
- Real-time event ingestion
- Splunk HEC configuration
- Dashboard design
- AI response structuring
- Cross-platform testing through ngrok
Building meaningful risk signals that remain interpretable to humans required multiple iterations.
Accomplishments that we're proud of
- Built an end-to-end AI safety monitoring platform
- Integrated real-time Splunk observability
- Created an AI Gateway capable of evaluating and blocking risky outputs
- Developed a mission-drift detection framework
- Designed a dashboard that converts complex AI behavior into actionable operational insights
What we learned
Through this project we learned that AI observability extends far beyond infrastructure monitoring.
The next generation of observability tools must understand not only whether an AI system is functioning, but whether it is functioning responsibly.
Splunk provides an excellent foundation for this future by enabling operational visibility across AI workflows and risk events.
What's next for LegacyLens
Future development includes:
- Real-time anomaly detection
- Multi-agent risk correlation
- Automated Splunk alerting
- Governance and compliance reporting
- Agent memory auditing
- Enterprise AI safety scorecards
- Predictive risk forecasting
Our vision is to become the observability layer for trustworthy autonomous AI systems.
Log in or sign up for Devpost to join the conversation.