Underwatch

Inspiration

AI coding agents can read files, execute commands, and modify code, but developers cannot always verify everything they do. Recently, a security firm reported that AI coding workflows had exposed over 13,000 internal screenshots associated with more than 300 organizations through public GitHub repositories. Such exposures can reveal sensitive business information that was never intended to be public.

We wanted to address this visibility gap. Instead of relying on an agent to explain its own behavior, we asked: What if developers could independently verify what their AI agents actually did?

What it does

Underwatch is an independent security and observability layer for AI coding agents. It compares agent-reported actions with independently observed system events to help developers:

Detect unexpected or unauthorized file access.
Investigate potential data exposure and trace execution paths.
Compare reported actions with observed behavior.
Evaluate whether security policies and enforcement mechanisms hold.
Investigate findings through evidence-backed, natural-language queries.

How we built it

We built a security investigation dashboard around independently observed system activity and agent-reported actions. The platform organizes suspicious behavior into concise findings, provides side-by-side comparisons of reported and observed events, and visualizes process relationships through an evidence graph backed by Neo4j.

The Safeguards interface distinguishes kernel-level enforcement from permission-rule enforcement, while Ask Witness provides a conversational interface for investigating findings with supporting evidence.

Challenges we ran into

Our main challenge was distinguishing what an agent said it did from what the system could independently verify. We needed to connect events across process execution paths, preserve their relationships, and present technical evidence in a way developers could investigate without digging through raw logs.

Another challenge was separating attempted access, successful access, and actual data exposure. These represent different security outcomes and cannot be treated as interchangeable.

Accomplishments that we're proud of

We're proud to have brought agent observability, evidence-based investigation, execution tracing, and safeguard evaluation into one workflow. The “Agent said → Underwatch observed” comparison makes discrepancies visible, while the evidence graph helps developers trace how an operation occurred. We also separate security outcomes from task outcomes, making it easier to understand both whether an agent completed its task and whether it respected security boundaries.

What we learned

An agent's explanation is not the same as independent evidence of its behavior. Effective AI security requires visibility into actual system events, the ability to trace operations to their origins, and clear distinctions between attempted actions, successful operations, and exposed data.

We also learned that security tools must make complex evidence understandable and actionable, not simply collect more logs.

What's next for Underwatch

We want to expand runtime coverage across more AI coding agents and execution environments, strengthen detection of sensitive-file access and data exfiltration, and improve real-time enforcement and alerting. We also aim to validate our detection mechanisms against broader attack scenarios and make Underwatch easier to integrate into existing developer workflows.

Our long-term goal is to make autonomous coding systems more auditable, so developers can benefit from their capabilities without giving up visibility and control.

Underwatch: See what your agent actually does.