• 

• 

• 

TraceOS

Inspiration

TraceOS started with an incident I still think about.

I watched an internal team respond to what initially looked like a manageable security event. The company had skilled people and plenty of security tools, but the investigation itself broke down. Evidence was scattered across different consoles. Teams handed partial context to one another. Important details were buried in screenshots and audit logs. Analysts lost time rebuilding timelines, while unverified assumptions began to look like facts.

By the time the organization understood the full scope of the incident, the damage had already compounded. Recovery, downtime, operational disruption, and the resulting business impact cost the company millions of dollars.

What stayed with me was that this was not simply a failure of one person or one alert. It was a systems failure. The team could collect evidence, but it did not have a reliable way to preserve context, coordinate specialists, challenge weak conclusions, and show exactly how a finding had been proven. During a fast-moving incident, those gaps became expensive.

I began TraceOS as an internal research project to explore a better model: an autonomous forensic fleet that can do the repetitive investigative work continuously, while keeping every consequential conclusion evidence-linked, independently verified, and understandable to a human responder. My goal is to harden this research and potentially roll it out internally so the same kind of failure does not happen again.

The use case behind TraceOS: replace scattered evidence and unverified assumptions with a controlled path from evidence to human action.

What it does

TraceOS is a zero-trust, multimodal forensic control plane for enterprise incident response. It is built for the Fortified Enterprise Fleet category and operates beyond a standard chatbot: it starts asynchronous investigations, delegates work to governed specialist agents, preserves state across days, enforces evidence boundaries, and produces a verified report.

An investigator begins a case or adds new evidence. TraceOS registers each source with provenance and a SHA-256 hash, then sends only the permitted scope to the appropriate agent. The visual agent extracts OCR and directly observable regions from image evidence. Identity, endpoint, network, and integrity specialists analyze their own evidence domains. Their outputs are observations—not findings.

The fleet then correlates those observations with Cloud Audit Logs and tests evidence-linked hypotheses. Only the independent verifier has permission to create a finding. If a hypothesis cites missing evidence or makes a claim the sources do not support, TraceOS stops it before it reaches the final report. In the canonical CASE-042 investigation, the unsupported EVID-019 reference is visibly rejected instead of being silently rewritten into a convincing answer.

The interface turns that process into an interactive evidence replay with four clear stages:

1. Evidence entered
2. Agents observed
3. Hypotheses tested
4. Finding verified

Selecting any node reveals the source, timestamp, evidence hash, observation, confidence, and verifier outcome. A keyboard-accessible 2D timeline provides the same information when WebGL or motion is unavailable.

TraceOS also demonstrates durable investigation memory. A case can pause, receive evidence later, and resume days afterward without losing verified facts or unresolved questions. The final Reporting Agent receives verified findings and an approved timeline—not unrestricted raw evidence. A separate one-way training branch uses Gemma to challenge the report, then can generate clearly labeled, non-evidentiary Veo and Lyria training material that cannot write back into the case.

For judges, the result is immediately testable: run the public evidence replay, follow the supported image-to-finding path, inspect the rejected claim, open the resumed memory, and confirm that only verified findings appear in the report.

The agent architecture separates delegation, observation, evidence-integrity checks, independent verification, reporting, and post-verification training.

How we built it

TraceOS uses a deliberately separated control plane, agent plane, evidence plane, and training plane.

The complete architecture connects authenticated evidence intake, durable state, managed agents, independent verification, replay, reporting, and an isolated training branch.

The public application is a scale-to-zero Cloud Run service combining a Next.js interface with a FastAPI backend. Firestore stores durable case and integration state; Pub/Sub delivers authenticated Cloud Audit events with deduplication; private, versioned Cloud Storage protects evidence; and Cloud Logging plus OpenTelemetry-compatible traces expose operational activity.

The forensic fleet uses the Google Agent Development Kit and a managed Gemini Enterprise Agent Platform Agent Runtime. Thirteen versioned manifests define each agent's identity, tools, scope, and outputs. The coordinator delegates bounded work to specialists, while typed handoffs prevent observation-producing agents from bypassing independent verification.

TraceOS uses Gemini 3.7 Flash for verified reports, Gemini 3.5 Flash in the managed ADK runtime, and Gemini vision for image evidence. Responses use strict schemas, bounded output, cached demo results, and explicit failure states. Images are validated and hashed before asynchronous analysis, and visual observations must correlate with Cloud Audit evidence before promotion.

Least-privilege scopes protect every handoff. The Reporting Agent is denied raw endpoint evidence and can access only verified_findings. The UI also reports integration states honestly: Cloud Audit Logs are CONNECTED, the managed runtime and identity are VERIFIED, Gateway is CONFIGURED, Model Armor is LOCAL_FALLBACK, and Security Command Center is NOT_CONNECTED.

After verification, an isolated one-way branch uses Gemma, Veo, and Lyria to create clearly labeled NOT EVIDENCE training material. Scale-to-zero hosting, cached runs, bounded output, and disabled-by-default media generation keep the project under its $5 limit. Tests cover evidence validation, replay ordering, rejected claims, verifier-only writes, duplicate events, model failure, state resume, accessibility, mobile layouts, and the non-WebGL fallback.

Challenges we ran into

• Making autonomy understandable: We needed to show what the agents were doing without turning the interface into an unreadable graph, which led to the four-lane evidence replay.
• Enforcing evidence boundaries: Specialists had to analyze multimodal evidence without gaining permission to create findings or bypass independent verification.
• Recovering without losing trust: Retries, duplicate events, failed model calls, and multi-day resumes all had to preserve the same evidence-linked case state.
• Representing cloud capabilities honestly: When managed controls were unavailable, TraceOS needed explicit states such as CONFIGURED, LOCAL_FALLBACK, and NOT_CONNECTED instead of overstating enforcement.

Accomplishments that we're proud of

• Failures remain visible: Unsupported evidence stops before verification, and rejected claims stay in the replay instead of being hidden or rewritten.
• Every finding is defensible: Only the independent verifier can create findings, and every report claim retains a path to its source evidence.
• Investigations survive across days: Durable memory lets the same case resume with verified facts, unresolved questions, and prior policy decisions intact.
• The system runs as a real cloud deployment: TraceOS combines Cloud Run, a managed ADK agent fleet, Firestore, Pub/Sub, Cloud Storage, and Gemini while remaining within a tightly controlled hosting budget.

What we learned

• More agents do not automatically create more trust: Enforceable responsibilities and permission boundaries matter more than adding agent personas.
• State management is part of reasoning quality: Typed handoffs, durable memory, and idempotent ingestion prevent retries and restarts from changing the investigation's facts.
• Independent verification must be architectural: Giving one agent sole finding-write authority proved more reliable than asking every worker to self-check.
• Transparent failure improves human trust: Showing missing evidence, policy denials, and unavailable integrations makes the system more credible and easier to operate.

What's next for TraceOS

• Validate with internal security workflows using approved, privacy-safe incident scenarios and responder feedback.
• Connect additional enterprise telemetry while preserving provenance, least-privilege access, and data-sovereignty requirements.
• Complete managed Gateway and Model Armor enforcement where project permissions and regional availability allow it.
• Measure operational impact and prepare a controlled rollout using investigation time, false-positive reduction, recovery outcomes, and human-review quality as success metrics.

Built With

• ai
• enterprise
• firestore
• gemini
• google-cloud
• google-cloud-run
• lyria
• react
• veo