Architecture Diagram

Monkey Tracer and Auditor

Selected track: Track #1 - Agent Launchpad: Design and Build Lightweight Agent Middleware

Inspiration

Working with agents is pain. "Make no mistakes" end up making 67000 mistakes. You see a giant chunk of text and traces, mental overload. Now you dont feel like caring. Then you become fed up, cyclical loop of "Just fix it!!!!!!!!". But end up not being useful because you do not know where it went wrong.

Got fed up, decided to POC whether this helps. (Spoiler, it helped with this project)

Middleware

Features:

Preserved Starter Baseline, to create Agent Runs
Tracer to trace Agent Runs (Tool call, Input/Output, Thought)
Scraper to scrape traces, processes (e.g. Redaciton, normalization) them for Auditor.
Auditor uses scraped results to Audit the Agent Run
Failsafe mechanisms (See "Try it yourself" for Scraper and Auditor)

Value Proposition:

Feedback of agent runs for agent development, providing traces to diagnose issues. We used this flow during development.
Visibility of run progress to ensure policies are not violated.
Loop feedback of agent runs for agent self checking.

Out of scope / Limitations:

Limitation	Remark	Remediation
Integration with log collection tools such as Loki	Log collection is simulated with our Scraper implementation.	Add a log sink or adapter that forwards runtime events to whatever log collector and point the Scraper to it. The Auditor just needs to have the normalized traces from whatever log store you use.
Bindings to other runtimes or custom agent runners	The current integration supports the runtimes and runners described in this README.	Implement adapters for additional runtime providers and custom agent runners. In our demo, we implemented the tracing of the Auditor. You can audit the Auditor
Auditing every chat run	Auditing all runs can be extremely costly. The auditor is intended for full-fledged auditing, such as agent development and security audits.	Use sampling, or a heuristic/classifier to select which runs require auditing.
Enforcement	We DO NOT automatically enforce policies. Mutation to the agent run creates side effects, it creates variability in intent. False positivie triggers can ruin your agent's performance.	You as the agent builder implement the policies and validate against the traces to determine whether it works
Runner Boundary	We treat the runners as owning the environment it is in. It will have access to the environment secrets. We have no control over it.	Similar to enforcement, you as the agent builder is responsible for this.

How we built it

Complained about the aforementioned problem at 8pm
Researched on existing tools and steal ideas
Drafted out plans (Yes we wrote our own plans, see the markdown files).
Summoned 4 human agents to orchestrate swarm of agents to vibe code it out, burning >200$ and 2b+ tokens throughout this hackathon. Very fun.
Cross validated progress by using Auditor traces for our coding agents to diagnose with. Recursive improvement!
Planned out e2e workflow tests to test against

Refer to README.md for implementation details.