Ledger

• 

  SIFT Workstation Diagram

• 

  Output Pipeline Diagram

Inspiration

Our team was drawn to the opportunity to experiment hands-on with how to improve frontier model performance for security automation. The goal was to increase our own understanding while contributing what we learned back to the community.

What it does

Our implementation focused on improving the CLAUDE.md file, as well as adding several scripts and a hook to the agent’s configuration to enforce audit logging for every command, validate report claims, and ensure quality report rendering.

How we built it

We started by evaluating the existing CLAUDE.md file using manual and AI-assisted analysis, researching opportunities for improvement.

Some of the most meaningful additions to the CLAUDE.md file include a phased workflow approach. By breaking each step in the DFIR process into a separate phase, the agent performs its work in a more structured and predictable manner, resulting in significantly more consistent output.

Additionally, we developed scripts to log all command execution and output into a ledger (as well as a hook to enforce this behavior), audit the ledger for completeness, check for any failed commands, validate the agent’s claims were backed by evidence and add confidence values to them, and check the final report PDF for visual errors (something we encountered quite frequently with the baseline configuration).

We tested our agent against another local agent built using the example sift-mcp (https://github.com/AppliedIR/sift-mcp/tree/main) project to compare how our basic implementation would fair against a more complex agent with a more structured and well-defined set of commands and definitions. Both agents were tested against the same evidence file multiple times to see how the outputs differed.

Interestingly, our implementation outperformed the more complex sift-mcp project in a few cases. In one instance, the sift-mcp agent halted execution and refused to continue running due to a usage policy violation. This type of error might be avoided by using more recent cyber-oriented frontier models with relaxed guardrails for validated cyber use cases. Still, it is interesting that our agent avoided running into such guardrails even while using a standard Claude subscription. In another instance, our agent surfaced a confirmed finding which the sift-mcp agent missed. Even when presented with the evidence, the sift-mcp agent insisted that our agent’s finding was a hallucination.

Challenges we ran into

One of the issues we encountered was a lack of realistic test case data that the agent’s performance could be objectively graded against. For this project, we chose to use the case data provided by SANS. This presented challenges since without complete knowledge of the input data, we could not verify that the agent found all IOCs. Also, in some cases the agent detected that the SANS data was a fictional case due to artifacts being left behind from case creation, which potentially biased its results.

We were able to largely resolve this by manually checking an evidence file to confirm the veracity of the report's claims. This enabled us to have an expected outcome for each run of that evidence file. While not perfect, it was sufficient for this project’s scope to demonstrate the improvements from our CLAUDE.md file, hook, and scripts.

Accomplishments that we're proud of

We are most proud of the improvements our implementation achieved in DFIR consistency and auditability. While there are minor differences in the output between runs, our additional scripts, hook, and CLAUDE.md improvements yielded much more consistent results than the baseline configuration.

What we learned

Our team learned valuable lessons related to prompt engineering and agent configuration. First, we observed that breaking up DFIR instructions into clear phases helps AI agents to perform them with more reliability and consistency. Second, we experienced the value of using agent hooks and scripts to more deterministically shape agent behavior.

Additionally, we learned that using an AI (such as ChatGPT Enterprise) to assist in the review and improvement of configurations of separate AI agents (e.g. our CLAUDE.md file) is immensely useful. Agent configurations for detailed use cases like DFIR can become verbose and complex very quickly. Incorporating AI into the development of these harnesses helps manage that scale and facilitates more rapid experimentation.

Unfortunately, in our view, AI is not yet ready for fully independent automated DFIR. Despite the improvements we made, there are still occasional hallucinations and inconsistencies which cause its performance to fall short of the rigor required for real investigations. It's possible this will improve as new models are released (we used Claude Opus 4.7 and occasionally Haiku 4.7). However, even in its current state, it shows promise as a valuable assistant to human analysts, accelerating their work by performing rapid first-pass analysis and report generation, which can in turn be validated and modified by the human collaborator.

What's next for Ledger

We intend to continue learning about AI agents and where they can be most helpful in cybersecurity workflows. We would like to learn more about how to write more effective CLAUDE.md files, skill files, hooks, etc. We are also interested in using and developing MCP servers to give agents more reliable access to security tooling.

Built With

• claude
• md
• python
• sh