AI Security Engineer

Plug and play AI security engineer - helps you identify security vulnerabilities in the code and also generate patches to fix them. Always on security - now possible!

Comment

Inspiration

Security is always an afterthought for a software engineer. But it doesn't have to be. With an AI security engineer, you've a teammate who is constantly looking for vulnerabilities and also creating patches to remediate them.

What it does

It performs different kinds of scanners on the code repository, e.g. static, sca or secrets. Once it identifies the vulnerabilities, it gives the user to auto-fix the vulnerability. That autofix will create a pull request which remediates the vulnerability.

How we built it

We used the Claude 4.5 API to build the auto-fix functionality. Other than that, we have integrated a few open-source security scanners to find vulnerabilities. We have built safeguards while building this functionality:

  1. We run the scanners after applying the patch to ensure we have remediated the vulnerability and have not added new vulnerabilities
  2. We also build and test the repository to ensure that we are not introducing bugs.
  3. We use LLM as a judge (Claude Opus 4.1) to get a verdict on the quality of PR.

Challenges we ran into

  1. Sometimes the code given by LLMs does not work, so we have built retries into it.
  2. There are many failure modes in the whole process, so currently we log all the failures so that we can analyse them.
  3. Developers do not understand security issues, so it requires explaining the issues to them.

Accomplishments that we're proud of

  1. We were able to find vulnerabilities and raise Pull Requests in almost all trending GitHub repositories. In total, we raised > 100 PRs.
  2. A lot of PRs got merged as well, and they thanked us for the contribution.

What we learned

  1. Creating an automated PR is still risky, as there can be unintended consequences of the code, so we have to continuously build guardrails.
  2. Developer community has not yet warmed to AI-created PRs, so sometimes they shrug it off even though the PR is genuine.
  3. Developers do not create about a security vulnerability until you show them how it can be exploited; otherwise, it is a nice-to-have feature.

What's next for an AI Security Engineer

  1. The space of code security does not have very good public benchmarks/evals. We would like to build a benchmark and contribute it to the open source.
  2. Take AI security engineer to early-mid stage startups to see how they can use it in their workflow.

Built With

Share this project:

Updates