FuzzGuard

• 

  FuzzGuard landing page

• 

  Repository status view

Inspiration

The increasingly popular usage of Large Language Models (LLMs) for software development has led to a surge in code creation, resulting in more and more lines of code being written at a faster pace than ever before.

However, as development speed continues increase, so does the speed at which potential vulnerabilities could be created. Our team wanted to explore possible solutions and work towards designing and developing something that could help identify potential vulnerabilities in this code that is also highly scalable. As we explored different options for code testing methodologies, we considered various different code analysis and testing approaches but eventually settled on fuzzing.

Fuzz testing, also known as fuzzing, is a dynamic software testing technique that involves providing invalid, unexpected, or random data as input to a computer program. The goal of fuzz testing is generally to discover coding errors and possible security loopholes in software.

Once we decided on a fuzzing-based tool as a potential way to address part of the larger problem we talked about earlier, we did some research on what existing tools, frameworks, etc. exist for automating fuzz testing. We found examples of implementing LLMs for fuzz test systems from both industry sources like Google (from a blog post detailing their usage of LLMs with their fuzzing tool OSS-Fuzz) and academic sources (we even sifted through a paper with UIUC involvement!

One thing we noticed with both sources was that documentation or general methodology was very complex and often for niche or specific use cases. This lead us to the groundwork for our project: we wanted to find a way to take some of the most useful concepts of these resources and apply it to the development of a tool that the everyday developer could easily actively benefit from.

What is FuzzGuard?

FuzzGuard is a Software-as-a-Service (SaaS) application that aims to enhance code security by leveraging fuzz testing techniques. Our platform takes your code repository, checks for code test coverage, and then utilizes an LLM to create numerous fuzz tests across the entire repository. These tests are then executed, and a feedback loop refines them in case any issues arise. The output data from these tests is aggregated and presented on our FuzzGuard site for ease of use.

On the FuzzGuard site, users sign into their GitHub accounts and view all of their repositories available to be fuzz tested. While our initial implementation supports Python, we aim to expand our language support in the future to encompass other programming languages.

How we built it

We built FuzzGuard using a combination of programming languages. For most of the code analysis, processing and test generation, and interfacing with the OpenAI API, we utilized Python. We utilized various libraries, including Google's open source fuzzing engine "Atheris".

The backend architecture followed an innovative and effective strategy, leveraging python coverage tools, Atheris, OpenAI, and abstract syntax tree analysis.

The initial analysis phase runs the following:

• Parse the AST of the selected repository, finding functions which benefit from fuzz testing
• Query LLM's to generate tests for the selected functions. To ensure quality, LLM's are provided with function context, example tests, and API references for Atheris

Following this is the execution phase:

• Execute generated tests, which provide coverage reports, inputs byte streams which lead to errors in code
• Due to the nature of LLM's, some tests fill no execute. These tests are fed back into the LLM in a feedback look, allowing more context for the LLM to create better tests

Finally, the reporting phase saves coverage reports, generated tests, exception reports, and troublesome input into Supabase to be viewed on the frontend.

Challenges we ran into

One of the primary challenges we encountered was integrating the LLM into our fuzz testing pipeline effectively. Since LLMs are still relatively new in the realm of software development, we faced some hurdles in optimizing their usage for generating meaningful fuzz tests. Additionally, ensuring compatibility and seamless integration with GitHub posed some challenges during the development process.

Accomplishments that we're proud of

Despite the challenges, we're proud to have developed a functional prototype of FuzzGuard that demonstrates the potential of leveraging LLMs for enhancing code security through fuzz testing. We're particularly pleased with the seamless integration with GitHub, as it enhances the accessibility and usability of our platform for developers.

What we learned

Through the development of FuzzGuard, we gained valuable insights into the complexities of integrating LLMs into software development workflows. We also deepened our understanding of fuzz testing techniques and their significance in identifying and mitigating potential vulnerabilities in codebases. Furthermore, the experience helped us refine our skills in web development and project management.

What's next for FuzzGuard

Moving forward, we plan to continue refining and enhancing the capabilities of FuzzGuard. Our immediate goals include expanding language support beyond Python to cater to a broader user base. Additionally, we aim to integrate FuzzGuard into GitHub actions, enabling users to automate fuzz testing at every commit or push to their repositories. We're committed to ongoing development and improvement to ensure that FuzzGuard remains a valuable tool for developers striving to enhance code security.

Built With

• atheris
• nextjs
• python
• supabase
• vercel