Patchwork

Inspiration

Asankhaya led product development on vulnerability remediation (using GPT-2) at Veracode, while I offered the same as a service at my previous start-up Scantist. The core value proposition at both companies was vulnerability scanning and not remediation, but our experience told us that this was a major pain point. When GPT-3.5 and open-source LLMs like StarCoder came out, we believed that the state of technology had evolved to a point where a product might be feasible - and we tested the idea out with a talk at FOSSASIA and subsequently a landing page.

After a few rounds of customer discovery and user feedback on a security-focused MVP, we quickly realized that fixing security alone wasn’t a hair-on-fire problem. Additionally, the state of LLMs isn’t ready for generalized autonomous systems and organizations aren’t fully comfortable with black-box AI systems touching their code. We then pivoted to an open-source framework that offers a much wider value proposition, while also allowing for transparency and user-specific tailoring.

What it does

Patchwork is an open-source framework that deterministically automates DevOps and maintenance tasks for development teams. It automates tasks like vulnerability fixing, code reviews, documentation, and dependency management out of the box, and can be extended to automate other DevOps tasks using a mix of natural language and code.

Users can either self-host a command line agent (patchwork), or use the managed service via the web (patched app). They can use their own LLM keys (for e.g. Google Gemini) or get one from our managed service. These modalities offer different levels of privacy and ease of use that the user can choose from.

Once set up, the automations are carried out by running the corresponding ‘patchflows’ during the software development lifecycle. A patchflow is a logical combination of ‘steps’ that carry out a specific task, typically with at least one step involving the use of an LLM.

Patchflows are written in Python, but are customizable using natural language prompts as well - again allowing for a tradeoff between specificity, flexibility, and ease of use. The same applies to the steps that make up the patchflows.

The result is a framework that provides a starter set of DevOps automations that can be extended and customized infinitely to provide tailored, deterministic outcomes for the end user.

How we built it

The entire framework is built in Python to ensure ease of readability and extensibility. We identified common DevOps actions that could benefit the most from LLM-enabled automation and built patchflows for them in a modular manner by defining reusable steps.

Challenges we ran into

Building a truly modular, reusable architecture for patchflows was our biggest challenge. Also, working out the best cost-to-performance ratio for our users when using LLMs.

Accomplishments that we're proud of

Feedback from early users and developers who have implemented this in their workflows over established solutions like Github Copilot.

What we learned

There is a strong preference for open-source solutions, especially in the software development world. The preference has only strengthened given that private data is extremely valuable in an AI-first world.

What's next for Patchwork

We are working on a GUI patch flow builder, as well as a standalone vector store for code repositories to expand the breadth and depth of automation possible with Patchwork. We are also looking to work with enterprise design partners who'd prefer to build their AI-enabled development lifecycle instead of relying on proprietary closed-source solutions.

Built With

• python