Inspiration
Modern CI/CD pipelines are great at catching syntax errors and running automated tests, but they often miss the legal and organizational compliance risks that can cause serious headaches down the road. It is easy for a developer to accidentally commit code with an incompatible open-source license, expose private credentials, or miss basic production readiness steps.
I built Jacob Compliance Agent (Agent Fresh) to act as an independent, intelligent auditor that automatically bridges this gap. Instead of just logging errors to a standalone dashboard, I wanted a tool that integrates directly into the development workflow by creating tracking issues right where the code lives.
What it does
Jacob Compliance Agent (Agent Fresh) is an autonomous development auditor designed to scan repositories and flag compliance issues before code goes live. Instead of just tracking static metrics on a disconnected dashboard, it actively participates in your development workflow.
When you target a public repository path from GitHub or GitLab, the agent runs a focused audit against your selected policy track, such as checking for license compliance (like spotting high-risk GPL-3.0 copyleft traps), scanning for exposed secrets, or evaluating production readiness baselines.
If the agent catches a structural or policy violation, it doesn't just print a report; it uses native function tools to automatically open a structured tracking issue directly inside the host repository. This ensures that critical fixes are logged immediately right where developers are already working.
How I built it
I engineered the application with a completely decoupled architecture, breaking the solution into lightweight, dedicated components built for speed and isolated security:
Backend API: Built using FastAPI to create a fast, asynchronous gateway. This layer parses audit payloads, manages system instruction parameters, and orchestrates the downstream tasks cleanly.
Intelligence Layer: Powered by the gemini-2.5-flash model via the Google GenAI SDK. I implemented native function calling directly into the model's execution config. This allows the AI to autonomously determine if a repository configuration violates a track policy and natively call my custom Python tools (create_gitlab_issue or create_github_issue) to push tracker updates back to the platforms.
User Interface: Developed a clean, responsive web dashboard using Streamlit.
Cloud Infrastructure: The backend API and frontend dashboard are hosted as separate, communicating web services on Render.com. All production access tokens and keys are securely isolated within cloud environment variables, meaning the live application runs with zero local configuration file dependencies.
Challenges I ran into
Handling Temporary API Load Spikes: During periods of high demand, the underlying AI engine would occasionally return a transient 503 Service Unavailable error. To prevent the application from crashing or showing a raw error to a user during evaluation, I implemented strict exception handling using the native SDK errors. This intercepts the server overload gracefully and translates it into a clear notification on the interface, advising the user to wait a few seconds and try again.
Billing and Environment Configuration: My initial setup relied on explicit service account configuration files to manage platform access. However, setting up a full Google Cloud billing account introduced immediate verification challenges due to local banking restrictions. To keep the project functional and completely free from billing blockers, I refactored the backend to authenticate directly through Google AI Studio using a standard API key, resolving the restriction while preserving the model's capabilities.
Git Tracking and Multi-Remote Syncing: I faced a common issue where local configuration keys were accidentally tracked by Git before my exclusion rules were finalized. This triggered automated security push protections, which I resolved by purging the files cleanly from Git's internal cache without altering my local directory. Furthermore, authenticating across both GitHub and GitLab required moving away from standard command-line credentials and embedding personal access tokens directly into the remote URL paths to achieve a seamless push to both codebases.
Accomplishments that we're proud of
Building a Fully Functional Autonomous Workflow: I am incredibly proud of successfully implementing deterministic function calling. Seeing the AI model analyze a repository, accurately identify a policy violation, and autonomously execute the exact code needed to open a real tracking issue on GitLab or GitHub without human intervention was a massive milestone.
Designing a Defensive, Resilient System: Instead of just building a simple "happy path" application that breaks under load, I successfully engineered robust error-handling mechanisms into the core pipeline. Intercepting transient 503 server overloads and translating them into helpful, real-time user guidance on the frontend makes the application feel like a stable, reliable tool. Pivoting to Overcome Real World Blockers: When local banking constraints threatened to halt development by blocking a full Google Cloud billing configuration, I successfully refactored the entire infrastructure authentication model on the fly to run completely through Google AI Studio. Finding a workaround that kept the project moving forward without losing any intelligence or feature capabilities was a great problem-solving win.
What we learned
Reliable Function Calling and Guardrails: I gained a much deeper understanding of how to structure system instructions so that an AI model can act as a dependable agent. I learned how to configure the model so it accurately executes external webhooks and custom Python tools only when specific policy thresholds are violated, keeping its autonomous behavior safe and predictable.
Decoupled System Communication: This project reinforced the value of moving away from monolithic designs. I learned how to build a clean FastAPI backend that manages heavy AI processing and external API communication independently from a frontend interface, allowing both services to deploy and scale seamlessly as separate web entities in the cloud.
Defensive UX Engineering: I learned that building an excellent developer tool means writing code that openly anticipates external infrastructure instability. Designing the application to handle transient server drops gracefully taught me how important it is to communicate clear, actionable feedback to the user on the frontend rather than allowing a system to fail silently or crash.
What's next for Jacob Compliance Agent (Agent Fresh)
Expanding the Agent Ecosystem: The ultimate goal is to evolve this project from a single compliance tool into a comprehensive suite of dedicated, specialized AI development agents. I plan to build and integrate new agent types into the pipeline such as, an automated code-refactoring agent that suggests structural rewrites for performance, and a documentation agent that automatically keeps repository readme(s) and wikis perfectly in sync with codebase changes.
Deeper CI/CD Pipeline Integration: I plan to move beyond manual repository URL scanning by packaging Agent Fresh as a native GitHub Action and GitLab CI/CD component. This will allow the agent to run completely automatically on every pull request or code commit, blocking compliance violations before they ever merge into main branches.
Advanced Custom Policy Frameworks: I want to give developers the ability to upload their own unique, localized company policy guidelines (such as specific internal security checklists or styling rules). The agent will then dynamically parse these documents to audit repositories against custom corporate standards, rather than relying solely on pre-configured tracking rules.
Interactive Issue Resolution: Right now, the agent excels at identifying problems and opening tracking issues. The next major feature upgrade will allow the agent to attach a recommended code patch or automated merge request directly to the issue it creates, allowing developers to review and approve the compliance fix with a single click.
Log in or sign up for Devpost to join the conversation.