Inspiration
Every developer knows the frustration of upgrading a dependency only to discover that previously working code suddenly breaks.
A simple package update can introduce removed APIs, incompatible function signatures, unexpected behavior, and failing tests. Fixing these problems often requires hours of debugging, reading migration guides, and manually rewriting code.
Existing dependency automation tools can identify outdated packages, but they don't always repair the application code affected by breaking changes.
This inspired UpgradePilot, an AI-powered dependency migration agent built around one important question:
What if AI could not only repair breaking dependency upgrades, but also provide evidence that its fixes are legitimate?
AI coding agents can sometimes make tests pass by skipping failures, weakening assertions, or changing test configurations. Such changes create the appearance of success without actually fixing the application.
Our goal was to build something different.
UpgradePilot: Don't just generate a fix. Verify it.
What it does
UpgradePilot is an autonomous Python dependency migration system designed to identify breaking changes, investigate failures, generate code repairs, and independently verify the results.
The workflow follows six stages:
- Baseline Testing: Run the original test suite to establish expected behavior before upgrading a dependency.
- Dependency Upgrade: Install the target package version and identify failing tests, exceptions, and compatibility issues.
- AI Investigation: Analyze error messages, relevant source code, and migration documentation retrieved through Tavily.
- Autonomous Repair: Use an AI coding agent to propose targeted modifications and evaluate them through repeated testing.
- Anti-Cheating Verification: Detect suspicious modifications such as skipped tests, weakened test configurations, or attempts to bypass the upgraded package.
- Evidence Generation: Record repair attempts, test outcomes, verification decisions, model information, and execution details.
UpgradePilot is designed to distinguish between genuine repairs, unsuccessful attempts, and modifications that compromise test integrity.
How we built it
We developed UpgradePilot primarily using Python, with a modular architecture separating dependency execution, AI reasoning, code repair, and verification.
Dependency Testing Harness
We designed a testing harness using isolated Python environments and pytest.
The harness establishes a baseline, upgrades the selected dependency, executes tests, and collects structured failure information for the repair agent.
AI-Powered Repair Agent
We built an LLM-driven workflow that examines dependency failures, identifies potentially incompatible APIs, and proposes source-code changes.
During development, we used NVIDIA NIM for model inference and designed a configurable provider architecture supporting integration with Nebius Token Factory.
Documentation Retrieval with Tavily
We incorporated Tavily into the workflow to retrieve migration guides, package documentation, and relevant technical information.
This helps the agent investigate breaking changes using external documentation rather than relying solely on model knowledge.
Anti-Cheating Guard
A major focus of UpgradePilot is preventing AI-generated patches from manipulating the evaluation process.
The guard is designed to identify suspicious modifications, including:
- Adding
skiporxfailmarkers. - Changing pytest configuration to exclude failures.
- Modifying protected test infrastructure.
- Creating files that shadow the upgraded dependency.
- Weakening verification instead of repairing application behavior.
Held-Out Behavioral Verification
We introduced the concept of independent behavioral assertions that remain hidden from the repair agent during patch generation.
These tests provide an additional verification layer beyond the original visible test suite.
Iterative Repair and Rollback
UpgradePilot evaluates repair attempts using actual test results.
The system is designed to preserve patches that reduce failures while rejecting invalid modifications and rolling back unsuccessful changes.
Developer Interface and Reporting
We worked on a command-line interface and structured result artifacts to make repair attempts inspectable.
The reporting workflow is designed to show test outcomes, repair decisions, provider information, and verification evidence.
Challenges we ran into
1. Reproducing Real Dependency Failures
Python dependency upgrades can behave differently depending on Python versions, package configurations, and transitive dependencies.
Creating reliable, repeatable testing environments was one of the most challenging engineering tasks.
2. Preventing AI Agents from Cheating
An AI-generated patch might make tests pass without preserving the intended behavior.
We had to design safeguards that distinguish legitimate source-code repairs from changes that manipulate test results.
3. Managing Multi-Step Repairs
Some dependency migrations require several coordinated changes.
An intermediate patch might fix multiple tests while leaving others broken. We explored ways to retain meaningful progress without accepting invalid modifications.
4. Protecting Held-Out Tests
Independent verification requires keeping certain behavioral assertions inaccessible to the repair agent.
Designing proper separation between the repair environment and verification environment introduced additional complexity.
5. Cloud Infrastructure and API Access
Integrating cloud-based models introduced challenges involving API access, billing configuration, and promotional credits.
Accessing Nebius Token Factory credits from India presented onboarding difficulties, reinforcing the importance of provider-independent architecture and transparent reporting.
6. Working Under Hackathon Constraints
As a solo student developer, balancing implementation, testing, debugging, documentation, and demonstration preparation within a limited timeframe required prioritizing reliability over unnecessary features.
Accomplishments that we're proud of
Building Around Trustworthy AI
Instead of focusing only on AI-generated patches, we designed UpgradePilot around verifiable autonomous software repair.
Developing an Anti-Cheating Architecture
We designed safeguards to identify attempts to bypass or weaken tests, making test integrity a central part of the system.
Introducing Independent Verification
Held-out behavioral checks provide an additional layer of evaluation beyond the tests available to the repair agent.
Creating a Reproducible Migration Workflow
We connected dependency installation, baseline testing, failure reproduction, repair attempts, and verification into a structured workflow.
Prioritizing Transparent Results
We focused on recording actual test outcomes, repair decisions, and model-provider information rather than presenting every attempt as successful.
Building an End-to-End AI Engineering Project
UpgradePilot brought together Python development, dependency management, LLM integration, automated testing, documentation retrieval, and AI-agent safety.
What we learned
1. Passing tests doesn't always mean a repair is correct.
Autonomous agents need independent verification because a successful-looking test result can sometimes hide broken behavior.
2. AI agents need clearly defined boundaries.
Allowing an agent to modify application code requires safeguards around test files, configuration, execution environments, and sensitive information.
3. Reproducibility is essential.
Meaningful evaluation requires tracking dependency versions, Python environments, test outcomes, and model providers.
4. Iterative feedback improves repair workflows.
Complex software migrations may require multiple attempts, making testing, rollback, and progress tracking important.
5. Failed experiments are valuable.
An unsuccessful repair can reveal weaknesses in the agent's reasoning, documentation context, or verification process.
6. Reliable AI systems require more than powerful models.
Testing infrastructure, isolation, observability, safety checks, and transparent reporting are essential parts of an autonomous engineering system.
What's next for UpgradePilot
Our long-term goal is to make UpgradePilot a practical autonomous maintenance assistant for real-world software repositories.
Planned improvements include:
- Broader Dependency Support: Expand the benchmark to additional Python packages and complex dependency migrations.
- Improved Agent Reasoning: Develop stronger multi-step repair strategies and context selection.
- Expanded Benchmarking: Evaluate more real-world repositories with reproducible dependency failures.
- Nebius Integration: Complete and validate the workflow using Nebius-hosted NVIDIA models.
- Automated Pull Requests: Generate reviewable dependency migration PRs with test evidence.
- CI/CD Integration: Explore automated dependency maintenance through repository workflows.
- Enhanced Observability: Provide richer execution timelines, repair comparisons, and verification reports.
- Stronger Safety Mechanisms: Improve held-out test isolation and expand anti-cheating detection.
Ultimately, UpgradePilot aims to reduce the manual effort required to maintain software dependencies while making autonomous AI-generated changes more transparent and trustworthy.
UpgradePilot — Because an AI-generated fix should come with evidence that it actually works.
Log in or sign up for Devpost to join the conversation.