DebugPilot AI

An AI-powered debugging copilot that analyzes logs, detects dependency and configuration issues, and provides actionable fixes for robotics and computer vision projects.

Comment

Inspiration

Robotics and computer vision projects often fail for reasons unrelated to the core algorithm. A developer may spend hours diagnosing incompatible CUDA versions, missing shared libraries, incorrect environment variables, ROS configuration problems, or conflicts between OpenCV, PyTorch, and system packages.

Traditional search-based debugging requires developers to manually extract useful lines from long terminal logs, identify the relevant subsystem, compare multiple solutions, and determine which commands are safe for their specific environment. This process is slow, repetitive, and particularly difficult for students and developers who are new to robotics.

We built DebugPilot AI to make this process faster and more systematic. Our goal is to provide developers with an AI-powered debugging copilot that can transform complex logs into clear diagnoses and actionable recovery steps.

What it does

DebugPilot AI analyzes build logs, runtime errors, environment information, and dependency configurations from robotics and computer vision projects.

The system can:

  • Detect important error messages inside long terminal logs
  • Classify problems by category, such as dependencies, networking, compilation, GPU configuration, ROS, Python, or shared libraries
  • Identify likely root causes instead of only repeating the visible error
  • Generate step-by-step diagnostic commands
  • Recommend environment-specific fixes
  • Explain why each fix is needed
  • Warn users before suggesting potentially destructive commands
  • Preserve debugging context across multiple troubleshooting steps

For example, when a user submits an error such as:

error while loading shared libraries: libopencv_core.so.4.2

DebugPilot AI does not simply recommend reinstalling OpenCV. It first checks the active library paths, installed OpenCV versions, executable dependencies, and environment variables. It then proposes targeted commands such as:

ldd ./application | grep "not found"
ldconfig -p | grep opencv
echo "$LD_LIBRARY_PATH"

Based on the results, it can distinguish between a missing installation, an incorrect library path, and a binary compiled against a different OpenCV version.

How we built it

DebugPilot AI was designed as a modular debugging pipeline.

1. Log preprocessing

The application accepts terminal output, compiler logs, stack traces, and basic system information. Before sending the input to the reasoning layer, the preprocessing module removes repeated lines, terminal formatting characters, progress bars, and unrelated output.

It then extracts high-value patterns such as:

fatal error
undefined reference
command not found
permission denied
connection reset
version mismatch
cannot open shared object file

This reduces noise and allows the system to focus on the most relevant evidence.

2. Error classification

The extracted information is classified into technical categories, including:

  • Build and compiler errors
  • Missing dependencies
  • Python environment conflicts
  • CUDA and GPU issues
  • ROS configuration errors
  • OpenCV and C++ ABI mismatches
  • Network, proxy, and TLS failures
  • Runtime library loading problems

The classification result determines which diagnostic workflow should be used.

3. Context-aware reasoning

A major design goal was to avoid generic advice. The system collects available context such as:

  • Operating system and version
  • Python version
  • CUDA version
  • ROS distribution
  • Compiler version
  • Virtual environment
  • Package manager
  • Project framework

The recommendation is then conditioned on this environment.

Conceptually, the system ranks a candidate solution (s) using:

$$ Score(s) = \alpha R(s) + \beta C(s) + \gamma S(s) - \delta D(s) $$

where:

  • (R(s)) represents relevance to the observed error
  • (C(s)) represents compatibility with the detected environment
  • (S(s)) represents solution safety
  • (D(s)) represents the risk of destructive or unnecessary changes

This ranking helps prioritize diagnostic commands before invasive fixes.

4. Structured response generation

Instead of returning an unstructured paragraph, DebugPilot AI generates a response containing:

  1. The likely root cause
  2. Evidence from the submitted log
  3. Diagnostic commands
  4. Recommended fixes
  5. Verification commands
  6. Alternative explanations when confidence is limited

This structure makes the output easier to follow and reduces the risk of users applying commands without understanding them.

5. User interface

We created a lightweight interface where users can paste logs, select their development environment, and receive a structured debugging report. The interface separates the diagnosis, commands, explanations, and warnings so that users can quickly identify the next action.

Challenges we faced

Distinguishing symptoms from root causes

Many errors are only secondary symptoms. For example, a package may fail to import because of a missing shared library, but the true cause may be an incompatible compiler ABI or an incorrectly activated Python environment.

We addressed this by designing diagnostic trees that begin with evidence-gathering commands before recommending installation or configuration changes.

Handling incomplete context

Users frequently submit only one line of an error message. Without operating-system, dependency, or version information, several explanations may be equally plausible.

To handle this, DebugPilot AI reports confidence levels and requests only the most relevant missing information. It also provides safe commands that users can run to collect that information.

Preventing unsafe recommendations

Debugging suggestions found online sometimes include destructive commands such as removing system packages, replacing symbolic links, or changing permissions globally.

We added command-risk checks and designed the system to prefer reversible actions. Potentially destructive operations are clearly marked and are not presented as the first solution.

Supporting fragmented robotics environments

Robotics projects often combine C++, Python, ROS, CUDA, OpenCV, deep-learning frameworks, hardware drivers, and third-party libraries. A change in one layer can break another.

Building useful workflows across these interconnected systems was one of the most difficult parts of the project. We therefore focused the prototype on common, high-impact error categories rather than attempting to support every possible development stack.

What we learned

We learned that effective AI debugging requires more than generating shell commands. A useful debugging assistant must reason about environment compatibility, collect evidence, distinguish root causes from symptoms, and communicate uncertainty.

We also learned the importance of structured output. Developers are more likely to trust and correctly apply a recommendation when they can see the supporting evidence, the expected result of each command, and a clear method for verifying the fix.

Finally, we learned that safety is an essential part of developer tooling. A technically valid command is not necessarily an appropriate command, especially when it modifies system-wide packages or configuration files.

Accomplishments that we are proud of

  • Built a focused debugging workflow for robotics and computer vision development
  • Converted noisy logs into structured technical diagnoses
  • Added environment-aware recommendations instead of generic fixes
  • Designed verification steps for confirming whether a problem has been resolved
  • Introduced safety checks for potentially destructive commands
  • Created a modular architecture that can be extended with additional debugging agents

What's next for DebugPilot AI

We plan to expand DebugPilot AI with:

  • Direct integration with VS Code
  • Automated collection of project and environment information
  • Support for Docker and remote development environments
  • Repository-level dependency analysis
  • Continuous monitoring of build and test pipelines
  • A searchable database of verified debugging cases
  • Team knowledge sharing for recurring internal issues
  • Automated generation of reproducible bug reports
  • Local models for private or offline debugging
  • Specialized agents for ROS, CUDA, Python, CMake, and DevOps

Our long-term vision is to turn DebugPilot AI into a development companion that not only fixes errors after they occur, but also detects configuration risks before they interrupt a project.

Built With

Share this project:

Updates