ParallelPilot

Inspiration

In data science and scientific computing, we often write Python code that runs perfectly but wastes hours because it runs on a single CPU core. Researchers and engineers are forced to choose between spending days rewriting code for parallelism or waiting hours for simulations to finish. Existing tools like Numba or Cython require manual intervention and deep knowledge of parallelism. We wanted to create a tool that brings high-performance computing (HPC) to every data scientist, regardless of their parallel programming expertise.

What it does

ParallelPilot is a CLI tool that takes serial Python code—typically used for simulations, Monte Carlo methods, or data preprocessing—analyzes it to identify parallelizable loops, and automatically generates an optimized version using multiprocessing or joblib. It runs a numerical verification cycle to ensure the parallel output exactly matches the serial version (within a user-defined tolerance) and measures the real speedup across different core counts. Finally, it generates a performance report with speedup curves and a detailed explanation of the parallelization strategy.

How we built it

We built ParallelPilot using a dual-agent architecture:

• Codex (headless mode): We leveraged Codex's autonomous code generation and testing capabilities. Codex iteratively generates parallelized code, executes it, verifies numerical correctness, and retries with different strategies if tests fail. This is not a simple one-shot prompt—Codex acts as an autonomous optimization engine that "works" until the code passes all checks.
• GPT-5.6: Used for high-level reasoning and strategy selection. GPT-5.6 analyzes the code's structure and the results of failed attempts to decide the next parallelization strategy (e.g., chunk size, process pool vs. thread pool). It also interprets the performance metrics and writes the final executive report in natural language.
• Core stack: Python, multiprocessing, joblib, click for CLI, pandas for report generation, and matplotlib for visual performance curves.

Challenges we ran into

• Numerical stability: Running parallel code can introduce floating-point errors due to non-deterministic execution order. We implemented a strict numerical verification loop with tolerance checks and fallback strategies.
• Loop dependency detection: Identifying which loops are truly independent and can be parallelized was a major challenge. We used GPT-5.6 to analyze control flow and data dependencies before handing the code to Codex.
• Codex iteration loop: Getting Codex to reliably retry when tests fail required careful prompt engineering and session management to keep the context window relevant without ballooning token usage.

Accomplishments that we're proud of

• Achieved real speedups of 4x–8x on tested simulation scripts without any manual code modification.
• Built a fully autonomous pipeline where Codex iterates until tests pass and automatically generates a professional performance report.
• Designed a clean, developer-friendly CLI that makes HPC accessible to any Python developer.
• Successfully integrated Codex and GPT-5.6 in a complementary workflow—Codex does the "heavy lifting" (code generation and testing), while GPT-5.6 handles the "intelligence" (strategy and reporting).

What we learned

• Codex is incredibly powerful for iterative code generation when paired with real execution feedback (headless mode).
• The combination of a "fast" reasoning model for code generation (Codex) and a "slow" reasoning model for strategy (GPT-5.6) creates a surprisingly effective agentic workflow.
• Parallelizing Python code automatically is harder than it looks, but with the right verification pipeline, it's entirely feasible.

What's next for ParallelPilot

• Adding support for CUDA GPU parallelization and distributed multi-node clusters.
• Creating a GitHub Action that automatically reviews PRs for parallelization opportunities.
• Expanding language support to C++ with OpenMP.
• Building a VS Code extension to bring ParallelPilot directly into the developer's IDE.

Built With

• automation
• cli
• codex
• devops
• hpc
• joblib
• machine-learning
• multiprocessing
• parallel-computing
• performance-benchmarking
• python