-
-
TuneProof results overview: 62.5 ms → 43.9 ms per image (1.43×) on AWS EC2, with no loss of detection quality
Inspiration
To cut cloud vision costs, teams shrink images, swap algorithms or reduce kernels, then ship. Nobody checks what silently broke. The first casualties are the small, faint features the pipeline exists to find. We wanted an agent that makes vision pipelines faster and is not allowed to make them wrong.
What it does
TuneProof profiles an OpenCV 5 pipeline stage by stage. It picks the next experiment from that evidence by attacking the stage that costs the most. It accepts a change only if the output stays within a declared quality guard and is measurably faster. The guard requires agreement with the baseline output of at least 0.95 F1, a ground-truth F1 drop of at most 0.03, and at least 2% faster. Rejected experiments are logged with reasons. On AWS EC2 it took a defect-detection pipeline from 62.5 ms to 43.9 ms per image (1.43×) in 5 experiments with no loss of detection quality. The guard rejected three tempting speedups: shrinking the image to 75%, a quarter-resolution background, and a smaller window.
How we built it
We used Python, OpenCV 5.0 and NumPy, with pinned dependencies and 13 unit tests. The workload is an anomaly-detection pipeline: large-window median background, residual z-score, morphology and connected components. The tuner is a measure → choose → verify → accept/reject loop. choose_next is the hook where an LLM planner (for example on Amazon Bedrock) can be swapped in. We ran the full install, tests and benchmark on AWS EC2 (t3.micro, Ubuntu 24.04), then terminated the instance. Every experiment is saved in results/coolpilot_aws.json.
Challenges we ran into
Setting a guard that blocks real damage without blocking everything.
An early image-repair prototype made dark images darker because of an inverted gamma exponent. We found it by tracing each action’s effect.
Getting honest timings (warm-up, medians), and then reporting that one accepted step (threads=1, +2.3%) is within noise.
Accomplishments that we're proud of
The 1.43× speedup came with no loss of detection quality (agreement 0.974, ground-truth F1 unchanged at 0.676).
An exhaustive 200-configuration grid found the same best configuration.
Random search with the same 5-experiment budget averaged only 1.07×.
Every decision, including rejections, is logged and reproducible with one command.
What we learned
Most of the speedup came from one stage, the background model (25.0 → 7.4 ms). The quality guard matters as much as the optimizer: the biggest “wins” were the ones it refused. Reporting limits openly makes the results easier to trust.
What's next for TuneProof
An LLM planner over the same tools (Amazon Bedrock, MCP).
Real image corpora and per-workload guards.
Reducing the residual stage, which is now about 64% of runtime.
Log in or sign up for Devpost to join the conversation.