• 

PerfPilot

Autonomous performance engineering, from signal to optimization.

Inspiration

Modern cloud applications change continuously, but performance testing is still often a periodic and manual activity. Engineers maintain fragile load-testing scripts, move information between disconnected tools, investigate regressions after deployment, and repeat the same analysis for every release. In multi-tenant systems, this becomes even harder because workloads vary by tenant and shared resources create unpredictable contention.

PerfPilot was inspired by a simple question: what if performance engineering could operate as a continuous, intelligent feedback loop instead of a collection of manual checkpoints?

We set out to build an agent-oriented system that can discover an important performance target, create a test, run it, evaluate its SLA and efficiency, and publish a traceable result. The longer-term vision is a self-improving platform that helps teams prevent regressions while using cloud resources more efficiently.

What it does

PerfPilot coordinates five specialized agents across an asynchronous workflow:

1. The Discovery Agent receives application and tenant context, validates it, and prioritizes performance-critical endpoints.
2. The Authoring Agent converts that context into a normalized test specification containing workload parameters and SLA thresholds.
3. The Runner Agent executes the test workflow. The hackathon prototype uses a deterministic simulator behind an interface designed for real JMeter, Locust, or Playwright adapters.
4. The Analyst Agent checks p95 latency and error-rate SLAs, identifies violations, and calculates throughput per watt as an energy-efficiency indicator.
5. The Publisher Agent produces a final pass/fail report and closes the feedback loop.

Every event carries a shared correlation ID. This creates an end-to-end audit trail showing which input produced a test, which measurements were analyzed, and why the final result passed or failed.

The current prototype includes a command-line demo, an in-memory asynchronous event bus, typed domain models, workflow tests, sample configuration, Python packaging, and a container image definition.

How we built it

We built PerfPilot as a modular Python 3.12 application with clear boundaries between orchestration, agents, transport, and domain models.

The workflow uses immutable event envelopes containing a topic, source, payload, timestamp, event ID, and correlation ID. An asynchronous publish and subscribe bus routes these events between agents:

Discovery -> Authoring -> Runner -> Analyst -> Publisher
     ^                                           |
     +---------------- feedback -----------------+

Test scenarios are represented as typed specifications containing the endpoint, tenant, concurrency, duration, and SLA. This canonical model is intended to become the source for generated JMeter, Locust, and Playwright artifacts.

We kept the prototype dependency-light and used the Python standard library for the initial orchestration. This made the core workflow quick to run and easy to test while preserving extension points for MCP servers, durable messaging, observability platforms, and external testing tools.

The project also includes:

• A perfpilot demo command for running the complete workflow
• Automated tests covering validation, publication, and trace correlation
• A sample discovery target for a multi-tenant API
• A Dockerfile running the application as a non-root user
• Environment-variable and secret-management placeholders

Challenges we ran into

The biggest design challenge was balancing autonomy with safety and explainability. A system that can generate workloads and act on cloud infrastructure must never become an opaque agent with unrestricted control. We therefore made correlation, immutable events, explicit SLAs, and bounded agent responsibilities part of the architecture from the beginning.

Another challenge was defining useful boundaries between agents. If each agent knows too much, the system becomes a monolith disguised as a multi-agent application. If responsibilities are divided too narrowly, coordination overhead becomes greater than the value of specialization. The five-stage model gave us a practical separation aligned with the real performance-engineering lifecycle.

We also had to distinguish a credible hackathon prototype from the complete production vision. Integrations with load generators, telemetry platforms, and enterprise tools introduce credentials, network access, failure recovery, and schema-evolution concerns. We isolated those behind replaceable boundaries and used deterministic execution for the prototype rather than pretending that every production integration was complete.

Finally, energy-aware performance engineering presents a measurement problem. Infrastructure providers commonly expose power estimates at a VM or node level, while performance decisions may concern one process or tenant. PerfPilot starts with throughput per watt as a transparent metric, but accurate tenant-level attribution will require richer telemetry.

Accomplishments that we're proud of

We are proud that PerfPilot is more than an architecture diagram: the complete five-agent pipeline runs end to end.

The prototype can take a tenant-specific API signal, generate a test specification, simulate a performance run, validate latency and error-rate SLAs, calculate energy efficiency, and publish a correlated report. Automated tests verify that all five workflow events share the same correlation ID, providing a foundation for explainable and auditable automation.

We are also proud of the project's extensibility. The simulated runner can be replaced with a real testing adapter without rewriting discovery or analysis. Likewise, the in-memory bus establishes a transport contract that can later use MCP-compatible tools or a durable event system.

Most importantly, the project treats performance, reliability, and sustainability as related optimization goals rather than separate dashboards.

What we learned

We learned that agentic systems benefit from strong contracts more than from unbounded autonomy. Typed test specifications, explicit events, correlation IDs, and narrowly scoped agents make the system easier to reason about, test, and trust.

We also learned that the feedback loop is the defining feature. Automating test execution alone saves time, but connecting discovery, authoring, analysis, and publication is what allows the system to improve future decisions.

Performance data needs context to become actionable. A latency spike is much more useful when it can be connected to a tenant, endpoint, code change, workload profile, and SLA. This is where an MCP-driven architecture can add significant value: it can give agents controlled access to the context and tools needed for a decision without tightly coupling every integration.

Finally, sustainability metrics should be included at design time. Adding them after a platform is mature makes it difficult to explain why an optimization improved latency but increased resource consumption.

What's next for PerfPilot

Our next milestone is to replace simulated execution with a real Locust or JMeter adapter and generate executable test artifacts from the canonical test specification.

After that, we plan to add:

• OpenAPI and Git-based discovery of changed or high-risk endpoints
• MCP tool integrations for controlled access to external systems
• Playwright generation for browser and user-experience performance testing
• Prometheus and OpenTelemetry ingestion for real runtime measurements
• Statistical anomaly detection using historical baselines
• Durable workflow state, retries, idempotency, and dead-letter handling
• Jira, Git, and Grafana publishing connectors
• Adaptive load control with safety limits and human approval gates
• Tenant-aware dashboards and more accurate energy attribution
• Cost- and carbon-aware scheduling for non-urgent test workloads

The long-term goal is for PerfPilot to become a trusted performance copilot that can continuously observe, test, explain, and recommend optimizations—and, where explicitly authorized, safely apply them.

Built With

• jmeter
• mcp
• openapi
• opentelemetry
• postgresql
• prometheus
• python
• redis