PerTaaS: Performance testing as a Service

This is an opinionated way of doing the performance test using virtual threads in Java. Quarkus/Vert.x & other frameworks are capable of giving a tough competition to wrk, wrk2, Nghttp2 & others.

Comment

Inspiration

This is an opinionated way of doing the performance test using virtual threads in Java. Virtual threads are lightweight threads that reduce the effort of writing, maintaining, and debugging high-throughput concurrent applications which has been introduced as part of Java 21 (LTS release). Quarkus started providing official support for Java 21 with 3.5.0 release as of October 23, 2023.

Currently, the performance tools that are used to run test are wrk, wrk2, plow, Nghttp2, jmeter and others. We lack a performance tool that can provide comparable performance to these tools using Java. Quarkus, Vert.x and other frameworks/toolset that have gained popularity over some years are based on lightweight, asynchronous and reactive netty framework and are capable of giving a tough competition to these performance tools.

What it does

The sample application provides basic operations to run the tests.

  1. There is a generic template that has been provided to run performance tests for basic needs. The current implementation of the test allows to test REST Api's running on HTTP 1.0/1.1/2 protocol. The source code for it is available at https://github.com/himanshumps/pertaas-job.git. The Readme.md file has the details about what type of tests can be executed. For any customized tests, like HMAC, OAuth or any other requirements, the source can be forked or a new repo can be created. The file HTTPRequestSupplierParameterized.java can be modified to suit the business requirement.
  2. Once the performance test code has been modified and pushed to github, the image for the same can be created using the custom image creator. This uses a docker strategy to build the code using tekton pipelines and push it as an image to the openshift registry.
  3. For the performance test to be available to other developers, the Helm job should be created which gives the flexibility of running the job from openshift console using helm. The Helm repository is available at https://github.com/himanshumps/pertaas-helm.git. The helm chart should provide all the details so that it is easy for developers to use the helm chart.
  4. The test can be started using the start test. The current start test page has been created to support the input parameters for the basic test. The various example of the json that can provided in the request json can be generated by modifying the file at examples. Once the test is submitted for start, it runs a tekton pipeline which in turn executes the helm chart by providing the values which have been provided as part of start test. The helm chart can also be executed from openshift helm chart by providing the values. See the chart with the name PerTaaS basic chart in the helm chart. The performance test send the metrics to console(can be read in kibana/splunk or any other logging framework), prometheus and couchbase.
  5. The helm chart created for pertaas basic test, refers to the image in the openshift image registry. It can be changed to refer performance test job image from external docker registries as well.
  6. The performance metrics for the test can be viewed at metrics page by providing the job id. There are lots of metrics that are captured as part of the scrape, but the primary ones have been displayed on the metrics page. To see the list of all the metrics that are available, refer to this page.
  7. If the job needs to be stopped in between due to any reason, the stop test can be used to stop the test. The summary metrics would not be generated in case the test is stopped in between.
  8. If a custom job ID is required for CI/CD integration or for Helm chart, it can be generated by calling the generate ID page.

How I built it

I have used openshift and redhat images for all the applications that are deployed on developer sandbox. These are the repo's and brief description about them.

Challenges I ran into

  • I started installing the openshift on my M1 Mac machine and realized that the images/operators that are available are not compatible with amd64 e.g. Couchbase Autonomous Operator
  • There are no java implementation for Helm charts.
  • There is no option for Helm Charts in openshift/k8s rest api. There is support for Helm repositories.
  • Micrometer is missing the support for clustered metrics.

Accomplishments that I am proud of

I was able to give wrk tool a run for it's money with such minimal code. There is an abundance of C/C++/Go/Rust based performance tool and the Java 21 virtual threads could be a game changer.

What we learned

I got an opportunity to work on openshift 4.x and couple of operators. The constant support on slack channel for basic queries helped in overcoming the bottlenecks while doing application development.

What's next for PerTaaS: Performance testing as a Service

  • Create more generic templates to support other protocols like HTTP/3 and websockets.
  • Look for improvement in the current code using flamegraphs.
  • Implement a cluster wide micrometer so that the load for the test can be distributed across multiple pods in the job
  • Use the json to form convertor to read the values.schema.json from Helm chart and generate the dynamic form.

Built With

Share this project:

Updates

posted an update —

We see that most of the performance tool uses shared resources and very heavy weight. The performance testing is also used very late in the project lifecycle leading to code changes. The PerTaaS allows us to use resources on demand and can be instrumental in doing the performance test very early while using the shift left approach. Thanks to the API driven approach, it can fit in any CI/CD tool in market. The k8s cronjobs can be use be used to trigger the performance test at scheduled frequency to ensure that there are no surprises when the code changes are pushed. Most of the companies uses performance testing tools running in a specified environment all the time irrespective of whether there are any test running and the cross environment testing introduces unwanted latency and produces incorrect metrics. The k8s environment makes it very easy to run the test for various scenarios like testing a server running in US from a application running in India and see various metrics or running a test which run the performance test from dev to dev region rather than prod to dev region. The running of test on demand makes it easy to schedule resources as they are needed and save cost at the same time.

Log in or sign up for Devpost to join the conversation.