Bench.it

Landing Page
Benchmarks
Documentation - 1
Documentation - 2
Workbench

Inspiration

We wanted to answer a question that comes up whenever someone says one implementation is faster than another. Does that still hold for any arbitrary input? Big-O notation tells us how the algorithm scales but not how it performs on various architectures and languages. We built Bench.it to make it easy to compile and run code on different languages and architectures and see their performance over a number of runs. Bench.it shows the whole comparison and see where performance changes and how the language, the inputs, and the internal things of the computer such as the cache and the processor influence it.

What it does

Bench.it is a browser-based workbench for comparing different language benchmarks side by side. It works with Go and Python for now. You can write pseudo-code and use our website to generate editable high-level language code, or even write the code yourself. You give various input values and architecture you want to test it on, and Bench.it runs every combination and plots the results as they come back. Bench.it displays runtime, page faults, cache misses, and branch misses for each run. You can compare different code bases and architectures with different input sizes without installing or setting up any environment, all from the browser.

How we built it

We built the workbench with React and TypeScript, using CodeMirror for the editor and ECharts for the charts. The browser sends each benchmark configuration to a Go proxy, which routes it to a configured Go or Python execution server with different architectures such as x86 and arm64. The profilers on the execution servers time the benchmark body separately from setup and cleanup allowing you to benchmark specific parts of the code instead of the whole process. They collect run times, page faults and hardware counters such as cache misses and branch mispredictions and report it back to the user for each combination.

Challenges we ran into

We had to be precise about what the timer was measuring so we chose the timestamp counters the CPU exposes rather than going with a generic OS level timer implementation. Generating an input array or checking the output can take enough time to change the result, so we gave setup, the benchmark body, and cleanup separate sections. These also helped us to allow user to benchmark a specific section of the code rather the the whole executable. Hardware counters were also a problem because the availability varies by operating system, CPU, and permissions, so we had to preserve timing results while clearly marking counters the profiler couldn’t read.

Accomplishments that we're proud of

Seeing the chart fill in as the runner worked was the moment the project came together! This project required creating profilers to execute Python and Go code from the psuedo-code provided by the user, which was a challenging task. We had to also spend a large amount of time to learn how to access hardware counters in different OSs.

What we learned

A benchmark is only as useful as the conditions you record. In our n-split merge sort experiment, we learned that while Big-O notation gives us an idea of how an algorithm scales with sizes, it is not always true in practical world because it makes a lot of assumptions of how machines work. n-split merge sort becomes more or equally efficient as n grows however, Bench.it allowed us to see that the performance actually degrades as the number of splits grow. This tool allows us to code, compile, run, and inspect the benchmarks of it performance on different architectures and languages.

What's next for Bench.it

These are the future plans:

Bench.it currently supports Linux with x86 and MacOs with Arm64 architectures. We plan on increasing the number of architectures supported including Windows.
Bench.it supports two languages - Go and Python. We plan on increasing the number of languages supported.
Bench.it supports standard compilers for the high level languages and we plan on increasing the compilers supported.
Bench.it allows execution on 2 types of CPU chips. We plan on increasing the number of hardware architectures in the future.