In this article, we will discuss benchmarking, provide a brief introduction to the Java Microbenchmark Harness with working examples
What is benchmarking?
Benchmarking is the practice of running code to compare the relative performance of different implementations of a data structure or algorithm.
Why benchmark?
Knowing which algorithms or data structures provide better performance for a given programming task allows you to design and implement higher performance code in general. Benchmarking helps you make those decisions.
The Java Microbenchmark Harness
What is the Java Microbenchmark Harness?
The Java Microbenchmark Harness, JMH, was developed as part of the OpenJDK project to provide a benchmarking tool for Java addressing these challenges. Due to the warmup time and dynamic optimization performed by the Java Virtual Machine, Java presents a challenging benchmarking environment. JMH abstracts away all the complexity and provides developers with a fairly straightforward API to perform benchmarking.
Using the Java Microbenchmark Harness
According to the official OpenJDK JMH document, the recommended way to set up a JMH benchmark project is to use the command line.
First create a pom file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.talentify</groupId>
<artifactId>benchmark-code</artifactId>
<version>0.0.1-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-archetype-plugin</artifactId>
<version>3.0.1</version>
</plugin>
</plugins>
</build>
<packaging>pom</packaging>
</project>
Run the following the command line in the directory with the pom file:
$ mvn archetype:org.apache.maven.plugins:maven-archetype-plugin:2.4:generate
-DinteractiveMode=false
-DarchetypeGroupId=org.openjdk.jmh
-DarchetypeArtifactId=jmh-java-benchmark-archetype
-DgroupId=com.talentify
-DartifactId=benchmark
-Dversion=1.0
The above command will insert the following into your pom file:
<modules>
<module>benchmark</module>
</modules>
It will also create the following subdirectories and files:
benchmark/pom.xml
benchmark/src/main/java/com/talentify/MyBenchmark.java
You must then implement a benchmark class by editing the generated MyBenchmark.java file (see below.) Once you have implemented your benchmark you can build and verify the project:
$ cd benchmark/
$ mvn clean verify
Finally run the benchmark:
java -jar target/benchmarks.jar
Implementing a Benchmark Class
The generated MyBenchmark.java file looks like this:
package com.talentify;
import org.openjdk.jmh.annotations.Benchmark;
public class MyBenchmark {
@Benchmark
public void testMethod() {
// This is a demo/sample template for building your JMH benchmarks. Edit as needed.
// Put your benchmark code here.
}
}
JMH makes heavy use of annotations. We see the @Benchmark annotation above. This indicates that this method will be benchmarked. Another critical annotation is BenchmarkMode, which indicates the types of benchmarks to run. It can be placed on a benchmark method or the entire benchmark class to run those benchmarks on all benchmark methods of that class.
The actual benchmarks are enumerated in the Mode enum. They include, for example, the average time per operation and operations per unit of time. We will illustrate how to use this annotation and several others in the example below.
Example: ArrayList vs LinkedList
Recall that the Java standard library includes both a LinkedList and an ArrayList implementation of the List interface. We will run a benchmark to compare the performance of the sort method on each of the classes.
We will use the following annotations:
- State We mark our class with the State annotation. This tells the JHM how our class’s state will be shared between the JHM’s worker threads.
- BenchmarkMode We already discussed the BenchmarkMode annotation above.
- Param This annotation indicates which parameters to configure for the benchmark. Note that the use of this annotation requires the State annotation on our class.
- Setup This annotation indicates which method to run before the benchmark. Note that the use of this annotation requires the State annotation on our class.
- Fork This annotation indicates forking parameters for the benchmark.
- Warmup This annotation indicates warmup parameters for the benchmark.
- OutputTimeUnit This annotation indicates the time unit in which the results will be present in.
- Benchmark We already discussed the Benchmark annotation above.
import java.util.ArrayList; import java.util.Comparator; import java.util.LinkedList; import java.util.List; import java.util.Random; import java.util.concurrent.TimeUnit; import java.util.stream.Collectors; import org.openjdk.jmh.annotations.BenchmarkMode; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.annotations.Fork; import org.openjdk.jmh.annotations.Level; import org.openjdk.jmh.annotations.Mode; import org.openjdk.jmh.annotations.OutputTimeUnit; import org.openjdk.jmh.annotations.Param; import org.openjdk.jmh.annotations.Scope; import org.openjdk.jmh.annotations.Setup; import org.openjdk.jmh.annotations.State; import org.openjdk.jmh.annotations.Warmup; // use the benchmark scope for this state class @State(Scope.Benchmark) // compute an average time benchmark @BenchmarkMode(Mode.AverageTime) public class MyBenchmark { // run three benchmarks with 1000, 10000 and 100000 size lists @Param({"1000", "10000", "100000"}) // the size of our lists public int listSize; // the array list to benchmark public ArrayList<Integer> arrayList; // the linked list to benchmark public LinkedList<Integer> linkedList; // our setup method @Setup(Level.Trial) public void setUp() { // create a list with random integers List list = new Random() .ints().limit(listSize).boxed().collect(Collectors.toList()); // initialize our array list with the values generated above arrayList = new ArrayList<>(list); // initialize our linked list with the values generated above linkedList = new LinkedList<>(list); } // create a single fork with a single warmup run @Fork(value = 1, warmups = 1) // perform a single warmup iteration @Warmup(iterations = 1) // output the results in microseconds @OutputTimeUnit(TimeUnit.MICROSECONDS) // indicate that this is a benchmark method @Benchmark public void testArrayListSort() { // sort our array list arrayList.sort(Comparator.naturalOrder()); } // create a single fork with a single warmup run @Fork(value = 1, warmups = 1) // perform a single warmup iteration @Warmup(iterations = 1) // output the results in microseconds @OutputTimeUnit(TimeUnit.MICROSECONDS) // indicate that this is a benchmark method @Benchmark public void testLinkedListSort() { // sort our linked list linkedList.sort(Comparator.naturalOrder()); } }
Running the benchmark will output (omitting considerable verbose details)
Benchmark (listSize) Mode Cnt Score Error Units
MyBenchmark.testArrayListSort 1000 avgt 5 2.994 ± 0.076 us/op
MyBenchmark.testArrayListSort 10000 avgt 5 43.363 ± 5.172 us/op
MyBenchmark.testArrayListSort 100000 avgt 5 844.443 ± 154.691 us/op
MyBenchmark.testLinkedListSort 1000 avgt 5 11.643 ± 3.248 us/op
MyBenchmark.testLinkedListSort 10000 avgt 5 136.231 ± 13.280 us/op
MyBenchmark.testLinkedListSort 100000 avgt 5 3125.115 ± 505.440 us/op
We observe that sorting an ArrayList is several times faster than sorting a LinkedList.
Conclusion
In this article, we briefly discussed the Java Microbenchmark Harness: JMH. The interested reader is encouraged to explore the JMH in more detail. An official list of examples can be found here. The official JMH article on avoiding pitfalls can be found here.