Bdtjtk

I was comparing the performance of JDK 8 and 11 using jmh 1.21 when I ran across some surprising numbers:

Java version: 1.8.0_192, vendor: Oracle Corporation



Benchmark                Mode  Cnt  Score   Error  Units

MyBenchmark.emptyMethod  avgt   25  0.362 ± 0.001  ns/op





Java version: 9.0.4, vendor: Oracle Corporation



Benchmark                Mode  Cnt  Score    Error  Units

MyBenchmark.emptyMethod  avgt   25  0.362 ±  0.001  ns/op





Java version: 10.0.2, vendor: Oracle Corporation



Benchmark                Mode  Cnt  Score   Error  Units

MyBenchmark.emptyMethod  avgt   25  0.723 ± 0.001  ns/op





Java version: 11.0.1, vendor: Oracle Corporation



Benchmark                Mode  Cnt  Score   Error  Units

MyBenchmark.emptyMethod  avgt   25  0.724 ± 0.002  ns/op

OpenJDK 11 and 12 perform similar to OracleJDK 11. I have omitted their numbers for the sake of brevity.

I understand that microbenchmarks do not indicate the performance behavior of real-life applications. Still, I'm curious where this difference is coming from. Any ideas?

Here is the benchmark in its entirety:

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>



    <groupId>jmh</groupId>

    <artifactId>empty-method</artifactId>

    <version>1.0-SNAPSHOT</version>

    <packaging>jar</packaging>

    <name>JMH benchmark sample: Java</name>



    <dependencies>

        <dependency>

            <groupId>org.openjdk.jmh</groupId>

            <artifactId>jmh-core</artifactId>

            <version>${jmh.version}</version>

        </dependency>

        <dependency>

            <groupId>org.openjdk.jmh</groupId>

            <artifactId>jmh-generator-annprocess</artifactId>

            <version>${jmh.version}</version>

            <scope>provided</scope>

        </dependency>

    </dependencies>



    <properties>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

        <jmh.version>1.21</jmh.version>

        <javac.target>1.8</javac.target>

        <uberjar.name>benchmarks</uberjar.name>

    </properties>



    <build>

        <plugins>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-enforcer-plugin</artifactId>

                <version>1.4.1</version>

                <executions>

                    <execution>

                        <id>enforce-versions</id>

                        <goals>

                            <goal>enforce</goal>

                        </goals>

                        <configuration>

                            <rules>

                                <requireMavenVersion>

                                    <version>3.0</version>

                                </requireMavenVersion>

                            </rules>

                        </configuration>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-compiler-plugin</artifactId>

                <version>3.8.0</version>

                <configuration>

                    <compilerVersion>${javac.target}</compilerVersion>

                    <source>${javac.target}</source>

                    <target>${javac.target}</target>

                </configuration>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-shade-plugin</artifactId>

                <version>3.2.1</version>

                <executions>

                    <execution>

                        <phase>package</phase>

                        <goals>

                            <goal>shade</goal>

                        </goals>

                        <configuration>

                            <finalName>${uberjar.name}</finalName>

                            <transformers>

                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

                                    <mainClass>org.openjdk.jmh.Main</mainClass>

                                </transformer>

                            </transformers>

                            <filters>

                                <filter>

                                    <!--

                                            Shading signed JARs will fail without this.

                                            http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar

                                    -->

                                    <artifact>*:*</artifact>

                                    <excludes>

                                        <exclude>META-INF/*.SF</exclude>

                                        <exclude>META-INF/*.DSA</exclude>

                                        <exclude>META-INF/*.RSA</exclude>

                                    </excludes>

                                </filter>

                            </filters>

                        </configuration>

                    </execution>

                </executions>

            </plugin>

        </plugins>

        <pluginManagement>

            <plugins>

                <plugin>

                    <artifactId>maven-clean-plugin</artifactId>

                    <version>2.6.1</version>

                </plugin>

                <plugin>

                    <artifactId>maven-deploy-plugin</artifactId>

                    <version>2.8.2</version>

                </plugin>

                <plugin>

                    <artifactId>maven-install-plugin</artifactId>

                    <version>2.5.2</version>

                </plugin>

                <plugin>

                    <artifactId>maven-jar-plugin</artifactId>

                    <version>3.1.0</version>

                </plugin>

                <plugin>

                    <artifactId>maven-javadoc-plugin</artifactId>

                    <version>3.0.0</version>

                </plugin>

                <plugin>

                    <artifactId>maven-resources-plugin</artifactId>

                    <version>3.1.0</version>

                </plugin>

                <plugin>

                    <artifactId>maven-site-plugin</artifactId>

                    <version>3.7.1</version>

                </plugin>

                <plugin>

                    <artifactId>maven-source-plugin</artifactId>

                    <version>3.0.1</version>

                </plugin>

                <plugin>

                    <artifactId>maven-surefire-plugin</artifactId>

                    <version>2.22.0</version>

                </plugin>

            </plugins>

        </pluginManagement>

    </build>

</project>

src/main/java/jmh/MyBenchmark.java:

package jmh;



import org.openjdk.jmh.annotations.Benchmark;

import org.openjdk.jmh.annotations.BenchmarkMode;

import org.openjdk.jmh.annotations.Mode;

import org.openjdk.jmh.annotations.OutputTimeUnit;



import java.util.concurrent.TimeUnit;



@BenchmarkMode(Mode.AverageTime)

@OutputTimeUnit(TimeUnit.NANOSECONDS)

public class MyBenchmark

{

    @Benchmark

    public void emptyMethod()

    {

    }

}

Here is the Windows-specific script I use. It should be trivial to translate it to other platforms:

set JAVA_HOME=C:Program FilesJavajdk1.8.0_192

call mvn -V -Djavac.target=1.8 clean install

"%JAVA_HOME%binjava" -jar targetbenchmarks.jar



set JAVA_HOME=C:Program FilesJavajdk-9.0.4

call mvn -V -Djavac.target=9 clean install

"%JAVA_HOME%binjava" -jar targetbenchmarks.jar



set JAVA_HOME=C:Program FilesJavajdk-10.0.2

call mvn -V -Djavac.target=10 clean install

"%JAVA_HOME%binjava" -jar targetbenchmarks.jar



set JAVA_HOME=C:Program FilesJavaoracle-11.0.1

call mvn -V -Djavac.target=11 clean install

"%JAVA_HOME%binjava" -jar targetbenchmarks.jar

My runtime environment is:

Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T14:41:47-04:00)

Maven home: C:Program Filesapache-maven-3.6.0bin..

Default locale: en_CA, platform encoding: Cp1252

OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

More specifically, I am running Microsoft Windows [Version 10.0.17763.195].

You are measuring empty benchmarks, not empty methods. In other words, measuring the minimal infrastructure code that handles the benchmark itself. This is easy to dissect, because you'd expect only a few instructions on the hot path. JMH's -prof perfasm or -prof xperfasm would give you those hottest instructions in seconds.

I think the effect is due to Thread-Local Handshakes (JEP 312), see:

8u191: 0.389 ± 0.029 ns/op
[so far so good]

  3.60%  ↗  ...a2: movzbl 0x94(%r8),%r10d

  0.63%  │  ...aa: add    $0x1,%rbp

 32.82%  │  ...ae: test   %eax,0x1765654c(%rip) ; global safepoint poll

 58.14%  │  ...b4: test   %r10d,%r10d

         ╰  ...b7: je     ...a2

11.0.2: 0.585 ± 0.014 ns/op [oops, regression]

  0.31%  ↗  ...70: movzbl 0x94(%r9),%r10d    

  0.19%  │  ...78: mov    0x108(%r15),%r11  ; reading the thread-local poll addr

 25.62%  │  ...7f: add    $0x1,%rbp          

 35.10%  │  ...83: test   %eax,(%r11)       ; thread-local safepoint poll

 34.91%  │  ...86: test   %r10d,%r10d

         ╰  ...89: je     ...70

11.0.2, -XX:-ThreadLocalHandshakes: 0.399 ± 0.048 ns/op [back to 8u perf]

  5.64%  ↗  ...62: movzbl 0x94(%r8),%r10d    

  0.91%  │  ...6a: add    $0x1,%rbp          

 34.36%  │  ...6e: test   %eax,0x179be88c(%rip) ; global safepoint poll

 54.79%  │  ...74: test   %r10d,%r10d

         ╰  ...77: je     ...62

I think this is largely visible mostly in tight loops like this one.

UPD: Hopefully, more details here.