Comparative performance and energy efficiency analysis of JVM variants and GraalVM in java applications

With the support of eight benchmark tests, this study investigates the performance and energy consumption of three selected JVM distributions (Amazon Corretto, Adopt OpenJDK, Zulu), and GraalVM.

In the past 20 years, platform independence is one of the most distinctive features that has made Java stand out compared with other programming languages. The applications of Java can be found in various places, including computers, smartphones, and even parking metres (Lestal, 2021). This was also the most important principle when James Gosling and a group of engineers at the United States’s Sun Microsystems decided to create it in 1991 (Ikedilo et al., 2021). It was later promoted as “Write once, run anywhere” (WORA) to draw the attention of the technology community at that time.

In his blog post, Kumar explained every component in this stack in detail. The JVM (Hotspot) is simply a JVM in the JDK (Kumar, 2021b). The JVM Compiler Interface (JVMCI) was introduced with Java 9, making it possible to write compilers as plugins that the JVM could use for dynamic compilation. It included an API and a protocol for creating compilers with customized implementations and optimisations. Noticeably, the Graal Compiler and Substrate VM offer memory management, garbage collection, thread scheduling, and other necessary VM functionality. Additionally, the GraalVM can build native images for a specific target OS/architecture utilising the Ahead-Of-Time application compilation mechanism. This feature is responsible for a reduced footprint, quick startups, and embeddable runtimes, therefore playing a pivotal role in cloud-native and serverless workloads. Based on the target OS, a native image is compiled which includes the application source code, dependencies, the JDK, and Substrate VM. This means the target OS does not need to install a specific JVM to get the application up and running as the image is executable for the target OS without any further requirements.

In fact, a study conducted by Ournani with eleven JVM distributions showed that, in most cases, GraalVM was the most energy efficient distribution (Ournani et al., 2021). Their results are in line with and corroborate the research presented in this paper.

Based on these hardware specifications, Joulemeter 1.2 was the ideal candidate to quantitatively measure and gather program execution duration and energy usage data. This tool is Windows-friendly, easy to use, and provides detailed data on the CPU, software applications, and the monitor (Kor et al., 2015).

Why do they use Windows at all? It does not make sense to me to execute the software under test inside of WSL and measure the energy consumption with a tool running on the Windows host system. It would be much easier to use a machine with native Linux and a more modern energy measurement tool.

As mentioned above, all experiments in this paper were conducted in a standalone environment, and the results were collected via Joulemeter as another process in Windows. Figure 11 shows the overall experiment design. Inside the Windows 11 host machine, there are three components: WSL2 with Ubuntu 20.04 Distribution, Joulemeter 1.2, and SSD storage to store CSV files

To assess the performance and power consumption of these JVM distributions, the Renaissance benchmark suite was used as a benchmark suite. It comprises 21 benchmarks created in popular Java and Scala frameworks that illustrate modern concurrency and parallelism workloads. The Renaissance benchmark suite was invented to provide a more efficient tool to identify new compiler optimisations compared with existing test benchmarks such as DaCapo (used by Ournani et al., 2021), ScalaBench, and SPECjvm2008 (Prokopec et al., 2019). This paper has considered all Apache Spark test cases in the benchmark to measure performance and power consumption.

How representative is the Renaissance benchmark suite for practical relevant use cases? It focuses only on workloads with a lot of concurrency, however, real-world applications are often single-threaded.

Eight benchmark tests were used to conduct experiments for five test candidates: OpenJDK 11.0.12, Amazon Corretto 11.0.14.1, Zulu 11.0.14, GraalVM 21.3.1 (build 11.0.14), and GraalVM 21.3.1 (build 11.0.14) with the native image.

Table 5. Aggregated average energy consumption per second (J/s)

Overall, this study has clearly demonstrated that, in most cases, GraalVM-based Java 11 applications provide better performance, consume less energy, and emit less carbon equivalent footprint than those utilising OpenJDK and two other popular JVM distributions in the market (Amazon Corretto and Zulu).

Figure 19. A Comparison of hardware energy consumption on chi-square

Figure 29. Comparison of hardware energy consumption on Naïve Bayes

Due to hardware and software limitations, this research was conducted in a non-dedicated environment. Thus, external factors could have interfered with the final results, such as phantom processes within the WSL2 environment, or CPU usage from other Windows background processes. For this reason, future work can utilize a more specialised environment with more advanced tools to measure performance, energy consumption, and carbon equivalent footprint emissions of the programs. Moreover, future work can also focus on other performance factors of the JVM such as VM startup time, heap size, and stack size to provide better optimizations for all JVM distributions.