Investigating differences in response time and error rate between a monolithic and a microservice based architecture

If implemented correctly, a microservice architecture can unlock a plethora of benefits, such as: increase in agility, developer productivity, resilience, scalability, reliability, maintainability, separation of concerns, and ease of deployment [6] [46]. However, the benefits come with challenges. As microservices are deployed independently, services communicate over the network. This creates a need for service discovery, increased security management, communication optimization and load balancing [23] [34]. The network latency is also much greater than that of memory, meaning intercommunication within a microservice architecture is slower as compared to a monolithic architecture that makes use of in-memory calls [48][47]. Moreover, decomposing distributed systems into independent granular components brings complexity and operational overhead. Meaning a microservice architecture does not simply remove complexity, rather the complexity shifts from inside of the application out into infrastructure.

To evaluate the differences between a microservice architecture and a monolithic architecture, two separate systems, S1 and S2, were developed and deployed. The two systems provide the same functionality but are built using two different system designs. S1 is a monolithic system whereas S2 is a microservice architecture. A set of performance tests were executed on both systems using Apache JMeter [4] to gather quantitative data for analysis. The performance tests measured response time as well as error rate under different workloads. To create realistic performance tests, the workloads were determined by use cases.

Figure 4: The flow of communication between client and the server in the microservice architecture.

As each container running on a node consumes computing resources, OpenShift allows to set minimum and maximum values for how much CPU and RAM a container can consume. The minimum amount of resources that can be allocated a container is referred to as “request” in OpenShift and the maximum is referred to as “limit”. Meaning, when a container is being deployed, it will be instantiated with the requested amount of CPU and RAM. With an increase in traffic to the container, OpenShift will allocate more resources to it, up until it reaches the limit. If a container attempts to exceed the specified limit for CPU, the system will throttle the container. If the container exceeds the specified memory limit, it will be terminated and restarted.

Table 2. The resources, CPU and memory (RAM), allocated for both systems, S1 and S2, for Use Case 1. Displaying both the requested amount of resources and the limits for each container.

To simplify the comparisons and analysis of the data gathered from the performance tests, auto-scaling was disabled. Therefore, each application was deployed as a single container, configured with the amount of CPU and memory presented in Table 2 and 3.

To measure response time, test plans were created to execute a constant workload of 1100 requests per minute during 5 minutes on each system, resulting in a total of 5500 requests. This created a steady workload on the systems without purposely making them crash.

The tests performed for Use Case 1 resulted in an average response time of 82 ms for the monolith architecture, S1, and 224 ms for the microservice architecture, S2. Meaning the monolith architecture had an average response time which were 64% faster than the microservice architecture. When inspecting the 90% line, meaning the response time of which 90% of the requests fall, the response time for the monolith architecture, S1, was 171 ms and for the microservice architecture, S2, it was 313 ms. Meaning 90% of the requests to the monolith architecture, S1, will have a 45% faster response time than the requests to the microservice architecture, S2. The results are presented in Table 4.

Table 4. Displays the data gathered from the performance test which measured response time for Use Case 1. The data displayed is sample size, the average response time and the 90% line response time.

To measure error rate, test plans were created to execute a constant workload of 4440 requests per minute during 5 minutes on each system, resulting in a total of 22200 requests. This created a heavy workload on the systems to purposely make the containers crash in order to see how the two different architectures behave when errors occur.

The results show that the response time for the microservice architecture is significantly slower when compared to the response time of the monolithic application. When evaluating error rate, the systems were exposed to a higher workload and the results show that the error rate is significantly lower in a microservice architecture than a monolithic architecture.

Moreover, the results also confirm and further strengthen the work of Ueda et. al. [47] and Villamizar et. al. [48], which reported that the performance of a microservice model is lower than that in a monolithic model.

Therefore, the complexity of the system was fairly low with a total of only three microservices. Each microservice was also fairly independent, meaning the need for intercommunication between microservices was close to non-existent. This opens up to debate if the system can be considered a distributed system at all. Because of the low level of complexity, circuit breakers and an event-driven architecture for data consistency, both considered key components in a successful microservice architecture, were not implemented.