Endress+Hauser

The start of a performance test journey

Endress+Hauser InfoServe, located in Germany and other smaller locations worldwide, is the internal IT provider to the organization. Within Endress+Hauser InfoServe, the Performance Team is a knowledge center in terms of performance, load- and stress-testing. It offers consulting and helps with troubleshooting.

Today, one of the most important Endress+ Hauser applications is its worldwide webpage, a portal that incorporates many useful tools to assist customers in their daily business with Endress+Hauser. The beginnings of the project described go back to the early 2000s, when selling engineering products was a face-to-face business and an online shop more a showcase of abilities. There were some online orders and visitors, but traffic remained low. This changed over the years and with the increasing load the correct setup and configuration of the systems became essential. OpenText Enterprise Performance Engineering was already in use and so Endress+Hauser decided to use OpenText Enterprise Performance Engineering to test its worldwide webpage. On weekends, all system experts were in a single room, looking at their own monitoring screens, the test started, load grew, and the test system crashed. When the system was up again, the process started again, often without yielding findings.

Know your system under test and set up useful monitoring

A lot has changed since then. Today, each load test starts with a longer and detailed analysis phase: what is the real architecture of the system, which components are integrated, which load balancer, application servers, databases, and web services are involved? With all the various applications monitored separately, it was decided to adopt a single consolidated monitoring to cover everything. The single system is a combination of Icinga, InfluxDB, and Grafana integrated with OpenText Enterprise Performance Engineering. When a separate monitoring is still needed, the results are sent to the central monitoring system. Leveraging Grafana, a dashboarding and metric visualization system, the team created a few specific dashboards reflecting the whole system under test.

Now, Endress+Hauser can steer performance testing much better and increase the webpage activity until one of the systems gets congested. Keeping the traffic just below load threshold allows a live analysis of the root cause and saves a lot of time. OpenText Enterprise Performance Engineering sends runtime data to InfluxDB and the central monitoring system provides visibility on everything: from client-side response times and the status of load generator, across the technical tiers of the application to the final backend system.

Using OpenText Enterprise Performance Engineering we deliver higher quality applications in a well-sized environment, fit for purpose. Our users really appreciate this, and our go-live experience has improved immensely.

A Reliable Load Generator setup

OpenText Enterprise Performance Engineering enables the visualization of client-side response times in its dashboards. This is an important KPI for load testing. A straight line at a low value is what the team wants. If webpage activity increases, the system gets under load; if the response times grow with increasing load, there is a bottle neck; if activity grows under stable load, the system is overloaded.

But what if the response times are a lot higher than experienced in real time, when a manual test with the system under test works as expected, while the response times in OpenText Enterprise Performance Engineering show problems? The test team started to analyze this situation, testing their OpenText Enterprise Performance Engineering setup itself. They discovered that running load generators (the machines that execute the tests) on a virtualized, shared environment was not effective: the load generators required so much CPU power that VMware reduced their CPU share to be fair to the other machines running on the same host. Consequently, fewer CPU time-slots on the client side means longer response times.

Fortunately, there is a stand-by computer center for emergencies with sufficient power and free resources. Changing to unshared load generators running within that computer center brought OpenText Enterprise Performance Engineering response times into the range of user experience and made the results reliable without showing false positive results.

Designing real-world test scenarios with OpenText Enterprise Performance Engineering

Endress+Hauser started to use real world business case descriptions as a basis for their load testing scripts. Simulating real user behaviors means also to simulate the times they need to read, think and validate, what they are doing. Then the test runs completely randomly, allowing only statistical evaluations and when the test shows a specific problem, it becomes hard to reproduce it.

The experience of Endress+Hauser InfoServe’s system integrator partner profi.com has been a game changer. The team learned that the most important goals are to produce a realistic load and to be most effective when doing load tests. The number of virtual users running is not that important.

A new step was introduced in the process: after collecting the business cases, the team identifies standard ‘Actions’, that are regularly used in the different business cases. Then the team retrieves and performs educated guesses on how often these actions are performed. The target for the load test is to reach the same amount of load while keeping a consistent relationship between the actions.

Finally, the test scripts and the test scenarios are defined. It follows the target to reach the same load, but it is much more load-intensive.

Now Endress+Hauser InfoServe is using a transactions-per-second KPI to track processed load, because each action is defined as a measurement transaction. The load test requires less time, is reproducible, and requires less virtual users.

Seamless Connection between Development and Testing Teams

At the load testing phase, the performance team joins the project team to define the test targets, the necessary tests, test data creation, and the test system setup. Rather than just running the test, followed by an analysis, and then a separate phase to resolve any issues, the team follows an iterative process of load testing, analysis, fixes, and then continuing with load testing.

“Profi.com manages test scripting and creating the load test,” comments Wolfram Wagner, Expert Systems Engineer Performance with Endress+Hauser InfoServe. “This keeps OpenText Enterprise Performance Engineering test execution and system analysis separate. The partner monitors the checks, the load generators, and the OpenText Enterprise Performance Engineering messages, while the system and performance analysis experts monitor the application and perform the analysis.”

High Quality Applications and Increased User Confidence

The team appreciates the flexibility of OpenText Enterprise Performance Engineering. As part of an ongoing DevOps effort, development teams use the Bamboo continuous integration and deployment tool which integrates with OpenText Enterprise Performance Engineering. For cloud-based Web applications, Endress+Hauser InfoServe uses Amazon Web Services (AWS) cloud load generators for easy scaling up of load levels as required without burdening the network.

Using OpenText Enterprise Performance Engineering we deliver higher quality applications in a well-sized environment, fit for purpose. Our users really appreciate this, and our go-live experience has improved immensely,” says Wolfram Wagner. “As an organization we have exciting plans for the future which will require OpenText Enterprise Performance Engineering expertise. We are opening a new warehouse and plan to test a handheld warehouse scanner system that integrates into SAP at the backend. We are also redesigning our ERP landscape to run in one SAP system which will require much testing.”

He concludes: “We really enjoy our OpenText Enterprise Performance Engineering experience, but more than that, we enjoy the partnership we have built with OpenText over the years. We are testing partners for new OpenText Enterprise Performance Engineering releases, and we are members of the customer advisory board. Seeing our suggestions converted into features in new releases is rewarding and boosts our confidence for the future.”

Over the past years not only the Endress+Hauser webpage has evolved into a high-performing platform which generates a lot of traffic, but digital capabilities and offerings are now becoming key drivers for the Endress+Hauser business. Thanks to OpenText Enterprise Performance Engineering load testing also keeps pace with the digital boost at Endress+Hauser.

We are testing partners for new OpenText Enterprise Performance Engineering releases, and we are members of the customer advisory board. Seeing our suggestions converted into features in new releases is rewarding and boosts our confidence for the future.