Australia Post

Seeking a holistic view of performance

Australia Post has a large and complex application landscape, serving employees, POS terminals, peripheral devices, corporate clients and retail customers. The organization runs many of its core business processes on SAP, digital, device and cloud platforms together with numerous integration services and middleware infrastructure. Many of these solutions are mission-critical: if events are not processed in time, postal deliveries will be delayed, with a negative impact both on the current business and the long-term brand reputation of Australia Post.

Donna Shepherd GM Technology, Enterprise Enablement at Australia Post, explains the challenges of application testing in this environment: “Much of our testing was project-based. We would make a change to an application, then run targeted performance testing to check that there was no negative impact on the application. But what we lacked was a holistic, end-to-end view of performance across numerous applications that interact with each other. For example: if multiple changes occur during the year across multiple applications, how could we identify the cumulative impact and ensure adequate testing to minimize risk to our production systems?”

OpenText™ LoadRunner™ Enterprise is a tool for enterprises with enterprise-scale challenges. If you’re trying to manage end-to-end performance and capacity planning across dozens of interacting applications, supporting millions of daily transactions, in our experience it’s the best tool in the market.

Managing unprecedented peaks

Performance testing release cycles at Australia Post are relatively short (at four to six weeks) and there are up to 40 testing projects (including spike, peak and soak testing) running concurrently before peak periods. Traditionally, Christmas was the single major peak, but Black Friday and Cyber Monday have now grown to practically the same levels of transactions, with peaks of 2.3 million parcels per day. Beyond the goal of supporting peak volumes without loss of performance or availability, the testing team works closely with capacity planning teams in the business to project future peak volumes and ensure that the appropriate infrastructure is in place to support them.

When the COVID pandemic hit Australia, plans to introduce end-to-end, full-portfolio performance testing had to be dramatically accelerated. Lockdowns and the closure of many retail stores caused a huge rise in online shopping and home deliveries, and Australia Post saw more than 30% growth in transactions almost overnight on its core business systems.

Jerome Anthonisamy, Head of NFT, APM, SRE & Cloud Assurance – ANZ Region at Wipro (Australia Post’s testing partner), comments: “COVID restrictions drove unprecedented volumes not only on internal systems but also on systems supporting corporate clients: the chains of retail outlets that Australia Post serves. In addition to supporting the business in keeping applications in good working order, we are helping with a major replat forming project which will see a majority of our event management services nearing end-of-life move to the Google Cloud Platform. These changes, along with other key upgrades to the digital and API platforms, were putting intense pressure on the relatively small team of performance consultants.”

Australia Post remained committed to its existing targeted high-volume testing but set out to introduce end-to-end performance testing capable of meeting these increased challenges.

Adopting true end-to-end performance testing

“The organization has been using Micro Focus (now OpenText) technologies for software testing and lifecycle management for a number of years,” says Donna Shepherd. “As we looked to change the way we test, it was clear that we could continue to use our existing technologies to address our new requirements. LoadRunner Enterprise remains the right tool because it gives us the functionality we need within a very stable product that is widely used. With LoadRunner Enterprise, we’ve transformed from a project-focused performance testing team to an end-to-end performance engineering service that can give application development, production and support teams the data they need to continuously improve their systems.”

In addition to LoadRunner Enterprise, Australia Post uses OpenText™ UFT One for functional testing and OpenText™ ALM Quality Center for managing the software lifecycle and storing manual test scripts.

Jerome Anthonisamy says: “In this challenging environment, having a quick turnaround for performance testing is vital. LoadRunner Enterprise enables this responsiveness through the ease of recording and customizing script functional parameters and the speed at which we can get detailed post-testing reports out to stakeholders.”

The enterprise scale of the Micro Focus (now OpenText) solution is another important plus for Australia Post:  LoadRunner Enterprise easily handles the high volumes and injection rates we test at. We run 130 scripts concurrently at high volumes and the solution runs seamlessly without errors.

Gaining scalability and flexibility

To improve its ability to scale up to future demands, Australia Post moved from an on-premises to a software-as-a-service implementation of LoadRunner Enterprise on the OpenText (formerly Micro Focus) cloud and adopted the Flex licensing model. “We can now scale up and add testing horsepower very rapidly thanks to the SaaS Flex licensing,” says Donna Shepherd.

Jerome Anthonisamy adds: “The flexibility of the new licensing helps a lot because levels of transactions are by no means consistent through the year. Micro Focus (now OpenText) gives us the ability to ramp up and down fast with little lead time, which also makes the solution cost-efficient because we don’t need to over-invest in permanent licenses that would only be used at peak times.”

A universal performance testing tool

Australia Post is using LoadRunner Enterprise to conduct large-scale, end-to-end performance testing across its internal-facing core business applications, the corporate-facing systems that enable retail clients to lodge their consignments, and customer-facing applications including online shops, mobile track-and-trace apps, and self-service kiosks.

When Australia Post built its new end-to-end Environment Health Check shakeout suite, the goal was to create a continuous capability rather than a tool used a couple of times a year in readiness for peaks. Given highly integrated and constantly evolving environments, the testing team needed to be sure that it could get the appropriate information to the right stakeholders quickly.

“We deconstructed the shakeout suite to take out some key scenarios, configured automated email notifications, and used the CI/CD integration feature to have the suite run every morning,” says Donna Shepherd. “At the top level, we get a report showing that perhaps of the 50 applications, 40 are green but 10 are red because something has broken due to changes made to the environment from the previous day. The testing and environments team looks into those 10 applications and focuses on fixing the issues first thing in the morning, so that by the time the testers and developers come in, they’re already working in a cleaner environment. We got great feedback on this solution from both the project and application support communities: environments are more available, developers have more information to fix issues, and delays to testing timelines are avoided.”

She adds: “ LoadRunner Enterprise is a truly universal performance testing tool at Australia Post. In addition to the core business systems, we support performance testing for around 4,000 in-branch kiosks, 2,800 self-serve label-printing kiosks, and 12,000 handheld scanners used by postmen and postwomen. During COVID lockdowns, the high volumes of transactions were largely driven by online activity and those handheld devices, so it was critically important to ensure excellent application performance.”

Cutting delays and boosting productivity

“When we needed to quickly set up this end-to-end performance testing capability in our complex environment, LoadRunner Enterprise proved to be the best tool,” says Donna Shepherd. “We’re supporting the business in confidently managing these huge peaks in transactional volumes, and that achievement is highly visible within the organization.”

The LoadRunner Enterprise integration with the company’s CI/CD platform – GitHub – has enabled Australia Post to conduct its Environment Health Check shakeout execution every day for the entire landscape. This has enabled the organization to reduce environment delays by at least 30% and proactively alert teams to application flow failures, helping to increase availability for test and development environments by 6%.

With OpenText™ Virtual User Generator (VuGen) supporting multiple protocols, manual effort around parametrizing and correlating test scripts is much lower. “We measured time-savings for scenarios where we migrated from JMeter to LoadRunner Enterprise,” says Jerome Anthonisamy. “Work that was taking 1.5 to 2 days with JMeter is taking a maximum of 3 to 6 hours with LoadRunner Enterprise. There is a relatively small team of 12 engineers focused on performance testing, so these time-savings help boost overall productivity in a very significant way.”

The time-scheduling and booking features of LoadRunner Enterprise 2021 have helped Australia Post to execute concurrent load testing and reservation at reduced cost and with greater collaboration between team-members. The organization anticipates a saving on licenses for Virtual Users, Load Generators and Controllers for concurrent versus separate operations.

New capabilities in software quality assurance

“The high scalability of LoadRunner Enterprise has enabled us to run end-to-end peak testing involving 22 applications and 87 business workflows with 4,700 concurrent users,” says Donna Shepherd. “That’s something that we simply weren’t doing before.”

Jerome Anthonisamy adds: “Not too many other Tier-1 organizations in Australia can do end-to-end performance testing like we can. With the number and variety of shared software assets in our infrastructure involving both synchronous and asynchronous transactions, it’s vital to have this end-to-end view rather than just testing applications in silos. We’re testing 35 to 48 applications with 132 test scenarios/ business flows across 30 test suites that run for around 6 to 8 weeks.”

Based on this performance testing, Australia Post can identify potential points of vulnerability to rising peak volumes and ensure that it makes the appropriate investments in optimizing the applications and augmenting the underlying infrastructure. Testing results also help the organization to optimize its usage of the existing infrastructure resources. Australia Post uses LoadRunner Enterprise to store a complete repository of test artefacts for every release, providing an auditable reference for senior management to understand how the organization is progressing in terms of software quality.

An enterprise tool for enterprise-scale challenges

Despite the extreme impact of COVID-related lockdowns on transactional volumes at Australia Post, the organization has recorded zero Priority One incidents related to application performance or stability issues in two consecutive years since starting the peak end-to-end testing. “Given all these challenges, we were able to keep applications stable and absorbing these huge volumes without any slowdowns or breakages,” says Jerome Anthonisamy. “Micro Focus (now OpenText) gives us consistently wide coverage of protocols in a single tool that is easy to manage, secure, and highly scalable and flexible. The advantage of using an enterprise tool like LoadRunner Enterprise is that we can focus on supporting the business rather than on managing up the tooling. The ticket price seems higher than for an open-source solution, but you need to consider how efficiently you can manage the tooling overhead: we think that LoadRunner Enterprise is more cost-effective overall.”

Donna Shepherd adds: “ LoadRunner Enterprise gives us powerful reporting right out of the box, with different teams able to access transparent information about the processes that matter to them. The built-in security is also very impressive across all the different parts of the solution. The popularity of LoadRunner Enterprise means we have easy access to skills and an online user community. Equally, the Micro Focus (now OpenText) support team has been fantastic from both the technical and the engagement perspectives. The Micro Focus (now OpenText) technical team works very closely with our performance consultants and responds very rapidly to queries. So we’re confident that things will be looked at quickly when we have an issue.”