Rabobank

For 13 years, the ICT team has been using Operations Orchestration (OO) to orchestrate end-­to-­end IT processes across Rabobank’s heterogeneous systems.

Automation in the early days

In the beginning, the team set out to automate IT operations with just three engineers. “With OO, the possibilities were endless,” said Marcel Wortel, automation specialist at Rabobank. “‘We can automate anything!’ became our slogan.” That slogan caught on quickly and requests came pouring in. Before long, there were too many to handle.

To meet the intense demand for automated processes, the team knew they had to shift orchestration into high gear—by changing the way they were developing OO workflows, maintaining the automation, and delivering IT services. One of the first steps was adopting the agile Scrum methodology. The team appointed product owners and scrum masters for DevOps teams across more than ten engineers. So began Rabobank’s high-­speed journey to best­-practices IT service delivery.

No more bouncing requests around departments and waiting weeks. We fulfill customer requests almost instantly, in a matter of minutes.

A new way to orchestrate— automation building blocks

Today, reusable drag­-and-­drop automation building blocks are ready for deployment— at any time, for nearly any automation need. Error-handling workflows manage exceptions consistently and simplify troubleshooting. A quality assurance workflow ensures workflows comply with design principles by checking attributes such as ROI settings and assignment of workflow categories.

Limiting customization work—with standard integrations that are ready to plug and play— is key to the team’s success. “Basically, we had to dumb down our workflows and make them generic,” said Wortel. “No more bouncing requests around departments and waiting weeks. We fulfill customer requests almost instantly, in a matter of minutes.”

The building blocks power Infrashop, a catalog-­based portal for all types of infrastructure needs—from virtual provisioning to rightsizing, storage extension to network firewall configuration, and user creation to starting monitoring services. Automation has dramatically reduced human errors and development times. And the team has been able to standardize the delivery of high­-quality services.

Given the success of the Infrashop, the team wanted to replicate that model for all non-­infrastructure-­related requests—for example, software provisioning, application performance monitoring, and identity and access management. “Since users were already managing their own content within our Marketplace portal, it made sense to have them build their own automations,” said Wortel. That’s how Marketplace’s catalog­-based do-­it­-yourself portal was born. Here, users design their own automation by selecting from a list of building blocks.

Inspired by all things orchestration, the team also implemented generic remediation OO flows that trigger domain-­specific scripts when events arise. The orchestrator directs the entire remediation process, picking up information from the event and passing on the context to scripts for automatic issue resolution. Thanks to the remediation flows, the team has seen a significant reduction in on-call support, mean­-time-­to-­resolution (MTTR), and incident impacts.

ROI keeps rising

The team measures ROI by calculating time saved with automated processes. “Last year, our ROI was 120,000 hours,” said Wortel. “Yearly, we have up to 75,000 flow runs. Since roughly 40 percent of the ROI comes from Infrashop flows, and since we automate more and more infrastructure processes, our ROI just keeps going up.”

According to Wortel, OO can do anything—but that that doesn’t mean it has to. His advice: use the strength of the solution to minimize customizations with reusable, generic building blocks. His team is spearheading an orchestration center of excellence that considers broad-based customer needs.

“We want to engage Cloud DevOps teams to reap the benefits of OO for cloud automation tasks,” he said. “Especially when a hybrid solution is needed to bridge the gap between cloud services and on-­premises infrastructure— that’s where OO really shines.”

Last year, our ROI was 120,000 hours. Yearly, we have up to 75,000 flow runs. … since we automate more and more infrastructure processes, our ROI just keeps going up.