SparkPost

Managing complexity at scale

Delivering email is complicated, especially at scale. As an everyday email user, you just click “send” and the message is gone—but when you’re running a platform that delivers more than a billion emails per day, the ability to monitor delivery delays and bounces becomes critical. Moreover, clients want to know more than just whether and when an email was delivered—they want to be able to see if the recipient opened it, how long they spent reading it, what they clicked on, and whether they unsubscribed or reported the message as spam.

As SparkPost developed its cloud email sending and delivery platform, it realized that its ability to help clients answer these kinds of questions could be a key differentiator in the market. However, building flexible ad hoc analytics tools that would enable real-time analysis of high-volume, high-velocity streams of email event data was a major technical challenge.

“Each of our clients might be running multiple campaigns, each sending different email templates from dozens of IP addresses to millions of email domains,” explains Chris McFadden, VP of Engineering at SparkPost. “It’s vital for their marketers to be able to drill down into this data to understand what’s working and what isn’t.”

Performing that type of analysis across more than 40 metrics is complicated enough with small volumes of data—but SparkPost needs to process more than 100,000 email events per second, and volumes are growing at approximately 40% per year. To unlock the power of real-time insight for its clients, SparkPost decided to rethink its approach and build a scalable analytics architecture from scratch.

We saw Vertica (now OpenText) as the most mature solution on the market, and it’s gotten even better over time. The support is great, the documentation is stellar, and the technology is extremely solid, especially in terms of real-time data loading and clustering.

Building for scale

SparkPost had previous experience of using the OpenText Vertica Analytics Platform in some of its earlier products and knew that OpenText™ Vertica™ offered all the features it needed to deliver real-time analytics across multiple metrics as well as ad hoc drill-down capabilities at massive scale.

“When we originally adopted it, we saw Vertica as the most mature solution on the market, and it’s gotten even better over time,” says Chris McFadden. “The support is great, the documentation is stellar, and the technology is extremely solid, especially in terms of real-time data loading and clustering.”

SparkPost’s new analytics architecture runs on the Amazon Web Services (AWS) cloud platform and has Vertica Analytics Platform at its core. It uses AWS Lambda to create a pipeline that loads data into the Vertica Analytics Platform in micro-batches, enabling real-time analysis. The Vertica Analytics Platform cluster can scale horizontally as data volumes increase, enabling SparkPost to deal with massive volumes of data at high velocity, without worrying about database performance.

Clients interact with the data in Vertica Analytics Platform via a user-friendly web application that makes it easy to monitor key metrics across marketing programs and campaigns, and understand the factors driving engagement, unsubscribes and spam reports. Beyond marketing, the tools also enable technical teams to troubleshoot any issues that are preventing delivery.

Unlocking the full potential of email marketing

As marketing becomes increasingly data-driven, companies want to understand the return on their investment in email marketing. SparkPost’s analytics platform makes it easier to evaluate the revenue generation potential of each campaign, while also optimizing its costs. The cost of analytics infrastructure for each email is a metric that SparkPost has been continuously improving over time.

“As our data volumes have grown, we’ve been able to use Vertica (now part of OpenText™) more efficiently instead of adding more and more servers to our cluster,” says Chris McFadden. “That’s contributed to a 50% reduction in CPM for analytics.”

The scalability of OpenText has also enabled SparkPost to stay ahead of rapidly increasing data volumes, and even integrate third-party data into its analytics to give clients unique insight into the relative performance of their email campaigns, compared to industry averages.

Chris McFadden concludes: “The analytics we can provide with Vertica [Vertica Analytics Platform] are an important differentiator for SparkPost, driving growth in active users and demonstrating the value we add for our clients.”

As our data volumes have grown, we’ve been able to use Vertica [Vertica Analytics Platform] more efficiently instead of adding more and more servers to our cluster. That’s contributed to a 50% reduction in CPM for analytics.