How Ticketmaster uses DevOps to better serve genuine customers while shutting out touts

Culture, standardisation around tools, and owning code in production are the keys, advises Ticketmaster's Simon Tarry

Few companies in the world have quite the same combination of challenges as Ticketmaster. The company has to meet sharp spikes in demand while shutting out touts, coordinate that across 21 countries, and consolidate a sprawling IT estate established over a period of 40 years.

This is partly why the company embarked on a shift to DevOps methodologies 18 months ago, according to Simon Tarry, engineering lead at Ticketmaster.

The company maintains a wide array of systems largely geared to ticketing and meeting spikes in demand for popular events. This IT estate has been fuelled by a combination of organic growth and overseas expansion.

"Ticketmaster was one of the original tech leaders in its field, and grew into a global giant simply by having some of the best ticketing technology. We still have mainframe platforms. You might not believe it, but we do," said Tarry at Computing's DevOps Summit 2016 this week.

"We've expanded across the globe, so we now have a ton of different technology platforms selling tickets on a whole range of different stacks."

And these days it's not just about selling tickets, but about connecting clients into the company's systems. Hence, if Madonna is on a world tour, her management needs to tap into Ticketmaster's systems wherever they are. These systems (and what's required to log-in to them) are currently different from one country to the next.

It was against this backdrop that the company started a shift towards DevOps as part of a bid to better manage its IT estate and an army of 200 or so developers across five locations worldwide.

"It's really high-traffic, high-intensity e-commerce. People expect to be able to buy their tickets from anywhere, and then there's the whole social aspect that we have to cater for," said Tarry.

In addition to dealing efficiently with fans keen to buy tickets for tours as soon as they are put on sale, the company has to prevent tickets being scooped up by high-tech touts.

"When you're buying tickets, it's not just other people that you are competing with, but automated bots hoovering up tickets and trying to resell them," said Tarry.

"So one of the big challenges is trying to figure out whether you are a real person or not. We're putting a lot of R&D into that side of things."

Ticketmaster embarked on its journey towards DevOps in early 2015, but one of the early lessons Tarry picked up was the importance of communication, not just from the top down but between team members. An important prerequisite, he believes, is a soft skills programme.

"We started about a year and a half ago. We had all these teams across different locations and we wanted to be able to release our products far more quickly than we were," explained Tarry.

"We have about 15 different platforms, most of which are selling tickets, which is a ridiculous waste, and we are trying to consolidate them. But what we really wanted to do was try to get stuff out the door more quickly."

Goals included delivering business value, and improving the efficiency of the IT team and the overall quality of development.

"That was around automated testing and getting the continuous integration pipelines working. The general reliability is all about our monitoring pipelines, health checks and alerting," said Tarry.

Perhaps most important, however, was delivering the service. "That was the culture ‘piece' about developers owning their code in production," said Tarry.

"So we came up with a number of maturity matrices with five different levels of maturity for all of those areas. That enabled each team to figure out where they were on this matrix and where they wanted to get to."

In other words, the matrices provided a mapping tool so that each team could see how far along the journey to full DevOps they had travelled and what else they needed to do.

How Ticketmaster uses DevOps to better serve genuine customers while shutting out touts

Culture, standardisation around tools, and owning code in production are the keys, advises Ticketmaster's Simon Tarry

One of the keys, according to Tarry, is standardisation of tooling. Some organisations like to give IT staff the freedom to propose and use their own tools, but Tarry is not convinced that a free-for-all is the right approach.

"Standardising on the tooling makes a hell of a lot of sense because you can share that kind of knowledge across the teams," he explained.

"So it was really important to get a common toolset in place. We have all of our source code in Git. We are using Gitlab and Jenkins for all our build pipelines, Nexus for our release packaging, SonarQube for quality reporting, Rundeck and Chef.

"And that's standardising across the Microsoft .NET platform, LAMP stacks, basically everything that we own runs that toolset."

This standardisation makes it easier for IT staff to understand how development and operations work across the organisation. Indeed, via Slack, the popular communications tool, IT staff in one part of the world can often be seen walking IT staff located elsewhere through challenges and problems.

And the standardisation means that tools developed in one part of the world to solve problems can often be deployed to solve the same problems elsewhere, again having a big impact on efficiency.

Despite all these tools and the benefits of standardisation, intensive feedback from the IT teams across the organisation zeroed in on one key flaw.

"All the tooling and all the strategy was great, but the piece that was missing was getting the people to talk to each other and getting that culture embedded," said Tarry.

Hence, the soft skills training and initiatives like weekly tech talks and simply getting IT teams together away from their screens.

"Culture eats strategy for breakfast," said Tarry. However good the strategy, changing the culture is the key enabler.