CIO interview: Running Comparethemarket.com's IT is no simples task

Comparethemarket.com used to be a poor number three in the insurance price-comparison market until the recruitment of Aleksandr Orlov, the Eastern European meerkat entrepreneur, to head the company's marketing campaign.

The popularity of Orlov helped to propel the company ahead of its rivals - who could only respond with annoying opera singers or badly drawn cartoon mascots (since retired).

But that initial campaign also brought the company face-to-face with another challenge of modern internet businesses: handling sudden and sharp spikes in demand, without bankrupting the company on investment in unnecessary hardware.

Back in January 2009, when Orlov made his first appearance, buying multi-gigabits of bandwidth, a ton of servers and some space to put them in - plus a bit extra, just in case - was the only way that the company felt it could reliably keep up with surging demand.

Today, though, Comparethemarket.com is running trials to see how well one of its comparison products can run in the cloud, partly to see how it will work and whether the service will be good enough, and partly to see how extensible such a service could be when the company experiences spikes in demand.

"What are you going to do? Are you going to create enough capacity to cater for that spike in traffic? You guess at that, then add a bit on top as well to be on the safe side. And then you can be stuck with all that unused computer resource that you are going to pay for," says James Lomas, information technology director at Comparethemarket.com.

At the moment, he says, it's not tenable to simply spin-up an extension to the corporate data centre on a cloud service in an ad hoc manner to satisfy sudden peaks in demand, but Lomas's IT infrastructure teams have nevertheless been engaged in a re-architecture project that may enable that to be done in the near future.

"One of our products will be wholeheartedly moved on to Amazon infrastructure. We will check that we get the benefits in terms of the three S's - stability, speed to market and scalability. Once we are comfortable that we can operate it, we won't have to prove the technology; it'll be proven," says Lomas.

At the moment, he continues, "it's about understanding it [the cloud] enough to be able to operate it successfully", but "that elastic nature of cloud is really interesting".

Those experiments with cloud are currently at an early stage. "What we need to do is measure and learn: let's see if it works - I'm pretty sure it will - let's makes sure we can operate it, and that the availability is four nines [99.99 per cent]. Then we can be confident," says Lomas.

Team Orlov

The lifeblood of an internet-facing organisation like Comparethemarket.com is always going to be its IT and its strength dependent on the quality of its IT staff.

Comparethemarket.com has around 100 or so "CTMers" - and growing - organised into product-focused teams, such as home insurance, car insurance, pet insurance etc, all organised into what he calls "two pizza" size teams. Teams no bigger than could comfortably share two large pizzas, a concept shamelessly stolen from Amazon.com.

These teams, he says, have a mix of skills, including IT engineers with experience in internet languages, such as Chef and Puppet, and knowledge of C#, MongoDB and RabbitMQ for messaging; as well as test engineers, a business analyst and maybe a project manager too, depending on the product.

These teams then "own" the "journey" of whatever they are working on for their product, from first click to last. They are responsible for ensuring that users get a responsive service and quotes in a timely fashion, as well as for devising, developing and introducing new ideas.

"We ask each of our teams to 'build it, run it, own it'. So they don't just write the code, they are accountable for the operation of that code and for the customer experience - they don't hand it off to anyone," says Lomas.

"We are also omni-channel, with about 50 per cent of our traffic now mobile," continues Lomas, which adds to the mix of skills and experience that IT team members need to have.

"We agree, kind-of, the ending [of the journey]. We have measures, such as key performance indicators, but the product teams are self-organising and write their own story. Those teams can innovate on a daily basis - it's quite incremental in nature," he says.

New features and ideas are depicted on a card wall, and make their way from one end to the other as they are developed, tested and go live. "If you are part of a product team, you can say 'I've got an awesome idea about how to improve our journey', watch it move from left to right on the card wall and into 'live' - it's as easy and low ceremony as that," says Lomas.

However, admits Lomas, attracting staff with the right mix of IT skills to work in the Peterborough area is proving increasingly challenging as the jobs market tightens. "One of the challenges we have is recruitment. We want to be more of a magnet for UK tech' talent," says Lomas. "You could say that being outside London is a bit of a barrier. But companies I speak to in London can get staff, but can they keep them? There's much more opportunity and much higher rates of churn."

As a result, it has widened its recruitment net further afield - even outside the UK - but Lomas is, nevertheless, keen to see the CVs of talented IT professionals who don't mind living in and around the Fens or East Midlands, rather than the bright lights of Orpington, Forest Hill, Epsom or other parts of Greater London.

What Lomas is keen to sell to would-be CTMers is what he calls its "culture of entrepreneurialism". That is to say, the autonomy that teams and team members have over the direction of their products, and the kind of innovations more normally associated with Silicon Valley technology companies, such as "five per cent" time.

Mr Five Per Cent

As part of that drive to encourage the IT teams to be more entrepreneurial, the company introduced the concept of five per cent time in 2012. One day a month, individual staff would be encouraged to work on projects of their own, which they felt might be useful or "cool" for the company or the products they work on.

This is not necessarily new - it has been implemented for some time at such companies as Amazon and Google (where it is "20 per cent time"), and has even led to the emergence of major products at Google, including Gmail and AdSense. However, Lomas found that it wasn't necessarily producing the American-style entrepreneurialism that he'd hoped for.

"3M [the manufacturing company] has done it for decades. We wanted to give people dedicated time so that they could just 'riff' with their cool ideas and just have some time away from their products," says Lomas. "We really thought that it would be as easy as that. But when we went back to it in 2013 to try and work out what had happened, we found that nothing had changed."

The reason? Quite simply, it wasn't what IT teams are measured on at work. Besides there was a moral obligation to other team members to make sure team projects were delivered on time and working smoothly. That five per cent time, therefore, became a useful day to catch up with overdue projects, rather than to work on esoteric individual ideas.

"Teams told us that they would love to do it, but that they have an 'engine room' creating new features for customers and are measured on how quickly their 'stories' move from left to right on the card wall. Doing something cool for a day slowed the team's pace and there was also a guilt element," says Lomas.

It was one of the teams that devised an alternative idea, he adds: that five per cent time should be done as a team, instead. "Now, every team uses that innovation time and it's not just about having ideas, but the opportunity to develop those ideas," he adds.

Of course, the IT teams aren't completely autonomous: the company's overall style needs to be respected, not just in terms of graphics, but also drop-down menus and other aspects of the site's usability. On the IT infrastructure side, the company has made a number of enterprise-wide choices, too.

"You have to have these strong, horizontal lines. For us, that would be our 'design authority process'. The danger is that multiple different architectures and coding standards emerge, which isn't really sensible. We also want people to be able to rotate around our product teams.

"Customer experience is one of those horizontals as well. So there's a certain tone, look and feel that you want to achieve. You want it to feel like Comparethemarket.com regardless of which product you use. But it's a difficult balance to achieve. We want autonomy and empowerment for our teams, which means you can't then go in, all guns blazing, saying 'the standard is this' in a top-down manner," says Lomas.

Architecture

That said, as part of a technology refresh currently under way, there are some changes afoot that will give Comparethemarket.com's IT teams even more autonomy.

"As part of the technology transformation that the company is currently going through, each product will have its own application and its own database, with a common SOA [service-oriented architecture] layer that all the teams will use," says Lomas.

That underlying application database isn't a standard Oracle or Microsoft SQL Server database, but a MongoDB NoSQL database, combined with RabbitMQ to provide a messaging infrastructure.

RabbitMQ is a cross-platform message broker, written in Erlang, that implements the open-source Advanced Message Queuing Protocol (AMQP), while MongoDB eschews the traditional table-based structure of relational databases in favour of the dynamic schemas modelled on JavaScript Object Notation (JSON).

MongoDB is preferred because it takes a lot of unnecessary complexity out of the hands of engineers. "The teams seem to love it; it hasn't slowed them down - in fact, it's speeded them up. The NoSQL database is amazing for transactions, but not very good for reporting," says Lomas. For reporting, therefore, the company uses Microsoft SQL Server.

While Comparethemarket.com certainly can't pass itself off as a Shoreditch-style tech start-up, not least because its base is closer to the Fens than Old Street roundabout, it is certainly using technology that ought to make any tech-savvy hipster stroke his beard approvingly.