Set the bar high: Lessons learned from a decade of outsourcing to India

Worldline CIO Ryan Bryers has been on a mission to build a high quality offshore DevOps team at low cost

Ryan Bryers is UK&I CTO and CIO at Worldline Global, the financial transactions service provider that's part of consulting giant ATOS. Having spent most of his career at ATOS, Bryers has learned many lessons from the great outsourcing boom of the early to mid-2000s, when among stakeholders in the company the attitude was very much: "if you're not using India you are going to struggle". The motivation was the bottom line: Indian developers could be had for the quarter of the price of those in the UK (at least on paper). It was also about fueling rapid growth.

This pressure remains to this day, although it has been somewhat tempered by experience, Bryers told delegates at the Computing DevOps Live event in Manchester earlier this week. Those earlier experiments in outsourcing to India had met with mixed results, for a number of basic reasons.

First, there was a lack of financial control, meaning that savings on paper were rarely achieved. Then there was the issue of time zones: it's difficult to work with people who are five hours ahead, especially if they are managed on an arm's length basis, which was very much the practice back then.

"I don't think we really knew what we should be expecting when went to India," said Bryers, describing the experience as "a hell of a challenge".

"The biggest problem we ran into was quality control. We hadn't thought about how to have quality controls outside of our office space for these guys 5,000 miles away."

Quantity over quality

The "Indian arms race" demanded that offshore teams be built quickly. This showed shareholders that the company was growing fast, remaining a contender in this hotly contested space. So teams doubled in size every year until the company had hundreds of contractors in Pune, India. Trouble was, said Bryers, the quality wasn't there.

"Technical debt is a silent killer. We had might look good on the surface when you rip off the cover is the code easily readable? Can it be changed quickly? Does it pass all the tests?"

In far too many cases the answer to these questions was ‘no'.

Good on paper

In addition to variable quality, a lack of financial controls meant that offshoring rarely achieved the anticipated savings. The costs would nudge up over time, meaning that, while it might still be cheaper to hire people in India than the UK, the headline "quarter of the price" figure which had originally caught the eye of the CFO was never realised. Costs could double or more without it ever being quite clear why.

The cultural challenge

The difference in time zones is really only a small part of the challenge of managing offshore teams. Culture and expectations play a much bigger part.

"The cultural challenge really shouldn't be underestimated," Bryers said. "We underestimated it, and it is still underestimated today."

The inevitable misunderstandings due to cultural difference were exacerbated by the distance-management style adopted by ATOS. In the six years before Bryers started to shake things up there had only been one visit to India by the management. This lack of communication enabled confusion and frustration to grow. Small wonder quality was low.

Keeping development in India at arm's length isn't the answer

"Keeping development in India at arm's length isn't the answer, you have to build relationships and build their capability," said Bryers.

Among the cultural differences Bryers pointed to was a reluctance on the part of the Indian developers to talk about everyday problems and challenges or to ask questions of their UK counterparts - in stark contrast to the combatitive ‘engineering culture' in Europe and the West.

This can lead to a situation where "you get something a bit different from what you asked for."

"The offshore people tend to make an assumption, and they're are not naturally inclined to ask for clarification. As a result, you get back their interpretation as opposed to what was requested."

Another factor is that Indian life is much more family-oriented than it tends to be in Britain.

"In India they are very much focused on relationships, and you get a lot out of Indian people if you understand how they work and also how they live. It's a culture that feels comfortable in a family environment, so being part of something that feels a bit like a family is very important."

An offshore DevOps team

A couple of years ago, Bryers started to build an offshore DevOps team with the aim of being able to work flexibly across time zones in an Agile way using the Scala programming language. He was determined not to repeat the mistakes of the past.

Can I get quality at the low cost? Because 10 years back we couldn't

"Can I get quality at the low cost? Because 10 years back we couldn't. We got low quality and low cost on the spreadsheet but with higher cost in reality," he said. "We wanted to build the capability so our Indian teams could be a bit more au fait with the way we like to work, with the tooling and the approach."

The process began by inviting three Indian employees to work in the UK for six months, to build relationships and to find out how adaptable they might be to learning new technologies like Scala (they were Java developers) and to a more Western-style of working where engineers are expected to ask more questions and raise objections if they don't agree with something. They were also there to teach UK staff about how better to relate to their Indian counterparts.

"We asked them for feedback, and they gave us ideas about how we could be more successful. After six months they went back and we then committed to visiting regularly."

Setting the bar high

Bryers went against management's preference for more of the same, building a team of 40 people offshore within two years. "I said I only wanted 10 and if it grows organically with the right people then we're happy with that. If it's still 10 people in 12 months that's OK."

This meant "setting the bar high" so that only the most capable engineers would be considered. He had his colleagues set up what they called the ‘Tech Studio' in India, in which people from different offices were encouraged to share skills and learn new languages from each other ("so we don't get the problem of ‘.Net is the answer, what's the question?'") under the watchful eye of mentors and coaches. Those who stood out were encouraged to spend three months seeing if they could "reach the bar" and join the core DevOps team.

Set the bar high: Lessons learned from a decade of outsourcing to India

Worldline CIO Ryan Bryers has been on a mission to build a high quality offshore DevOps team at low cost

"The expectation was on them was for constant delivery and just to be prepared to have the odd disagreement because actually we need that in forming the teams."

Those that didn't make it had at least learned some new skills that should stand them in good stead, said Bryers.

The system had an early success when the new team won a global challenge to create a chatbot: "no one from the UK ever won that award," Bryers said.

The main lessons learned

Bryers has now recruited his core team of offshore engineers. There are around 10 individuals in the team, and he is happy to keep it at that level unless it makes sense to grow organically. The bar will remain high, he says, and most won't make it. He told the audience at DevOps Live about some of the lessons learned.

Pune is one of India's hi-tech hubs

Recruiting and retaining the right people is hard

The first stage of the recruitment process was dogged by applicants' CVs being "not quite as accurate as you'd like them to be," and by promising candidates being poached by the big names. In fact, out of 44 people interviewed initially, none made the final cut.

"When you're looking for a DevOps Scala engineer most people with those skills were going off to work with Apple and Google, and they would come to us as a sort of safety net: if I get an offer from you I can go off and find an offer from someone else."

Fortunately the internal Tech Studio eventually produced a small number of suitable engineers, and Bryers found recruiting graduates direct from universities to be a useful source of talent.

"We grab them straight off the conveyor belt once they've had their boot camp training, so to speak, so straight into a DevOps approach and straight into Scala development. So then they can be developed under the guidance of the lead engineer we have in India, and we maintain that control."

Coaching continues a daily basis giving guidance on approach and methodology. The company offers a competitive if not Google-style salary, with perks for good performance, and Worldline tries to find other ways to keep the developers happy, Bryers explained.

"It is the good remuneration; it's the challenge, give them some training, give them some difficult work, make them feel part of the family. You do have to accept that there is a risk they will go to Google if you train them as DevOps engineers with skills in Scala. You have to accept that."

Elite groups and the talisman

The bar for entry is high, and expectations remain at that level. A lot of it is about attitude.

I often get slammed for calling them elite groups

"I often get slammed for calling them elite groups," said Bryers. "But let me explain. When I say elite it's a pure sporting reference, it's the challenge of becoming the best you can possibly be, not necessarily being better than someone else."

Having an elite group gives other engineers something to aim for he believes, and the practice can also filter out to others among WorldPay's 450 staff in Pune.

What's also required to make a cohesive team is a talisman, as all football managers know.

"You have to have one person to build around. And that person needs training, so you need to guide them to be the sort of person we want them to be."

This sort of training requires open channels of communication. Bryers and his colleagues now visit India regularly, up to six times per year rather than once every six years as happened before. In addition, he tries to use video and collaboration tools as much as possible so that the teams are in touch at all times, and is regularly on the phone to India at 6am.

Are we DevOps yet?

Certain Agile and DevOps practices are hard to maintain across time zones, for example, the UK ‘morning stand-up meeting' happens in the afternoon in Pune. While the team is making strides, Bryers says there is a bit more of the journey to go, is prepared to take time to get there.

"For me, it's quality over quantity every day of the week, and I'm prepared to take longer to get something perfect."