How EA built FIFA with DevOps

Joshua Nixdorf, technical director at Electronic Arts, explains how the gaming giant revolutionised its development process

Few people may know it, but England actually won the last Fifa World Cup.

Held in London in May 2017, England's top player of Fifa 18, a football simulation game from Electronic Arts (EA), beat his German opponent to achieve a feat which has eluded players of the real-life game for over 50 years.

The Fifa series is phenomenally popular, having sold well over 100 million units since its first release in 1993. And the game is evolving, with each annual iteration bringing new players, tactics and ways of playing.

This complexity is in part made possible by new software development practises, including DevOps.

Joshua Nixdorf is a technical director at the firm, tasked to run the development teams. He began his career at EA twelve years ago as a build engineer working on NBA Live, a basketball series.

"Build engineer was the worst job, you're the first in and last out," says Nixdorf. "I had to make sure that QA [Quality Assurance] had builds [working code], or there would have been a hundred testers sitting there with nothing to do. Also we were releasing games all on Blu Ray. If we weren't sure we were burning the right thing to the disc, we'd have been wasting thousands of dollars on media.

"We didn't have automation in place, there was no continuous integration in those days, so we were kind of blind," he adds.

He explains that the gaming industry suffers from a traditional "boom or bust" working model, where teams regularly put in mammoth working shifts to get products finished on time.

And even with this phenomenal effort, it sometimes still can feel insufficient.

"We do lots of QA, both manual and automated today. You can easily have a million or two hours of testing put into a game before launch, which sounds a lot. But with a big game, the player base have played it way more than that within a couple of hours of launch."

The DevOps Journey

Discussing the DevOps journey EA have been on, he explains that it became apparent some years ago that the old method of "throwing code over the fence" at the testers or operational people wasn't going to work as games increased in size, but game development cycles didn't necessarily increase with them.

"As we scaled up, we realised we needed to change the business model to be less boom and bust. We had to change the way we develop, and DevOps has been a huge piece of how that's happened."

Most coders will be familiar with branches, where software is duplicated so that many versions can be modified simultaneously. The idea is to save time - teams can work on multiple different parts, before synchronising it all at the end.

However, this isn't often practical at EA, according to Nixdorf.

"At EA, development generally happens in very few branches. QA are playing games in development all the time, and they need to see the entire product. That means we're very much trying to keep everything on the main trunk all the time. And that's hugely risky, someone could check in bad code and then suddenly everybody's broken all the time."

How EA built FIFA with DevOps

Joshua Nixdorf, technical director at Electronic Arts, explains how the gaming giant revolutionised its development process

One answer to this problem is automation, specifically automated code validation.

Nixdorf and his team started introducing automation capabilities into EA, with a ‘build it and they will come' philosophy. However, the early system wasn't perfectly reliable.

"We saw what happens if you don't have perfect reliability. But even with a perfect solution to the automation problem, people won't depend on it until they've learnt to trust it. You need to try it, then watch it not explode before people will use it."

He argues that making DevOps work in an organisation is far more about culture than tools. That was becoming obvious, as it was increasingly hard to hire for his old role, the much-maligned Build Engineer.

"It's hard to hire engineers to work on the side-lines, not on cool features you can tell your friends about. Then during the final year of a game's development, you pull the build engineer into bug fixing, and after that they don't want to be build engineer any more.

"So we recognised that that was never going to work, and needed to break out of that pattern."

Further problems with scale then started rearing their heads, with games still on Blu Ray, with huge artefacts (source code compiled for testing) and binaries (object code), which require long durations to build.

"And we needed different configurations for testing all the various releases for different consoles and other machines, so suddenly our developers needed to do 12 builds after each piece of development just to check they didn't break anything. And that's just for a game that only ships in one region!"

Once development teams saw the time that could be saved with automation, it took off "like wildfire".

"But we had no big data centres and this was still pre-cloud, so we started investing in virtualisation," Nixdorf explains.

EA started using VMWare for virtualisation, Chef and Puppet to handle configuration management, and NetApp filers (a storage device) to move builds around faster.

"That's where Jenkins [a continuous integration and continuous delivery tool from CloudBees] entered the picture for us," he adds. "At this point we weren't identifying ourselves as a dev team, or focusing on the infrastructure layer, but learning about operations, so we broke out of that dev mindset and into ops. Jenkins helped us solve problems like recruitment, because we have this thriving community we can share with. Where something doesn't give us competitive advantage, we share it, and push that community culture."

But even Jenkins wasn't a cut and dried choice; there were five other continuous integration solutions in place initially in different parts of EA.

"Each faction had standardised on a tool to solve each problem, so the first thing we did was to have a holy war over which one to use. Jenkins won, due to its open source nature. That means we could continue to develop with it, but framed in a way to be shared back. So we're not just developing for ourselves but for the whole community.

"That also gave us a framework for our own maturity. So if we decide to develop a plugin that doesn't exist, the first question to ask is: why doesn't it exist? It's like having guardrails."

Problems of Scale

Some source code files that EA needs to maintain are upward of 500 Gigabytes in size, and these need to be kept for significant periods of time. Nixdorf explains that the firm used to spend so much time crafting environments to perform builds, that when the environments broke they'd just ignore it, as the effort required to create a new one was so huge.

Today EA uses Chef as a way to treat these environments as "cattle rather than pets", in other words, make them disposable.

"Now we're at a stage where we can throw it away if it looks suspicious, as with automation, environment setup isn't nearly so much of a bottleneck."

How EA built FIFA with DevOps

Joshua Nixdorf, technical director at Electronic Arts, explains how the gaming giant revolutionised its development process

And with a series like Fifa, the scale of the challenges with the complexity of the more recent iterations, and size of the teams involved, means that development would be practically impossible without a DevOps culture.

"The scale that Fifa operates at is massive. It ships in multiple regions across three main platforms and four configurations. There are 120 50GB artefacts, which is a huge amount that QA need to have access to. The dev team is massive, and spread throughout the world, and there's a very small number of branches, so you need a huge amount of automation and source control.

"It's a complicated group dance, and continuous integration and the tenets of the DevOps culture is almost calling out the moves so everyone stays co-ordinated."

Nixdorf adds that this co-ordination is crucial in order for trust to exist between different studios working on the same title.

"The build environments need to the same across different studios. If you lack this co-ordination you introduce bugs which one studio misses, but perhaps another one finds. Then they blame the first studio for introducing the bug and trust goes out of the window. That leads to some toxic situations and it can be brutal.

"So you take those problems away from the studios and give a consistent environment across all of them."

This shift to a DevOps culture has also enabled the modern games industry business model. Whereas previously a title would be worked on for up to three years, with all development finishing on the day of release, today's games have a far longer tail. This means both paid-for and free content being released semi-regularly, often for many years after the original release.

"Maintaining that model takes a lot of ops," says Nixdorf. "You need release engineering [the compilation, assembly, and delivery of source code into finished software] and continuous delivery principles. Where you used to finish stuff and throw it over the fence [to operations], now you shift it all left into those development processes. Similarly, running a live title used to be divorced from the dev process, but now it's just part of it."

Next steps and lessons learnt

The next step for Nixdorf and EA, he explains, is to transition game servers into the cloud.

"Just think of the benefits of having the dev process in the cloud," he begins. "It's pretty standard for other industries, but we have massive scale problems. It's not cheap to pull 50GB artefacts off Amazon."

It's this issue of cost that Nixdorf is currently struggling with before he reaches the dream of a cloud-based dev process, though he claims to be confident that a solution is in reach.

"If I'm paying for data transfer, how do I minimise the volume of data I need to shift around? If I put a new endpoint somewhere, do I need to worry about 1,000 build machines synching across the cloud? Or what if we switch to PC-based instead of console testing? If we make that change it'll all be about the results, not the artefacts, and I can leave those in the cloud, so I won't have to pay for those transfers."

Nixdorf finishes by saying what he'd do differently were he starting the DevOps journey afresh today. As is often the case, he says that recognising how difficult it can be to change culture is the biggest lesson.

"When we started I assumed if we built good tech people would just use it, but I found that culture doesn't change overnight. There are a lot of steps in going from classic software development to DevOps, and you need to go through each, and mindfully. So the biggest challenge has been around culture and mindset, and in finding people with the flexibility to ask the right questions."

Both the FIFA 18, and the real-life World Cup are fast approaching once again, with both occurring over the next couple of months. England are once again in the running for victory in the former, something which seems unlikely in the latter - at least for another four years.