Software development 2.0: the five pillars of sustainable software development

'Much more than a technical framework, the five pillars are a vision for how we collaborate as a function'

In the second of a two-part series (s ee part 1), Alan Reed, head of Sports Development at Hillside Technology, bet365's technology business, unveils five pillars of development for quicker more consistent delivery of higher quality code.

The process of finding a workable solution to the increasing complex work of developing code for a multi-tenanted, super-fragmented code base started with an education programme.

Systemic change was needed. Sports Development couldn't develop a solution in isolation. It was, therefore, imperative to understand the challenge from the perspective of all the different departments involved.

We embarked on a roadshow of the business and found that two issues were consistently coming up: (1) lack of visibility and (2) a lack of appreciation of what was happening from department to department. Improving both across the software development lifecycle (SDLC) was essential.

A workable solution would require that everyone involved in developing and delivering the code understood the impact their work had on everyone else in the chain. While there's always collaboration between departments much of our work is completed in isolation. A project passing from one team to the next as responsibilities begin and end.

It was clear this was no longer sustainable.

First, we had to accept that we'd be maintaining many different versions of our systems, in parallel, forever. Then the key question was, how do we help Sports Development consider their wider environment and all the stakeholders necessary for moving code from development to live.

Each team needed to be aware of what the other was doing and that communication had to be clear and manageable. It was this realisation that led to what would become known as the five pillars of sustainable software development.

The five pillars of sustainable software development

Much more than a technical framework, the five pillars are a vision for how we collaborate as a function.

1. Diagnostics

Clear communication and an ability to drive collaboration was critical. However, we found that different parts of the business had developed their own diagnostic tools to reflect the way they worked.

This was fine when everyone was working in isolation. Now that there was a need for a broader coalition, it was essential we create a common language through which everyone could monitor and measure their area but also communicate and collaborate.

Developers are most effective when they are fixing bugs, not looking for them

It wouldn't reduce the complexity of writing the code, but it would speed up the development and debugging process. Developers are most effective when they are fixing bugs. Not looking for them. Everything we can do to reduce triage, the quicker we can test, get things live and debug them when they go wrong.

We haven't changed our development methodology. This isn't test driven development. It's the use of tooling to join the teams up and create efficiency of work as the code makes its way down the SDLC and out into live.

Through the diagnostics and tooling, we are creating a unified culture that shares effectiveness, efficiency and efficacy.

2. Hot releases

The QA cycle is typically slow, cumbersome and autocratic. Necessarily so. We couldn't expect it to change. If we were going to find efficiency, the change would have to start in Sports Development. This realisation forced us to look at our own processes and we found that by breaking the code into bitesize pieces everything was more streamlined.

This saw the birth of our Hot Release programme, an incredibly fast delivery mechanism that has ensured our code can get live within minutes. And because they are small modular releases, it gives us the flexibility to release region-specific code when and where necessary. If something fails, all we have to do is replace the part that has failed.

The result is the same amount of code is released but the volume of releases is increased, which are then orchestrated through Kubernetes. Using containers as our transportation and orchestration method, Hot Releases gives us the ability to manage a multi-faceted system that's on different versions in different regions.

3. Automation

We identified two issues with our approach to test. First, we were leaving it to too late in the development cycle and second, we were far too reliant on manual processes. I don't know many developers that enjoy testing, so if you can automate the process and build an incremental analysis of that change you free up development time and reduce the risk of bugs further down the SDLC.

There was less need for manual QA when it reached Test, which opened the door to automating it

Now that our monitoring and diagnostics were synchronised through the SDLC, the QA process was effectively taking place all the way through the lifecycle of the code. This meant there was less need for manual QA when it reached Test, which opened the door to automating it. A process that could be undertaken overnight and any relevant fixes completed the next day.

What we've found is our Hot Release programme, combined with greater automation is giving us such a high quality that less code needs to be tested manually. We can now push code out to live, run the automated test overnight and if it passes, it can remain live. If it doesn't, we can spot the fix needed quickly and push out new code.

This also means that due to the different timescales at which products go live around the world, we can send it out to our regional stakeholders to audit way in advance of its live date.

4. Virtualisation

We currently have a large enough physical estate to handle double the data volume generated during our busiest periods. As we move to a multi-tenanted, multi-instance estate, however, we will need a lot more. But, because we don't know what the end game will look like, it's impossible to estimate how much hardware you are going to need to run it.

We need to virtualise our infrastructure, and moving to Linux and use of containers and Kubenetes is enabling us to do this. We're not sure if we are going to group all the functions together or whether we will split by region. We believe it will be some mix of the two and so we have to give ourselves the maximum flexibility we possibly can.

Moving to a virtualised platform, we are going to be far more efficient in how we use that hosted space. It gives us options for managing our peak load that reflects the sporting calendar, time zones and regions, and it also gives us the fastest and most efficient way to scale our products to our customers.

As we understand the way we are going to virtualise better, we can be more specific when directing the developers on how to optimise their code. Something that is already happening as Golang gains adoption in Sports Development and beyond into the wider SDLC. It is such an efficient language that we are already seeing optimisation gains.

5. Encryption, obfuscation & efficiency

We live in a world of data security. Development has a commitment to hosting safe, secure systems that customers should feel confident in using. Our data usage and consumption are paramount to understanding how to optimise our performance.

The twin goals of security and performance should never be considered as a trade-off. They are amongst the foundations of quality and are part of our code beliefs. It is for this reason that encryption and obfuscation remains a key pillar.

Rather than a repercussion of our need to change our stance, security is omnipresent and must always be considered.

The path of least disruption

It has been quite a journey and one where we have learned a tremendous amount. Three lessons have been key.

We've learnt that portability is as key as performance

First, the development function has a much greater awareness of our entire ecosystem and considers the impact the systems we write has on everybody else down the chain. Second, we've learnt that portability is as key as performance. The code we write now isn't just performant but it's also extremely transportable.

Third, you can never put too much focus on the way your code is consumed. We've learned that collaboration and communication are vital. To facilitate the speed and efficiency needed to get our code out and distributed at pace to each of our external stakeholders, we must work together and come together as a single function.

We know that we can't control everything. But we also know that if we focus on these five pillars we will be on the path to least disruption. Which will mean that fewer things will come back down the SDLC once they've moved out of Development.

Where we are now

Today we're no longer using technology to drive the process. We are using culture to drive our thinking and behaviour. That's not to say that technology hasn't been a big part of the process. It has. It's been the enabler.

Golang has been key here because it has driven the modernisation and miniaturisation of our estate. It has moved us onto the Linux platform, which in turn has enabled us to introduce Kubernetes. Because it's a simple language to use, it has reduced the amount of code we write, enabled us to split the code into bitesize pieces, made communication between teams far easier and enabled us to introduce consistency into the tooling we use.

With Golang the better you get the less code you write

One of the interesting things about other languages is the more proficient you become at writing it, the harder it is for others to read it. You become lost in art of that particular platform. You know the shortcuts, how to bend the rules and the art of bending it to your will. Introducing someone else into this world is much more difficult. Golang is the opposite of this. The better you get, the less code you write.

It's delivered optimisation and enabled us to leverage the things we were already experimenting with in our core systems.

These are not technologies that have sprung up overnight but tools that have been introduced over time that in a wonderful bit of synchronicity have ensured that the five pillars work.