How Tesco took an agile approach to rebuilding its data warehouse

Tesco's head of BI, Andy Ruckley, explains how the company needed to shift from a waterfall approach to agile methodologies in order to overhaul the firm's failing data warehouse

When Tesco's head of business intelligence (BI), Andy Ruckley, asked whether the retail giant's data warehouse could keep up with the pace of change in the industry, he stumbled upon some interesting findings.

He'd always regarded Tesco as a data-driven company, but after a full review of people, processes and technology, he found that reports were frequently delayed and that a lot of the data architecture at the company was over 15 years old. He also found that when business users asked for a specific dataset, it wasn't easy to locate.

The quick solution, Ruckley said, was to upgrade the hardware - but he told the business that a bigger box couldn't be the only answer.

"We had to do something really fundamental with our people and processes.... [new hardware] wasn't going to solve the underlying issues we had," he explained at Teradata Universe EMEA in Amsterdam.

His team wanted to take a 'big bang' approach, but Ruckley realised this was the wrong strategy.

"We were trying to recreate a new warehouse in one big go; you can't fulfil data warehouse requirements very easily and you'd never get the business to sign it off, and even if you did it would take you months to do so in which time it is already out of date - so our waterfall approach just wasn't working," he said.

After speaking to Teradata Professional Services, Ruckley said that the company was ready to trial the agile methodology. He said what needed to change was that when business users asked for data they would gain insight rather than receive a spreadsheet with data presented on it.

The first thing his team did was to reduce uncertainty with the data it had.

"We had the data and the right people, so we made some prototypes in which we could question ‘what if I wanted that?', and because it was a prototype, we could grab data from anywhere and build using it, and that helped us deliver a rapid big data solution," he said.

But Ruckley stopped short of giving his "gold stamp" of approval to the data itself, labelling the data as "okay". However, he said that the solution was necessary to help refine the company's requirements and objectives moving forward.

The next phase of the project was to switch from a waterfall approach to an agile way of working, thereby reducing the complexity of obtaining the data.

The team created a discovery stage and a build stage. During the discovery stage, his team would define what exactly they wanted to get and send it over to the build team who would manage the integration and data quality aspects of the model and then start building the production version.

"Because we did some prototyping, we knew what types of reports the business wanted, we threw these to the build team and they started producing those front ends... so we were slimming down the waterfall approach," he explained.

The firm completed a proof of concept with Teradata professional services over six months, and the teams worked in 30-day workflows; once the pre-planning for the build solution was done, they would go on to the next one.

Where to start when rebuilding a data warehouse?

Ruckley looked at the areas within Tesco that overlapped the most when deciding how to go about rebuilding the firm's data warehouse.

"The key area of overlap was with commercial sales, price and products and our stores - so if I'm going to rebuild our data warehouse that looked like a good place to start," he said.

His team initially looked at one country in Asia and decided to overhaul each data warehouse business area in "sprints".

"We analysed across a number of dimensions, across a number of prototypes... and then in the second sprint we looked at sales and prices, and in the third we looked at forecasts," he said.

Ruckley needed a combination of skillsets in the rebuilding project. The team had a business analyst to prioritise the business's needs, a design team of five people to prototype what the team wanted to do, and architects within the business using agile and scrum methodologies looking at dimensions and pragmatic modelling using WhereScape Red and MicroStrategy.

Once they completed their work, they handed it on to a small team dubbed the ‘integration layer', who did all of the development and testing.

During the POC, the firm used scrum methodologies but Ruckley said having a time-box approach didn't suit the developers.

"You looked at the developers and what they were doing - and they were focusing on getting something delivered by Friday so they would cut [things] out, and that reduces the quality of what you're doing and I didn't want that," Ruckley explained.

Therefore, he took away the time element.

"Once you're done you pick up the next one - it's still agile, but it's between waterfall and scrum," he said.

"Then on the access layer I flip back to scrum because I want it in 30 days," he added.

Key findings

Tesco has now rolled out agile methodology across the whole of IT, and Ruckley said that it has been cultural shift for the organisation. People were familiar with plans and milestones, he said, whereas with this way of working it was more about picking up the next bit of prioritised work and working through it.

He said that business users had gone from being completely uninterested to being extremely engaged.

"At the beginning, we had to encourage business users to come to these meetings but by the end of it, they were the first ones there, driving our actions, and they were last ones to leave, taking photographs of everything, because they really wanted to be part of this project," he said.

Ruckley believes his team is working at a much faster rate but that other departments were still using a waterfall approach, and so when his team were ready for integration, the other departments were delayed because they still needed sign-off.

He advised companies who were rebuilding their data warehouse to not just use a waterfall approach, but implement different methodologies.

"Don't just think it is scrum, scrum is just one of the agile development practices. Think about the core principles of agile," he urged.