Have you tried to build your data team backwards?

David Palmer, Market Intelligence Partner & Future Skills expert at BPP sets out the pitfalls of building your data teams backwards.

2023 is proving difficult time for UK businesses to be building data teams. With unprecedented demand for data engineering roles, new AI technologies to grapple with, and a talent shortage gripping the UK labour market, hiring people with the right skills is harder than ever before.

Part of the issue in the UK is that businesses have been building their data teams ‘backwards' — hiring data scientists before hiring the data engineers who construct and maintain the systems that give data scientists something to analyse.

By looking back at the way that the data profession has developed in the UK over the last decade, we can learn some lessons that will help us to build effective data teams in the future: teams with the right mix of people and the right mix of skills.

Building backwards

The work of a data team is usually quite structured: first, data engineers built a data pipeline, then data analysts perform relatively straightforward analysis while data scientists conduct advanced analysis that - if they're lucky - includes opportunities to experiment with interesting machine learning algorithms and neural networks.

Simply put, data passes from data engineers to data analysts and data scientists. UK businesses, however, appear to have hired data scientists first, and only started to hire large volumes of data engineers in the last two years

The data science goldrush

Between 2012 and 2018, businesses went through a period of frantic data scientist hiring. In 2019, the Royal Society published an analysis indicating there had been a 1,287% increase in postings for data scientists between 2013 and 2018.

This is understandable. in the 2010s the field of data science had taken centre stage, with articles in Harvard Business Review dubbing it The sexiest job of the 21st century. At the same time, advances in neural networks were helping data scientists coach algorithms to find videos of cats on YouTube whilst IBM Warson was off winning Jeopardy.

Understandably, companies went on a data scientist hiring spree.

These new data scientists, however, quickly found themselves confronted with data pipelines that needed work and data quality that needed fixing.

For the last decade it's been common knowledge in the profession that data analysts and data scientists should expect to spend a considerable proportion of their time cleaning data which is not the best use of a data scientist's skills. A more efficient data team structure would see data engineers maintaining data pipelines so that data scientists are free to discover valuable insights that help inform business decisions.

Data engineers: here to save data quality

The increase in demand for data engineers over the last few years has been remarkable: businesses seem to be trying to correct course. Looking at recent labour market data to see how postings for roles have developed over time, we can see that data engineer postings haven't just seen a post-pandemic bounce, they've exploded.

Data engineer postings have gone from fewer than 1,000 per year to over 32,000, meaning the demand for data engineers has now outstripped demand for data scientists.

For the data analysts and data scientists who have spent a significant amount of time building serviceable data pipelines, this market correction may be a huge relief. For UK businesses, however, the demand for data engineers now outpaces the supply of talent, which presents a problem.

What should businesses do?

The lesson for businesses is that they put greater emphasis on structuring their data teams so that there are specific roles for each stage of the data process; engineers to do the engineering, analysts to do the analysis, and data scientists to do the advanced analytics.

For the last decade, the line between these roles has blurred as a single analyst or data scientist was expected to perform every step of the process, offering no efficiency gains for the business through specialisation.

The UK's talent shortages are, however, particularly acute in technology roles It is unlikely that businesses will be able to recruit the data professionals they need - especially for an emerging role like data engineering - so ‘building' rather than ‘buying' talent is a practical solution.

Building data teams through professional apprenticeships

At BPP, we spend our time developing data apprenticeships that give data analysts, data scientists, and data engineers the skills they need. Far from only being of use for entry-level roles, professional apprenticeships can train people up to and including Level 7 MSc qualifications.

Additionally, we are part of a ‘trailblazer group' that is designing a new data engineer apprenticeship standard, which will - for the first time in the UK - provide a blueprint for training capable data engineers.

The idea is simple: if employers are struggling to hire data professionals, we can help by training data engineers, data analysts, and data scientists, using training funded by the apprenticeship levy - a government funding scheme specifically designed to provide training funding to address skills gaps.