The big problem in big data: a lack of skills

A lack of skilled analysts has hamstrung the industry at a time when the demand to adopt AI and big data analytics has never been higher, says Aaron Auld

IT decision makers at all levels cite a lack of skills to make the most out of the digital revolution as a frequent pain point for business, and yet, the clamour and need to adopt technologies like machine learning, AI and big data analytics is only growing stronger. This has led to these technologies failing to achieve their full potential, and even putting them perpetually out of reach for some companies.

The IT skills gap is a perennial problem for the industry that is only getting worse. A lack of properly skilled and qualified analysts has hamstrung the industry at large, at a time when the demand to adopt technologies such as AI and big data analytics has never been higher.

There are a lot of contributing factors worsening the current skills gap, ranging from a lack of quality graduates to an overreliance on expensive multi-skilled contractors. On a fundamental level though, the available pool of properly trained and experienced analysts simply has not kept pace with the recent explosion of new technologies.

Many companies are hiring what they think to be unicorns, only to find they've accidentally adopted a donkey

One of the most in demand roles is, of course, the data scientist. Genuine data scientists are highly sought after because they are often seen as the forward-thinking, innovative, problem-solving superheroes of the industry. In many ways, they are the unicorns of the analytics business. Unfortunately, with the current drought of new talent struggling to fill an ever-increasing number of roles, many companies are hiring what they think to be unicorns, only to find they've accidentally adopted a donkey.

As Computing's Big Data & IoT Review 2017 research points out, there is little consensus of what the core skills of data scientists are - and specifically what distinguishes them from data analysts. There is some agreement that a data analyst is somebody that looks at data and derives insight from it, and data scientists are notable for their creativity and ability to delve deeper to find insights on a broader scale.

For businesses, being able to store, access and model data is only one part of the process. When they do not have people with the right skills to convert that data into actionable business intelligence, return on investment will continually fall short of projections, no matter how incredible the technology.

So what are businesses supposed to do to avoid being trapped somewhere in the widening skills gap? Put simply, if data really is going to be "the new oil" then we need more oil well drillers. Turning to multi-skilled contractors, who are naturally expensive and in high demand, is little more than a stop gap measure that will exacerbate the problem in the long run. A more practical solution is to upskill current employees, but once they have been re-assigned, their old responsibilities will still need filling, effectively moving the problem down the line. In the long term what is needed is a focus on creating a data literate workforce from the bottom level up.

At EXASOL we are currently providing placements to students at The Data School, an institution offering a training programme for the next generation of aspiring data scientists, teaching them the skills they need to succeed in analytics using tools such as Tableau, Alteryx and EXASOL. Over the last five years, the team at The Data School has trained thousands of people on these tools, affording trainees a deep understanding of how to store, manage, prepare and visualise data in a business setting.

A key problem they address early on in their training is the need for soft skills, which are essential for conveying the value of big data analytics. Finding the insights is one thing, being able to present what you found to the rest of the business is something else. Data scientists need to have the whole package, from data analysis to presentation skills; a mix of the scientific and the creative. If they can take a holistic approach to drawing out actionable insights from the data they gather then this will equip them to better demonstrate the value of big data to businesses that are not necessarily technology literate.

Looking towards the future, it is clear that we need more initiatives such as The Data School to train our next generation of data scientists. We also need Government initiatives offering grants and incentives to help close the gap. Without action, as the industry continues to grow so too will the gap, continuing to outpace the rate of new skilled individuals coming in. Only when we have created a data literate workforce from the ground up will we see the digital economy realise its potential.

Aaron Auld is CEO of EXASOL