Harnessing the winds of change: How Shell is working on next-gen machine learning to meet future energy demand

'If you can have machine learning operating in real-time and in stream you really can change the way the whole business operates,' says analytics head Dan Jeavons

Shell is an enormous, diverse organisation, one that is changing fast to keep ahead of an energy market that is moving in many new directions. By 2050 energy demand is predicted to double from its 2000 level but CO₂ emissions must be half of what they are today to avoid the worst outcomes of climate change. New power sources are being brought online, with a significant swing towards natural gas and renewables, and in all parts of the business - from exploration, extraction, production, transport and retail - there is necessarily a strong focus on operational efficiency.

So, like many organisations, Shell needs to be more agile in its operations, but given its size, global reach and the number of physical assets under its purview, for Shell this is more challenging than for most. In practical terms the need to move fast means investing more in the growth areas and particularly in R&D, ensuring that successful experiments are moved into production. On the software side, which is where much innovation currently resides, promising pilots need to scale up so that they are applicable to as many parts of the business as possible.

Image credit: Shell

For Dan Jeavons, general manager for advanced analytics at Shell, the challenge can be summarised by having to do everything simultaneously faster, at a larger scale and with greater flexibility. Software and algorithms must be capable of rapid scaling, but they also need to be adaptable since a one-size-fits-all rarely works, which means an increasing reliance on machine learning (ML). However, there are real difficulties when embedding machine learning (ML) into real-time systems. They need to be robust enough to be able to handle exceptions. Surprises, such as an unforeseen data format coming through the pipeline in real-time, can cause serious problems.

"Real-time systems are a pain to operate and we've been very much learning as we go," says Jeavons, speaking about an online offers system he's been working on for the firm's retail division. "Just keeping a real-time system running is hard, they tend to be quite fragile and if the slightest thing goes wrong it can have a big impact."

It's an area that's only just emerging from the labs so there's not always very much to fall back on, he explained.

"I don't think anyone has managed yet to embed machine learning algorithms at scale in real-time systems, at least I haven't really seen many good examples. It's something we are investing in quite heavily because that's going to be the next frontier, and the opportunity to use that to drive business process change is very exciting because if you can have [ML] operating in real-time and in stream you really can change the way the whole business operates."

On the asset management side of the business Jeavon's team is involved in helping to optimise equipment utilisation be predicting when a drill, engine or bearing is about to fail.

"We're doing a lot of work on predictive asset maintenance, trying to develop a bunch of algorithms that run on a near real-time basis with an alerting mechanism to try and provide several hours of early warning before a failure."

And real-time algorithms increasingly come into play in the supply chain, he went on, where knowing exactly where everything is at a given moment in time is important to understanding where things might be dome in a more efficient way.

[Turn to next page]

Harnessing the winds of change: How Shell is working on next-gen machine learning to meet future energy demand

'If you can have machine learning operating in real-time and in stream you really can change the way the whole business operates,' says analytics head Dan Jeavons

Taking algorithms from R&D into production is one of the key selling points of Apache Spark, it enables code written in many common languages to scale up using the same tools: if it works on a laptop it should work on a distributed cluster. So, it's no surprise that this is one of the tools Jeavons' team makes good use of.

"We started using the free version of Spark and spent four or five months playing around with it, but we then moved to the Databricks platform because of the support and the framework they put around Spark which really helps us scale it up much more quickly."

Databricks' recently announced Delta data management system service is of interest he said.

"The combination of real-time and batch and the management of delta for very large datasets is certainly interesting for some of our use cases.

The company is also a "heavy user" of the Alteryx self-service analytics tools which it uses to help non-technical users understand their data and knowledge graph AI startup Maana, in which Shell has an equity stake.

"They are particularly interested in dealing with a variety of data sources and integrating those into a consolidated knowledge graph."

Data is, of course, central to the process of building new services. Jeavons (pictured) mentioned that the data produced from the weather stations attached to Shell's recently acquired wind farms could well be a case in point.

"We need to improve the way we forecast wind. If we have good weather data coming from those sites we believe we can improve and augment the base weather forecasting services, which are readily available, using our own weather monitoring technologies."

At first, these forecasts would be used to optimise the wind farms, but there is potential to take it further, for example, to provide a broader weather forecasting service in partnership with existing providers.

All such new ideas need to undergo a rigorous evaluation, however, before the data scientists are let loose on them. Maintaining the right balance between experimentation and direction is very important.

"There's very much a funnel approach where we ensure that opportunities are framed very clearly, that they have the right business support before we then move them forward," he said.

"It's early days, as our New Energies business is not very old, so we're still thinking through the implications and what the right strategy is."