Storage at scale: picking the right options for Kubernetes

Managing storage for large, complex distributed containerised applications is a whole different game; industry insiders discuss how the options stack up

In the old days, applications and storage sat in separate domains. Application developers didn't need to worry too much about storage - that was someone else's job; and anyway, the choice of backends was restricted to a small number of well-defined options: a handful of databases, data warehouses, and file, object and block storage infrastructure.

Not so now. The DevOps way of doing things has seen infrastructure abstracted into code. Storage, and more broadly data flows, are now very much within the developer's remit.

So long as things remain small this is not that much of an issue; the main challenge is choosing between the dizzying array of options now available. And if the first choice doesn't perform it's generally not too hard to swap it out for another.

But as the scale heads towards large, some inevitable trade-offs hove into view.

Scaling up

The ability to scale applications up (and down) is a key driver for the adoption of microservices and containers, and of Kubernetes which automates many of the management tasks. Indeed, the ‘cloud-native' movement was to born out of a wish to create applications that can be deployed across multiple datacentres and also pushed down into tiny IoT devices. But as applications grow larger and more complex with greater levels of concurrency and requirements for lower latency, the role of storage and related services becomes more and more central.

Data infrastructure is now the focus, Patrick McFadin, DataStax

"Developers want to use containers to build their apps, they think about using Kubernetes to mask some of the complexity around how to manage that app… and then they think about their data," said Patrick McFadin, VP developer relations at DataStax, the data management company built on Apache Cassandra.

"Data infrastructure is now the focus. It had survived for years as less of a problem than application infrastructure, but now operators are anxious to get all infrastructure under one umbrella."

Kubernetes supports a whole host of storage volumes (28 are listed here and there are many more) which pods (groups of containers) can use to manage their data and state, so there's plenty to choose from, but as applications grow developers may discover their first choice was not the right one.

Matt Yonkovit, chief experience officer at vendor-independent database-as-a-service provider Percona, says developers should think carefully about how the application will be used in the foreseeable future, and not just default to a familiar option.

"Often, developers build the app and then think ‘which database am I most familiar with? I'll use that again', he said.

"Kubernetes can help you make the most of that familiar database and automate processes, but it might not be the right option all the time. There are so many more options around cloud hosting, around managed services, around different versions of the same database that it's worth spending the time upfront to consider what you are putting together and what you want it to achieve.

"Automating something that performs to a lesser level is fine, but it's not as good as a properly tuned and configured set-up. Automating your database when it is properly tuned can really help with your performance and scalability over time."

Managing state

When containers first arrived on the scene they weren't much concerned with state and stored data because they tended to be used in places where these things weren't a primary concern, but as they've grown more popular so the range of use cases has expanded. Log files, transactions records and state all need to be stored somewhere.

There is no such thing as a stateless application, Alex Chircop, StorageOS

"There is no such thing as a stateless application - all applications need to store state somewhere," commented Alex Chircop, founder and CTO of containerised storage platform StorageOS.

Now you see me

Storage infrastructure for Kubernetes must be highly available and agile because pods are distributed and ephemeral. Kubernetes might kill or regenerate an individual pod at any moment in order to optimise the performance of the application as a whole. Pods are also portable: an individual pod may be moved to or recreated on a new node, perhaps in a different cloud environment. When this happens it's important the data follows, said Chircop.

"The application state and dataset must also be portable and available just like the application, otherwise if a container fails, you'll lose access to your data when the container is restarted.

"The storage system needs to be able to cope with high rates of change and cloud-native storage solutions are built from the ground up to handle this."

Data at scale

Large-scale containerised applications are rarely standalone and complexity can arise where they interface with other platforms. Data-intensive use cases such as machine learning are becoming the norm, but current containerised methodologies show some limitations at scale and not just with respect to storage, said Brad Shimmin, distinguished analyst enterprise technology at technology research firm Omdia.

"While you might think that serverless computing technologies like AWS Lambda seem ideally matched to ML/DL modelling use cases, unfortunately, that's not the case. The sheer size of many real-time deep learning inference models, which can be several gigabytes, simply outstrips the computing power available to serverless technologies right now."

Shimmin continued: "Kubernetes and containers themselves are geared around asynchronous compute power (hence the 'cloud-first' notion). They are not tuned to managing the deployment and management of resources that include both asynchronous code - code you'd run using Lambda - with synchronous data resources that are intimately tied to that code.

Database - or something else?

Database companies are rushing to position their products as the ideal primary storage solution to Kubernetes storage issues, and in many cases, they could be a good choice, said Omdia's Shimmin.

"Databases are highly optimised to gather, store and serve such information and to do so securely and at scale. They provide predictable performance, they do auto-scaling quite well, they offer numerous opportunities for global distribution, and they just so happen to come equipped with a rich set of interfaces for query and execution. For that reason, we're seeing a number of prominent database vendors jumping on this notion in support of workloads like ML inference at scale."

Unsurprisingly, DataStax's McFadin sits firmly in the database camp: "State and storage are your primary concern as you deploy containers. How do you manage state in your application while the application serving tier is ephemeral? Pushing state down to the data layer has been an established solution and using a database like Cassandra with the uptime and scaling features make it a perfect choice."

Then again, there are many types of database - graph, time series, key-value - and column stores like Cassandra, each with its own strengths and weaknesses. There are Kubernetes operators available for many well-known databases including MongoDB and MySQL and Oracle is making a play for hands-off operations with its autonomous database. Depending on the use case, some organisations might prefer to store data in a purpose-built layer like StorageOS from where it can be served to whichever database fits the bill. There are many factors to consider.

Transactional vs operational

If the application is primarily transactional in nature, data will need to be committed to multiple nodes to ensure consistency, whereas for more operational workloads low latency in terms of reads and writes might be more important. So far so standard, but when applications are distributed across multiple clusters, these qualities must scale too.

There isn't one storage system that's good for everything, Alex Chircop, StorageOS

"Developers need to understand what their application needs are to choose the appropriate storage system," said StorageOS's Chircop.

"Some storage systems might be optimised for very large data sets or scale for analytics, and others may be optimised to reduce latency for transaction performance like database use cases. There isn't one storage system that's good for everything, that's why there are multiple options, it's up to the application developer to understand the options that are available to them."

Cloud agnosticism / data agnosticism

Kubernetes opens the door to genuine cloud agnosticism. Containerised applications can be pushed out across different cloud platforms and even over bare metal. Any storage platform must be able to follow suit.

"You want to keep your data close to your application, as that reduces latency. You don't want great application performance and then everything slows down as soon as you call for data. At the same time, you also want to avoid being locked into any specific cloud service," McFadin said.

However, storage choices may not be so agnostic. Indeed, databases are a common cause of de facto lock-in.

"You have to consider this sort of trend [database as primary data store] as a trade-off between gaining access to [the advantages databases offer] and losing the ability to move from platform to platform openly," Shimmin said.

Wrapping up

So should developers of large-scale distributed containerised apps opt for a database as the primary data store, a purpose-built cloud-native storage layer - or something else? The unsatisfying answer is ‘it depends'.

Any potential solution must have one key characteristic, however: it must be flexible enough to change with the times and able to operate with the minimum amount of manual intervention.

"Making your database autonomous means that it can self-heal if something goes wrong, and backups are taken automatically and regularly, avoiding single points of failure. This is a goal that everyone should be working towards and Kubernetes can help make that happen," said Yonkovit.

Kubernetes is now farly widely adopted but it, and the ecosystem around it, is still changing fast.

"Kubernetes is like a giant that doesn't know its own strength yet," said McFadin. "It is so powerful that it needs care in how it is handled, but if you can make it work for you then you deliver a great result when it comes to scale."

McFadin continued: "As you start deploying the data tier in containers, storage is the next problem to manage. Making sure to use things like Docker Volumes to externalise your database storage. When orchestrated properly, you can build an elastic application stack from ingress to data that will scale as you need."

In the meantime, there is a need for standardisation to avoid the creation of new silos, argued Shimmin: "I feel that we should, as an industry, look for some sort of common ground between database and cloud platform as an application repository with both adhering to the same set of containerised standards."