Database shakedown: Five reasons why there's a revolution going on

10gen CEO Max Schireson explains why there is room in the market for a new generation of providers

Max Schireson, CEO of database firm 10gen, seems accustomed to taking the long view. This is no doubt a useful attribute when your main ambition is to challenge Oracle in its home territory - the notoriously conservative £20bn world of the enterprise database.

"It may take a decade to get there," he told Computing. "But we just want to get to the point with the market that the customer can pick the technology that suits their use case best. When that happens, a large number will choose document-oriented databases over relational ones."

10gen is the distributor of the open source NoSQL database MongoDB. Schireson offered a number of reasons why he believes the document-oriented distributed data model represented by MongoDB is more suitable than the relational one for a growing number of use cases. But first some history.

"The relational database was invented in 1970. You have a technology that was invented over 40 years ago to support accounting-type applications with very regular tabular data, with schema that got updated every couple of years, that were deployed to dozens of users, that ran on a single computer. Now people are trying to use this technology to manage everything from social media to customer contacts and contracts, often being deployed to millions of end users and clusters of machines - and they want to update them every day. If you started off with that goal you never would have designed that [relational] technology, but that's what people are using."

Which leads us to our first reason why a shakedown is happening in the database world.

Reason 1: The advent of big data

The document-oriented, distributed model is better able to cope with large volumes of disparate data that is changing very rapidly.

NoSQL databases are designed to run on clusters of commodity servers and can be easily scaled up by adding new hardware. Data is stored in a flexible format such as JSON rather than the rigid rows and columns of relational database tables, allowing formats to be changed on the fly and new data sources to be incorporated with the minimum of disruption.

"Originally 10gen had a vision of building a database that would be more agile for web-style development and more scalable to be able to run on clusters of inexpensive commodity servers. We built that and people have responded very well to it," said Schireson.

Reason 2: Priorities have changed

All technology design involves trade-offs. When storage was expensive data models were designed to minimise the footprint on disk. However, development time is now a much more valuable commodity than storage, Schireson explained.

"When hard drives that could store megabytes cost as much as a house, it was virtually criminal to waste any storage," he said. "Now developer productivity is much more important, so using something like JSON - which is a data storage format that's self-describing and very simple for developers to work with - might be more important than one that's 20 or 30 per cent more storage efficient by not having to store all of the labels that go with all the fields.

"Knowing that the phone number is always the third column might be more storage efficient because you don't need the label, but it's less flexible. If I want to add a cell number, Skype, all these various contact points, I have to go and edit the table to make room for these new things. That's not well suited to actual development."

Reason 3: Developers have more power

Shireson believes that the last 10 years have seen a sea change in the way that decisions are made within the IT department.

"The CIO is an incredibly important role, but nowadays the CIO has much less control over technology selection than they did a decade ago," he said. "Some of that has to do with open source becoming a mainstream approach, with developers downloading stuff and trying it for themselves. Some is about a cultural sense of empowerment: where skills are scarce developers are speaking up more about what they want to use. This means they can choose technologies that make them more productive."

[Turn to next page]

Database shakedown: Five reasons why there's a revolution going on

10gen CEO Max Schireson explains why there is room in the market for a new generation of providers

Reason 4: New applications are better suited to the document model

Schireson acknowledges there will always be a place for relational databases, but he believes the document-oriented model is a better fit for many modern applications.

"The majority of new applications are going to be better suited to a document-oriented model," he said. "Relational is great for a broad set of applications but most of those applications - accounts receivable, accounts payable, payroll and the rest - have been built already. That's not where a lot of application development is. For new projects the application development that we see is more interactive. It pulls in a lot of product information, customer information, social media information... This sort of application often tends to fit better with the document-oriented model. The most important thing is that the customer has a choice and can use relational or document models, whichever fits best."

Reason 5: People want alternatives

"[The database sector] is a boring market because it has been so dominated by the biggest players for so long, and they are some of the biggest tech companies out there," said Schireson.

"When we started we were really only used by web companies, but we're seeing a lot more traction with traditional enterprises now - banks, telcos, governments and insurance companies are doing a lot with Mongo. That's the biggest change over the last year."

Schireson went on to describe how US insurer MetLife had 70 different customer service systems. As a result they had to deploy 24 different customer service lines, because they couldn't train one agent to use 70 different systems. Metlife spent years trying - and failing - to unify the data in a relational database, whereas with MongoDB it took a couple of months to get all the data in one place and develop a customer service system that could interrogate it all at once.

The long march

So why are organisations still using relational databases if they are unsuitable in many circumstances? Why is it such a tough market to crack?

According to Shireson this inertia is down to the core structural importance of databases within the enterprise. The wide breadth of use cases for enterprise databases means they support many important functions - functions that are very difficult to migrate elsewhere.

"It's the level of investment that users make," he said. "When developers choose a database they develop applications specifically for that database, they invest a great deal in the training and the code. This ‘bakes in' that layer of the stack and it doesn't move easily. Even with an open standard like SQL it's hard. For example, there are plenty of banks that chose Sybase 25 years ago and would love to change, but the effort to move to another provider is too great."

There are historical reasons, too. In the 1980s, the idea was to standardise on one database for all applications. When weaknesses of this monolithic strategy became apparent, a split occurred between operational databases, which support real-time applications, and data warehousing solutions, where newer players such as Greenplum, Teradata and Netezza have emerged. Schireson believes the time is ripe for another schism to split the market, this time in the operational database field.

"People don't want to have too many tools in their toolbag," he said. "No-one wants to have 17 databases - that would be too hard to support. But I think in the future people will have two choices for operational development: a relational choice and a document-oriented choice. This will become mainstream. The bang for the buck will be there."