How open source software took over the world

Open source software now dominates vast swathes of computing

Right in the beginning of computing, almost all software was, effectively, open source.

That's because computing was dominated by a clericy of scientists and engineers who shared and adapted each other's code (in punch card or paper tape format) to suit their particular area of research.

Later, when computing left the lab for the enterprise, companies such as IBM, DEC and Hewlett-Packard sought to lock-in their IP by make software proprietary and charging a hefty licence fee for its use.

The precedent was set and up until five years ago, generally speaking that was the way things went. Proprietary software ruled the roost and even in the enlightened environs of the Computing office mention of open source was invariably accompanied by jibes about sandals and stripy tanktops, basement-dwelling geeks and hairy hippies. But now the hippies are wearing suits, open source is the default choice of business and even the arch nemesis Microsoft has declared its undying love for collaborative coding.

But how did we get to here from there? Join Computing as we take a trip along the open source timeline, stopping off at points of interest on the way, and consulting a few folks whose lives or careers were changed by open source software.

The GNU project

The GNU Project (for GNU's not Unix - a typically in-jokey open source monicker, it's recursive don't you know?) was created by archetypal hairy coder and the man widely regarded as the father of open source Richard Stallman in 1983. GNU aimed to replace the proprietary UNIX operating system with one composed entirely of free software - meaning code that could be used or adapted without having to seek permission.

Stallman also started the Free Software Foundation to support coders, litigate against those such as Cisco who broke the license terms and defend open-source projects against attack from commercial vendors. And in his spare time Stallman also wrote the GNU General Public License (GNU GPL), a "copyleft" license, which means that derivative work can only be distributed under the same license terms - in 1989. Now on its third iteration GPLv3, it remains the most popular way of licensing open source software. Under the terms of the GPL, code may be used for any purpose, including commercial uses, and even as a tool for creating proprietary software.

Amendment: Richard Stallman asked us to provide the following clarification.

I have never advocated "open source", because it rejects the ethical core of what I stand for - free (libre) software. Free as in freedom that is. See my article Why open source misses the point of free software.

PGP

Pretty Good Privacy encryption was created in 1991 by anti-nuclear activist Phil Zimmerman, who was rightly concerned about the security of online bulletin boards where he conversed with fellow protesters. Zimmerman decided to give his invention out for free. Unfortunately for him it was deployed outside of his native USA, a fact that nearly landed him with a prison sentence, digital encryption being classed as a munition and therefore subject to export regulations. However the ever-resourceful Mr Zimmerman challenged the case against him by reproducing his source code in the form of a decidedly undigital hardback book which users could scan using OCR. Common sense eventually won the day and PGP now underpins much modern communications technology including chat, email and VPNs.

"PGP represents the democratisation of privacy," commented Anzen Data CIO and developer of security software, Gary Mawdsley.

Linux

In 1991 Finnish student and misanthrope Linus Torvalds created a Unix-like kernel based on some educational operating system software called MINIX as a hobby project. He opened up his project so that others could comment. And from that tiny egg, a mighty penguin grew.

Certainly he could never have never anticipated being elevated to the position of open-source Messiah. Unlike Stallman, Torvalds, who has said many times that he's not a "people person" or a natural collaborator (indeed recent comments have made him seem more like a dictator - albeit a benevolent one), was not driven by a vision or an ideology. Making Linux open source was almost an accident.

"I did not start Linux as a collaborative project, I started it for myself," Torvalds said in a TED talk. "I needed the end result but I also enjoyed programming. I made it publicly available but I had no intention to use the open-source methodology, I just wanted to have comments on the work."

Nevertheless, like Stallman, the Torvalds name is pretty much synonymous with open source and Linux quickly became the server operating system of choice, also providing the basis of Google's Android and Chrome OS.

"Linux was and is an absolute game-changer," says Chris Cooper of compliance software firm KnowNow. "It was the first real evidence that open could be as good as paid for software and it was the death knell of the OS having a value that IT teams would fight over. It also meant that the OS was no longer a key driver of architectural decisions: the application layer is where the computing investment is now made."

Red Hat

Red Hat, established in 1995, was among the first proper enterprise open source companies. Red Hat went public in 1999 with a highly successful IPO. Because it was willing to bet big on the success of open source at a time when others were not, Red Hat is the most financially buoyant open source vendor, achieving a turnover of $1bn 13 years later. Red Hat's business model revolves around offering services and certification around its own Linux distribution plus middleware and other open source enterprise software.

"Red Hat became successful by making open source stable, reliable and secure for the enterprise," said Jan Wildeboer, open source affairs evangelist at the firm.

How open source software took over the world

Open source software now dominates vast swathes of computing

FreeBSD

Arriving around the same time as Linux, FreeBSD is another Unix-like OS but one that's comprised of a full software suite rather than just the kernel. FreeBSD forms the basis of Apple's MacOS.

"FreeBSD led to the gift that was post 1997 Apple," commented Anzen Data's Mawdsley.

Apache web server

Just as Linux came to conquer the physical server, so Apache HTTP Server quickly became the web server of choice. Just a year after its 1995 release it was the most popular web server - albeit the field was pretty empty back then. More impressively, it's a position it holds to this day. The name allegedly comes from "a patchy" webserver - a set of patches to the NCSA httpd webserver that it soon displaced, although the Apache Foundation has since quashed this story (as well it might).

"I remember comparing the branded web server with the Apache version. Branded version was slower, demanded a higher compute overhead and more bells and whistles than I could use. Apache version worked, was simpler, faster and more reliable," commented KnowNow's Cooper.

As of July 2016, Apache served an estimated 46 per cent of all active websites and 43 per cent of the top million websites. However, its status as top dog may be under threat from another open source web server Nginx, which was written with one aim: to outperform Apache.

MySQL and PostgeSQL

Two open source SQL databases MySQL and PostgreSQL were among the earliest examples of open source enterprise software. MySQL, developed in Sweden by Monty Widneus and David Axmark, was slightly the earlier with a public release in 1996, while PostgreSQL or Postgres emerged from the University of California, Berkeley in as an open source project in 1997, having been developed as an alternative to the Ingres database (post-Ingres = Postgres, geddit? Groan).

MySQL became part of the open source LAMP stack (Linux, Apache, MySQL, PHP) which underpins database-powered web sites globally, while Postgres offered a viable alternative to the expensive Oracle enterprise databases.

"Postgres, like other successful open source platforms, has not stood still," said Dave Page of EnterpriseDB who has been working with PostgreSQL since 1998 and is one of six members of the Postgres Core Team. "The community and vendors have continuously added sophistication to Postgres, which is making it even more attractive today in mission critical environments. It has helped that there is a good understanding between the community and commercial vendors respecting the differing needs of pure open source projects and enterprise IT environments."

Both databases remain widely used to this day, although unhappy with the direction in which Oracle was taking his baby (My is his daughter's name), MySQL founder Monty Widneus has since forked its code to create MariaDB, named after another of his daughters.

Python and Perl

These days pretty much all programming languages are open

source meaning that their compilers and libraries are open to inspection and are free to use - who would pay to use a language these days when there are hundreds out there to choose from? But it wasn't always thus. Microsoft opening up the source code to the C# compiler in 2014 is surely a significant milestone, but scripting languages Python (creator Guido van Rossum pictured) and Perl both began life in the open in the late 1980s. Apple's Swift started as a proprietary project in keeping with its famous ‘walled-garden' approach, but in 2015, in belated recognition that progress happens quicker with more eyes on the code, even Apple bowed to the inevitable.

Wikipedia

Anyone remember Microsoft Encarta? I do, mostly because I worked on it briefly in the ‘90s. Encarta was going to be the encyclopaedia to beat all encyclopaedias, a glittering cornucopia of knowledge that would be sold at a suitably impressive price. That was the plan. Within a few years Microsoft could barely give it away, and by 2009 it was dead.

The reason? Well part of it was the web, the full significance of which Microsoft famously failed to anticipate, but in the main what did for Encarta was a new rival with a model custom made for the web in which anyone could contribute - Wikipedia.

The secret to Wkipedia's success was sharing. Its software is licensed under GPL V.2, as is the documentation, and the text itself is under a Creative Commons Licence.

Founder Jimmy Wales said the open source philosophy was central to the project from the start.

"We chose the free licensing model or open source licensing model really from the beginning," he said during a TED talk. "That was the whole point of the project in many ways because with a freely licensed encyclopaedia, people can take and modify and adapt and use it for whatever they want… It was so core to the original concept that there wasn't even a question about it."

How open source software took over the world

Open source software now dominates vast swathes of computing

Open Office

With Microsoft Office rapidly taking over offices all over the world Sun Microsystems did something drastic. In 1999 it purchased German firm StarOffice and released the code for free. Like Microsoft's offering Open Office was a fully featured suite with word processor, spreadsheet, database, presentation and drawing modules. What's more it ran on Windows, Linux and Solaris, and its OpenDocument format was widely compatible with other office software, eventually even Office itself recognised it. While it never really threatened Microsoft's ubiquitous suite its successor, the fork LibreOffice, is widely used as a free alternative and it comes as standard with most Linux distributions.Mozilla

Once upon a time the web was ruled by only one browser, Netscape Navigator. Netscape's dominion was absolute (if you discount text-only creations like Lynx) but short-lived. It was not long before it was permanently knocked off its perch by Microsoft's Internet Explorer (IE), which controversially came bundled with Windows. By 2003 Netscape

was no more and IE enjoyed a 95 per cent share of the browser market, but then Navigator got its revenge when its source code was taken up by Mozilla and turned into the Firefox browser. Firefox immediately began gnawing away at IE's lead eventually peaking at around 30 per cent of all users, since when it has been oustripped by Chrome. Firefox's aging Gecko engine is being replaced by Quantum, which Mozilla hopes will see a leap in its fortunes.

Github

Many open source software projects are worked on by developers based all over the world. Such collaboration is only possible with an efficient central platform to host and manage the code, which is where GitHub (founded 2008) comes in. Ironically not open source itself, Github allows individual contributors to create a copy or fork of a project to work on, before issuing a ‘pull request' for their changes to be integrated into the main body of the code. Github uses the Git version control language created by Linus Torvalds for development of the Linux kernel. GitHub is the world's most popular site for hosting open source software projects, overtaking earlier efforts such as SourceForge.

Android

While Android is based on the Linux kernel it deserves a slot in its

own right simply because of the way that smartphones have changed everything. Google took Linux and wrapped a proprietary layer around it, meaning the Android is not, strictly-speaking open source. However, Google's decision to build its OS on Linux and to give it away to phone vendors for free means that Linux is now the dominant kernel on mobile platforms and supercomputers as well as being a major force on servers. There is a fully open-source Apache 2.0-licensed version of Android, LineageOS, which was forked from earlier effort CyanogenMod in December last year. Google, in fact, is the originator of many open source projects particularly in the fields of big data and AI.

Chromium

Mozilla's Firefox was had it all its own way as open-source browser of choice until Chromium came along in 2009. Chromium is the open-source version of Chrome and is available on most Linux distributions. As well as Chrome, Chromium is also the foundation of other popular browsers such as Opera, Vivaldi and Firefox- originator Brendan Eich's new browser Brave.

Chrome OS

While Linux may have conquered the server and the smartphone, there are relatively few Linux desktops. However, Chromebooks, running Google's Linux-based Chrome OS, which is built from the open source Chromium OS, are increasing in popularity particularly in education where cheap, easy to use and maintain machines with a lightweight operating system and cloud storage are well suited to the requirements of students.

Xen

Born in the University of Cambridge in 2003, open source hypervisor Xen was soon adopted by Red Hat, Novell, and Sun as their virtualisation solution of choice and became the basis of Amazon's EC2 platform three years later. Given the size that Amazons cloud operations subsequently grew to this fact makes it worthy of a place on this list on its own, but really it deserves a slot because it was the first generally available open source hypervisor, giving companies an alternative to the proprietary VMware and, later, Microsoft's Hyper-V.

Also important was the Xen project's realisation of paravirtualisation, which can simplify the hosting of guest operating systems. Now a self-governing project under the auspices of the Linux Foundation, contributors include Amazon, AMD, Bromium, CA Technologies, Calxeda, Cisco, Citrix, Google, Intel, Oracle, Samsung and Verizon.

How open source software took over the world

Open source software now dominates vast swathes of computing

Amazon Web Services

Amazon is a company that has used a platform strategy to great effect to become

dominant first in books then general retail and now cloud. It encourages third parties to base their businesses on its platforms and thus expands its reach. When Amazon launched AWS a decade ago, no doubt the same principles were very much on its mind. Rather than adopting another tech company's proprietary systems, it started with Linux and Xen and built up from there. Linux, Xen and other open source projects remain the core technologies behind AWS, which is now the largest public cloud platform, and by some considerable margin.

Bitcoin

The pioneering work of the mysterious Satoshi Nakamoto in 2008 on the decentralised maths-based currency Bitcoin - and more specifically its supporting cryptography and ledger blockchain - introduced a genuinely new paradigm for distributed computing. A blockchain is a 'trustless' immutable ledger over which no single party has party has control. Data can only be added to it, never amended or deleted. It is an area of great interest to banks and privacy activists alike, and may also prove useful for IoT applications. Bitcoin has also spawned hundreds of other crypto-currencies, some of which are the basis of the internal economies in the new peer-to-peer networks that are starting to appear.

Big data

The technologies synonymous with big data pretty are pretty much all open-source projects. There's Hadoop, which was open-sourced by Yahoo in 2009 in order to get more developers on board, and the Hortonworks distribution is comprised only of open source projects from the Apache stable, which remains a breeding ground for new software in this field, including notably Spark. NoSQL databases Cassandra, Redis, MongoDB, Basho, Couchbase and numerous others are open source, as are data integration technologies like Talend and Pentaho.

There are exceptions of course, not only among older firms like SAS and Oracle, but database vendor Marklogic also prefers the closed source route. It's also a fact that many open source providers add proprietary enterprise features to the basic community edition as a way of earning a buck. Nevertheless, it's doubtful if the explosion of analytical power could have ever happened in a closed environment.

Microsoft loves Linux

In the past Microsoft was openly hostile to the open source movement, with CEO Steve Ballmer once memorably calling it "a cancer", but cloud computing, especially hybrid cloud, has changed the rules of the game. It is no longer possible for one company to monopolise the underlying computing platform, as Microsoft once did with Windows, making it difficult for rival operating systems to co-exist with it.

Microsoft's decision to enable users to run Linux on it's Azure cloud platform dates back to 2012 when it added support for Ubuntu, Suse and CentOS distributions - having discovered that business users of Amazon's EC2 were deploying Linux instances in greater numbers than Windows ones.

At about the same time Microsoft forged a partnership with Hadoop start up Hortonworks to certify the Hortonworks Data Platform (HDP) for Azure. Nowadays any number of open source services are available on Azure, and Microsoft supports many thousands of open source projects on GitHub - and indeed develops much of its own code there too.

OpenStack

OpenStack is an example of another key feature of the open source picture - collaboration between competing vendors. Originally created by Rackspace and NASA in 2010 it now features contributions from hundreds of vendors including HP, Red Hat, Intel, Dell, AT&T and IBM, as well as thousands of individual developers. The fact that it has support from so many major technology companies is significant, in that it is designed, by virtue of its APIs, to be as platform-agnostic as possible, enabling it to scale out over a multitude of disparate environments. It also suggests that the vendors realise that it is a mistake to paint themselves into their own proprietary corner when it comes to something as big as cloud computing.

Docker

Docker is another open source project that's been making waves in

the software development world. Docker and other containers allow applications to run on any Linux server, without the need to provision virtual machines.

Docker makes use of features of the Linux kernel to create containers that hold the application and all of its dependencies. These "containerised" applications can be transported from Linux machine to Linux machine allowing the applications to function without any further tweaking being necessary. As well as saving on virtualisation software this sits very nicely with the microservices approach to software development.

Kubernetes is also worthy of mention, enabling the deployment management and scaling of containerised workloads.

"Kubernetes puts you in control of your cloud's elasticity in a vendor neutral way," said Mawdsley.

DevOps

As with big data, it's hard to imagine DevOps without open source. The companies that kicked off the movement like Chef, Ansible and Puppet, continuous integration project Jenkins, and issue tracking tool Jira are all open source and widely used. In fact at this point this feature risks becoming a long list of hundreds of tools and plugins so universal has open source software and its associated platforms become.

From its genesis in the intellectual musings of visionary thinkers and the fevered tinkerings of hobbyists, open source is now quite simply the way software is done. The world's biggest companies, not just the Googles and Amazons but Bosch, GE, Accenture and many others are all heavily invested in the open source way of doing things. The wheel has turned full circle.

Next stop: open source hardware.