Fastly blames software bug for massive internet outage this week

The company has apologised to people for the inconvenience caused as a result of the glitch

A major internet blackout that left many of the world's top websites offline for a brief period on Tuesday was caused due to a software bug which got triggered after a customer updated their settings, CDN provider Fastly has disclosed.

"We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change," Nick Rockwell, Fastly's senior vice president of engineering and infrastructure, said in a blog post on Tuesday.

According to Rockwell, the company began a software deployment on 12 May, which "introduced a bug that could be triggered by a specific customer configuration under specific circumstances."

When a customer pushed a valid configuration change on 8 June, the bug was triggered, causing 85 per cent of Fastly's network to return errors.

Rockwell said the issue was detected within one minute. The engineers identified and isolated its cause, and disabled the configuration.

The outage occurred at 0947 GMT, and engineers worked out the cause at 1027 GMT, Rockewell said. Once the configuration setting was disabled, most of the company's network quickly recovered.

"Within 49 minutes, 95% of our network was operating as normal," he said, adding that a permanent software fix was initiated to roll out at 1725 GMT.

Rockwell apologised to "customers and everyone who relies on them" for the inconvenience caused as a result of the major outage.

"Even though there were specific conditions that triggered this outage, we should have anticipated it," Fastly's senior executive noted.

"We provide mission-critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority."

Rockwell promised to examine why Fastly failed to identify the software flaw during its own testing process.

Meanwhile, on Tuesday, several websites, including Amazon, Twitch, Reddit, Spotify, HBO Max, PayPal, Quora, Vimeo, BBC, the New York Times, and Bloomberg, were either working improperly or not loading. Users who attempted to access the sites were shown error messages.

Fastly, a content delivery network (CDN) company, sits between users and websites and helps them to view digital content in a faster way. Apart from CDN services, the company also provides security, video delivery, and other edge-computing services.

Fastly's CDN system is one of the biggest on the internet, along with similar networks operated by Cloudflare, Akamai, and Amazon's CloudFront.

Tuesday's outage raised questions about the risks associated with dependence of the internet on a handful of infrastructure companies.

Fastly's recovery led to an increase in its share price, which was up 12 per cent on Tuesday. This could be because the firm demonstrated that it was able to implement a quick and effective incident response plan, or because the glitch made investors more aware of the size of Fastly's customer base.