11 hours ago
Google Cloud Stumbled and Unplugged the Internet: Here's How the 'Crash Loop' Began
Tobias Schwarz/CNET
Last Thursday, a massive internet outage took down vast swaths of the online world, kicked off by issues with Google Cloud services and affecting sites including Spotify, Discord, Snapchat, UPS, Pokemon and many of Google's own Workspace offerings.
Google has since acknowledged -- in a lengthy post on its Cloud status page at the end of the Friday workday -- the technical issues that spurred the outage, which stemmed from a series of failures in how a new system management feature was set up and deployed. It also described the steps it is taking to prevent similar problems from occurring in the future.
By about 3 hours after it began, the issue had been resolved and most sites and services had returned to normal operations. Late Thursday evening, Google issued a preliminary incident report on its Cloud status page, tying the incident to disruptions in its API management system. And apologized: "We are deeply sorry for the impact to all of our users and their customers."
The Google Cloud platform supports a wide array of fundamental internet services, from networking to storage to AI, that businesses rely on.
IT services company Cloudflare, describing how the outage affected its customers, said that "the proximate cause (or trigger) for this outage was a third-party vendor failure."
Earlier in the day, as we all found ourselves blocked from accessing favorite services or necessary work tools, Google was noting that its engineers had "identified the root cause" and had taken steps to mitigate the issues. Still, some areas were not responding as fast as others. "Our infrastructure has recovered in all regions except us-central1," Google said.
Affected companies began noting that they were seeing recovery. By 2 p.m. PT Thursday, the Downdetector service was showing that the spikes in outages were largely past their peaks and quickly heading toward zero reports of problems. (Disclosure: Downdetector is owned by Ziff Davis, which is also the parent company of CNET.)
Google noted at the time on its Workspace status page that its services were back up and running.
"The problem with Gmail, Google Calendar, Google Chat, Google Cloud Search, Google Docs, Google Drive, Google Meet, Google Tasks, and Google Voice has been resolved," it posted at 12:53 p.m. PT. "We apologize for the inconvenience and thank you for your patience and continued support."
See below for the latest updates and notes on individual sites and services.
