Crowdstricken! A Lesson in Technology Risks from CrowdStrike

Crowdstricken! There it is. I'm surprised nobody has used that headline yet. The CrowdStrike(out) events of the past week, in which a botched software or "content" update from the cybersecurity firm resulted in a major shutdown of global IT systems, holds messages about the risks in technology markets.

The CrowdStrike disaster is now being referenced by some experts as the largest IT failure in history, having shut down transportation services, healthcare systems, and financial systems, among others. There are many lessons from the CrowdStrike shutdown. It’s also instructive on human nature. We are naturally bad at calculating risk—risk to systems, to business, and to markets. We’ll dive into the risks and lessons, but first let’s look at what happened.

Here are a few of the impacts:

• A faulty upgrade to CrowdStrike’s Falcon Sensor antivirus software, beginning on July 16, caused shutdowns and “blue screens of death” (BSOD) on millions of Microsoft-powered devices worldwide. It did not affect systems with non-Window operating systems (OSs), such as Linux and Apple. Some organizations are still struggling to get systems back online a week later.
• A chain reaction caused outages in many cloud systems, including a major outage in Microsoft Azure and reported outages in some other cloud services, including Google Cloud.
• The U.S. airline industry was broadly disrupted, with thousands of flights being cancelled. Delta Airlines reported as recently as Tuesday that 41% of its flights were cancelled for the day. Disruptions are still being experienced.
• Many other industries were affected, including critical areas such as healthcare and financial services. One critical example is widespread problems at the UK's National Health Service (NHS).
• Supply chain firm Interos estimates that nearly 700,000 customer relationships were affected by the failure. Microsoft said that 8.5 million Windows devices failed.

Now, this catastrophic failure will be analyzed for months, if not years, but early returns are not good for CrowdStrike. Many software experts immediately criticized the company for not having basic continuous integration/continuous deployment (CI/CD) risk controls in its software delivery process. CrowdStrike said there was a mistake in validating the software upgrade.

Vendor Concentration and Supply-Chain Risk