CenturyLink is in the Hot Seat

An outage at CenturyLink (CTL) has drawn attention to the service provider’s security infrastructure — and raised questions about Internet routing using Border Gateway Protocol (BGP) overall.

CenturyLink, which provides internet and managed services based on software-defined wide-area networking (SD-WAN) acknowledged that on Sunday, August 30, a router issue caused an outage at one of its data centers in Ontario.

In a disturbing cascade of outages, peered routers attached to CenturyLink’s infrastructure went haywire. AWS, Cloudflare, Twitter, Hulu, Microsoft Xbox, Reddit, OpenDNS, and others, along with a range of interdependent e-commerce sites worldwide, took performance hits. Various sources say that 3% to 5% of global Web traffic was affected.

How CenturyLink Routers Misfired

The error at CenturyLink was traced to a faulty configuration involving Border Gateway Protocol (BGP) flow specification (flowspec), a feature used to reroute traffic among peer routers on the Internet in order to mitigate distributed denial of service (DDoS) attacks and other security woes.

One of the entities affected, Web content and security service provider Cloudflare (NET), nabbed flowspec as the culprit early on, and Cloudflare CEO Matthew Prince blogged about the issue shortly after it occurred:

At Cloudflare, early in our history, we used to use Flowspec ourselves to push out firewall rules in order to, for instance, mitigate large network-layer DDoS attacks. We suffered our own Flowspec-induced outage more than 7 years ago. We no longer use Flowspec ourselves, but it remains a common protocol for pushing out network firewall rules.

Outage Raises Issues of IP Routing

Problems with current routing protocols have been the target for a range of software-defined networking (SDN) and security innovations, such as a new network operating system from Arrcus; virtual networking platforms from Alkira and DriveNets; virtual “e-mesh” from Elisity; remote access and cloud security solutions from Zscaler (ZS), as well as many others. As 5G networking brings more attention to the network edge, vendors will be under pressure to provide more solutions for distributed routing.

“The Internet is growing, MPLS is slow, it’s expensive. LTE and wireless is growing,” said Amir Khan, the co-founder and CEO of Alkira, in an interview with Futuriom in July 2020. "[W]e need to create a fabric across multiple types of transport which is ubiquitous and secure and can span anywhere to anywhere without worrying about the underlying transport.”

CenturyLink in Hot Seat

CenturyLink didn’t need the extra scrutiny it drew Sunday. Less than two years ago, another outage drew ire when 911 calls, government sites, ATM withdrawals, and other resources were disrupted for over 24 hours, leading to a federal investigation of CenturyLink.

It doesn’t stop there. Court papers filed Monday, August 31, settled a class action suit with customers (terms undisclosed) for accidentally exposing the personally identifiable information of 2.8 million subscribers.

And here come the shareholders: Today, litigators announced a lawsuit pending against CenturyLink executives for failing their fiduciary duties, in part related to the company’s acquisition of Level 3 Communications in 2017.

Of course, all service providers, and all public companies, must fight litigation regularly. But for companies competing in today’s increasingly complex digital environment, mistakes draw unwanted attention.

In a sense, CenturyLink is serving an awkward but hopefully temporary role as a cybersecurity canary in the coal mine, pointing the way to the pitfalls looming ahead, and the changes needed to avoid them.