On-Prem

Networks

Don't turn it off and on again: Expired Cisco cert cripples vEdge SD-WAN kit

Updates said to be rolling out now... if your gateway hasn't already bricked itself


An expired security certificate is threatening to wreak havoc with Cisco customers' wide-area networks. For a change, turning the equipment off and back on again will only make things worse.

In a bulletin published this week, Cisco warned that customers using vEdge SD-WAN appliances could experience complete loss of service if their device is reloaded, updated, or if new templates are pushed.

The culprit: a cryptographic certificate, affecting the SD-WAN appliance's control plane, expired Tuesday, May 9. “If left unaddressed, this could impact data plane connections and result in SD-WAN downtime,” the Cisco bulletin reads.

It's understood this hardware-level certificate is stored in the devices' TPM. And bear in mind, even if you don't manually restart or update your equipment, there are timers in the devices that will, by default, start a reload that will trigger disruption as a result of the now-dead cert.

'Time bomb'

This surprise expiry could have wide sweeping implications for enterprises that rely on Cisco’s Viptela SD-WAN products for communication between their satellite offices, headquarters, and datacenters. While the scope of the snafu isn't clear, plenty of netizens have reported outages as a result of the cert expiry.

"All vEdge based SD-WAN customers are sitting on a time bomb, watching the clock with sweaty palms, waiting for their companies' WAN to implode and/or figuring out how to re-architect their WAN to maintain connectivity," as one put it.

In addition to service disruptions, Cisco said organizations could experience other failures, including:

As of publication, it appears Cisco has released a patch resolving the issue. Posting to Twitter Wednesday morning, Daniel Dib, a senior network architect at Cisco partner Conscia Sverige, shared a (gated) link to a software update to address the disruption, and said additional updates would be rolling out soon:

Based on the documentation, the patch likely amounts to certificate replacement. Unfortunately it doesn’t appear that the update will do much good for devices that have already been rendered inoperable by the expired certs. Cisco recommends customers with bricked gateways contact Cisco for assistance.

The Register has reached out to our contacts at Cisco for comment on how the certificate was allowed to lapse, and what the IT giant is doing to help folks hit by the blunder. The networking goliath declined to comment further.

This isn’t the first time this has happened. As we reported back in 2018, a very similar issue took out Cisco VPNs for customers using the manufacturer's delightfully named Application Policy Infrastructure Controller Enterprise Module (APIC-EM).

That SDN controller relied on an SSL certificate that Cisco neglected to renew, causing all manner of headaches for network administrators trying to provision connections to branch offices and hubs.

While you might think companies would keep tabs on when certificates are set to expire as to avoid these kinds of costly, not mention confidence shaking, mishaps, they aren't uncommon. A dive into El Reg's archives reveals plenty of examples, including several that borked features in Microsoft Windows. So, at least Cisco has company. ®

Send us news
15 Comments

C-suite execs not immune to downsizing drama at Cisco

Maria Martinez, chief operating officer, is out after role was 'eliminated'

Virgin Media to stand up rival network operator to BT Openreach

NetCo hoping to eat some of the pie by opening network plumbing to ISPs

Cisco wields axe again as results season swings around

In an industry addicted to job cuts, 34,000 staff roles vanished in first six weeks of 2024

WTF is 'deployment phasing'? One reason Cisco revenue just went backwards, is what

Splunk deal may close early, but AI is a way off turning into a money fountain. Meanwhile, Cisco waits for you to finish projects

Cisco cuts 5% of workforce amid cautious enterprise spending

$800M charge facing network giant as customers work way through existing inventory

Security is hard because it has to be right all the time? Yeah, like everything else

It takes only one bottleneck or single point of failure to ruin your week

Nutanix doesn't expect a rush of VMware refugees – maybe for years

Beats guidance as renewals grow and waits for Broadcom and Cisco to bring more bucks

Juniper sued over HPE buyout after allegedly ginning up execs' wallets

Material information withheld from shareholders, it's claimed

Cisco, Nvidia expand collab to push Ethernet into AI clusters

InfiniBand dominates in GPU-boosted servers while Big E gains steam

5G network slicing finally shown to be more than pipe dream

Telcos demonstrate configuration in action at research facility. Now to find customers that want to buy one

Curious tale of broken VPNs, the Year 2038, and certs that expired 100 years ago

It’s not NTP. There’s no way it’s NTP. It was NTP

Nearly 4-year-old Cisco vuln linked to recent Akira ransomware attacks

Evidence mounts of an exploit gatekept within Russia's borders