Don't turn it off and on again: Expired Cisco cert cripples vEdge SD-WAN kit

An expired security certificate is threatening to wreak havoc with Cisco customers' wide-area networks. For a change, turning the equipment off and back on again will only make things worse.

In a bulletin published this week, Cisco warned that customers using vEdge SD-WAN appliances could experience complete loss of service if their device is reloaded, updated, or if new templates are pushed.

The culprit: a cryptographic certificate, affecting the SD-WAN appliance's control plane, expired Tuesday, May 9. “If left unaddressed, this could impact data plane connections and result in SD-WAN downtime,” the Cisco bulletin reads.

It's understood this hardware-level certificate is stored in the devices' TPM. And bear in mind, even if you don't manually restart or update your equipment, there are timers in the devices that will, by default, start a reload that will trigger disruption as a result of the now-dead cert.

'Time bomb'

This surprise expiry could have wide sweeping implications for enterprises that rely on Cisco’s Viptela SD-WAN products for communication between their satellite offices, headquarters, and datacenters. While the scope of the snafu isn't clear, plenty of netizens have reported outages as a result of the cert expiry.

"All vEdge based SD-WAN customers are sitting on a time bomb, watching the clock with sweaty palms, waiting for their companies' WAN to implode and/or figuring out how to re-architect their WAN to maintain connectivity," as one put it.

In addition to service disruptions, Cisco said organizations could experience other failures, including:

• Loss of connections to vSmart and/or vManage
• Port-hopping in some way impacted
• Control policy changes affected, including topology changes
• Interface flapping

As of publication, it appears Cisco has released a patch resolving the issue. Posting to Twitter Wednesday morning, Daniel Dib, a senior network architect at Cisco partner Conscia Sverige, shared a (gated) link to a software update to address the disruption, and said additional updates would be rolling out soon:

Based on the documentation, the patch likely amounts to certificate replacement. Unfortunately it doesn’t appear that the update will do much good for devices that have already been rendered inoperable by the expired certs. Cisco recommends customers with bricked gateways contact Cisco for assistance.

• Cisco: Don't use 'blind spot' – and do use 'feed two birds with one scone'
• Dump these insecure phone adapters because we're not fixing them, says Cisco
• Russian snoops just love invading unpatched Cisco gear, America and UK warn
• Cisco Moscow trashed offices as it quit Putin's putrid pariah state

The Register has reached out to our contacts at Cisco for comment on how the certificate was allowed to lapse, and what the IT giant is doing to help folks hit by the blunder. The networking goliath declined to comment further.

This isn’t the first time this has happened. As we reported back in 2018, a very similar issue took out Cisco VPNs for customers using the manufacturer's delightfully named Application Policy Infrastructure Controller Enterprise Module (APIC-EM).

That SDN controller relied on an SSL certificate that Cisco neglected to renew, causing all manner of headaches for network administrators trying to provision connections to branch offices and hubs.

While you might think companies would keep tabs on when certificates are set to expire as to avoid these kinds of costly, not mention confidence shaking, mishaps, they aren't uncommon. A dive into El Reg's archives reveals plenty of examples, including several that borked features in Microsoft Windows. So, at least Cisco has company. ®