Ubiquiti got a lesson in never blindly trusting external input this month.
Its intrusion detection and prevention system (IDS/IPS) feature on its gateway hardware fetched a set of rules from an outside source that were broken, and rather than ignore the invalid data and fall back to known-valid data, it simply silently stopped working.
Thus users were none the wiser the network security mechanism had failed, and was no longer doing its job of alerting folks to malicious activity, in the case of IDS, or blocking it, in the case of IPS.
The IDS/IPS functionality is said to be built on the open-source Suricata tool, and is branded by Ubiquiti as its Threat Management.
This is a beta service for the network gear maker's UniFi Dream Machine (UDM) and Security Gateway (USG) products. Its job is to flag up, or block, any network activity that matches a set of rules that is refreshed each day. These rules define known malicious network traffic, such as common connections made by malware, so if any of these rules are triggered by packets on the network, it's likely something bad is happening.
Those rules are sourced from Proofpoint's Emerging Threats intelligence service, which pools together a bunch of suspect network packet signatures into one place.
Unfortunately, from Friday to last night, a collection of those rules – ranging from worm and trojan detection to rogue external netblocks – contained invalid data, which caused Ubiquiti's device software to ignore those rules completely. And thus, any malicious traffic that would have matched the rules will have potentially sailed through silently.
You spoke, we didn't listen: Ubiquiti says UniFi routers will beam performance data back to mothership automaticallyREAD MORE
Specifically, Ubiquiti's back-end systems fetch the rules from Proofpoint, and then distribute the rules to UDM and UGS devices. What should have happened is that Ubiquiti should have validated the rules before passing them to customer equipment, and the firmware on the gadgets should not have thrown away its previous rules before installing, and failing on, the latest ones. If either step had been taken, Threat Management would have continued operating as expected albeit using out of date rules.
Ubiquiti promised it would fix the issue within its back-end, and on Wednesday night, the rules were corrected. Devices should automatically download and install the working rule set.
It's all detailed here in the Ubiquiti support forums.
"The feature is failing until Ubiquiti Inc puts out a fix," wrote user AlphaTango, who raised the alarm. "You are not currently protected from the rules ... The rules coming from Proofpoint aren't validated and are then distributed to all users. The lack of rule validation on the update scripts results in bad rules being loaded.
"Right now it appears Proofpoint has inadvertently caused a widespread denial-of-service attack on the Ubiquiti IDS/IPS protection. Rule number one of secure programming, never trust external input."
The Register has yet to hear back on the matter from Ubiquiti, though a support forum rep called Marcus acknowledged the blunder, and said Threat Management was now back to normal.
"This was caused by a change on Emerging Threats open signatures repository that, unfortunately, we have no control over it," said Marcus. "Our firmware team is already working on a fix that should be available soon and will not require any firmware updates since it will be handled on our cloud side."
He then followed up: "The issue is resolved. The IPS/IDS auto-update rules daily, so it should fix itself. However, if you want to make this faster, please feel free to disable/enable IPS/IDS via GUI [to reload the rules] or manually update rules via SSH."
We note that AlphaTango tried to report the issue via Ubiquiti's bug-bounty program, but was denied a reward. The gateway maker's rep said this was because Threat Management is still considered a beta service and out of scope. Marcus also said the device firmware's has other protection mechanisms as well as the IPS/IDS. ®
Thanks to the anonymous reader who pointed us to the Ubiquiti forums. Got a tip to share? You can contact us securely.