It's every sysadmin's worst nightmare: discovering that someone has planted a device in your network, among all your servers, and you have no idea where it came from nor what it does. What do you do?
Well, one IT manager at a college in Austria decided the best bet was to get on Reddit and see what the tech hive mind could figure out.
User geek_at posted pictures of the alleged suspicious gizmo over the weekend, and the sysadmin subreddit has been providing its particular brand of priceless expertise mixed with uninformed idiocy ever since.
"Rogue Raspberry Pi found in network closet. Need your help to find out what it does," geek_at claimed along with a few pictures of the device – a Raspberry Pi with an unidentified USB dongle stuck into it.
The post included some intriguing details: the network closet it was found in is always locked, requires a key, and very few people have the keys. The Linux-powered Pi was trying to connect to a nearby wireless network. It included Docker containers that were updated every 10 hours. And it connected via a VPN to the Balena platform – which is typically used for large internet-of-things system.
In short, it looked a lot like someone had installed a sophisticated network-bugging device, and was sending whatever it gathered to somewhere outside the facility, presumably via the VPN. Copies of the Pi's file system were taken and uploaded, though no obvious smoking gun was immediately apparent.
The big question was: why? And why hadn't it been more effectively disguised – attaching a loose Raspberry Pi to a rack-mounted switch is bound to attract attention eventually. It wasn't even dressed up in a little box with a "Production DNS" label on it, which would have scared off most techies from poking around it too much.
Asked which industry they worked in – and so the likelihood of it being industrial espionage – geek_at revealed that they were the IT manager at a college with around 1,000 people so the data flowing through the systems in that sense was of "no value."
Testing, testing, one, two, one, two
Various Redditors' first conclusion was that the techie had stumbled on an independent penetration test ordered by management to check the security of its systems, though that explanation fell away after geek_at approached management to reveal the device.
The first breakthrough came when a Reddit poster ID'ed the USB dongle as a pretty powerful IoT device that contains both Bluetooth and Wi-Fi functionality. The nRF52832-MDK costs around $30. The Bluetooth functionality could mean a number of different things, posters mused: it could be connecting to other devices, such as keyboards, and logging their activity; it could be used by someone walking close by to download data from the device, and so on.
"Wireless key logger?" suggested another poster. "Someone could have an inline key logger that dumps data to this box over Bluetooth, then this box ships it out on port 443. Bluetooth is low range, I would check all the PCs within 50 feet for a key logger."
Security guard cost bank millions by hitting emergency Off buttonREAD MORE
Another user who claims to be a professional penetration tester concurred: "Completely agree. I did something similar to this in a Pen test my company was hired to do. Managed to do it to the CEO and CFO; went weeks before anyone spotted it. Lots of juicy information was given out to me thru this. Fortunately it went to me and not the bad guy."
Intriguingly, geek_at revealed in a subsequent update that the switch room is "only feet away from secretary/CEO office" and that a program he had identified as running on the device was called – yep – "logger." That could be a key or network activity logger, of course.
Intrigue then turned to who had managed to get into the room: only geek_at, their boss, and the cleaning staff had access. At which point, the "other Reddit" – the one that tried to identify bombers at the Boston Marathon in 2013 and did a dangerously terrible job of it – kicked in.
Out came the instant experts: it was the cleaners! Suddenly a vast global conspiracy started brewing: a Jason Bourne character dressed as a cleaner sneaking in to do… who knows what? Nuclear war maybe? Certainly an assassination of some kind.
Here we go
"This is a serious problem," railed one user who is a walking advert for the expression "a little knowledge is a dangerous thing." Filled with visions, they continued: "Who had the access and/or authority to put anything in your rack? If you aren't king of the hill, you need to be going to the top on this one."
Great advice. But wait there's more: "Lock it the hell down! Secure the door to your rack, change the lock IMMEDIATELY. No entry allowed. I'd also consider putting a dummy device in place and see who comes to try to retrieve it. Take total control of that server room, no one in without you standing over their shoulder."
America! Fuck yeah! "You need to be informing the FBI," the clearly very experienced poster noted.
Being based in Austria, calling the FBI wasn't something that geek_at felt was going to be terribly useful, however. But on it went, the eternal online battle between people who know what they're talking about, and those who believe they do.
Despite numerous treatises to put the college into full lockdown and call the cops – who presumably would turn up within minutes in tinted black trucks bellowing information-laded instructions to one another, geek_at instead took the issue to the higher ups based with what he had.
And what's the latest? It turns out that there was someone else was able to get into the room: a former employee that "still has a key because of some deal with management," geek_at informed Redditors.
It also turns out that the IT bod was able to identify the username of that former employee – and he had been seen attempting to log into the system just a few minutes before the device was spotted poking the organization's DNS server.
So what we have is a former employee who for some reason had access to a secure server room in the heart of the organization, without the IT manager being informed, and who installed a fairly sophisticated bit of kit seemingly designed to sniff wired and nearby wireless network traffic and/or connect to and log Bluetooth devices, such as keyboards, and fire all the gathered intelligence back to base via a VPN.
"Still no idea what it actually does," explained geek_at in their latest update. And so we've contacted the netizen to see what the latest is, but the lesson appears to be – gasp! – don't give old employees access to your server room.
And, possible, maybe, on occasion, that Reddit can actually be useful. ®
PS: If you like these sorts of tales, check out the kid who found devices hidden around his college library, the housemate who plugged an ad-injecting backdoor into a home router on the promise of $15 a month, or better yet, our long-running series On Call, which IT staff reveal the horrors and hacks they've witnessed over the years.