Black Hat Neil Wyler and Bart Stump are responsible for managing what is probably the world’s most-attacked wireless network.
The two friends, veterans among a team of two dozen, are at the time of writing knee deep in the task of running the network at Black Hat, the security event where the world reveals the latest security messes.
The event kicks off with three days of training, then unleashes tempered anarchy as the conference proper gets under way.
Wyler, better known as Grifter (@grifter801), heads the NoC – the network operations centre – at Black Hat, an event he has loved since he was 12 years old. “I literally grew up among the community,” he says.
Bart (@stumper55) shares the job.
Wyler's day job is working for RSA's incident response team while Stumper is an engineer with Optiv, but their Black Hat and DEF CON experience trumps their professional status. Wyler has worked with Black Hat for 14 years and DEF CON for 17 years, while Stump has chalked up nine years with both hacker meets.
Together with an army of capable network engineers and hackers, they have operated two of the few hacker conference networks that delegates and journalists are advised to avoid.
Rightly so; over the next week the world’s talented hacker contingent will flood Las Vegas for Black Hat and DEF CON, the biggest infosec party week of the year. The diverse talents – and ethics – of the attending masses render everything from nearby ATMs to medical implants potentially hostile and not-to-be-trusted.
Some 23 network and security types operate the Black Hat NoC and are responsible for policing that particular conference's network, which they helped create. Come August each member loosens the strict defensive mindset they uphold in their day jobs as system administrators and security defenders to let the partying hackers launch all but the nastiest attacks over their network.
“We will sit back and monitor attacks as they happen," Wyler tells The Register from his home in the US. "It's not your average security job."
The crew operates with the conference din as a background, sometimes to cheers as speakers pull off showy hacks or offer impressive technical demos in rotating shifts. In the Black Hat NoC, some laugh, some sleep, and all work in a darkness broken by the glow of LEDs and computer screens. Their score is a backdrop of crunching cheese Nachos, old hacker movies, and electronic music.
"Picture it in the movies, and that's what it's like," Stump says, commiserating with your Australia-based scribe's Vegas absence. "It'll be quite a sight, you'll be missing something."
Delegates need not. The Black Hat NoC will again be housed in The Fish Bowl, a glass den housing the crew and mascots Lyle the stuffed ape and Helga the inflatable sheep. Delegates are welcome to gawk.
The Black Hat NoC operators need to check their defensive reflexes at the door in part to allow a user base consisting almost entirely of hackers to pull pranks and spar, and in part to allow presenters to legitimately demonstrate the black arts of malware.
"When you see traffic like that, you immediately go into mitigation mode to respond to that threat," Wyler says. "Black Hat is a very interesting network because you can't do that – we have to ask if we are about to ruin some guy's demonstration on stage in front of 4,000 people."
Stump recalls intruding on a training session to claim the scalp of a Black Hat black hat slinging around the infamous Zeus banking trojan on the network: "The presenter says 'it's all good, we are just sending it up to AWS for our labs' and we had a laugh; I couldn't take the normal security approach and simply block crazy shit like this."
Flipping malware will get you noticed and monitored by one of the Black Hat NoC's eager operators who will watch to see if things escalate beyond what's expected of a normal demonstration.
If legitimate attacks are seeping out of a training room, the sight of Wyler, Stump, or any other NoC cop wordlessly entering with a walkie-talkie clipped to hip and a laptop under arm is enough for the Black Hat activity to cease. "It is part of the fun for us," Wyler says. "Being able to track attacks to a location and have a chat."
Targeting the Black Hat network itself will immediately anger the NoC, however.
The team has found all manner of malware pinging command and control servers over its network, some intentional, and some from unwittingly infected delegates. "We'll burst in and say anyone who's MAC address ends with this, clean up your machine," Stump says.
Training is by far the most expensive part of a hacker conference. Of the 71 training sessions running over the weekend past ahead of the Black Hat main conference, each cost between US$2500 (£1887, A$3287) and US$5300 (£4000, A$6966) with many students having the charge covered by generous bosses.
So it was to this writer's initial incredulity that most of the sea of "weird porn" flowing through the Black Hat pipes stems from randy training students. "It is more than it should ever be," Wyler says of the Vegas con's porn obsession. "While you are at a training class – I mean it's not even during lunch."
The titillating tidbit was noticed when one Black Hat NoC cop hacked together a script to pull and project random images from the network traffic on Fish Bowl monitors. A barrage of flesh sent the shocked operators into laughing fits of alt-tab. Another moment was captured when Stump was filmed for on CNN Money and a shopper's blowup doll appeared with perfect timing.
Black Hat's NoC started as an effective but hacked-together effort by a group of friends just ahead of the conference. Think Security Onion, intrusion detection running on Kali, and Openbsd boxes.
Now they have brought on security and network muscle, some recruited from a cruise through the expo floor, including two one-gigabit pipes from CenturyLink with both running about 600Mbps on each. "We were used to being a group of friends hanging out where a lot of stuff happened on site, and now we've brought in outsiders," Stump says.
Ruckus Wireless, Fortinet, RSA and CenturyLink are now some of the vendors that help cater to Black Hat's more than 70 independent networks. "It's shenanigans," Wyler says. "But we love it."
The pair do not and cannot work on the DEF CON networks since they are still being built during Black Hat, but they volunteer nonetheless leading and helping out with events, parties, and demo labs. "I feel a responsibility to give back to the community which feeds me," Wyler says. "That's why we put in the late nights." ®
PS: There's some more technical info on DEF CON's network – which is, of course, separate to Black Hat's – over here.