This article is more than 1 year old

Pwned so many times - but saved by the incident response plan

In IT paranoia is never a bad thing

Sysadmin blog Companies that are more proficient with technology are more likely to believe that their security is "very effective". Is this a form of contempt born of familiarity, or a true understanding of the risks? The bigger the company, the harder they fall, and no organisation – not even the US state department – has proven impenetrable.

Survey after survey is conducted and each time it seems that the"early adopters" of whatever tomorrow's next big thing is are the companies who think they have got this security thing taped. Companies who felt ahead of the curve because they were all cloudified were riding high on the knowledge that public cloud providers have better security than most.

Until they realised you're only as secure as your stupidest mistake.

Internet of Things and wearable adopters don't seem to even be thinking that far. Lots of widgets are being deployed with security threats that will never be patched and companies are gleefully putting these on the same networks as their primary data. In IT, the hard lessons learned face-first by so many other companies in the past never seem to be retained.

It's easy to think that large companies with huge resources have security down. Small business admins are constantly told by their better resourced brethren about how little they know, how being an enterprise admin playing with enterprise toys means that the SMB admin's skillset is old, rusty and will never amount to anything.

I disagree.

Unhealthy security

An off the record discussion with a source of mine within the US health care insurance industry revealed the shocking fact his very large employer had thousands of servers with no anti=malware, intrusion detection or other defenses installed. These aren't any old servers either - we're talking database farms filled with the most sensitive of customer data. The entire organisation relies on eggshell security: crunchy defences on the very outer edge, but the inside is soft.

Worse, being in the industry they talk to others in the industry, and it turns out that this is fairly standard. It is alleged that some of these companies even cover it up in order to pass audits. This, BTW, looks like this remains the case after the ridiculous hacks against Premera. Which, if you'd followed the Anthem hacks, is no big surprise. And that's before we get into revenge hacking by health care sysadmins.

After some digging around, I found a common theme to the "why" of this: the CIOs were under immense pressure to cut costs, and they didn’t feel the risk of getting Sony-ed was all that high. Besides, their companies themselves have insurance in case of a breach, so really, why should they care? The consequences just aren't high enough.

Fortunately, this is changing.

I got pwned. Right pwned. Several times

Paranoia is proper

I've been hacked. After 20 years in IT, my systems have probably been penetrated dozens of times. It happens a few times a year now, so I've honestly lost count.

By admitting this, I am of course setting myself up for a right pantsing in the comments section for being wholly inadequate to live. Sysadmin machismo and brogrammer culture demand that all technologists be infinitely capable, infinitely knowledgeable and under no circumstances ever admit fault.

Well screw that. I got pwned. Right pwned. Several times. Some of them were clearly my fault (I done goofed) and some of them were the result of completely asymmetric resources deployed against me. I've seen stuff turn up on my personal systems that qualifies as state-level (seriously, they hid the malware in the video bios!)

I've also had attacks against systems I thought were "secure" that were so spectacularly beyond my level of capability that months later I still don't have the foggiest idea in hell how they got in. Those systems had no known exploits, the attackers danced past every security feature, from fail2ban to layers of intrusion detection. To this day, I'm completely baffled.

I have come to accept being pwned as a matter of course. I don't have the resources of the US State Department. I don't have the resources of a health care company, Sony, or a cloud services provider. They can't – or don't, or won't – put the time and effort into fending off black hat hackers, so in what universe do I get off thinking I'm going to?

And that's before we talk about the bit where using public cloud computing is essentially handing your data over to governments that are decidedly hostile to even their own constitutional guarantees regarding individual civil liberties.

Those breaches however, aren't "fatal". Why? Paranoia. Defence in depth. A realisation quite some time ago that eggshell computing is a really, really bad plan.

Separation anxiety

We must assume that any system on our network can be – and in the fullness of time will be - compromised. Data and applications need to be segmented. Different applications need to be separated. Pwning one system cannot be allowed to grant an attacker access to all other systems, nor can it allow them the ability to nuke the only copy of vital data.

As I write this, I am dealing with a pwned webserver on a client site. Reasonably well built server, but some putz walked through some ancient and unmaintained script in the client's custom-built (and publicly facing) web-based middleware. They thoroughly owned the CentOS VM running that workload (using an off-the-shelf package that, by the looks of it, was designed to compromise Debian,) and used it to try to send spam.

Which must have been frustrating for the attackers, as there's a lovely little hardware appliance monitoring every packet out of the host that system is attached to, and it is squelching any and all email traffic. Oh, and the website files are loaded read only. And it doesn't have access to deeper layers of the network because the routers squelch any attempts (and then freak out and email when they detect it.)

I've taken down the compromised script, shaken my finger at the devs and the VM is being cloned for some later forensics. Somewhere around the point where I finish writing this article, I'll revert to the "known good" snapshot, run updates and we're back to clean living.

Sysadmin enough to admit you need help

I learned a long time ago that I'm not the IT messiah. I was once pretty decent at providing enterprise-level IT on a practically impossible budget, but the older I get the less stamina I have to play that game. IT is bigger today than it was 20 years ago. No one human being can fit all that needs be known about our industry into their brain.

So I simply gave up. Every now and again I need an adult. I am not going to build a better firewall than F5. Even if it is all I did all year long, they are smarter, more experienced and there are simply more of them than there are of me. Hell, technology is moving so fast today that I can barely keep up with rebuilding my anti-spam server's applications and definitions up to date. Every 6 months or so ClamAV makes some change that borks everything and I have to rebuild it.

As with all things in IT, plan for failure. Including your own.

Security appliances are required. Professional security services are required. External audits, additional pairs of eyes and above all else engineering your network expecting to be compromised are required.

You aren't as good a sysadmin as you think. I don't care if you fly to work every day on your personal helicopter that you handcrafted from mahogany while getting 50,000 steps a day on your Fitbit and bench pressing a 747 full of lesser sysadmins, I promise you that you aren't hot shit enough defend your home network against a well-resourced attack, let alone an enterprise IT network.

Accept it. Deal with it. And as with all things in IT, plan for failure. Including your own.

Prevention will not stop all security threats. Detection will miss some incidents. You, and I, and all of us need to be spending time on mitigating the inevitable, and preparing incidence response plans for when breaches happen anyways. ®

More about


Send us news

Other stories you might like