Several weeks ago, security researcher Lawrence Baldwin dispatched an urgent email to abuse handlers at OptimumOnline, the broadband provider owned by Cablevision, warning that one of its customers stood to lose more than $60,000 to cyber crooks.

"He's got a keylogger on his system . . . below is a log of the miscreant viewing the info that was logged from his system while accessing his [Bank of America] accounts," Baldwin's email read. "Looks like he's got nearly $60K in there, so a lot at stake. Can you get someone to phone me that might be able to establish contact with this customer?"

The email, which was addressed to a specific handler's email address and was also copied to OptimumOnline's abuse desk, went on to provide the user's IP address and enough specifics to suggest Baldwin's claim of a keylogger was probably accurate. Yet, more than three weeks later, Baldwin still hasn't heard back from the company.

"Normally, I don't bother because I think this is going to be a complete waste of time," says Baldwin, who is chief forensics officer for myNetWatchman.com. "The abuse and security department at an ISP is the bastard step-child component of a service provider. In some sense, they're doomed to failure by design."

Absentee Landlords

Talk to anyone who makes a living sniffing out online fraud, and you'll hear the same story over and over. Researcher uncovers the source of a massive amount of spam, identifies an IP address that is part of a botnet or stumbles upon a phishing site that's spoofing a trusted online brand. Researcher dutifully reports the incident to the internet service provider whose network is being used, only to find the bad behavior continues unabated for days, weeks and even months.

A lack of engagement from ISPs is nothing new, but it has continued even as the malware scourge makes steady gains.

No one really knows exactly how many infected PCs are out there, but just about everyone agrees the number is high and growing. Accepting even conservative estimates that 10 percent of machines are part of a botnet means that tens of millions of systems are actively sending spam, launching denial-of-service attacks, and spewing all sorts of other malicious traffic across networks owned by the world's biggest ISPs.

According to figures from researcher Peter Gutmann, the Storm Worm alone is believed to comprise from 1m to 10m CPUs, creating one of the world's most powerful computers.

"This may be the first time that a top 10 supercomputer has been controlled not by a government or mega-corporation but by criminals," Gutmann says.

To be fair, legal liability and economic realities sometimes make it hard for ISPs to respond to the threat in a meaningful way. But in light of the surging malware problem, their frequent inaction looks more and more like complicity.

Although some ISPs are more active than others in policing their networks, absentee abuse departments and a lack of enforcement seems to be the rule. The Register spent several weeks calling ISPs large and small, including Comcast, OpimumOnline, Verizon, Earthlink and Road Runner. Many didn't bother to return our repeated calls. And all declined our requests for an interview with a member of their security team to discuss what steps they take to ensure their networks are not used as a launch pad for computer attacks.

The Worst of the Lot

The criticisms go well beyond abuse handlers who don't answer their email. According to this list from antispam organization Spamhaus, Deutsche Telekom users accounted for an estimated 2.2 percent of all compromised systems on the internet. The dubious distinction ranks the German ISP as the 11th most bot-infested provider, just narrowly edging out Verizon, which accounted for an estimated 1.97 percent. (Spamhaus Figures, which change frequently, were current as of time of writing).

Other European and US-based providers with unfavorable ratings include Telecom Italia, Comcast, Arcor, France Telecom and Road Runner, which together provided net access for an estimated 4.2 percent of the world's infected hosts.

Take a gander at other lists that track spam origins and you'll find many of the same names. According to the Trend Micro's Network reputation ranking, subscribers from Verizon, Telecom Italia, France Telecom, BT, Road Runner, Telefonica Data Espana and Tiscoli, AT&T, Cableinet Telewest Broadband and Comcast are some of the most prodigious senders of spam.

Because almost all spam is generated by bot-infected PCs, the rankings are a strong indication that those networks are home to a large number of zombies under the control of criminal gangs. Comcast and Road Runner declined to comment. Verizon turned down requests for an interview with a security engineer, but a spokeswoman said officials are aware of the rankings and are working to put new measures in place by the end of the year to curb the spam flowing out of its network. "We are concerned about it," the spokeswoman, Bobbi Hensen, said. "We don't like spam. We are aggressively working on that."

A chief cause of rampant spam is the refusal of many ISPs to block port 25, which is commonly used for traffic being sent to remote mail servers. Baldwin, of myNetWatchman.com, says his own experience with Comcast is illustrative of the problem. As a security researcher, he regularly runs malware that sends Spam over the ISP's network.

"It was very depressing because I would purposely let things run for days and I would call Comcast abuse on myself," he explained. And yet, even after telling support people he had reason to believe he himself was sending huge amounts of spam, Baldwin was told there were no issues.

Finally, Baldwin woke up one morning to find his test machines could no longer send spam through the ISP, a development he saw as "an extremely positive step for Comcast." Alas, the change didn't last. Comcast inexplicably stopped the block, leaving Baldwin's machines free to spam once more.

Into the Rubber Room

One name you won't see rise to the top of any of these lists is Cox Communications, a US-based provider with 3.5m high-speed customers. In much the same way that hospitals put deranged patients in rubber rooms to protect them from doing harm to themselves or others, Cox quarantines infected customers into environments where internet access is severely limited.

That allows the customer to download antivirus software and other applications designed to clean up their systems, but prevents them from sending spam or connecting with nefarious servers that may be trying to siphon personal information.

"When you get a customer on the phone, sure they're angry at first that they're taken off line, but once they realize that someone else was in control of their computer - pulling their social security number and credit card number off their computer - they're generally pretty grateful," says Matt Carothers, a senior security engineer for Cox. "Taking people off line seems a little harsh, but when you get down to it, you're doing it for their own good, and most customers recognize that."

In 2004, Cox put about 22,500 customers into one of these padded rooms, compared with 8,000 in 2005 and 2,000 last year. The sharp decline is largely the result of mechanisms Cox has put in place that prevent many Trojans from being able to phone home to command and control servers. Cox only disconnects customers whose infections manifest in abusive behavior.

Another ISP that takes an active role in patrolling its network is Internet Texoma. With fewer than 10,000 subscribers, the managers from the rural North Texas provider are able to dote personalized attention on their customers in a way the eludes its larger competitors.

Several weeks ago, for instance, the company received data indicating that six of its subscribers were infected with malware related to Storm Worm that was causing them to send spam and actively try to infect others. By the end of the day, managers had helped two of them to disinfect their machines. The other four were not able to be reached, so Texoma disconnected those machines.

It's all part of Texoma's zero-tolerance approach when it comes to malware. "The ISPs should do everything possible to prevent the transmission of malware through its network," says Larry Vaden, a co-founder of Texoma. "It is not good for our upstream friends to notice us. It's like having a cousin who robbed a bank. You don't want that sort of family member."

The Money Argument

Listening to Vaden wax on about the responsibility of ISPs is like stepping into a Utopian world where providers have unlimited resources to lavish on any customer who needs it. The reality is that these days most ISPs are barely eking out a profit. For many, asking them to play custodian to the malware-riddled PCs of millions of customers scattered over large geographic locations is untenable.

"They can't play traffic cop, cleanup expert and mother to people who are using their services," says Bill Stearns, a security expert who also works as an incident handler for the SANS Internet Storm Center. He says he likes the idea of ISPs collectively combating the malware menace but says such an approach is fraught with problems.

For one, about the only way to disinfect a badly contaminated machine is to reformat the hard drive and reinstall the operating system - a laborious task for those who are technically inclined that is beyond the ability of average users. Asking already-struggling ISPs to take on such a Herculean task simply isn't realistic.

One of the few other options for ISPs is to simply disconnect customers or confine them to a highly restricted environment. That is fraught with liability, since more and more customers depend on their net connections for access to emergency services and other essential services. Pulling the plug on infected machines also requires ISPs to turn away paying customers.

But those who absolve ISPs for their inaction may also be ignoring financial realities. Botnets are the single largest threat facing ISP infrastructure, according to a recently conducted survey conducted by Arbor Networks of security engineers for network operators. As such, they represent a huge liability. They translate into other substantial costs that result from lost bandwidth and ISPs getting blacklisted by other providers.

While largely defending ISPs' lack of involvement, Stearns also laments it as the loss of a key opportunity. "The frustrating part is they're one of the few places where we can put in filters and automatic detection tools to identify zombies," he says. "Part of me says if we could only get the bigger ISPs to put in blocks for certain types of malicious activity, that'd be great. The other part says, who gets to say what's malicious?"

ISPs are also uniquely positioned to provide protection to infected net users because they have the name and contact details of their customers.

Randal Vaughn, a professor of information systems at Baylor University and a specialist in tracking and shutting down sources of malware, also admits to being torn over the issue. On the one hand, he says, the magnitude of the malware problem "kind of puts the impetus into the ISPs' lap to do something."

But he quickly adds it's not that simple. "We've got tainted water going through the pipe, and we're blaming the pipe," he says. "The ISPs and the networks aren't the problem. The problem is we've got a tainted water supply. ISPs can't really be a solution. They might be able to play a part in the solution, but how are they going to pay for it?" ®

Please direct news tips, story ideas, inside scuttlebutt and other security-related intelligence to this reporter by using this link. Confidentiality assured.