White Fir Design Blog

The Internet has lots of bad information on website security floating around. In dealing with many websites that have been hacked, we see the harmful impact this has due to it leading to bad security practices and making it harder to get people to take the measures that will actually keep websites secure. Much of the bad information comes from companies providing security tools and services, whom you would expect would know what they are talking about. We looked at an example of bad security research by Cisco on Friday that lead to bad security reporting by Ars Technica and by today they have both pulled back from their claims.

Cisco has struck through most of their post and included this update:

This post’s focus relates to a malicious redirection campaign driven by unauthorized access to thousands of websites. The observation of affected hosts running Linux kernel 2.6 is anecdotal and in no way reflects a universal condition among all of the compromised websites. Accordingly, we have adjusted the title for clarity. We have not identified the initial exploit vector for the stage zero URIs. It was not our intention to conflate our anecdotal observations with the technical facts provided in the listed URIs or other demonstrable data, and the below strike through annotations reflect that. We also want to thank the community for the timely feedback.

Ars Technica has added an update to their post, included below, which doesn’t explain why they went beyond the claims in Cisco’s post or why they repeated Cisco’s claim without doing basic research that would have shown the research was highly flawed.

The Cisco blog post has been updated to change a key finding Ars reported in the following post. Contrary to Cisco’s earlier reporting, the update says not all the servers compromised in the attack were running Linux version 2.6. “We have not identified the initial exploit vector for the stage zero URIs,” the update stated. “It was not our intention to conflate our anecdotal observations with the technical facts provided in the listed URIs or other demonstrable data, and the below strike through annotations reflect that. We also want to thank the community for the timely feedback.”

Considering how colossally bad Cisco’s findings were we want to expand on how they got it so wrong, so that it might point security reporters in the direction of better vetting security research before repeating its claims in the future.

One of their key findings was that all of the websites were running an old version of the Linux kernel:

All of the affected web servers that we have examined use the Linux 2.6 kernel. Many of the affected servers are using Linux kernel versions first released in 2007 or earlier.

They then raised the possibility that this was what allowed the hack.

It is possible that attackers have identified a vulnerability on the platform and have been able to take advantage of the fact that these are older systems that may not be continuously patched by administrators.

The original title of the post, Mass Compromise of the Obsolete, also implied that the hack was related to obsolete software.

What we brought up on Friday was that not all of the websites on their list of affected websites were even running Linux, much less the Linux 2.6 kernel. Cisco’s explanation for this discrepancy is that their claim that all of the examined websites were using the Linux 2.6 kernel was anecdotal. We don’t how you can square the claim you examined the websites, but your finding was anecdotal. It seems either they didn’t look at their whole list of websites or they used a faulty tool that determined websites not running Linux were using the Linux 2.6 kernel, neither of which we would describe as being anecdotal. Asking Cisco how they determined the website were all running the Linux 2.6 kernel and what there sample set was would have been something that should have been done before journalists repeated their claims. Incorrectly identifying a set of hacked websites as having a common software version is something that we have seen repeatedly from security companies (a couple of examples), so reporters should look carefully at the evidence and probably get a second opinion on the evidence.

While their original post doesn’t spell out what versions they are referring to by the “many of the affected servers are using Linux kernel versions first released in 2007 or earlier”, a comment by one of the authors of the post says that “version 2.6.18 appeared to be particularly prevalent”. If the Cisco researchers had look into why this version was rather prevalent they should have realized they were going down the wrong path. Why would Linux 2.6.18 be rather prevalent? Well for one thing, it happens to be the Linux kernel version used by Red Hat Enterprise Linux (RHEL) 5 and it derivatives, the most prominent being CentOS 5. A little further checking would have shown them that RHEL 5 will continue to be supported for some time, so servers using the Linux 2.6 kernel would not necessarily be obsolete or insecure. This is something that Cisco should be aware of since the server powering the Cisco Blog is using RHEL 5:

Because we often see people saying otherwise, it is important to note that just because there is a newer version of software available it doesn’t mean that an older version is not safe and secure, as long as the older version continues to receive security updates.

What ultimately would have prevented this mess is if Cisco had taken the basic step of determining how the websites were hacked instead of jumping to conclusions based on data that was not reliable. Security reporters should understand that determining how a website has been hacked is an integral to dealing with them and if somebody isn’t explaining that, it should be a huge red flag that the information being given might not reliable.