Tag: Tolly Group

When it comes to cyber security, it has been clear to us for some time that most of the companies in the field don’t really care about security. Just yesterday we discussed a cyber security company that doesn’t even bother to keep the software running their websites up to date, despite that being a really basic security measure (that is far from the first time we have spotted that type of situation either). One of the areas where we see this lack of care about security is shown by the fact that security companies services and products often are focused not on things that would actual prevent systems from being hacked in the first place, but on detecting the system has been hacked after the fact.

That brings us to a recent post on the web security company SiteLock’s blog. The post uses the results of a test they recently had done by the Tolly Group to argue their product is better at protecting against threats to website than another product of a different type. As we discussed last week the test was, at best, quite poor, but might be accurately describe as being rigged. The test involved testing if their product and another product could detect malicious code on a website and SiteLock not only had access to the samples being tested, but provided the sample code that was tested. Not surprisingly they were able to detect 100 percent of it (the developer of the other product wasn’t provided the sample code). To make things even ridiculous they then promoted the testing as having been independent, despite the fact that they even provided the samples to be tested.

First off, the post really could have used some editing, as it has some quite bad statements such as one in this paragraph:

In recent years, though, informal blogging environments, such as WordPress, have blossomed into full-blown web application platforms. Commercial and community developers contribute blocks of codes, known as “plugins” to enable just about any type of functionality that you can imagine. (A Google search on “WordPress Plugins” shows over 11 million hits.)

If you want to measure how many WordPress plugins there are, you could look at the homepage of the official Plugin Directory, where most WordPress plugins are made available, as that provides a count of plugins available through that, currently 47,146. If SiteLock was as familiar with WordPress as they promote themselves, they should have known that.

Explaining the basis of the test you can see what is so wrong with the view that SiteLock appears to agree with:

The basis of the test was the assertion that traditional endpoint security solutions are not designed to detect web application threats and, therefore, would have a low detection rate when scanning for such threats.

The actual threats against web applications would be vulnerabilites in the software, not malicious code that can be added by exploiting those. But the testing instead looked at the end results of threats being exploited:

A corpus of nearly 3,000 web-based malware samples, defined by SiteLock, was run through a prominent “traditional” endpoint security solution to illustrate SiteLock’s point.

The conclusion on the post is:

The point, really, is not the absolute percentage of malware detected. The point is to illustrate that there is an entirely new set of threats “out there” that traditional endpoint solutions have not been designed to detect. And, those new threats clearly require an additional, “next-gen” endpoint security solution in place to provide protection.

The reality from dealing with many hacked websites that many of those could have been prevented by taking basic security measures and many other could have prevent if other security practices were improved. From what we have seen of automated methods for trying to detect and clean malicious code, they produce poor results. Also, websites don’t just get hacked to place malicious code on them, so leaving a website vulnerable and trying to detect malicious code added to it, would among other things, allow for the possibility of sensitive data being extracted from it on a repeated basis.

While the post was written by the found of the Tolly Group, it isn’t just a situation that SiteLock posted someone else’s words with this very wrong view on the security, our past experience has shown that SiteLock view is in line with this. For example we have found that they label websites as being secure when they are using outdated software with known vulnerabilites and they don’t make sure that the software on a website is upgraded when they are cleaning up after a hack.

Recently for several different reasons we have taken a look at claims made about quite a few web security services and products and noticed that there are often bold claims that seem highly unlikely to be true. There also is a decided lack of evidence put forward to back those claims, much less independent third-party evidence to back them. So we were surprised today to see a press release from the web security company SiteLock, Independent Testing Shows SiteLock Web-Based Malware Protection Outperforms Traditional Endpoint Solutions. Considering their track record, we wouldn’t have expected them to be the ones actually having independent testing done. After taking a quick look over the details of the testing, we found that, not surprisingly, the testing was actually the opposite of independent.

The test compared their ability to detect malicious code on websites against McAfee Complete Endpoint Protection ability to do that. It appears the testing was intended to show that products of the type that McAfee have are not well suited to doing this, but since there is not a comparison of multiple products in each category, the value that could be gleamed from the test is limited.

One finding of the test did stick out to us though:

SiteLock SMART provided 100 percent PHP and JavaScript coverage versus less than 6 percent coverage by McAfee

Considering the poor quality we have seen with SiteLock’s detection in the past, whether it be missing malicious code or falsely identifying non-malicious code as malicious, we had a hard time believing that that they would possibly detect 100 percent of the tested malicious code in a reasonable test.

A key question to determine the quality of the testing was how the malicious code being used in the test was picked. When the items tested are mentioned in the press release there source isn’t mentioned either time, here:

Testing a group of nearly 3,000 web-based malware examples to determine effectiveness

and here:

Tests showed that the SiteLock solution detected and cleaned 100 percent of the samples provided.

Looking at the complete report (PDF) by the Tolly Group, there is the answer:

The test corpus was provided by SiteLock and consisted of 2,972 samples that included PHP, JavaScript and other executables that contained a variety of attacks.

So SiteLock provided the items that would be tested against. The test clearly isn’t independent at that point and the real story would have been if SiteLock didn’t detect them all.

The complete report also makes no claim to the testing being independent.

While SiteLock knew ahead of time what would be tested, since they provided the items being tested, the company behind the other product was not allowed to see them:

In accordance with Tolly’s Fair Testing Charter, Tolly personnel invited representatives from McAfee/Intel to participate in the testing. Because of internal guidelines, McAfee/Intel was unable to review the test corpus in detail and had no further comment.

Based on that you wonder what else might be in their internal guidelines that could skew the test results.