Category Archives: Website Security

Ars Technica has been running a series of articles about the recent hack of several US Federal Trade Commission websites hosted at Media Temple. The latest reporting indicates that “critical vulnerability in some versions of Parallels’ Plesk Panel control panel software appears to have been key to the recent penetration”. A patch for the vulnerability was released in September, though the article mentions that Parallels didn’t send email alerting customers to the critical nature of the vulnerability until February. Media Temple’s failure to keep Plesk up to date goes back years and we unsuccessfully tried to get them to address the issue back in July of 2010.

Media Temple and Parallels are both part of the Hosting Security Forum, which is supposed to “share critical security information in order to protect the integrity of a customer’s data, their web presence, and online availability”. That organization’s website is running an outdated version of WordPress, 3.2.1, which might be a good indication as to the group’s level of dedication to security. Media Temple, as well as Dreamhost and other members of that organization, is also running outdated software on their website.

Warning Media Temple

Back in July 2010, during a period of hacks that were targeting Media Temple customers (but that Media Temple claimed was not due to their security failings), Media Temple  made some long needed security improvements and asked for people to contact them if they were missing any security measures.

While cleaning up a hacked website running on their Dedicated-Virtual service we noticed that Media Temple was using a nearly two year old version of Plesk, which also meant that the other software that comes with Plesk was also two years old. We contacted Media Temple alert them to the need to keep that and other software running on their systems up to date and at the time we hoped that they would quickly resolve the issue as they were publicly claiming to want to improve their security.

Media Temple’s response was that there hadn’t been a known vulnerability in Plesk since 2007 and therefore they were secure. We don’t why they felt they didn’t need to keep their up to date just because there had not been known vulnerabilities, but in any case it wasn’t the whole truth. While the version of Plesk didn’t have any known vulnerabilities other software that came with that version of Plesk did have known vulnerabilities. We specifically brought to their attention there were apparently security issues in at least the versions of ProFTPd, Ruby, and phpMyAdmin that came with it. We never received any response after we brought that up. Overall, their response seemed to be more focused on creating the impression that they cared about security then about actually making sure they and their customers were secure.

During the email exchange they claimed they had recently put in place a policy that “requires us to patch any software with security flaws within 30 days of a patch being made available.  For the most critical issues, such as a kernel exploit, we will patch immediately. ” We certainly would describe the vulnerability in Plesk as being a critical issue, but for some reason they didn’t feel the need to apply the patch immediately or even in the 30 day window.

Blame the Customer

When the issue of the hack was first surfacing Media Temple was quick to blame their customer for the hack and criticize them for running outdated software. While it’s true that many hacks are due to issues which the customer is responsible for and many of those are due to outdated software, it is irresponsible to claim that it was the customer’s fault without actual evidence to support that, especially to do that publicly. For Media Temple’s to do this is worse as during our email exchange they excused not keeping Plesk up to date on there not being known vulnerabilities, so they certainly should understand that just running outdated software doesn’t mean that it is vulnerable. To be criticizing a customer for running outdated when they don’t keep the software they are responsible for up to date makes the response appalling.

Unfortunately, Media Temple’s response to this incident isn’t out of line with the usual response that customer’ with hacking issues receive when contacting their web host about a hacking issue.

Media Temple Runs Outdated Web Software

Based on the rest of Media Temple’s actions it isn’t surprising that they fail to keep software running on their website up to date (while criticizing others for doing the same).

Their blog is running WordPress 3.3. The latest version, 3.3.1, was release nearly two months ago and included a “fix for a cross-site scripting vulnerability that affected version 3.3“.

The Media Temple Community Wiki is running a version of MediaWiki, 1.16.x, that hasn’t been supported for nearly three months. They also failed to apply the last three updates, all of which included security fixes, for 1.16.x. The oldest update they failed to apply, 1.16.3, was release over ten months ago.

You can get alerts for outdated web software, like the one the ones in the screenshots above, with Meta Generator Version Check extension (available for Firefox and Chrome).

We don’t believe, as others do, that web host’s have a responsibility to clean up their customer’s website if it’s are hacked, unless it was caused by something on the host’s end. They do have responsibility to properly handle the security of their own systems, which as of just a few weeks ago Dreamhost didn’t do in a basic way. We also think they have a responsibility to not be giving bad security advice to their customers or the public at large, as Dreamhost does. It does a serious disservice to their customers, including the potential that customer’s hacked websites could remain vulnerable to being re-hacked because of their bad advice. The bad security advice, which certainly isn’t limited to web hosts, also makes it harder for people that deal with security issues, and take security seriously, to do their job. We have to spend to much time explaining to clients that the information they are seeing and being told is in most cases wrong. What makes Dreamhost worth focusing on is that in addition to the bad advice they give, they don’t bother following the good advice they give others.

In this post will focus mostly on comments they made in a recent post on their blog and some communication between them and one of their customers, whose website we recently cleaned.

From the blog post:

Based on the sites we have cleaned up already, these attacks have almost universally been due to insecure website software running on the site in question. You could have the best passwords in the world, but if the apps you’ve installed on your server have any security vulnerabilities or aren’t kept up to date, attackers can still find their way in.

Based on the websites we clean up and other data we collect the claim that attacks are “almost universally been due to insecure website software running on the site in question” is completely false. Vulnerabilities in web software are one of the top two things, but the other, compromised FTP credentials, has nothing to do with the software running on the website. From what we see, which one of issues is the biggest changes from time to time. For example, we might see a spike in web software attacks after a new vulnerability is discovered. There are other things that are lesser but not insignificant causes of attacks, like security vulnerabilities at web hosts.

What could explain the discrepancy between what we and Dreamhost see? It could be that they have systems in place that are effective at preventing other types of hacks. It could be that they only look for certain types of hacks, so they only find what they are looking for. Or it could be that don’t actual bother looking at the cause of attacks at all.

Determining if Outdated Web Software is the Cause of a Hack

Recently we were cleaning up a hacked website hosted at Dreamhost. From reviewing the information we had available it didn’t look the hack came through a security vulnerability in web software. Unfortunately, we couldn’t review the necessary logs to be able to determine how the hack occurred because of poor log retention at Dreamhost.

To determine if the website was hacked due to a vulnerability in web software, we would have reviewed the log of HTTP activity. That show all the requests made to the website, so it would show any exploit of software running on the website. Unfortunately, those are only stored for a very short time at Dreamhost. If Dreamhost actually wanted to determine that websites were in fact being hacked by vulnerabilities in web software they would also need to look at that as well. Except for cases where the hack is found immediately, how could they make a strong determination that the hack was due to a vulnerability in web software? The answer seems to be they don’t even try, as will get to in a little bit.

The other thing we wanted to look at was the log of FTP activity. If this is available for the time period of the hack it is easy to see whether the hack came through FTP by reviewing the log. Our client was told that the logs are only stored for two weeks and because the hack occurred before then the log wasn’t helpful. The short retention of the FTP log also means that Dreamhost couldn’t check if the hack came through FTP if it occurred more than two weeks before, which would limit their ability to know that hacks are not coming through FTP as they seem to think is the case. This also makes another claim in the post seem questionable. In regards to a recent hack of Dreamhost they stated that “An extensive investigation has revealed that no customer FTP or SSH user accounts have been maliciously accessed due to this password breach.” If you don’t keep logs longer than two weeks how could you claim to do an extensive investigation, as you can’t look at data older than two weeks once you discovered the issue?

While we couldn’t make a strong determination on the source of the hack, Dreamhost didn’t have a problem doing that. They said:

Also, “hacks” usually occur due to outdated web software, and since you’re running joomla, it’s safe to say, the hack occurred there.

There hasn’t been the kind of vulnerability in Joomla that has led to the software being hacked in a long time. That isn’t to say that something couldn’t be found, but if somebody is telling you that a hack likely occurred from that or that Joomla is not secure then they don’t know what they are talking about. From the hacked websites that we see that are running Joomla, the issues that lead to hacks are vulnerable extensions, compromised FTP credentials, a hack of the hosting provider, and recently we have had a number of clients hit by a dictionary attack campaign that is targeting Joomla websites. It clearly isn’t safe to say the hack occurred because they were running Joomla, as Dreamhost, unless they have done a thorough investigation and found evidence to back that up.

Let’s say you did believe that the cause of a hack was due to a Joomla vulnerability, there would be more work to do after that. You would need to determine if the vulnerability has been fixed in Joomla before you tell somebody “simply reinstall your software, and make sure you’re always keeping it up to date”, as Dreamhost did, because doing that would have no impact if the vulnerability hasn’t been fixed in Joomla.

If the hack is on a website was running the latest version of Joomla it would mean that the vulnerability hasn’t been fixed or more likely that it wasn’t a vulnerability in Joomla.

For a website running an older version you would need to compare what information you had about the hack to the vulnerabilities that have been fixed in subsequent versions to see if the probable vulnerability has been fixed. For example, there was recently fixed vulnerability that could make it easier for some to guess a user’s password after causing it to be reset. To check for that you would ask the client if the password stopped working recently, which would have happened if it was reset. If the logs were available, you could also check those for additional information. If you don’t find a corresponding vulnerability it would mean that the vulnerability hasn’t been fixed or more likely that it wasn’t a vulnerability in Joomla that caused the hack.

Dreamhost didn’t inquire into what version of Joomla was running when the website was hacked or doing anything else, that is another good indication that don’t know what they are doing or they just don’t care. If this customer interaction is any indication of how they handle this type of situation for other customers, it doesn’t appear they do any checking at all and instead just assume a hack must have been caused by software on the website.

Failure to Warn About Vulnerable Software

When it comes to dealing with outdated web software that is actually vulnerable to being exploited, and has been exploited, Dreamhost fails their customers. Dreamhost has tool that scans a customer’s files to find the web software in the account and warn if it is out of date. As of a couple days ago it was listing OpenX 2.8.7 as being “Up-to-date.”, despite the fact that OpenX 2.8.8 was released back in November. OpenX 2.8.7 isn’t just out of date, it contains a serious security vulnerability that was being exploited even before version 2.8.8 was released. You have wonder how they wouldn’t be aware of this if they were actually determining how websites were being hacked or just keeping track of security issues in web software.

Dreamhost Runs Outdated Web Software

In another post by the same author they recommended “upgrading website software as soon as there is an update available”, which is actually good advice. It just isn’t advice that Dreamhost follows.

If you visit the Dreamhost blog with our Meta Generator Version Check extension (available for Firefox and Chrome) you would get an alert the blog is running an outdated version of WordPress, 3.2.1. That version is not that far out of date, it’s only one major version out of date and two months out of date. (Also, contrary to recent claims by Websense there are not known vulnerabilities in 3.2.1.) But considering that Dreamhost is telling people how important it is to keep software up to date and they don’t do it themselves then it seems like Dreamhost either does not care about the security of their website or they don’t believe what they are saying.

What make this more ridiculous is that the person that wrote both of those posts, who is described as Dreamhost’s Web Application and Security administrator, will be giving a talk  titled “Security 101″ at a WordPress event.  Should you be telling other people about WordPress security if you don’t even bother to take the basic step of keeping WordPress up to date?

It’s not just that blog that is running outdated software, the Dreamhost Wiki is running an outdated version of MediaWiki, 1.16.5. Support ended for that major release, 1.16.x, two and half months ago. Like the WordPress installation that is not far out of date, but for whatever reason Dreamhost doesn’t feel it is important to keep the software running on their website up to date.