GoDaddy’s Bad Response to the Drupal 7 Vulnerability

Back on October 15, Drupal released Drupal 7.32, which resolved a highly critical security vulnerability that existed in prior Drupal 7 versions. Unlike security vulnerabilities that have been fixed in recent years in Drupal and other major software, this vulnerability was easily exploitable. By the next day we were seeing attempts to exploit the vulnerability and we have been seeing a steady pickup of people contacting us about cleaning up their hacked Drupal websites, which were hit due to this. The speed and scope of the exploitation points to the need to improve how security vulnerabilities are handled in Drupal and more broadly.

Since making software that is completely free of vulnerabilities is next to impossible the best solution for this type of situation is to introduce an improved upgrade mechanism, like the one in WordPress. Starting with WordPress 3.7, security updates are automatically applied without requiring any user intervention. Checks for new versions are normally done every 12 hours and the WordPress server can instruct them to happen more frequently ahead of a planed update, so within a day most websites will have the security update applied. Not only does this protect those websites, but it makes the remaining vulnerable websites a less attractive target since the success rate of trying to exploit the vulnerability is much smaller (that doesn’t mean that they won’t get hacked though).

In the meantime, getting the word out on the need to update or take remedial as soon as possible is important to lessening the severity of security vulnerabilities. In this case Drupal recommend restoring a backup from before October 15 if you didn’t upgrade right away and every day that passes makes it harder to do that. Web hosts could play an important role in getting the word out. At least some of the largest web hosts already have the capability to detect what software and in some cases what version is in use, so they can inform impacted customer of the situation. GoDaddy has attempted to do this for the Drupal 7 vulnerability, though their implementation leaves a lot to be desired.

Today we have had a number of people contacting us saying that they had just been informed of they needed to upgrade Drupal due to a security issue. Considering that the last security update for Drupal 7 was 7.32 and that was released in the middle of October we were wondering what was causing this. It turns out that GoDaddy has just been letting people know of the vulnerability. While rather late, what was more problematic was what they said in the email.

The first big problem is that based on their email you would think that that is just occurred:

A few days ago Drupal announced a “Highly Critical Public Service announcement” that affects all Drupal users. In short, there’s a major security vulnerability that attackers can leverage against your visitors.

The Highly Critical – Public Service announcement was released back on October 29, not a few days ago. Even before that was released it was widely known that there was urgent need to update as the details of the vulnerability were disclosed with the release of the update on October 15.

The email gets more problematic from there. The beginning of the emails indicates that the required action is updating:

Action required:

Update your Drupal website

Later it says:

It’s extremely important that you update your site immediately to ensure you’re not putting your customers at risk.

At this point if you have Drupal 7 website still running a version below 7.32, that hasn’t been otherwise protected against the vulnerability, you should assume that it has been hacked, so just updating isn’t an appropriate resolution. That isn’t mentioned in the email at all, despite the public service announcement they cite being very clear about that.

Drupal Did Not Recommend This

After reading over this we were curious to see if GoDaddy was spreading this bad information on their website as well. It looks like they only got around to mentioning the issue on November 6, but at least in that case they provided accurate information. Another article linked to from that page those highly inaccurate though. Specifically the section Manually Remove Backdoors, which says:

If you do not have a backup of either your website or database (or both), you must manually remove any backdoors from your Drupal installation.

To do this for you, we offer an Expert Service for $79. With this service, we will perform all of the work for you to make our best effort to remove all backdoors using the procedures identified by Drupal. This service does not guarantee your website is free from compromise, but it is as close to compromise-free as anything can come if your Drupal installation wasn’t upgraded before the first reported compromises or restored from a backup created before Oct. 15, 2015 at 11pm UTC.

To purchase the Expert Service, contact customer support.

You can also manually remove any backdoors yourself using the Drupal-recommended procedure outlined here. This procedure is very complicated and requires an advanced understanding of the technologies Drupal uses (PHP, MySQL) to use effectively. Not all steps listed in the procedure are applicable to shared hosting environments, but completing what you can from this list will provide you the greatest likelihood of removing backdoors from your site.

If you follow the link “the procedures identified by Drupal”, you will find that it isn’t actual something from Drupal. Later GoDaddy links to the same page and describes it as the “Drupal-recommended procedure”, but if you actually look at the page it says “The official recommendation is: Restore from pre-October-15-backups.”. You really have to wonder about a company trying to sell you on an “Expert Service”, for which they don’t have the expertise to actually understand a basic, that is they they are citing something that isn’t actually procedures identified by Drupal or recommend by them.

Drupal 7.32 Usage Reached 24 Percent in Second Week

With a highly critical vulnerability found Drupal 7 versions prior to 7.32 and with Drupal providing data on usage of various versions, a good opportunity to see at what speed websites are being updated when there is a pressing need to do so is available now. Last week we found that in the first week only 12.46% of Drupal 7 websites had been updated to Drupal 7.32. In the second week it has now increased to 23.77%. We should note that it is possible to patch the vulnerability without doing a full upgrade, so not all websites that have not been upgraded are necessarily vulnerable at this point.

drupal-7-usage

Drupal 7.32 usage has now surpassed to 7.31 usage, which was also a security update. What is rather troubling is that 59.28% of Drupal 7 websites haven’t been updated to at least 7.31 despite it being released nearly three months ago.

For those in need of an upgrade we provide one time Drupal upgrades and ongoing upgrades for Drupal 7 (with security updates in a day). You can use our Drupal Version Check chrome extension and Up to Date? Chrome app to keep track of the update status of Drupal websites you manage.

Exploit Attempts of Drupal 7 Vulnerability Are Reminder That Hiding Software Versions in Use Isn’t a Security Measure

When it comes to securing website there is lots of bad advice out there, much of it coming from people that claim to have security expertise. A prime example of this bad advice is the claim that hiding version of software in use on the website will somehow protect you being hacked. While there a number of reason it won’t protect you, the central issue is that most hackers won’t bother checking if you are using software, much less what version of the software is in use. Instead they will simply try to exploit the vulnerability without checking anything first. That means that no matter how hard you try to hide the version information it won’t protect you if you are running a vulnerable version. Seeing as people continue to believe and tell others that hiding versions information is a security measures we thought it would be a good time show a real world example of this in action.

Recently a highly critical vulnerability was found in Drupal 7 versions below 7.32 and shortly afterward attempts to exploit it were happening. Below is the series of requests from one of those attempts that occurred the day after the vulnerability was fixed:

62.76.191.119 – – [16/Oct/2014:21:03:25 -0400] “POST /?q=node&destination=node HTTP/1.1” 200 4929 “-” “Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0”
62.76.191.119 – – [16/Oct/2014:21:03:26 -0400] “POST /user HTTP/1.1” 500 3566 “-” “Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0”
62.76.191.119 – – [16/Oct/2014:21:03:27 -0400] “GET /?q=ocyuys HTTP/1.1” 404 6035 “-” “Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0”
62.76.191.119 – – [16/Oct/2014:21:03:27 -0400] “GET /ocyuys HTTP/1.1” 404 6035 “-” “Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0”
62.76.191.119 – – [16/Oct/2014:21:03:28 -0400] “GET /modules/profile/mhtd.php HTTP/1.1” 404 6035 “-” “Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0”

The IP address the requests came from, 62.76.191.119, appears to have been used for a widespread attempt to exploit the vulnerability.

Looking at the requests the first thing that sticks out is what isn’t requested. If you were going to check what version of Drupal is in use the first thing to request would be the file CHANGELOG.txt, which on Drupal websites will contain the Drupal version in use, if the file hasn’t been deleted. Since our website is running Drupal and we haven’t deleted the file the hacker have seen that we were running version 7.32 and therefore were not vulnerable. If the CHANGELOG.txt file doesn’t exist then you could check other files to get some idea of what version is in use. In this case the vulnerability only exists in Drupal 7, so a hacker might check for a file that exists in that version and not in Drupal 6.

Instead of doing any checks first, the hacker first request appears to be an attempt to exploit the vulnerability. The log of requests doesn’t include the POST data, so we don’t exactly what was done, but it was likely something similar to the POC #1 listed here. The rest of the requests seem to be checking if the first request was successful.

Beyond this being a reminder that hiding software is not a security measure, it is an important reminder that you need to keep the software running up to date. Something people running Drupal 7 websites have not been good at doing. It also highlight the fact that many people dealing with security have little understanding of it and in many cases are not doing the work they should do. Determining how a website has been hacked is one of the key things to properly cleaning up a hacked website, yet we see over and over we are hired to re-clean hacked websites that person hired before hadn’t done that. If people were doing that then they would know that hackers are not checking for the versions of software in use and therefore wouldn’t be telling people that hiding software versions is security measure.

Only 12 Percent of Drupal 7 Websites Upgraded to 7.32 In First Week

One of the basic measures needed to keep websites secure is to keep the software on them up to date, as that prevents them from being hacked due to a known vulnerability in the software. That unfortunately isn’t something that people are very good at doing. With Drupal you can get fairly good data on how well people are at keeping up to date due to the developers publicly releasing data on what version are in use. When we looked at the data in late March we found that only 33 percent of Drupal 7 websites were up to date and only 72 percent of Drupal 7 websites were up to date or less than year out of date.

With a highly critical vulnerability fixed in Drupal 7.32, which hackers started exploiting shortly after the fix being released, we are curious see how fast websites are being updated. The first week of data, for the week of October 19, has now been released and it shows that only 12 percent of websites have been upgraded to 7.32. We should note that it is possible to patch the vulnerability without doing a full upgrade, so not all websites that have not been upgraded are necessarily vulnerable at this point.

That number is better than with a previous, less severe, security update. We previously look at the rate that Drupal 7 was updated to 7.27, which fixed a moderately critical vulnerability. In that case after a week only about 5 percent had been upgraded and it was only up to about 15 percent at two weeks.

What is more troubling is how few websites have been upgraded to the previous release, Drupal 7.31, which fixed a moderately critical vulnerability. That version was released on August 6, so there has been plenty of time for an upgrade. As shown in the chart below, at this point 62 percent of Drupal 7 websites are still running a version below that:

Drupal 7.32: 12.46%, Drupal 7.31: 25.74%, Drupal 7.30 or older: 61.80%

For those in need of an upgrade we provide one time Drupal upgrades and ongoing upgrades for Drupal 7 (with security updates in a day). You can use our Drupal Version Check chrome extension and Up to Date? Chrome app to keep track of the update status of Drupal websites you manage.

Drupal Websites Not Receiving Security Updates in a Timely Manner

At the end of March we took a look at Drupal’s usage statistics and found that two months after new versions of Drupal 6 & 7, which included security updates, were released only 33 percent of Drupal 7 websites were running the latest version and only 19 percent of Drupal 6 websites were running the latest version. That obviously isn’t what you would like to see if you care about security.

It has now been two months since another set of security updates, 6.31 and 7.27, have been released. The percentage of websites that have updated to at least those versions isn’t much different from what we saw with the last set of updates. 29 percent of Drupal 7 websites are running at least 7.27 and 17 percent of Drupal 6 websites are running 6.31.

For those interested we have graphed the percentage of websites that have been upgraded over time:

Drupal 7 Update Pace Graph

drupal-6-update-pace-graph

For both Drupal 6 & 7 the graphs show that during the first two weeks after a new version is released there is pretty quick uptake and then it slows down.

With drupal.org still running 7.27 a month after 7.28 was released that might indicate that the upgrade process could be improved:

drupal.org is Running Drupal 7.27

 

With our Up to Date? Chrome app you can keep track of the Drupal versions (as well other web apps) on all of the websites you manage in one place, so you can easily check if they are in need of an upgrade.

 

Another Major University is Running Outdated and Insecure Version of Drupal

Last week we spotlighted the fact that only a third of websites running Drupal 7 are up to date. As keeping the software running a website up to date being an important security measure and with the most recent version of Drupal 7 being a security update that obviously is a problem (though certainly not a problem limited to Drupal). What makes this more troubling is that it isn’t just small websites that are not keeping their software up to date, but large institutions that are more than capable of doing the upgrades. In gets worse when you see institutions that have departments focused on the technology security that are failing to keep their software up to date. Last month we looked at the fact that the University of Cambridge was running an outdated version of Drupal, while the blog of their Security Group was running on a very out of date version of WordPress. They unfortunately are not alone.

Using our Drupal Version Check web browser extension, available for Firefox and Chrome, we can see that the Rutgers University website is still running Drupal 7.21:

The Rutgers University Website is Running Drupal Version 7.21That version is now a year out of date and two security updates have been missed (7.24 and 7.26). Making sure the website is kept up to date is something that you would hope that Rutger’s University Information Protection and Security Division would be on top of, but they are not even keeping their website up to date:

The RU Secure Website is Running Drupal Version 6.29That website is less out of date than the main Rutgers website as the current version of Drupal 6, 6.30, was released in January, but it was a security update so they should have gotten it upgraded by now.

For those reading this and realizing they need to get their Drupal installation up to date, you can find the upgrade instructions here.

Only One-Third of Drupal 7 Websites Are Up-To-Date

Earlier this month we looked at some data from our tools showing that large percentages of Joomla, WordPress, and MediaWiki websites checked with them were running outdated versions of the software. For Drupal, there is much more comprehensive set of data publicly available that comes from the Update status/Update manger module. To get a better idea of how well webmasters are at making sure Drupal websites are being kept up to date we have analyzed the data reported for March 16, 2014, which has data on over a million websites. Making sure the software running websites is a basic security measure and when they are not it can lead to them being hacked if the vulnerability can be used for that (as we have been seeing recently with a vulnerability in older versions of Joomla).

At this point a large majority of the websites, 79 percent, using Drupal are using version 7. Of those only 33 percent are running the latest version, 7.26. This is troubling as this version was a security update, so websites running older versions are potentially vulnerable to being hacked. This version was released on January 15, so even websites that need extensive testing before apply an upgrade should have been updated by now. Looking beyond that, 72 percent of the websites are either up to date or less than a year out of date so the majority of websites are probably getting updated, if somewhat infrequently.

Drupal 7 Version Freshness: Up To Date 32.75%, Less Than 1 Year Out of Date 39.72%, More Than 1 Year Out of Date 22.53%, More Than 2 Years Out of Date 5.00%

For Drupal 6 the situation is worse. The latest version of Drupal 6, 6.30, was released alongside of 7.26 on January 15, but so far only 19 percent of websites have been updated to that version. The situation in terms of somewhat recent updated websites is also worse, with only 64 percent of website being up to date or less than a year out of date. 20 percent are at least two years out of date, which means they have missed at least four security updates.

Drupal 6 Version Freshness: Up To Date 18.62%, Less Than 1 Year Out of Date 45.06%, More Than a Year Out of Date 16.59%, More Than 2 Years Out of Date 11.34%, More Than 3 Years Out of Date 6.46%, More Than 4 Years Out of Date 1.90%, More Than 5 Years Out of Date 0.02%

To make it easier to check for Drupal websites in need of an update we have made the web browser extension Drupal Version Check, available for Firefox and Chrome, which in most cases will identify what version of Drupal is in use and in others indicate if the website is using an outdated version of Drupal.

If you are in need of a Drupal upgrade we can do that for you or we can also handle upgrades on an ongoing basis, so you don’t have to worry about taking care of this.

Another ING US Website Running Outdated and Insecure Version of Drupal

Yesterday, as part our series of posts highlighting the fact that even high profile websites are not taking the basic security measure of keeping the software running them up to date, we highlighted the fact that ING US was using outdated and insecure versions of Drupal on their website. Today we have a few quick follow-ups.

First it was brought to our attention that the fact that ING was using Drupal was a big enough deal for the creator of Drupal to highlight it, saying in part

You know when a piece of software is mature when it starts being adopted by financial services organizations.

The fact that such high profile user isn’t keeping Drupal up to date in light of the security need of doing so either means that that Drupal is too hard to keep up to date, which we strongly disagree with based on keeping our own installation up to date and handling plenty of upgrades for clients, or there is more general problem with security practices for websites.

In the aforementioned post another ING US website was highlight as running Drupal and that website unfortunately has also not been kept up to date:

The ING Global Perspective Website is Running a Drupal Version 6.22That version is over two years and they have failed to apply five security updates (6.23, 6.27, 6.28, 6.29, and 6.30).

At the bottom on that website is a link to a Web Site Security page, which in part advises keeping the software on your computer update:

Take care of your computer

  • Update your computer by installing the latest software and patches to prevent hackers or viruses from exploiting any known weaknesses in your computer.

It would great if ING, as well as everyone else running a website, took that advice and applied it to their websites.

 

ING US and Voya Financial Websites Running Outdated and Insecure Versions of Drupal

When it comes to keeping websites secure one of the basic things that needs to be done is to keep the software running the website up to date. This prevents the website from being exploited through a known vulnerability in old versions of the software that has been fixed in a subsequent release. We know that many websites are not doing this, which is troubling, but what is more troubling is that the major institutions are not even doing this with their websites. Last week we looked at major security software provider not doing it and if you go back in this blog, you can find other examples. Today let’s look at example of a major financial institution in the same boat. ING US, which in the process of rebranding as Voya Financial, reports having $511 billion of assets under management and administration and serving approximately 13 million customers. They use Drupal for main portion of the ING US website. Using our Drupal Version Check web browser extension, available for Firefox and Chrome, you can check if it is up to date:

The ING US Website is Running a Drupal Version Below 6.28You can see that they are not. With a little further checking we were able to determine they are using Drupal 6.19. That means they haven’t updated the software in over three years and they have failed to apply five six security updates (6.21, 6.23, 6.27, 6.28, 6.29, and 6.30). It is important to note that account access portions of their website are separate from the main website, so they are not directly impacted by this lax security. Though it does raise the question of how well they secure the other portions of their website if they are not doing something this basic. Also, if someone could exploit one of the vulnerabilities in the version of Drupal on the main website they could change the links directing people to the account access portion of the website to another location and use that to gather login credentials.

It isn’t just the ING US website that has an out of date version of Drupal in use. The website for their new name, Voya Financial, also is using an outdated Drupal version:

The Voya Financial Website is Running a Drupal Version Below 7.25With a little further checking we were able to determine they are using a version no newer than Drupal 7.21. That means that they haven’t updated the software in nearly a year and they have missed at least two security updates (7.24 and 7.26).

Tech News Websites Not Taking Basic Security Measure With Their Websites

When it comes to improving the security websites one of the biggest problems we see is that there is so much bad information available on the Internet, especially the information coming from companies trying to sell security products and services. We would hope that news organizations would provide the public with a source for better information, but most of the security reporting we see in technology news websites is just as bad as anywhere else. Their lack of security knowledge also impacts their own websites as we see that they are not taking basic security measures with their websites and therefore leaving them vulnerable.

We found three prominent technology news websites that are running very out of date versions of the Drupal software. Keeping software up to date on a website prevents known vulnerability being exploited and we have found that when vulnerabilities in website software are exploited it almost always due to a vulnerability that has already been patched in a newer release of the software.

ITworld

ITworld is Running Drupal 6.19ITworld is running a version of Drupal that is nearly three years out of date – the next version was release in December of 2010 – and they have missed three security releases.

InfoWorld

InfoWorld is Running Drupal 6.16InfoWorld is running a version of Drupal that is nearly three and half years out of date – the next version was release in June of 2010 – and they have missed four security releases.

Network WorldNetwork World is Running Drupal 5.14

Network World is in much worse shape than the other two organizations as they are using Drupal 5, for which support ended back at the beginning of 2011. They haven’t even bothered to at least make sure they are running the most recent version of Drupal 5. In fact they haven’t updated it in over four and half years – the next version was released in January of 2009 – and they missed the last nine security releases for Drupal 5.