At the end of March we took a look at Drupal’s usage statistics and found that two months after new versions of Drupal 6 & 7, which included security updates, were released only 33 percent of Drupal 7 websites were running the latest version and only 19 percent of Drupal 6 websites were running the latest version. That obviously isn’t what you would like to see if you care about security.
It has now been two months since another set of security updates, 6.31 and 7.27, have been released. The percentage of websites that have updated to at least those versions isn’t much different from what we saw with the last set of updates. 29 percent of Drupal 7 websites are running at least 7.27 and 17 percent of Drupal 6 websites are running 6.31.
For those interested we have graphed the percentage of websites that have been upgraded over time:
For both Drupal 6 & 7 the graphs show that during the first two weeks after a new version is released there is pretty quick uptake and then it slows down.
With drupal.org still running 7.27 a month after 7.28 was released that might indicate that the upgrade process could be improved:
With our Up to Date? Chrome app you can keep track of the Drupal versions (as well other web apps) on all of the websites you manage in one place, so you can easily check if they are in need of an upgrade.
When it comes to IT security companies, what we see over and over is that they have little to no concern for security (and also often have little to no understanding of proper security practices). So it isn’t surprising that despite billions being spent on IT security, IT security continues to be in such poor shape. This leads to situation like the massive breach of Target’s systems last year. While that was big news, what didn’t get much attention was the company who declared Target compliant with standards for handling credit card transactions shortly before the breach, Trustwave. Trustwave has a history of declaring companies compliant shortly before they suffer major breaches and for being lax in their assessments.
We recently spotted another example of their highly questionable practices of Trustwave. We were contacted about doing a migration of a Joomla-based website still running version 1.5, for which support ended in September 2012. While taking a look at the website, we noticed a seal for Trustwave Trusted Commerce:
Considering that the website is running software that is no longer supported and therefore cannot be considered secure, we were curious to see if Trustwave was claiming it was secure. It would be quite easy for them to find that the website is running Joomla 1.5 if they wanted to as the source code of every page on the website the following line is included:
<meta name=”generator” content=”Joomla! 1.5 – Open Source Content Management” />
If you click on the seal you get this page:
At the top of the page Trustwave proclaims that “Your credit card and identity information are secure.”, which they shouldn’t be saying for a website that is running unsupported software.
As we looked closer we noticed the small text disclaimer at the bottom of the page were they say “Trustwave Holdings, Inc. makes no representation or warranty as to whether [redacted] systems are secure from either an internal or external attack or whether cardholder data is at risk of being compromised.”. So they are basically telling you that despite saying “your credit card and identity information are secure”, there not actually saying that at all.
It is highly inappropriate for them to mislead the public like they are doing with this seal, but unfortunately our experience is that this kind of thing is considered acceptable in the security industry.
Two years ago we created a plugin that list any installed WordPress plugins that have been removed from the WordPress.org Plugin Directory. There are a number of reasons that plugins are removed, the most important being when it has a security vulnerability. For that reason WordPress should alert when a plugin has been removed, until that occurs our plugin can be used to check if any installed plugins have been removed.
Recently it was suggested that the plugin also list plugins that have not been updated by their developers for over two years as well. On the Plugin Directory website, plugins that have not been updated in over two years have a banner at the top of the page that states “This plugin hasn’t been updated in over 2 years. It may no longer be maintained or supported and may have compatibility issues when used with more recent versions of WordPress.”. Even plugins that have not needed any changes should have been updated periodically to indicate that they are compatible with new versions of WordPress. For the reasons listed in the banner it would be helpful to those already running the plugin to be aware the situation as well.
One of things we looked at before deciding to add a listing of plugins that had not been updated in over two years was how many plugins fall in to this category. We first checked a small sample and found that many of the plugins fell in this category. When we looked at all of the plugins we found that was still the case. As of yesterday a total of 12,703 plugins had not been updated in over two years. That is 30 percent of all the plugins that have entries in the Subversion Repository for the Plugin Directory. Not all plugin entries have actually been used, so the percentage of plugins that people could being using is probably higher.
Below we have charted what year these plugins were last updated. The number for 2012 is lower as plugins last updated after March 9 of 2012 would still under two years out of date.
Today we released a new version of the plugin that lists installed plugins that have not been updated in over two years. If you want to check if you are using any plugins that have not been updated in over two years install our No Longer in Directory plugin (available through the Add new page in the WordPress admin area) and then in the admin area go to the plugin’s page in the Plugins menu. The page will list any installed plugins that have been removed for the directory first and then any plugins that have not been updated in over two years.
Recently we have had a lot of blog posts highlighting major organizations running outdated and insecure versions of Drupal, but we don’t want to give the impression that it is only with Drupal based websites that major organizations are failing to keep the software up to date on. So we wanted to find an example of a website running Joomla to highlight as well and we quickly found a very concerning example. The third website listed on Joomla’s showcase of websites running Joomla is the website of Guaranty Trust bank, which is Nigeria’s largest bank and has assets of over 12 billion USD. As you can see with our Joomla Version Check web browser extension, available for Firefox and Chrome, their websites is running a fairly out of date version of Joomla:
That version is over two years out of date and there have been twelve subsequent updates with security fixes. One of the security vulnerabilities fixed in a subsequent version is of particular concern. The vulnerability, which we discussed before, allows a new user account to be created with “Administrator” privileges through privilege escalation. If user registration is disabled this will not work, but in this case it does appear that user registration is enabled. It is important to note that account access portions of Guaranty Trust Banks’ website are separate from the main website, so they are not directly impacted by the lax security of the main website. But it does raise the question of how well they secure the other portions of their website if they are not doing something this basic. Also, if someone could exploit one of the vulnerabilities in the version of Joomla on the main website they could change the links directing people to the account access portion of the website to another location and use that to gather login credentials.
Due to how potentially serious the security issue with their website is we attempted to contact Guaranty Trust Bank as soon as we saw the version they are running, but we were unable to get far. For one of their listed email addresses we got back message that the mail box was full. For the other we were told to “liaise with our Corporate Affairs Unit at the head office”, but our reply asking how to do that was met with a message that the email address we were replying to did not exist.
In WordPress 3.7 a new feature was introduced that causes WordPress to automatically apply minor updates to WordPress (for example, going from 3.7 to 3.7.1). The underlying system that handles that also supports doing automatic updates for major updates, plugin updates, and theme updates as well. The reason for limiting it to minor updates by default makes a lot of sense, because there is a process in place for minor updates to make sure there is a little chance as possible for something going wrong. The lead developer of WordPress describes the process as:
Background updates are incredibly, incredibly safe. Sites already running WordPress 3.7 have attempted more than 110,000 updates without a single critical failure, thanks to a number of verification steps that have made updates that much more reliable. A background update for a minor or security release (which is all they are enabled for, by default) means downloading and copying over just a few files. We’ve gotten really good at that. We’ve also spent years honing our craft of shipping stable and targeted fixes in minor releases — we don’t indiscriminately backport bug fixes. They must be serious bugs, and fixes go through additional levels of review, including at least two lead developers. And, we have the ability to roll out an automatic update over a period of hours or days. For 3.7.1, we’ll likely see how a few hours of user-initiated updates go before telling about 1% of sites to update, then steadily increase that percentage.
Depending on the situation of an individual website enabling other updates to occur automatically as well makes sense. Keeping plugins up to date is an important as it prevents the website from being exploited due to a vulnerability in an outdated version of the plugin. At this point we haven’t seen a vulnerability in WordPress that is likely to lead to the average website being hacked in years, but with plugins that isn’t the case. So keeping plugins up to date is at least as important as keeping WordPress up to date. If you use plugins that have a good track record of not breaking after an update (we haven’t had any issues with the plugins we use on our blogs over many years) then it can make sense to turn on automatic updates for plugins.
Turning on automatic plugin updates can be accomplished by adding the following line to functions.php of your current theme:
add_filter( 'auto_update_plugin', '__return_true' );
If you prefer not to modify your theme you have another option with our new plugin, Automatic Plugin Updates, which enables automatic plugin updates along with providing a couple of additional features. The lesser additional feature is that it turns on email notifications for those automatic plugin updates (this can be disabled). The bigger feature is that it enables disabling automatic updates for selected plugins. This can be useful if you have modified plugins in use or if you have plugins that you are more concerned that an update could cause problems with the website. While you can roll your own code to do this as well, with our plugin you don’t have to worry about changes being made in the process of handling excluding plugin from automatic updates. As of the current beta of WordPress 3.9 the process has changed from previous versions, causing code written for the old versions to not stop the automatic update from happening, and our plugin is already ready to handle that if the change remains in the production release of 3.9. Both of the additional features can be accessed on the plugin’s setting page:
Fairly often we have people contacting us about doing an upgrade of software on a website, which they are hoping will resolve a hacking or other security issue. Unfortunately in most instances they don’t tell us that is why they want an upgrade done. In the worst case this could cause the upgrade to get messed up if the hack has made modifications to things that are affected by the upgrade. In many cases the upgrade isn’t going to fully resolve the issue and may not have any impact at all. For example, if a website was running Joomla 1.6, 1.7, or 2.5.0-2.5.2 and it got hacked due to the privilege escalation vulnerability in those versions, which allows someone registering a new account to escalate their account to “Administrator” level, upgrading would prevent new accounts with those privileges from being created but the existing accounts would still exist. Those accounts can be deleted, but you have to know they exist to do that. The upgrade also might overwrite other modifications the hacker(s) made to the website, but it might not.
For website still running Joomla 1.5 the website cannot be upgraded to a newer version. Instead a more complicated migration, which move content from the Joomla 1.5 installation to a new install of Joomla 2.5 or 3. Despite support for that version ending in September of 2012, the version is still widely used and we recently have been contacted about a lot of hacked website that are still running Joomla 1.5. Since the migration leaves a lot of the website behind it would reasonable to wonder if a migration will resolve the hack. A website we were just dealing is reminder that isn’t the case.
There are three major areas where parts of the hack could move over during the migration. First a common place for placing malicious files is in a website’s images folder, which is something that will move over to new website. Another common area where malicious code is placed is in theme files. While Joomla 1.5 themes are not directly compatible with newer versions of Joomla, some can be easily converted to the new version either by hand or with automated tools. The third area, which is where malicious code was in this situation, is the in the database. In this case the malicious HTML code had been added to the content of a number of articles, which was moved over during the migration.
Earlier this week we mentioned that Revive Adserver 3.0.3 has introduced a pretty serious bug that caused new statistics data to not show up. At that point two bug reports for this issue had been created and then they were closed by one of the developers. With those reports maybe you could argue that there were not enough details given to identify what was going on, though there isn’t much to tell beyond the fact that the statistics are not being updated (we didn’t see message in the debug.log for this). Also at that point, a third bug report had been filed that contained more details on what was going on.
Earlier today the third bug report was report was closed and listed as being a duplicate of one of the previous bug reports. That previous bug report was closed, without the bug being addressed, and the reporter was directed to the Revive Adserver forum. On the forum the problem causing the updated statistics to not show had already been identified. At this point the bug just needs to be fixed in the software and a new version released, but that can’t happen if the developers keep closing the bug reports without doing anything. After years of neglect by OpenX, it is unfortunate that the new maintainers of the software are acting so oddly. Hopefully this will get resolved soon and a situation like this doesn’t occur again.
For those waiting on an official fix, the easiest way to resolve this for now is to go to the file /lib/RV.php and change the line:
require_once RV_PATH . '/lib/pear/PEAR.php';
Update (April 7, 2014): The bug has now been fixed and the fix will be included in Revive Adserver 3.0.4.
When it comes to the poor state of website security we have a situation where basic security measures often are not taken but security products of questionable value are proliferating. What we see in dealing with websites that have been hacked is that it is harder to get someone to make sure that proper security practices are being taken then it would be to sell them on these questionable security products. One of the things we see that really sells people on these security products is that they will highlight that they have stopped numerous attempts to exploit a website. It sounds impressive, but in reality they may not be stopping anything at all. Let’s take look at an example of why this information doesn’t mean much.
JCE is a popular extension for Joomla. Older versions had a serious security vulnerability that was fixed in August of 2011. For people that used JCE the simple way to protect themselves against the vulnerability was to keep their software up to date. For anyone not running JCE they didn’t need to do anything since it wouldn’t impact them at all. We would fall in to that second category as we haven’t used Joomla on our website, so there is no chance that we would be running JCE. That hasn’t stopped hackers from attempting to exploit the vulnerability on our website. In March for example our logs recorded 16365 attempts (or about 528 attempts per day) to exploit the vulnerability. Here are the numbers of attempts our logs show for the last six months:
Even in the month with the lowest attempts we had an average about 9 attempts a day. If you are not familiar with the vulnerability and the fact that unless a website was running a fairly out of date version of JCE there is no chance of being exploited then it would certainly sound scary that there were so many attempts. It also wouldn’t seem unreasonable that you would recommend the product to others.
While we wouldn’t recommend security products for most websites (those basic security measures are all that you will need), what you should look at when considering these products is if they have independent testing results showing that they do in have fact protect against vulnerabilities that wouldn’t be stopped by taking basic security measures. You should also consider if they will even protect against threats you face. For example a product designed to protect against exploiting software on your website isn’t going to stop someone from getting in via FTP or from exploiting a web host’s poor security.
It has been a little over six months since the software formerly known as OpenX changed hands and became Revive Adserver. We thought now would be a good chance to look at an important improvement that has occurred and pretty serious problem that has popped up.
Before we get to that we should note that anyone still running OpenX should upgrade as soon as possible as Revive Adserver has fixed several security vulnerabilities. Other than the bug we will get to later in the post, the upgrade should be rather seamless.
The story of the later years of OpenX was a series of security problems and a lack of concern for security that lead to at least some of those problems. In one instance their systems were breached and someone was able to modify the OpenX downloads so that malicious code was included. In another their systems were breached and used in conjunction with a vulnerability in OpenX, that the OpenX developers had been warned about, to gain access to individual OpenX installations. Another ongoing issue was that OpenX was not releasing the details of what changes were being made in releases. Doing this is important when security vulnerabilities are fixed as it allows others to double check that the issue has been resolved and it also helps people cleaning up hacked ad servers (like us) to know if the vulnerability that was exploited is an old vulnerability that has been fixed or a new vulnerability that would need to be reported to the developer to get fixed.
So far the people behind Revive Adserver have been handling things much better. For the last security vulnerability found in the software, which existed long before it became Revive Adserver, it was promptly fixed and a security advisory with details of it was released.
Our own experience with reporting a security issue to the OpenX and Revive Adserver teams showed the dramatic difference between the two. In June of 2012 a vulnerability was discovered in the XmlRpc component of the Zend Framework. Shortly afterward we sent an email to OpenX’s security address alerting them to the vulnerability in the component, which was included OpenX. We never heard anything back from them and the vulnerable component was never fixed. After the software became Revive Adserver we remembered the issue and decided to try reporting the issue again. Not only did we get a prompt response, but it was clear that they had actually looked into the scope of the vulnerability. As the components were no longer used in Revive Adserver the only way the vulnerability could be exploited is if a plugin used them. To resolve the issue they removed those components in the next release of Revive Adserver, 3.0.3.
A Serious Bug Unfixed
The latest release of Revive Adserver, 3.0.3, has a serious bug that causes new statistics data to stop showing up. Since statistics are important function of the ad server this is something that should have been promptly fixed, but about three weeks later it hasn’t. There were a couple of threads started on their forum (one of which is currently labeled as being HOT) shortly after the release raising the issue and identifying the problem. There were then a couple of bug reports filed, which were closed with a developer directing people to the forum. The latest bug report was filed a week ago and has yet to receive a response from the developers. This situation seems to indicate that improvements could be made in the handling of bug reports and that better pre-release testing might be needed, so that this type of bug can be spotted before it gets into a released version.
For those waiting on an official fix, the easiest way to resolve this for now is to go to the file /lib/RV.php and change the line:
require_once RV_PATH . '/lib/pear/PEAR.php';
Last week we spotlighted the fact that only a third of websites running Drupal 7 are up to date. As keeping the software running a website up to date being an important security measure and with the most recent version of Drupal 7 being a security update that obviously is a problem (though certainly not a problem limited to Drupal). What makes this more troubling is that it isn’t just small websites that are not keeping their software up to date, but large institutions that are more than capable of doing the upgrades. In gets worse when you see institutions that have departments focused on the technology security that are failing to keep their software up to date. Last month we looked at the fact that the University of Cambridge was running an outdated version of Drupal, while the blog of their Security Group was running on a very out of date version of WordPress. They unfortunately are not alone.
Using our Drupal Version Check web browser extension, available for Firefox and Chrome, we can see that the Rutgers University website is still running Drupal 7.21:
That version is now a year out of date and two security updates have been missed (7.24 and 7.26). Making sure the website is kept up to date is something that you would hope that Rutger’s University Information Protection and Security Division would be on top of, but they are not even keeping their website up to date:
That website is less out of date than the main Rutgers website as the current version of Drupal 6, 6.30, was released in January, but it was a security update so they should have gotten it upgraded by now.
For those reading this and realizing they need to get their Drupal installation up to date, you can find the upgrade instructions here.