Are Your Websites Up to Date?You can keep track of what versions of concrete5, Drupal, Joomla, Magento, MediaWiki, Moodle, PrestaShop, Revive Adserver, TYPO3, SPIP, WordPress, and Zen Cart are running on all of the websites you manage with our Up to Date? Chrome app.
Search This Blog
- Poor Security In Automattic Sponsored WordPress Plugin
- What to Watch For When Upgrading to Magento 1.9.1
- The Upgrade From Joomla 2.5 to 3.x Often Isn’t A One-Click Update
- FAQ: Should I Upgrade Magento and Change My Theme at the Same Time?
- Automattic’s Responsibility for the Security of WordPress Plugins
Web Software Updates
WordPress VersionWe are running WordPress 4.1 and despite what many supposed "security experts" claim letting you know what version we are running does not make us less secure.
Did We Make a Mistake?While it seems to be acceptable for blogs discussing web security to contain numerous factual mistakes, we hold ourselves to a higher standard. We only write about things that we actually understand and only after we have double checked the information. So if you see a mistake in one of our posts please leave a comment on the post or contact us so that we can add a correction.
Category Archives: Bad Security
When it comes to the security of websites, keeping the software running them up to date is an important. While web hosts make a point of emphasizing the need to keep the user added software up to date, up to point of often incorrectly jumping to the conclusion that a website must have been hacked due to outdated software, they often fail to their part by keeping the software running the server up to date. In the case of DreamHost, this now not only means that their servers are not properly secured, but also that recent software can’t be used.
The latest version of Moodle, 2.7, requires at least version 5.5.31 of MySQL. This shouldn’t be a problem as MySQL 5.5 is currently the oldest series supported and version 5.5.31 was released 16 months ago. Unfortunately, while we preparing to do a Moodle upgrade for a client hosted with DreamHost we found that they are still on version 5.1.56. Our client contacted them about this and didn’t get any movement on getting this updated. They were not first, as the issue was brought up in May on a thread on DreamHost forum requesting that MySQL be updated. A DreamHost representative replied in the thread before and after that so they should have be aware that it was mentioned.
While the inability to use the latest version of Moodle is of concern, the larger issue is just how out of date DreamHost leaves the software running on their servers. Support for MySQL 5.1 ended at the end of last year, so they have been running an unsupported version for eight months. If they needed to stick to MySQL 5.1 for some reason, then you would expect that would be running the last version of 5.1, but there not. Instead they are running a version that is over three years out of date (5.1.57 was released in May of 2011) and they didn’t update after either of two subsequent releases with security updates were put out (5.1.62 and 5.1.63).
On this blog we focus a lot on the large problem of software on websites not being kept up to date. But the importance of keeping software up to date is misunderstood or misused, leading to more security problems. What we often see with web hosts, and to a lesser degree security companies, is that they tell people that their hacked website must have been hacked due to outdated software. There are a couple of major problems with this. First, websites are often are hacked due to reasons other than outdated software. It could be caused by malware on the computer of someone involved in the website, poor security at the web host, a vulnerability that even exist in the latest version of software, or a variety of other issues. The second major problem is that if you assume that the website was hacked due to outdated software and it wasn’t then the vulnerability doesn’t get fixed and the website could get hacked again (which based on the people that come to us to re-clean hacked websites, happens often). Below we dive into more detail of several of the important points on understanding what role outdated software plays in hacks.
Most Vulnerabilities Are Not Likely to Lead to Your Website Being Hacked
If you look at popular software like Drupal, Joomla, and WordPress they release security updates on a fairly regular basis. While you should be applying those security updates, it is important when dealing with a hacked website to understand that most security vulnerabilities fixed in software are not likely to lead to your website being hacked. For the average website, hackers will only try to hack it using very basic hacks that don’t rely on human interaction, so vulnerabilities that would require targeting your website are unlikely to be used. There are other vulnerabilities that would need to be combined with another vulnerability to be successfully exploited and yet other security vulnerabilities that couldn’t be used to hack your website, for example an old WordPress vulnerability allowed users to view other user’s trashed posts.
When it comes to Drupal, Joomla, and WordPress, only with Joomla have we seen a new vulnerability in the software successfully be exploited in the past few years. So with Drupal and WordPress if somebody is telling you an outdated version caused the hack chances are they are wrong. The vulnerabilities in Joomla could impact websites running 1.6.x, 1.7.x, and 2.50-2.5.2 if user registration is enabled or versions 1.5.x, 1.6.x, 1.7.x, 2.5.0-2.5.13, 3.0.x, and 3.1.0-3.1.3 if untrusted users are allowed to upload files.
When hiring someone to deal with a hacked website, finding someone with expertise with the software you use can be important for understanding what impact the security vulnerabilities in an outdated version of it potentially have and if they could have lead to the website being hacked.
You Need to Determine How the Website Was Hacked
Our experience is that many companies provide hack cleanup services don’t actually do the important task of determining how the website got hacked. While you might get lucky and the vulnerability is fixed without determining what it was first or the hacker doesn’t come back, you shouldn’t bet on that. We often have people comes that had previously had someone else clean up the website and then in short order it gets hacked again. Our first question in those situation is if the source of the originally hacked was determined and we have someone answer that it was, the usual response is that determining the source of the hack was never even brought up.
When it comes to saying that your website must have been hacked due to outdated software, what we have seen is this often not based on any evidence. In fact, in some cases we have seen web hosts blaming outdated software despite the software being up to date at the time of the hack. If somebody tells you that it is the cause they should be able to tell you what the vulnerability is and provide evidence that supports the claim. If the logs of access to the website are available they should be able to show you the relevant log entries showing when the hack was exploited. Unfortunately, in too many cases web hosts do not have good log retention policies so the logs are gone once the hack is discovered, but someone who knows what they are doing should be able to explain why the evidence still available matches exploitation of the vulnerability.
Before you hire someone to clean up a hacked website make sure that determining the source of the hack is part of their service, if it isn’t they are not doing things properly.
You Can Be Up to Date Without Running the Latest Version of Software
We often see people confusing the need to keep software up to date with the need to be running the latest version of the software. While they are the same in some cases when the developers only support one version of the software at a time, in other cases you only need to be running an up to date version of one of the supported versions to be secure. For example, Drupal currently supports versions 6 & 7, so at the moment you should be running 6.31 or 7.28. While newer versions may include security improvements over an older version, the older version should still be secure against hacking as long as it is receiving security updates. Using Drupal as an example, Drupal 7 introduced better password hashing, which improves security but would only have impact on it in a situation where someone has gained access to the database, which they shouldn’t if things are secure.
For those in charge of managing numerous websites you can use our Up to Date? Chrome app to keep track of the update status of websites running Drupal, Joomla, WordPress, and other software all in one place.
When it comes to the security of websites, what we see over and over is that the basics are not even being handled by people that shouldn’t have a problem doing it. If you are running a WordPress website then part of Security 101 is keeping WordPress up to date, as it prevents your website from being hacked due to a known vulnerability in an older version of WordPress. Unfortunately, that isn’t being done in many cases as can been seen in the fact that only 40 percent of WordPress websites were running the latest series of WordPress in the data set we looked at in March.
You would think that providing better management tools would help this situation, though the example of one of the providers of such a tool would say otherwise. ManageWP describes its services as providing you the ability to “Manage all your WordPress sites from one place – including updates, backups, security and more.” You would certainly expect they would be keeping the WordPress installation powering their website up to date, but they’re not:
ManageWP’s failure to take handle a basic security task is sharp contrast to their claims of security. For example, they claim
Securing ManageWP and the sites we interact with has always been our highest priority. We use state-of-the-art encryption and security standards that go above and beyond what WordPress, itself, offers, to ensure that your sites are protected.
On another page they make a series of claims about their security:
How ManageWP Is Secure
- We have a full-time security specialist
- We regularly perform penetration testing
- No credit card information stored
- No WordPress passwords stored
- OpenSSL encryption
- ManageWP is built on top of WordPress
- Account password encryption
- White hat reward program
If you are security specialist who fails to make sure such a basic security measure is taken then you probably should find another profession.
When it comes to IT security companies, what we see over and over is that they have little to no concern for security (and also often have little to no understanding of proper security practices). So it isn’t surprising that despite billions being spent on IT security, IT security continues to be in such poor shape. This leads to situation like the massive breach of Target’s systems last year. While that was big news, what didn’t get much attention was the company who declared Target compliant with standards for handling credit card transactions shortly before the breach, Trustwave. Trustwave has a history of declaring companies compliant shortly before they suffer major breaches and for being lax in their assessments.
We recently spotted another example of their highly questionable practices of Trustwave. We were contacted about doing a migration of a Joomla-based website still running version 1.5, for which support ended in September 2012. While taking a look at the website, we noticed a seal for Trustwave Trusted Commerce:
Considering that the website is running software that is no longer supported and therefore cannot be considered secure, we were curious to see if Trustwave was claiming it was secure. It would be quite easy for them to find that the website is running Joomla 1.5 if they wanted to as the source code of every page on the website the following line is included:
<meta name=”generator” content=”Joomla! 1.5 – Open Source Content Management” />
If you click on the seal you get this page:
At the top of the page Trustwave proclaims that “Your credit card and identity information are secure.”, which they shouldn’t be saying for a website that is running unsupported software.
As we looked closer we noticed the small text disclaimer at the bottom of the page were they say “Trustwave Holdings, Inc. makes no representation or warranty as to whether [redacted] systems are secure from either an internal or external attack or whether cardholder data is at risk of being compromised.”. So they are basically telling you that despite saying “your credit card and identity information are secure”, there not actually saying that at all.
It is highly inappropriate for them to mislead the public like they are doing with this seal, but unfortunately our experience is that this kind of thing is considered acceptable in the security industry.
Recently we have had a lot of blog posts highlighting major organizations running outdated and insecure versions of Drupal, but we don’t want to give the impression that it is only with Drupal based websites that major organizations are failing to keep the software up to date on. So we wanted to find an example of a website running Joomla to highlight as well and we quickly found a very concerning example. The third website listed on Joomla’s showcase of websites running Joomla is the website of Guaranty Trust bank, which is Nigeria’s largest bank and has assets of over 12 billion USD. As you can see with our Joomla Version Check web browser extension, available for Firefox and Chrome, their websites is running a fairly out of date version of Joomla:
That version is over two years out of date and there have been twelve subsequent updates with security fixes. One of the security vulnerabilities fixed in a subsequent version is of particular concern. The vulnerability, which we discussed before, allows a new user account to be created with “Administrator” privileges through privilege escalation. If user registration is disabled this will not work, but in this case it does appear that user registration is enabled. It is important to note that account access portions of Guaranty Trust Banks’ website are separate from the main website, so they are not directly impacted by the lax security of the main website. But it does raise the question of how well they secure the other portions of their website if they are not doing something this basic. Also, if someone could exploit one of the vulnerabilities in the version of Joomla on the main website they could change the links directing people to the account access portion of the website to another location and use that to gather login credentials.
Due to how potentially serious the security issue with their website is we attempted to contact Guaranty Trust Bank as soon as we saw the version they are running, but we were unable to get far. For one of their listed email addresses we got back message that the mail box was full. For the other we were told to “liaise with our Corporate Affairs Unit at the head office”, but our reply asking how to do that was met with a message that the email address we were replying to did not exist.
Last week we spotlighted the fact that only a third of websites running Drupal 7 are up to date. As keeping the software running a website up to date being an important security measure and with the most recent version of Drupal 7 being a security update that obviously is a problem (though certainly not a problem limited to Drupal). What makes this more troubling is that it isn’t just small websites that are not keeping their software up to date, but large institutions that are more than capable of doing the upgrades. In gets worse when you see institutions that have departments focused on the technology security that are failing to keep their software up to date. Last month we looked at the fact that the University of Cambridge was running an outdated version of Drupal, while the blog of their Security Group was running on a very out of date version of WordPress. They unfortunately are not alone.
That version is now a year out of date and two security updates have been missed (7.24 and 7.26). Making sure the website is kept up to date is something that you would hope that Rutger’s University Information Protection and Security Division would be on top of, but they are not even keeping their website up to date:
That website is less out of date than the main Rutgers website as the current version of Drupal 6, 6.30, was released in January, but it was a security update so they should have gotten it upgraded by now.
For those reading this and realizing they need to get their Drupal installation up to date, you can find the upgrade instructions here.
Yesterday, as part our series of posts highlighting the fact that even high profile websites are not taking the basic security measure of keeping the software running them up to date, we highlighted the fact that ING US was using outdated and insecure versions of Drupal on their website. Today we have a few quick follow-ups.
First it was brought to our attention that the fact that ING was using Drupal was a big enough deal for the creator of Drupal to highlight it, saying in part
You know when a piece of software is mature when it starts being adopted by financial services organizations.
The fact that such high profile user isn’t keeping Drupal up to date in light of the security need of doing so either means that that Drupal is too hard to keep up to date, which we strongly disagree with based on keeping our own installation up to date and handling plenty of upgrades for clients, or there is more general problem with security practices for websites.
In the aforementioned post another ING US website was highlight as running Drupal and that website unfortunately has also not been kept up to date:
At the bottom on that website is a link to a Web Site Security page, which in part advises keeping the software on your computer update:
Take care of your computer
Update your computer by installing the latest software and patches to prevent hackers or viruses from exploiting any known weaknesses in your computer.
It would great if ING, as well as everyone else running a website, took that advice and applied it to their websites.
When it comes to keeping websites secure one of the basic things that needs to be done is to keep the software running the website up to date. This prevents the website from being exploited through a known vulnerability in old versions of the software that has been fixed in a subsequent release. We know that many websites are not doing this, which is troubling, but what is more troubling is that the major institutions are not even doing this with their websites. Last week we looked at major security software provider not doing it and if you go back in this blog, you can find other examples. Today let’s look at example of a major financial institution in the same boat. ING US, which in the process of rebranding as Voya Financial, reports having $511 billion of assets under management and administration and serving approximately 13 million customers. They use Drupal for main portion of the ING US website. Using our Drupal Version Check web browser extension, available for Firefox and Chrome, you can check if it is up to date:
You can see that they are not. With a little further checking we were able to determine they are using Drupal 6.19. That means they haven’t updated the software in over three years and they have failed to apply
five six security updates (6.21, 6.23, 6.27, 6.28, 6.29, and 6.30). It is important to note that account access portions of their website are separate from the main website, so they are not directly impacted by this lax security. Though it does raise the question of how well they secure the other portions of their website if they are not doing something this basic. Also, if someone could exploit one of the vulnerabilities in the version of Drupal on the main website they could change the links directing people to the account access portion of the website to another location and use that to gather login credentials.
It isn’t just the ING US website that has an out of date version of Drupal in use. The website for their new name, Voya Financial, also is using an outdated Drupal version:
With a little further checking we were able to determine they are using a version no newer than Drupal 7.21. That means that they haven’t updated the software in nearly a year and they have missed at least two security updates (7.24 and 7.26).
The Internet has lots of bad information on website security floating around. In dealing with many websites that have been hacked, we see the harmful impact this has due to it leading to bad security practices and making it harder to get people to take the measures that will actually keep websites secure. Much of the bad information comes from companies providing security tools and services, whom you would expect would know what they are talking about. We looked at an example of bad security research by Cisco on Friday that lead to bad security reporting by Ars Technica and by today they have both pulled back from their claims.
Cisco has struck through most of their post and included this update:
This post’s focus relates to a malicious redirection campaign driven by unauthorized access to thousands of websites. The observation of affected hosts running Linux kernel 2.6 is anecdotal and in no way reflects a universal condition among all of the compromised websites. Accordingly, we have adjusted the title for clarity. We have not identified the initial exploit vector for the stage zero URIs. It was not our intention to conflate our anecdotal observations with the technical facts provided in the listed URIs or other demonstrable data, and the below strike through annotations reflect that. We also want to thank the community for the timely feedback.
Ars Technica has added an update to their post, included below, which doesn’t explain why they went beyond the claims in Cisco’s post or why they repeated Cisco’s claim without doing basic research that would have shown the research was highly flawed.
The Cisco blog post has been updated to change a key finding Ars reported in the following post. Contrary to Cisco’s earlier reporting, the update says not all the servers compromised in the attack were running Linux version 2.6. “We have not identified the initial exploit vector for the stage zero URIs,” the update stated. “It was not our intention to conflate our anecdotal observations with the technical facts provided in the listed URIs or other demonstrable data, and the below strike through annotations reflect that. We also want to thank the community for the timely feedback.”
Considering how colossally bad Cisco’s findings were we want to expand on how they got it so wrong, so that it might point security reporters in the direction of better vetting security research before repeating its claims in the future.
One of their key findings was that all of the websites were running an old version of the Linux kernel:
All of the affected web servers that we have examined use the Linux 2.6 kernel. Many of the affected servers are using Linux kernel versions first released in 2007 or earlier.
They then raised the possibility that this was what allowed the hack.
It is possible that attackers have identified a vulnerability on the platform and have been able to take advantage of the fact that these are older systems that may not be continuously patched by administrators.
The original title of the post, Mass Compromise of the Obsolete, also implied that the hack was related to obsolete software.
What we brought up on Friday was that not all of the websites on their list of affected websites were even running Linux, much less the Linux 2.6 kernel. Cisco’s explanation for this discrepancy is that their claim that all of the examined websites were using the Linux 2.6 kernel was anecdotal. We don’t how you can square the claim you examined the websites, but your finding was anecdotal. It seems either they didn’t look at their whole list of websites or they used a faulty tool that determined websites not running Linux were using the Linux 2.6 kernel, neither of which we would describe as being anecdotal. Asking Cisco how they determined the website were all running the Linux 2.6 kernel and what there sample set was would have been something that should have been done before journalists repeated their claims. Incorrectly identifying a set of hacked websites as having a common software version is something that we have seen repeatedly from security companies (a couple of examples), so reporters should look carefully at the evidence and probably get a second opinion on the evidence.
While their original post doesn’t spell out what versions they are referring to by the “many of the affected servers are using Linux kernel versions first released in 2007 or earlier”, a comment by one of the authors of the post says that “version 2.6.18 appeared to be particularly prevalent”. If the Cisco researchers had look into why this version was rather prevalent they should have realized they were going down the wrong path. Why would Linux 2.6.18 be rather prevalent? Well for one thing, it happens to be the Linux kernel version used by Red Hat Enterprise Linux (RHEL) 5 and it derivatives, the most prominent being CentOS 5. A little further checking would have shown them that RHEL 5 will continue to be supported for some time, so servers using the Linux 2.6 kernel would not necessarily be obsolete or insecure. This is something that Cisco should be aware of since the server powering the Cisco Blog is using RHEL 5:
Because we often see people saying otherwise, it is important to note that just because there is a newer version of software available it doesn’t mean that an older version is not safe and secure, as long as the older version continues to receive security updates.
What ultimately would have prevented this mess is if Cisco had taken the basic step of determining how the websites were hacked instead of jumping to conclusions based on data that was not reliable. Security reporters should understand that determining how a website has been hacked is an integral to dealing with them and if somebody isn’t explaining that, it should be a huge red flag that the information being given might not reliable.
On Tuesday we looked at example of the poor state of security journalism. In that case a hack was tied to a specific version of TYPO3, despite fact that websites not running that version of TYPO3 or running TYPO3 had been hacked. There was also the larger issue that no evidence was provided as to how the websites were hacked, which would have been what would be needed to actually tie the hack to a specific version of TYPO3 and would allow people to make sure the protected their websites against it. Just a few days later we have spotted another very similar example worth highlighting. Ars Technica today put out an article “Ancient Linux servers: The blighted slum houses of the Internet” that states:
Now comes word of a new mass compromise that preys on even more neglected Web severs, some running versions of the Linux operating system kernel first released in 2007. According to a blog post published late Thursday by researchers from Cisco, the people behind the attack appear to have identified a vulnerability that has since been patched in later Linux releases that allows them to dish malicious content to unsuspecting people who visit the site.
If you read Cisco’s blog though they only state “it is possible” that a “vulnerability that has since been patched in later Linux release” was the source of the hack, while Ars Technica says that it “appears” to be the case. Here is the relevant section of Cisco’s post:
That turns out to be less of an issue then the fact that the websites are not even all running Linux, much less the Linux 2.6 Kernel. Some websites provide information on the software running the in HTTP headers served with the page. Our Server Details web browser extension, available for Chrome and Opera, can parse those HTTP headers to provide the details in them and warn for outdated software. Using those headers we started going through the Cisco’s list of compromised websites and second compromised websites. For each we have listed below the first five websites we found not running Linux and what operating system they are running:
archive.mrpools.co.uk Windows Server 2003
blueprintbowling.com Windows Server 2008 R2
hwy65mx.com Windows Server 2003
jandjpoolspa.com Windows Server 2003
mussotra.com Windows Server 2003
Second Compromised Websites
Cisco provides no evidence of how the websites were hacked, which is the really important thing to prevent more websites from being hacked. If they had actually determined how it was hacked before jumping to speculation then they wouldn’t have tried to connect this to Linux, which it seems pretty likely it doesn’t have anything to do with. Cisco also has provided no evidence this has anything to do with outdated software, if we were to make an educated guess based on the evidence provide so far we would say it is more likely due to compromised FTP credentials, which could easily be checked for by reviewing the FTP logs for the websites.
We should also note that the use of the Linux 2.6 kernel is does not indicate that website using obsolete software, as distributions including Debian, Ubuntu, and Red Hat still have supported releases that use that version of the Linux kernel.