Cloudways Isn’t a Great Option For Hosting a Magento Website if You Want to Be Able to Upgrade It

We were recently hired to do a Magento upgrade of a website hosted with a web host named Cloudways. Our experience with what they were providing and their support when what they provide doesn’t work was bad. That is despite claiming to provide the best managed Magento hosting platform:

Experience faster than ever Magento cloud hosting designed with a fully optimized stack for the best performance, reliability, and security, with 24/7 active support.

The most serious issues was the first big one we ran into. While working on the test of the upgrade, running the update command for Composer managed to cause the server to hang, and it had to be restarted. That shouldn’t happen and being able to run that command is a core part of the upgrade process, so that was a big problem. When the server was restarted, it had a different issue with hanging.

Cloudway’s support was no help in trying to understand what caused the initial hang or how it could be avoided. Instead, they simply suggested running it again and monitoring what happened.

Further usage of the command produced inconsistent results. In some cases, running it errored out and others ran without issue. That isn’t ideal when trying to handle doing an upgrade, where you are trying to carefully manage things.

Whatever is the underlying problem with their server environment and Composer, in our limited experience it didn’t just impact upgrades. After the upgrade, our client was trying to install a new extension through Composer and that also failed because a lack of “memory or swap”. Re-running the process later ran without issue.

An entry on Cloudway’s user suggestion forum indicates that people have been experiencing issues with taking basic actions through Composer since at least 2016.

Hackers Using Insecure Magento Extensions to Access Magento Admin Login Page Without Knowing Normal Address

While reviewing the log files while cleaning up a hacked Magento website recently we ran across a reminder that a common security practice isn’t full proof. With the Magento software and some other software the software has a built in capability to use a non-standard address for the login page for the admin portion of the website. With other software that is commonly promoted security feature to be implemented with an add-on.

The value of that is limited as while there are widespread claims that there are frequent brute force attacks against admin passwords, in truth what is going on are dictionary attacks, which involved trying to log in using common passwords and that can easily be prevented from being successful by using a strong password. There is the possibility of some value of doing this in a far more limited situation where the hacker has access to valid login credentials for the website, but it turns out that there can be various ways to get access to login page without knowing that address.

In the logs of this hacked website we were seeing many POST requests to this address:

/index.php/magenotification/adminhtml_feedback/index/

When visiting the page we saw that the Magento admin login page was showing. In looking into if that all there was that we found that this is something that hackers are looking around with pages like that generated by various extensions.

Magento added protection against this issue with SUPEE-6788, which was part of Magento 1.9.2.2, but by default the protection is not enabled.

Hacker Using SQL Injection Vulnerability to Add “magentoupdate” Admin Account to Magento Websites

As is a common occurrence, we were recently hired to re-clean a hacked website that the security company Sucuri, which is owned by GoDaddy, had repeatedly failed to properly clean. This time it was a Magento based ecommerce website we were cleaning. As is standard issue in those situations they had missed malicious code that should have been easy to find. What we also found was that the hacker had been able to add an additional admin account, unfortunately that had occurred prior to the time period logging was still available, so we didn’t have evidence of how that had been done.

In a situation where we haven’t been able to determine how the hacker has gotten access, part of our cleanup process is to recheck things for a couple of weeks to see if the hacker tries to get back in. In this case the admin account returned a couple of days later.

For others dealing with the admin account in this situation had these details:

  • User Name: magentoupdate
  • Email: support@media.com
  • First Name: support
  • Last Name: support:

With the logging available from when this occurred we found a log entry where one of the URL parameters was this:

');insert%20into%20%60admin_user%60%20(firstname,lastname,email,username,password,created,lognum,reload_acl_flag,is_active,extra,rp_token,rp_token_created_at)%20values%20('support','support','support@media.com','magentoupdate','8df1e8abd8ce4761633042eb8958db97:rp',NOW(),0,0,1,'N;',NULL,NOW());INSERT%20INTO%20%60admin_role%60%20(parent_id,tree_level,sort_order,role_type,user_id,role_name)%20VALUES%20(1,2,0,%22U%22,(SELECT%20user_id%20FROM%20admin_user%20WHERE%20username%20=%20'magentoupdate'),'support');

That is SQL code that generates that admin user, which would be exploited through a SQL injection vulnerability. In this case it involved exploiting a SQL injection vulnerability in an extension on the website, which we then patched up.

Security Journalist’s Bad Focus and Flashpoint’s Questionable Business Risk Intelligence

When it comes to why website security is in such bad shape there are lots of parties that play a role. Journalists could play a critical role is shining a light on what is wrong with the security industry, but for the most part they instead act as stenographers for claims made by security companies without a concern for the accuracy of the claims or if they are newsworthy.

An example of that this week that we happened across (there are in all likelihood plenty of others just this week) involved what seems like an insignificant claim. Multiple outlets including The Register, SecurityWeek, and Bleeping Computer ran with a story of a claim that a thousand Magento websites had been hacked. A hack of that size alone doesn’t seem all that significant and highlighting it might not be helpful if it leads people to think of Magento being less secure than other solutions that are in fact less secure. What might make this newsworthy is if the method of the hacking was significant, say a new vulnerability was being exploited. As we will get to in a bit, it seems like the claimed source of the hacks might not be accurate, but if true, it doesn’t seem all that significant and isn’t Magento related.

It isn’t that security journalist haven’t had anything to cover recently were there could be some real journalism done when it comes to hacked websites. Recently we have been discussing a situation where a relationship between a web hosting company and a security company looks to have lead to the web hosting company ignoring hackers targeting their customers and possibly ignoring that their systems have an insecurity that is leading to the hackings. Questioning those companies could provide more insight on the situation and might lead to corrective action being taken.

The claims about the Magento websites being hacked came from a company named Flashpoint, which promotes itself delivering business risk intelligence, which they describe thusly:

Business Risk Intelligence (BRI) broadens the scope of intelligence beyond threat detection in the cyber domain to provide relevant context to business units not traditionally afforded the benefits of intelligence from the Deep & Dark Web. By informing decision-making and improving preparation, BRI mitigates risk across the enterprise.

BRI can not only bolster cybersecurity but also confront fraud, detect insider threats, enhance physical security, assess M&A opportunities, and address vendor risk and supply chain integrity. The results are better decisions that protect a company’s ability to operate.

To us that sounds like something that probably isn’t of much value when so often security basics are still failing to be done, but it seems to be easier to sell people on more advanced things than on what they really need to be doing.

If their claims about Magento websites being hacked are any indication that intelligence seems to be of little value. Here is the kind of insight that provides:

Researchers at Flashpoint are aware of the compromise of at least 1,000 Magento admin panels, and said that interest in the platform has continued unabated on entry-level and top-tier Deep & Dark Web forums since 2016. Attackers have also demonstrated continued interest in other popular ecommerce-processing content management systems such as Powerfront CMS and OpenCart.

Hackers being interested in hacking websites shouldn’t be news to those in the security industry. That they would interested in popular software also shouldn’t be news. What might useful intelligence is if it was discovered that hackers were exploiting a zero-day vulnerability, which is a vulnerability being exploited before the developers of the software are aware of it, but that isn’t the case here.

Here is the claimed cause of the hacks:

The Magento sites are being compromised through brute-force attacks using common and known default Magento credentials. Brute-force attacks such as these are simplified when admins fail to change the credentials upon installation of the platform. Attackers, meanwhile, can build simple automated scripts loaded with known credentials to facilitate access of the panels.

There isn’t an explanation of how they determined that was the case and we can’t think of what “known default Magento credentials” would refer to, so that leaves us to wonder if they actually know how the websites were hacked (or have even a basic understanding of Magento).

Later they made additional reference to “default credential usage” usage that seems unconnected to Magento:

In the meantime, the rash of attacks resurrects the epidemic of default credential usage among admins. Default credentials were at the core of the 2016 Mirai attacks where hackers were able to access connected devices such as security cameras, DVRs and routers using known and common default passwords. The compromised IoT devices were corralled into a massive botnet that was pointed at a number of high-value targets including DNS provider Dyn, French webhost OVH, and journalist Brian Krebs’ website in order to carry out crippling distributed denial-of-service attacks.

The Mirai attacks involved trying to gain access to Internet of things (IoT) devices using factory default usernname/passwords, but Magento doesn’t have a default username/password like that, so we are at loss to understand what they are talking about (it might be that they have no idea what they are talking about).

For a moment let’s assume what they are claiming about trying to log in using common passwords is true, since that is certainly a real issue. What seems to be the biggest take away from that is that security basics (like using a strong password) are still not being done, while companies like Flashpoint are selling people on the need for additional security services. You would hope that these companies might consider that they might be part of the problem instead of the solution for the poor state of security, but we have seen no indication that they do.

What also sticks out to us is that Flashpoint doesn’t seem to understand basic security terminology, as they are mixing up two different types of attacks. What they are talking about, trying to log in using common passwords, is a dictionary attack. They instead refer to it as a brute force attack, which actually refers to trying to log in using all possible passwords. How you would protect against those is very different, so understanding the difference is important for those in the security industry. That the Wikipedia manages to get this right and Flashpoint doesn’t seems like an indication that they don’t have the best grasp of security, which might explain them trying to sell people on business risk intelligence instead of something that will have a better effectiveness at improving security.

Sucuri Claimed Customer’s Website Was Clean Despite It Comprising Credit Card Info Entered on It

Back in June of 2012 we wrote a post mentioning that looking at false positives produced by a malware scanner would give an idea of the quality of the scanner. In that post we looked at a rather bad false positive from web security company Sucuri’s scanner. Moving forward nearly five years it is clear that Sucuri hasn’t improved the quality of their scanner as a month ago we looked at them falsely claiming our website was defaced because we have a page named “Hacked Website Cleanup”. When your scanner is that bad, it doesn’t seem all together surprising that it would manage to miss things that it should catch as well and a recent situation we were brought in to deal with confirmed that. But much worse, it also reconfirmed everything we have seen in the past that Sucuri is company that either really doesn’t have much clue about what they are doing or doesn’t care to do things right, and in this situation that lead to people’s credit card information being compromised.

A week after we wrote the post about Sucuri falsely labeling our website as being defaced we were contacted by someone with Magento website that was having credit card information entered on it compromised. Sucuri, who they had brought on while before to deal with the situation, was telling them that website was clean, despite the compromises continuing to happen. Since that claim that the website was clean was pretty clearly not true, the person behind the website was then looking for someone competent to properly resolve the situation.

If credit card information is being compromised when entered on a website, the default assumption should be that the website is hacked. About the only other possibility we can think of is if the payment processor is compromised (which is lot less likely). So upon believing it was clean, Sucuri should have realized they were missing something and figured out what they were doing wrong, but they didn’t.

One of the questions we asked about the situation right after being contacted was who is the payment processor, if it was a major one then the payment processor could be ruled out as the source. It was a major one.

At that point we assumed that code causing the credit card info must be well hidden seeing as Sucuri couldn’t find anything. But after getting the response about the payment processor, we did quick check of the website from the outside and we immediately ran across part of the problem. It wasn’t even detected using any highly advanced proprietary technology, but off the shelf tools.

What we noticed was that there was JavaScript being loaded from the domain adyenweb.com through this script tag:

That was clearly meant to look like it was loading some type of tracking code.

The code in the file being loaded from that was highly obfuscated (when we ran through a tool to deobfuscate it, all it could pull out is that the code was requesting another file that was an encryption library for JavaScript):

At that point, considering the code didn’t look legitimate, instead of spending a lot of time trying to get a more complete deobfuscation before moving forward, we did a few other quick checks to try to assess the legitimacy of the domain the code was being loaded from.

First, we tried to trace where the server the domain was hosted on was, but found that it traced back to Cloudflare, which could have pointed to this being legitimate or it could have been someone with malicious intentions protecting themselves through Cloudflare (which is apparently a fairly common thing).

Second, we looked at the domain name registration, which didn’t look all that suspicious, but the domain was only registered on March 17.

Finally we tried to take a look at the website, but we found that there was nothing served at http://adyenweb.com or http://www.adyenweb.com. There also was nothing that came up for it in a Google search.

At that point we could safely say that this was at least part of problem. At the same time we noticed that despite something fairly obviously malicious being on the website Sucuri was telling the public the opposite about the website, as the website had this badge claiming it was “Secured by Sucuri” at the bottom:

Clicking that brought up this:

Not only did they claim the website was clean, but that their service “provides peace of mind that the website is not infected”, despite that not being true.

After we got access to the logins, we found that script tag shown earlier was stored in Magento’s settings in the database (as shown from phpMyAdmin):

 

This turned out to not be the only fairly hard to miss portion of the hack that Sucuri missed. In the root directory of the website was the backdoor script that the hacker was using to take actions on the website. That was something that Sucuri should have noticed at multiple points. Those points being during a visual inspection of the filesystem (since you need to get an understanding of what all is part of the website when first assessing the situation), during the reviewing the website files for malicious code (it wasn’t something that was obfuscated in a way that would make detection difficult), and when reviewing the log files to try to determine the source of the hack. In looking at the logs we found that the backdoor script had most recently been accessed two days after adyenweb.com was registered.

The backdoor script looks like it might have been on the website for nearly a couple of years, so we were not able to say what was the source of that was, but continued reviewing of the logs files showed that after it was removed and the various logins changed the hacker no longer had access to the website. So getting this resolved was rather simple for a competent company, which this incident shows Sucuri is far from.

The Company Upgrading Your Magento Store Might Not Have Any Prior Knowledge of How to Do That

We feel that it is important that we only provide services were we have the necessary expertise to properly handle the work, that isn’t something that is shared by every other web development company, as were recently reminded in a situation involving a Magento upgrade.

We were contacted by a web development company about doing a Magento upgrade for them (the company portrays themselves as being based in New York, despite it being rather obvious they are actually based out of India due to things like the companies name ending “IN” after the rest of the name was in lower case). After sending a few emails back and forth they were supposed to be getting us the login details to get started working on the upgrade.

Four days later they get back to us and say they had now upgraded the Magento software, but needed help upgrading the extension. We were rather perplexed by this, as it is much easier to upgrade the extensions than Magento, so how is that they could handle upgrading Magento, but not the extensions? When we asked them that we didn’t get a direct answer, instead they replied with the steps they took to upgrade Magento.

The steps they took indicated that they didn’t not actually know how to upgrade Magento, that started with it looking like they had just copied the steps from some random website. That is concerning since they apparently were not even familiar enough with Magento to even know about the official upgrade instructions. Of more concern is that they were not doing things right, as they skipped doing a test of the upgrade, which is very important to having a successful upgrade. Something their customer is unfortunately fairly likely to find out down the road when something doesn’t work right on the upgraded website.

Cart2Cart’s Strange Magento Security Announcement

Last October we wrote a post about strong circumstantial evidence pointing to the fact FTP credentials provided to the company Cart2Cart had compromised on their end. In the past few days we became aware of a security announcement they put out that either obliquely notifies their customers of that compromise of FTP credentials or indicates they really have no clue when it comes to security (while feeling it appropriate to be giving out security advice).

They are sending out the following email to customers:

We’re writing to inform you that our security audit has revealed an unpleasant vulnerability of certain Magento stores. Considering the fact that Magento shops are being attacked by hackers more often lately, we strongly recommend you to double-check the security of your e-store.

Please, contact your developer team, so that they could scan your Magento source code in order to ensure that your shop is not under the threat of being abused. Read more info here:
http://www.shopping-cart-migration.com/important-security-announcement-for-magento-store-owners

If you need technical assistance regarding this, reply to this email and we will check your store from our side.

Following the link mentioned there, you get a page that starts out:

After performing an audit, we’ve revealed an unpleasant vulnerability of certain Magento stores that may have a negative impact on the security of your customers’ personal data.

To ensure the finest security of your Magento retailer, we strongly recommend you to contact your developer team and check the source code for the presence of any suspicious customizations.

The link to “an unpleasant vulnerability” discusses not a vulnerability, but the end result of a vulnerability being exploited, code added to one of Magento’s files that sends credit card information entered on the hacked website to a third-party. The distinction is quite important because when a website is hacked, if you don’t find and fix the vulnerability that allowed it to be hacked the website can remain vulnerable to being hacked again.

Cart2Cart’s email and page never mention what the code they are mentioning does, instead saying “First of all, there’s no need to panic. You can eliminate any possible risks simply by revealing and deleting the code, if there’s any.”.

The next thing they say leads us to believe this could be a reference to their being compromised (or it shows they have no idea what they are talking about) as it suggest doing two things if you have the code on your website:

  1. Delete the following code inside and save the changes:
  2. Change your FTP account credentials

Those steps would suggest that the hack happened through compromised FTP credentials, since they want to change those credentials. But the FTP credentials would have to have been compromised somehow, yet they don’t suggest doing anything to stop them from being compromised again. That could be because the compromised happened on their end and has now been fixed, or it could suggest they have no clue what they are talking about.

The last section of the page would certainly lend some credence to them not having much clue when it comes to the security of websites. They provide this list of security tips:

Useful Security Tips

  1. Use up-to-date antivirus software

  2. Don’t store your FTP account passwords in programs like Total Commander, Filezilla, etc.

  3. Change your FTP account password periodically e.g. once a month, especially after granting access to any service providers

  4. Limit the FTP access to specific IP addresses

  5. Change the administration panel login “admin” to a custom one

  6. Hire certified developers, designers or other staff you can trust to only

  7. Use repository for a proper and secure workflow

Notable missing from their list of security tips is keeping the software on your website up to date. Not only is this a basic security measure, but it is particularly relevant with Magento based websites right now, since most of the hacked Magento based websites we are cleaning these days have been hacked due to the software not being kept up to date (or not having the security patches for older versions applied).

If you do find code added /app/code/core/Mage/Payment/Model/Method/cc.php, removing the code and changing the FTP credentials is not enough. You will want to also:

  • review the rest of the websites files for malicious code and remove any found
  • check for Magento extensions added by a hacker
  • check for additional Magento admin users
  • get Magento secured by either upgrading it to the latest version of 1.9, currently 1.9.2.3, or applying the security patches for older versions
  • rename the Magento admin directory to something other than “admin”
  • change the passwords for other logins associated with the website (database, Magento admin, etc), in case they were compromised
  • try to determine how the website was hacked and make sure that is fixed

Magento Still Failing To Take Basic and Important Security Measure with Their Software

Last week Magento had a blog post about a spate of false reports of security issues in the Magento software. We will take a closer look at the role that bad security companies and bad security journalism play in that sort of situation in an upcoming post, but something else that stood out to us with that was the fact that they feel the need to put out a post to refute non-existent security issues while still failing to take a basic and important security measure with their software. That security measure being that the current version of your software shouldn’t have known security vulnerabilities in it. This has unfortunately has been the case again for Magento, this time for over a month and a half and counting.

Back on February 9, Magento released a security patch for a very serious vulnerability. It wasn’t until May 1 that they released a new version 1.9.1.1, which included the security fix built-in, meaning that for nearly three months someone downloading the latest version would be getting something known to be insecure. Then on May 14 another security patch, SUPEE-5994, was released, which is they describe as:

This patch addresses multiple security vulnerabilities in Magento Community Edition software, including issues that can put customer information at risk.

While a major new version, 1.9.2, is expected shortly, as of now the latest version is still 1.9.1.1, which doesn’t include the security fixes included in SUPEE-5994. If Magento truly means it when they say in that blog post that “The security of the Magento platform is our top priority”, this practice needs to change going forward.

Google’s Bad Answers

Recently we wrote a post on how Google was placing bad instruction for upgrading Zen Cart directly in the search results. We have run across another example of where Google isn’t providing a good answer. If you do a search for “Magento PHP 5.5” currently you get the following answer above the normal search results:

This link says that Magento is compatible with PHP 5.2.13 - 5.3.24, but when you run the PHP script to check server requirements, it says that is okay to run on 5.4 and even 5.5. But I've seen some issues with 5.4 over the internet.Aug 23, 2013

Unlike the Zen Cart upgrade example, the information isn’t wrong, it just out of date. If you following the link referenced in that answer you are taken to the Magento System Requirements page which now lists the latest version of Magento, 1.9.1, as being compatible PHP 5.4 and 5.5 (as we mentioned in a previous post, as of Magento 1.9.1 the bare minimum it will allow being run on is 5.3.0).

The Magento System Requirements page was the first result when we did the search:

magento-php-5-5-google-first-result

So excluding a direct answer would have produced a better result in this case (by comparison the page Google took their answer from was ranked 7th).

Preparing to Have Your Magento Website Upgraded

Upgrading is Magento is no small task. It is something that we recommend you hire someone do for you, not because we provide Magento upgrades, but because we know from plenty of experience how much trouble it can be. Whether you hire us or someone else do your upgrade there are a number of things we have found are important to do and consider when preparing for the upgrade:

Upgrading Usually Won’t Fix an Existing Problem

One thing that comes up often with upgrades of Magento, as well as other software, is clients hoping that the upgrade will fix some problem with the website. In most cases the upgrade won’t fix the problem and in some cases the problem can instead cause additional problems during or after the upgrade. Your best bet is to ask the person you are contacting about doing the upgrade if the upgrade will fix the issue and if it won’t, finding out what else they suggest should be done to resolve the issue. In many cases we have dealt with, the fix for the problem requires doing much less work than doing an upgrade.

A Test of The Upgrade is a Must

Almost no Magento upgrade is going to be without issues, this is the biggest reason to hire someone who does Magento upgrade regularly to handle the upgrade as they should be aware of how to handle most issues that come up and will be better equipped to work through issues they haven’t seen before. What is key to making sure the issues that will occur don’t impact the normal function of your website is doing a test of the upgrade first. This allows the issues to be spotted before the production website is upgraded and while you can compare how things were working in the old version.

In addition to dealing with the issues that come up, the test will also allow you to adjust to any changes that have been made between the version of Magento you use now and the version you are upgrading to. We find that many of the things that clients bring up to us during the testing relate to these types of changes.

Test in a Matching Server Environment

To insure you don’t run in to any unexpected problems when the upgrade is applied to the production website you should make sure to do the testing in a server environment that matches the production environment. This can usually be best accomplished by placing the test version of the upgrade in a directory on the website or by using a staging server configured the same as the production server. If you don’t do this you may run into problems. For example, one time we found during the testing that the shipping module that was in use wasn’t working in the new version of Magento due to a PHP module not being enabled on the server. When our client contacted their web host to see about getting it enabled the web host first claimed that the modules was enabled and it ultimately took several days of back and forth for them to finally get it enabled. If this had only been discovered after the production website was upgraded it would have been a big problem.

Test, Test, Test

We can’t emphasize enough the importance of checking over everything in the test before upgrading the production website as it much easier to resolve any issues at this stage of the process than once the production website has been upgraded.

Mention Any Failed Upgrade Attempts

If there has previously been a failed attempt to upgrade Magento it is important to mention that when discussing the upgrade as this can create complications during the upgrade. The biggest of these being that sometimes after the failed upgrade the database is restored over the upgraded database, which then causes the database upgrade portion of the second upgrade attempt to fail when it tries to add new tables to the database that already exist because they were added during the previous upgrade attempt.

Changing Your Theme

When discussing upgrades we are often asked about changing the theme in use along with the Magento upgrade. Our suggestion is to split up the upgrade and the theme change as that makes it easier to deal with any issues that come up during either one. We also suggest doing the upgrade first and then using the test of the upgrade to test out the new theme.

Handling PHP Changes

One of the big changes that came with Magento 1.9.1 is that Magento will not function on versions of PHP below 5.3 (the listed minimum PHP is even higher, PHP 5.4). Some hosting environments making switching PHP versions quite easy, but in some cases it can require moving to a different server so this will be something you will want to discuss with the person handling the upgrade ahead of time. Also, if you are still running Magento 1.3 the process for handling this will be more involved since that version of Magento was not designed for PHP 5.3 or higher.

Clean Up the Database

The Magento database can grow rather large in size due to long term storage of data that doesn’t actually need to be stored on a long term basis. This can negatively impact website performance when interacting with the database and can cause the website to go over its disk usage quota. During the upgrade process is a perfect time to check out if this is the case with your website as cleaning out the excess data can significantly speed up the database upgrade portion of the upgrade and people that handle upgrades are usually familiar with this issue due to that. Due to a change in Magento 1.9.1 that makes sending most emails reliant on the Magento cron job being set and enabled, the person handling the upgrade will need to insure the cron job is set correctly, so most of the work to enable automated log cleaning that takes care of much of the excess data problem going forward is already being done as well.