Today we're excited to announce that we've added a black belt to our cyber defenses: BitNinja.
Compared to many of our competitors, our hosting is already much more secure. We were one of the first hosts to introduce account isolation technology and we have a heap of other custom security systems under the hood to keep customers' web sites safe. One of the most groundbreaking enhancements we made to server security was the introduction of our Patchman vulnerability and malware scanner, which has now prevented thousands of web sites from being hacked.
However, in spite of the robust security, with the increasing rate and intelligence of attacks, some of our customers sites that ran on vulnerable code or out of date software, were nonetheless being compromised. The reason for this is largely due to the growing sophistication of the attacks, making them increasingly difficult to detect.
To understand how this works, it's important to understand the nature of a hacker.
Know your Enemy
Dispel from your mind any notion of a human hacker spending time to intentionally target your web site. For the majority of hackers, their motivation is money, and their business is all about volume. The more e-mail they can harvest, the more spam they can spread, the more sites they can compromise, the more money they can earn. To this end, almost all hacks are automated by bots. In fact, over 30% of the traffic to our servers are from 'bad bots' - pieces of software designed to probe your web site for vulnerabilities and attempt to embed malicious software.Now for many years our servers have run firewalls which are capable of detecting and blocking attacks from many 'bad bots'. If we saw a large number of requests coming in from a bad bot, that IP would be blocked automatically from our entire fleet of servers. Ding ding, round one to us.
But the fight is not nearly over. Hackers are smart. Launching an attack from a single source that generates thousands of requests to our server is easy for us to identify. To avoid easy detection, the hacker had to evolve. With countless bots under their control, rather than try and attack from just one source, they'd instead use thousands of compromised computers, all controlled from a 'command and control' centre. Each of those bots may only make one or two attempts to hack your site before handing over to a different bot, in a different part of the world. The speed of the hack attempts, whether from a single source or from a dispersed source can have a devastating effect on an unprotected server.
The graph below shows the load spikes on a server under such attacks:-
Each load spike can cause slowness or potentially outages if the attack is severe enough. Fortunately we can usually detect many of these attacks, as more often than not, though dispersed geographically, the requests all come in very quickly and our firewall can act to block the attack. Unfortunately if the attacks are very dispersed, it can be difficult to differentiate between a human visitor and a bot, and so sometimes the only option is to disable the page for a few minutes until the bot attack subsides. Though effective, this is of course undesirable and is a simple act of last resort.
More recently, we've observed a new evolution - now the attacks are often time delayed, so the command and control centre will permit each bot to make just a handful of connections, it will then pause the attack before launching a few more connections with a different bot. With only a couple of requests coming in from each bot, over a long period of time, that's now significantly harder for us to identify. These kind of attacks are often absolutely indistinguishable from genuine human visitors, but over time, little by little, the hacker will be attempting to gain access to your site. Round two to the hacker.
Thinking like a hacker.
To overcome this, we have to think like the hacker, and increase our own arsenal. Our latest weapon is called BitNinja, which is like our own anti-hacker command and control system. Once operational, BitNinja inspects connections to our servers, looking out for obvious malicious activity, verifying whether any bots accessing the server are 'good bots' or 'bad bots', and logging all suspicious incident in a central database. This analysis is done in an intelligent way with lots of checks and balances that help ensure there are virtually zero false positives.Once suspicious activity is identified, the IP address is added to a 'greylist' which identifies it as potentially harmful. The log of the suspicious activity is recorded and the type of attack is categorised. We can identify with around 99.99% confidence that a greylisted IP is harmful, and so to prevent malicious activity reaching the site, any subsequent connections from a greylisted IP are presented with a captcha page. This concept of 'greylisting' is what makes BitNinja particularly effective. Traditionally, a firewall will only ban or allow an IP address from reaching the server. The problem therein, is that if there is even the smallest chance of a false positive, and the visitor was human, they will no longer be able to access the web site. Instead, greylisted IPs are presented with a 'captcha' page which allows them to instantly unblock their IP address, which looks as follows:-
A bot will typically not be able to complete the captcha, but just in case, the captcha page is also laced with tempting hidden links and 'honeypots' that only bad bots would follow or try to exploit. The use of a captcha allows us to ensure that human visitors can continue to access the site, whilst also allowing us to build more information on the reputation of the IP should it get tempted by our honeypots. All the information collected is processed centrally and shared across all our servers in real time. This data is also shared more widely with thousands of other servers at hundreds of other web hosts, as their data is shared with us. This collective sharing of IP reputation data allows us to build a very clear idea of malicious IPs.
Let's look at the following example:-
This log extract shows an IP address which attempted to perform a brute force login attack on a WordPress installation. This attack was first detected on our server on November 19th, and the IP was added to BitNinja's global greylist. The IP is then unable to connect to the WordPress login page on any BitNinja protected server. Over the course of the next 9 days, the captcha page that appeared on BitNinja protected servers recorded subsequent attacks directed at WordPress and other types of CMS, totalling 573 attempted attacks.
All of our hosted web sites are thereby protected from the attack, and we've been able to build a great picture on just how malicious this IP is. Even if the bot connects only a few times to our servers, BitNinja will be able to see the bigger picture and will know that the IP is connecting maliciously to many servers.
Once BitNinja collects enough data, it will report the attack automatically to the network administrator of the IP address so that they may have an opportunity to cleanse the bot - most likely it was placed there by the hacker without the knowledge of owner of the network or server.
Eventually, if the attacks continue, the IP will be completely blacklisted so that it won't even be able to connect to BitNinja protected servers whatsoever. The IP will periodically be added back to the greylist so that we can verify whether it's still malicious, and will be removed if the attacks have stopped, or added back to the blacklist if they continue.
BitNinja has millions upon millions of known, bad bots in its database. This means that your web site is now protected from a huge range of threats and is far less likely to be hacked.
What about false positives?
As with any security system, there is the chance of a false positive, though our evidence to date shows that this is in less than 0.01% of cases. We have introduced BitNinja very slowly across our fleet of servers, starting with our own test servers, and moving onto production servers. In the time that it's been introduced, we've seen close to zero false positives. The cases that have been reported to us have generally fallen into three categories:- Cases where customers are connecting using a greylisted IP address which was previously controlled by a bad bot. Thanks to the historical logging on BitNinja, we can easily see that the IP has in the past been maliciously connecting. BitNinja will always remove greylisted IPs if they show no further signs of malicious activity, so typically this activity would needed to have been very recent. Most commonly this could occur on shared wifi networks where multiple computers may access the internet, of which one may be infected, or on mobile phones where the IP address a device is assigned changes frequently, and may have been used on an infected device in the recent past. In either case, the user can easily de-list by completing the captcha form and they will be able to immediately browse the site.
- Cases where customers devices are actually infected. Your customers computer or device may actually be infected with a bot. They will be able to de-list their IP, but they may find that they are greylisted again if BitNinja detects more harmful activity. They may need to run a malware or virus scan on their computer.
- Cases where we've incorrectly determined malicious activity. Such scenarios will be few and far between - and in fact we've only seen one such case. Our systems have already been adjusted to account for the single false positive we've experienced to date, but should this occur, your customer will still be able to manually de-list by completing the captcha form. If they let you know we can also take measures to ensure this doesn't happen again.
It's important to remember that it will be extremely rare that your customer will ever see the captcha page, but if it does happen, it provides them with a method of de-listing immediately. By contrast, other security systems cannot do this and will simply block the IP address all together.
The net result of implementing BitNinja has meant that we are now seeing near to a 100% elimination of hacked sites and spam originating from our servers. What's more, our servers are operating with greater stability, as they're no longer having to deal with the myriad of problems that result from bot scans and attacks.
If your site is hosted with Kualo on a shared or reseller account, the risk of it being hacked has now dramatically decreased. If you're not hosting with Kualo, and have had problems with hacked sites - why not take a look at one of our hosting plans - we'd love to welcome you!
Remember, this is a war that constantly evolves. We will continue to improve our security precautions, but always ensure that you also do your part by keeping your installed applications up to date, just in case.
If you have any questions about BitNinja, please hit up the comments below or contact us directly via our helpdesk. BitNinja can also be added to any virtual or dedicated server, please contact us for more details.
