Reflections on Internet security after an eventful week.
· 5 min read
This turned out to be an hell of a week for internet security. It is one of those things where we look for patterns in co-incidence. Till now internet security was only of a passing concern for me because we never thought we were big enough (yet) to be attacked. But it turned out, I was wrong.
We got hacked.
It was a crude attack. The hacker used a standard php (php is the technology that powers a lot of standard applications like forums, website management tools etc) exploit to enter in our system and tried to steal some data and damage our site. Thankfully what saved us was that since we have built our own infrastructure, the hacker was not able to understand our architecture (apparently he did not checkout our code, or did not understand it) so he (I am assuming its a he, but could be a she as well) ended up causing limited damage. Not to say that the hacker failed. We are pretty sure he stole some data, but thankfully he was not able to locate client data so we got lucky. By the time he came back for more, we had already plugged the hole.
The irony is that we do not use php, but for some reason, one of our past team members had installed phpMyAdmin (a utility to browse databases) and we had deleted that application, but forgot to switch off the php module.
So who was this hacker and what was the intent?
Here is my theory, it seems that the hacker was looking for credit card information, since we sell something online or was looking for data he could sell, like verified email address (that can be spammed) or he was looking to install malware on our server that he can then use for other purposes. We identified at least four different IP address the hacker used. They were from Khakhastan, Egypt, Pakistan and China. I am sure these are proxies and some of them belong to Internet Service Providers (ISPs) that use NAT (a technology where they assign IPs dynamically to their customers, so its hard to say which customer was using it unless you have their logs).
Just to clarify, we do not store any credit card information of our customers or their passwords, most our transactions happen via PayPal and they have the credit card information. Each customer database is separate and all private data is stored separately on individual databases.
Based on what we saw, we believe that the hacker was most likely Indian, because the intrusion script that he used, had some Indian names. We need to find out the source of that intrusion script and maybe then we can be able to locate more clearly where this hacker came from. When we searched on the net, this exploit was used to hack many popular web applications and such compromises happen routinely.
Either ways, this incident came as a wake up call to us. We had always heard that a counter culture exists that thinks its cool to attack companies and governments, but this was a first hand encounter. We will now take security much more seriously.
The second event of note this week was that Hacker News (the #1 forum for technology) was full stories regarding a startling leak that the United States Government is engaged in widespread surveillance of mostly non-US and US citizens through major services like telcos, Google, Facebook and others. A wild debate raged on civil liberties and whether the US government has crossed some kind of a line and if this was true, then there was no such thing privacy on the internet.
They way it works is this, if you have been “black listed” as a potential harm to US interests, the US security agency is going to go after all your data. For non-US citizens using US services like Google and Facebook, such requests do not even require basic scrutiny. If you are a US citizen, does not mean they will allow your privacy, but there are some checks and balances in place to minimize misuse. So by your bad luck, or bad chance you exchanged some emails with some black listed accounts, you are probably under surveillance.
So what does this mean for us, specially non-US users? We use Google services all the time and GMail is particularly the weak link. I think in the near future, it should become pertinent for all of us to use a variety of services for mail rather than just GMail. We need to setup our own email boxes with 3rd party backups so that we do not handover full control of our lives to Google. I think our mailboxes are our most critical digital assets and we should be taking better care of them. This may sound paranoid, but is at least worthy of a thought.
Getting Serious about Security
This brings me to the third incident this week. Since we won a prize few weeks back by creating a better visualization of Indian Government data, we got in touch with a few of the engineers behind the open data initiative (data.gov.in). The first thing a users sees when going on the site is that the SSL certificate is not verified by the browser. When I asked the engineers about this, they told me that Mozilla and Chrome does not identify the Indian government agency as a valid certifying agency and they told that people like me should be lobbying.
Now I felt this to be a bit odd because Mozilla is a well respected non-profit and there should not be any such trouble. I did a few web searches and realized that there is a procedure that needs to be followed to apply as a certifying agency with Mozilla. So if the government is serious about this, it should follow the steps.
This also shows that the Indian government is not really serious about web security. Here we are talking about setting up our own mail servers, and thousands of Government officials use Google and Microsoft email accounts to transact official work. This is ridiculous and its time the Indian government wakes up to take charge of its own digital assets and sets its house in order.
This week I got a clearer idea of what internet security means. Digital assets are just like physical assets and are prone to theft and damage. If you let your physical assets be stolen, most likely the will be. We usually take commonsense precautions why protecting physical assets, and the same has to be with digital assets. If you are storing digital assets for someone else, you are more likely to be attacked and hence you need to be even more cautious. Some basic precautions are:
- Maintaining a minimal and secure server setup. The more services that you run, the more likely they are to be compromised.
- Adding multiple layers of security, from the operating system, to inside the application.
- Creating self-contained architectures so that you have to rely on minimum number 3rd party services.
- Finally, the hardest one, using commonsense to ensure there are no gaping holes for anyone to exploit.
In the end, this is a continuous process of learning and adaption. But as we grow, it is likely to be a critical part of our service and should become a reason why customers would want us to manage their digital assets.
Rushabh is a software developer and founder of ERPNext. He usually writes about the startup experience, open source and the technologies he is working on.
Interesting post, although sorry for the fact that you'll had to go through this experience to wr