No News is....er, Good News?

[manyagem]

>> We're hoping this forum will become much more active and you'll be able to get all the fun and excitement you need from our forums. <<

That's a quote I found from Pat Fread whilst looking (for the first time) through these forum posts. Well, the past weekend brought a little more excitement and not so much fun as I care for. Since this is the first opportunity I've had to share it with anyone, well why not here?

I've had at least one website hosted here for the past 5 years or more, first with a reseller who let me down badly then with another reseller who looked after me with great care. When he decided to chuck it in, the simplest way forward was for me to become a reseller myself. Then I started taking on a few friends and one or two clients and hosting two or three e-shops for myself.

It all went remarkably well. Even the ownership changes from VO to Data393 to WebHSP passed me by with little more than a curious comment from my accountant.

Then I woke up on Saturday 12th and called up one of my e-shops. Shock, horror, it's been hacked! I called up another one of my own sites: same potted diatribe on Turkish history, muslim resentments and backlash against Denmark. And, even worse, it was the same with my clients' sites.

You know that sinking feeling you get when it first dawns on you: Oh Sh*t, what have I done wrong? Who's found my password? What else have they found, and what can they do to me next? Yep, that's just how it felt. Then, after a few minutes panic, you start groping around for answers. Where do I find support? Heck, where's that password for the support desk login? How can I get hold of it when there's no e-mail coming in? Luckily, a colleague also has reseller hosting with WebHSP so I called him up (at 7.0am Sat morning) and told him the good news - we may have a problem, or it may be just me, ....please tell me it's not just me! But he couldn't. His sites are on a different server so they were unaffected. But he did at least login to support and find some sort of hint that things weren't running smoothly for megaman. What you need in these situations is to get that slightly warmer fuzzy glow of knowing you aren't alone and that however bad it looks, others are in the same boat, so someone must be on the case. I don't often ring the US from here, especially when I know most good people will be asleep, but this was an emergency. I looked it up and rang the help number at WebHSP. I got confirmation, the hack attack was general, it affected most Index pages on megaman. Well, that's some reassurance at least. Now, what to do?

Fortunately, at that time I was still able to connect to most sites and overwrite the defacement. So, at least my clients wouldn't be on my back. But by the time I got to my own domain, all the space quotas had gone through the roof and it wouldn't accept any more uploads. Worse than that, a login to cpanel showed the same Turkish hack page so I couldn't even do much to suspend accounts till the storm died down. Oh well, just have to sit back and put faith in those guys across the Pond to put things right.

And you did! There were a few ups and downs, mail was erratic, I tried joining this forum but didn't get the authorisation e-mail till now (Mon morning). But hey, here I am. And everything more or less back to normal. I've had my own share of tech support nightmares in a previous career, so I can imagine the sort of turmoil you must have been going through trying to restore everyone back to normality. Well done, guys!

Now then, having come this far in joining your community, what's the reassurance it won't happen again? And, if I can offer just one small word of advice: how about a newsflash posted somewhere prominent on the main site just to reassure people like me that you know what's wrong and you are on the case for recovery. Thanks for listening.

John Kerr

[WHSP-Jarrod]

Hi John,

Firstly, thanks very much for taking the time to detail your experiences with WebHSP, even if they aren't the best ones, we appreciate any and all feedback sent our way.

As for what exactly happened, it looks like a hacker or "script kiddie" took advantage of the Megaman server using a Kernel exploit, and ran only one script, to change the contents of any page with the strings index, home or main in it's name to display the hacked message.  Thankfully, the Data Center was on the case shortly after the reports started coming in and patched the Kernel on Megaman, then began the lengthy restoration process.

In total, the System Engineers found around 30,000 files that were either modified with this hacker message or new files that were created in directories without an index page.  These should all be cleaned up now, however we're still running checks to see if we can find any others that may have been missed in the restoration process.

Also worth mentioning, some resellers on Megaman may have received disk space quota warnings while the restoration was going on.  This was due to the file compare/restore script that the System Engineers had been running, and should now be corrected.  The backup data that was not restored to the various sites on Megaman has been removed, returning disk space quotas on all accounts back to normal.

As things stand now, Megaman should be back to normal and all of our servers have had their Kernel patched so that they can not be similarly exploited.  All of us at WebHSP do sincerely apologize for any inconvenience experienced by our resellers and their customers from this exploit.  We are continuing to work with the Data Center to ensure that Megaman (and all the rest of our servers, for that matter) is running all updated, upgraded and secure software, so that other exploits such as this won't happen in the future.

If you have any further questions about what happened this weekend, or if there is anything else we can do for you at all, please just let us know.

Jarrod,
WebHSP Support Team

[Pat]

Hi John,

I just thought I would add my comments.  Yes, it was rather an uncomfortable and stressful weekend for all of us.  The scope of the issue was large and impacted everyone on the server.    

Although all of the other services were running normally, the process to re-secure the server created a couple of short outages.  Also, because of the added backup files, many accounts went over quota, which would impact email.  

In an attempt to address the over quota issue as quickly as possible, we doubled all resellers and accounts over quota temporarily.   That process took some time but was quicker than waiting for all of the files to be restored and the backups deleted off the server.

We will go through the accounts and change those quotas back to their original setting by the end of this week.

All of these activities were posted in our announcements section, which John personally kept updated throughout the weekend.  

Keeping customers informed is very important to us and we realize the importance for our resellers getting the information so they can communicate with their customers.  If you have not signed up to receive announcements, you can do so through your Customer Control Center (CCC) at https://ccc.webhsp.com and the first post will be emailed to you with the link to the post so that you can directly link to the post as needed.  We realize that in some cases email can be impacted. You can also link to the announcements through your CCC.

Keeping our servers secure is very important.  Our kernels are updated within a short time of a security release and the kernel on the megaman server had recently been updated within the last several weeks. However, we will be working with our team to review the chain of events to ensure that all processes and procedures for keeping the servers secure were followed and/or if there are other measures that need to be added to ensure we continue to provide the quality of service expected of our customers.

Thank you all for your patience during this unfortunate event and we hope you all have a good week.

Pat

[manyagem]

Thanks for the replies. I feel like I'm in safe hands - even tho there's an ocean between us.

John K

[WHSP-Mark M]

Thanks for the replies. I feel like I'm in safe hands - even tho there's an ocean between us.

John K

[snapback]419[/snapback]

Hi John,

You certainly are .If you should ever need us, we are always around.

Wishing you all the best.

Regards,
Mark