Ihelm Enterprises Limited - Website Server Status

System Status Log - Current Status - DORA is online - MARVIN is online

23/02/2008 - 12:30pm - Dora is now working properly again. There may be a few intermittent issues but these will be looked at this evening.

23/02/2008 - 9:00am - Dora is having problems sending e-mails - this has occurred after a recent software upgrade. We are currently looking into it and will have it fixed as soon as possible.

04/06/2008 - 11:30am - All websites should now be back and able to be seen.

04/06/2008 - 10:40am - There now appears to be a few network issues as the server is online but cannot be accessed via the web. We are currently waiting to speak to a support technician to find out what the problem is and when it will be resolved.

04/06/2008 - 7:00am - HAL (ns3.ihelm-enterprises.co.uk and ns4.ihelm-enterprises.co.uk) came back online late last night GMT time. We began to initiate our plans for preventing our customers from suffering such an outage again, however, at 3am we decided to halt work until today. If you notice that there is limited access to your website and other services, it is because we are starting to initiate the plans again. If you notice that your website is broken - please do not panic - you will be receiving an e-mail from us shortly and your website will be back up and running as quickly as possible. We are updating our server page - www.ihelm-enterprises.co.uk/serverstatus.php - with continued updates. We appreciate your cooperation and understanding during this difficult time.

03/06/2008 - 12:15am - HAL is now back online. Due to the temporary nature of the solution (a new temporary generator is being used), we are not currently allowing full access to the server. This will enable us to take up to date backups of all sites and data in case of further outages.

03/06/2008 - 12:58pm - Around 2:20 AM CDT, the backup generator being used to power H1 Phase | experienced an electrical issue resulting in service loss for Phase I; Phase II remains unaffected at this time. Our data center operations and facilities teams immediately began investigating the cause of the failure to restore power to the Computer Room Air Conditioner (CRAC) units and Power Distribution Units (PDUs) for Phase I.

The staff successfully tested the 2 megawatt generator without load, so they began powering up the CRAC units and PDUs to restore service to Phase I. While working through this power restoration, the generator's breakers were tripped by their internal electronics. The generator is rated to handle more than the load required to power the phase, and the generator itself is fully functional, but the breaker system must be replaced to guarantee stable power distribution.

We have attempted to locate a replacement generator and are evaluating the time necessary to repair the breakers on the current generator so we can restore power as quickly as possible. We do not have an ETA for power restoration, but we will be updating you hourly with our current status or sooner, as developments warrant.

03/06/2008 - 8:34am - Due to an issue with one of the backup generators, there is inconsistent power distribution to some key components that are fundamental to server racks, so customers may note some downtime currently. The company's data center operations and facilities teams checking the generators, CRACs, PDUs and racks to restore connectivity.

02/06/2008 - 9:33pm - HAL is now back online and functioning. At present all systems appear to be working with no problems. The power is a temporary fix. We will inform all of our client's by e-mail and this page when we have information of any planned outages for remedial work to be done. We thank you for your patience.

02/06/2008 - 12:43 pm - They are continuing to power on servers andit is looking likely our server is in the more damaged area. However they hope to have power on later today to those servers still. There are still some of the original 6000 that are not powered on but we do not want to raise expectations falsely.

02/06/2008 - 7:20am - The company are continuing to work around the clock to restore power to all parts of the building. They are going to be powering up 2/3rd (6000) of the servers today, however 1/3 (3000) of the servers are in a more damaged part of the building and are not likely to have power until tomorrow and are expected to suffer further outages due to work required. We do not know which part of the building our server is physically located at this time.

01/06/2008 10:59pm - The company is planning to start tests of their air conditioning system and other internal systems on some form of power shortly and depending upon the results of those tests, which will last approximately 4 hours, we should be able to update you with further information. We are unlikely to update this page until the early hours of Monday morning due to our staff signing off for the night and the fact that the tests will take approximately 4 hours.

01/06/2008 6:40pm - The damage to the data center's power utility is extensive and there is no estimated time from the power companies due to the damage. As a result we are still unable to give you any further information. We are getting updates hourly and we will pass on any significant information. We thank you for your patience.

01/06/2008 1:32pm - The server provider has started the recovery and installation of electrical equipment into the data center.

01/06/2008 - We received the e-mail below from our server provider for HAL. We will update the site as we get more information.

Dear Valued Customers:

This evening at 4:55 in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and apologize for the impact.

Sincerely,

Douglas J. Erwin
Chairman & Chief Executive Officer

25th May 2007 - HAL is now fully working after a 2nd new hard-drive was installed on Tuesday night.

25th April 2007 - All Eddie Customers were transferred to our new server DORA in January and there haven't been any problems. Eddie was fully retired in February 07.

6th December - 13:38pm - Full access is restored to Hal. At this time it appears to have been a routing issue.

6th December - 13:30pm - The AOL issues that we were experiencing on Monday appear to have been fixed. We are experiencing connectivity issues to Hal.

4th December - 9:00pm - We are experiencing intermittent problems with e-mails being sent to AOL. We have contacted AOL and it is in the process of being resolved.

16th August - 8:40pm - Hal is now fully working. Problem with a bind dns server which is a rare problem and a software update has fixed the problem.

16th August - 4:10pm - Hal unexpectedly went down. We are currently trying to fix this and will keep everyone posted.

10th August - 8:15am - Eddie is now completely back online, however there may be a bit of a delay whilst e-mails are processed. We expect this delay to last about 20 minutes. Unfortunately the cause of the error has not been found - however we will keep our eye on it.

10th August - discovered at 8am that Eddie (one of our servers) had stopped responding a couple of hours earlier - we are still working to resolve this issue.

*site upgrade Ihelm Enterprises is upgrading its site over the next few hours - customer services will not be effected. 6th August 2006

*resolved* Database is now up and running

The mysql database on our servier eddie went down last night during the nightly back-up - we are working on determining the cause of the fault, at the moment it looks like some of the tables are corrupted - more updates will be posted soon.

16th February 2006 - the memory upgrade has been completed and the server is now back up and running. Total downtime was 11 minutes.

16th February 2006 - we would like to inform all of our customers that are hosted on server eddie that the memory is being upgraded over the next 24 hours and you may experience a period of downtime of no longer than 20 minutes. We would remind all customers to ensure that they have a current backup of their website.

15th March 2005 - we now have anti-phishing programs installed - this will attempt to prevent emails pretending to be banks from being delivered to our customers

14th March 2005 - mail server was down for 10 minutes while anti-virus and anti-spam technologies were updated - all went fine no mail lost and everything is working.

14th March 2005 - spam e-mails are no longer hitting our servers this incident is closed

Over the weekend 11th march to 12th march we suffered a massive influx of spam - the server has processed over 200,000 emails. We have been in touch with the relevant people and this has decreased - we suffered delays in emails for approximately 12 hours until the queue had been processed.