Thursday, December 8, 2016
We are writing to inform you that we appear to have been alerted to several drives in a failing state on node18 in New York, NY, and will be taking it offline for at least 24 hours to rebuild the disk array.
We apologize for the inconvenience but feel that this is the best method to preclude data loss by reducing read/write stress on the array.
If you would like to reduce your downtime, we can re-create your VPS in the same location with your same IP on a different server. Please open a ticket if you desire this. Note that this new VPS would indeed be empty, but it will be online and available for your use immediately.
Again we apologize for the short notice on this necessary outage. Please don't hesitate to open a ticket if you have any questions or inquiries regarding the situation. Thank you for your time.
------------------------------
Update:
The server had 3 failing drives. Two of them were in one span of the RAID50 array and one of them was on the other span. We were able to successfully save one of the failing disks in the span by cloning it to a new, healthy disk, sector by sector. This process took around 18 hours.
Afterwards, we were left with only one failed disk per span. These disks have also been replaced with new, healthy ones. At this time the RAID array is being rebuilt. We are anticipating this would take another 30 - 35 hours since these are 5TB disks. We are keeping the VMs offline at this time and we are rebuilding one drive at a time to reduce the stress on the other disks.
We are doing everything in our power to prevent data loss on this node. Your patience is very much appreciated while we work through this delicate matter. With the help of some luck, we are anticipating your VM will be online by Monday.
We sincerely apologize for the inconvenience caused by this hardware problem on the node.
Update:
As you may know from our last email, we have experienced a multi-disk failure on the hardware node of your virtual server. The node was running a RAID50 which can protect against multiple disk failures as long as they are on different spans, unfortunately we experienced additional disk failure while rebuilding the arrays. This new failure happened on the same span.
In summary, we originally had 3 failing drives in total out of 12 drives. One of them was in one span and then two of them in the other span. Then we have immediately taken the server offline to avoid data corruption. Then cloned one of the failing drives to a brand new drive, and installed it on the server. At this time we had one failing drive in each span. Later we replaced both of the failing drives with brand new drives, and started the array rebuilding process on the first span. Very unfortunately, at 86% of the rebuild process, another disk has failed on the same span. Therefore the array became in failed state and even after 48 hours of trying various methods, we couldn't get it back online without data corruption.
Unfortunately it's our conclusion that the data is unrecoverable. Because of this, we are going to have to recreate your VPS on a new node preserving your current IP.
To avoid future reliability and performance issues, we are replacing the node that your VPS was on with a newer generation node that has faster CPUs, much more memory, faster drives, and with a better RAID controller. Also to provide more reliability we are going to use RAID60. As a gesture of goodwill we will be issuing the replacement services with an extra 1GB of RAM (totalling 1.5GB), and will credit two months worth of service.
We understand if this is unacceptable and if you wish to discontinue service with us. In that case we will issue a refund of the past two months worth of payments. If you would like to opt for the refund, just open a ticket to let us know. Otherwise we will have the newer, faster and more reliable services all setup by Friday, December 16th.
We will email you once your VPS is setup and ready to go.
We are terribly sorry for this misfortunate event and we sincerely appreciate your understanding and continued business.
Should you have any questions or comments regarding this, please get in touch with us by submitting a support ticket in your client area.
Update:
We are writing to inform you that further to our previous communication we have successfully re-created your virtual machine on our node18 replacement.
The node18 replacement is a brand new server with the latest technology for the utmost in stability, utilizing new enterprise hard drives in RAID 60. We are exceptionally pleased with the performance of this new server and we are confident that you will appreciate it too.
Your virtual machine has been re-provisioned with the last OS that you installed on your virtual machine. Your virtual machine's IP remains the same, and we have sent you an email entitled "New Backupsy VPS - Provisioning Started" that contains your temporary root password. Please change this password as soon as possible.
If your previous virtual machine had the OS installed manually, you will need to complete this again using the console we provide in the client area. The ISO for the latest version of your selected OS has been mounted and is waiting for you to proceed with the installation.
If you have purchased a "Configure it for me" addon or a R1Soft license with installation service, a new ticket has been opened to you automatically. We'll be contacting you very soon.
As promised in our prior message, a credit equivalent to two months' worth of service has been applied to your account as compensation for this issue. Additionally, all virtual machines on this node have been provisioned with a default allocation of 1.5 gigabytes of RAM instead of the usual 512 megabytes, at no extra charge.
We truly apologize for any inconvenience you may have suffered as a result of this issue. We are committed to providing you the best service at a competitive price, and we look forward to continuing provide you with upcoming years of excellent service. If you have any questions or comments, please don't hesitate to get in touch with us via our ticket system at any time and we'll be happy to field your input.
Login
Register
Forgot Password?