[SOLVED] node18 outage

Saturday, January 30, 2016

We have seen issues with one of the arrays on this node. We are currently running an fsck on the filesystem.

Update: Disk has been replaced and the array is rebuilding. There is a possiblity of corruption

01 February 2016 | 3:00 PM MST:

Unfortunately, there was a failing disk that was showing healthy until recently and was silently corrupting the data. We were able to detect this disk after running through lots of options. Once we found it, we have immediately replaced it and rebuilt the raid array again.

We are currently running fsck on the volumes. As you may know fsck is a lengthy process which will take hours on big raid arrays like deployed on our servers. Once the fsck process is done, we will have more data about the current situation of the raid array.

We expect some data corruption. We are fairly sure that most of the VPS's will come back online without much issue. However, our previous data shows that there would be some fully corrupted VPS's. Please prepare yourself for some data loss. Unfortunately, we will not know which VPS are corrupted until the fsck is complete.

If you require a new VPS to backup your data and don't need the contents of your current VPS, please reply and we'll create it in another node/location to replace this one to get you back online sooner.

We are sorry for the inconvenience this issue has caused.