Outage: hsvz16.dal

[matta]

Today a drive in the hsvz16.dal.tektonic.net's RAID-10 array went bad. We asked our datacenter to swap out the bad drive with a new one. What happened next is that the ports were mis-labeled by the datacenter and they swapped out the other drive of the RAID-1 subunit, causing a server crash.

We are attempting what we can to salvage the data and get hsvz16.dal.tektonic.net back up and operational.

UPDATE: It was the SuperMicro drive backplane that had the wires twisted. This will be checked on all setups in the future.

[psalzman]

The backplane had wires twisted and this caused the harddrive to short? Or did the drive fail and someone ignore which one had an amber light and which one was green?

I assume the RAID-10 volume is fine, but what was on the RAID-1 stripe that was lost?

[matta]

This setup (3Ware 9550SX) doesn't do an amber led, but it does do drive activity and that should have looked for (the one that isn't blinking).

The wires were crossed inside the chassis and that caused the confusion in the mapping of ports.

1 RAID-1 set of a RAID-10 set is completely gone which has destroyed the RAID-0 stripe. Even putting all the drives back in exactly as they were lists it as degraded and unusable.

[psalzman]

Crap, OK, I misread RAID-1 for RAID-0. I see what you're talking about now. I'm not familiar with the 9550SX cards, but I've found in the past with some Dell Perc and Adaptec you occasionally have to do a power cycle to get them to find the valid information w/ the failed drive removed. It'll be degraded until its finished rebuilding, but not corrupted.

I just performed a server-side backup a few days ago, but I'm guessing those were stored on the local disks too?

[matta]

We are performing a lot of different configurations to try to find one that will boot. If we don't within 15-20 minutes we've exhausted what we can try and everything is gone. We will then at least attempt to have the VPS's re-created as quickly as possible so those with backups can start restoring.

Server side backups are stored on the server, yes. This is the primary difference between the server-side and client-side backups in the HSPc CP.

To those who do not have a backup strategy in place please contact us and we can discuss various options and help you to develop your own backup strategy for any possible future issues similar to this.

[matta]

All VPS's have been re-created based off the information contained in HSPComplete. If you cannot access your VPS please contact support to determine the cause. We are very sorry this this problem.

[apn3a]

Hello
You mean we lost everything? I can't access virtuosso, plesk and all my sites are dead!

[gluek]

Originally Posted by apn3a

Hello
You mean we lost everything? I can't access virtuosso, plesk and all my sites are dead!

My Control Panel work, but ssh password wasn't accepted. And restore of backup procedure doen't work!

[matta]

Regarding access to the server the root passwords may not be what you think they are. If you contact our support via support@tektonic.net or Live Chat @ www.tektonic.net they can reset your root password.

HSPComplete is still showing the server-side backups are being there -- this is a database issue only, the actual backups are not there and cannot be restored.

[apn3a]

Originally Posted by matta

Regarding access to the server the root passwords may not be what you think they are. If you contact our support via support@tektonic.net or Live Chat @ www.tektonic.net they can reset your root password.

HSPComplete is still showing the server-side backups are being there -- this is a database issue only, the actual backups are not there and cannot be restored.

I did already contact (twice live chat) and 2 emails and still i'm in waiting list in order to know my password. Plesk not installed yet as well. I have a DEAD forum with 13.000 member because of YOUR fault and not the opportunity to setup a domain with a index page to inform them