Recovering data from my broken Terastation Pro II

A few days ago my Buffalo Terastation Pro II showed a strange error message in the webinterface while I wanted to empty the trash (the Terastation has a trashbox for the samba shares). The error message was:

[BufCommonFileShareInfo.save]File Close Error - /var/lock/cgi_check_gate

The error was only affecting the webinterface and I could not issue any configuration changes any longer. But as it was only an error caused by a lock file I thought a reboot of the Terastation might fix the error condition.

Terastation Pro II Rackmount

But I was wrong. I rebootet the Terastation and everything I got was this message on the LCD:

System booting ...

I thought it it might be doing a filesystem check or something like that, but after about 6 hours (which should be more than enough to check 700 GB) I unplugged the power cord to cold-reset the Terastation. Now thing got a bit worse and together with a red light and loud beep the LCD showed:

System Error
E04 Can't load Krnl!

The advise from Buffalo for this error is to reflash the firmware of the Terastion with a newer version. I could have done this, but I don't fully trust the vendor and the device to not loose my data on a firmware update.

It's my fault that I don't had a backup of the data, which was stored on the Terastation but now I had to go another route of recovering the data without flashing the firmware of the Terastation.

Recovery Theory

My plan was to put the drives from the Terastation into another system, boot any Linux distribution on the system and reassemble the RAID on that system to copy the data somewhere else.

I know that the Terastation uses the XFS filesystem and that the RAID is a Linux software raid, but Linux software raid is not endian agnostic, so one important point was to find out if I need a PowerPC system or if an x86 system will be enough.

Older Terastations use a PowerPC cpu and are therefor big-endian systems. You need a PPC system to reassemble the RAID. After a long search I found a post in the Buffalo support forum, where someone wrote, that the newer ARM-based Terastations are little-endian systems and therefor an x86 system should be used to reassemble the RAID.

I have a few x86 system lying around, but either they don't have enough Sata ports or they don't have a CD-Rom/DVD-Rom to boot Linux from CD. So I needed to prepare an USB-Stick to boot Linux on a system where I could attach all four disks at once.

Preparing a bootable Linux USB-Stick on OS X

As I currently don't have access to a Linux Workstation one tricky part was to create a USB-Stick, which will boot Linux, on my OS X Workstation.

Many Linux distributions only distribute ISO images and document how to build a bootable USB-Stick from them. All manuals I read involve syslinux at a point, which I couldn't get to compile on OS X.

After searching for a bit I found that Archlinux has ready-made images for USB-Sticks for their releases. After downloading the image just has to be copied to the Stick block-wise. As a note to myself, here are the steps necessary to dd an image to a USB-Stick on OS X:

That's it, now I have a bootable USB-Stick, which will boot an Archlinux 2009.08, ready to recover the RAID from the Terastation.

Recovering the Data

Having a bootable USB-Stick with Linux and an x86 System with at least 4 Sata ports I could start the recovery of the Terastation RAID.

I attached the disks to the onboard Sata controller and inserted the USB-Stick into one of the onboard USB ports and booted the system. Luckily the BIOS of the onboard Sata controller told me that it detected four disks, so none of the disks was completely bad.

After booting the Archlinux to a shell, I looked at the dmesg output and the kernel had detected the four disks with their partitions and everything looked right.

Using fdisk I double checked that all disks had the right paritions and identified that /dev/sdX6 (with X = a,b,c,d) was the data partition, as it was the biggest one and I only had one big Raid5 Array configured on the Terastation.

Using mdadm I inspected all 4 data partitions with mdadm --examine /dev/sdX6 and it reported that they all belong to a clean Raid5 Array. That meant that the Array does not got damaged and I could reassemble it with mdadm:

mdadm --assemble /dev/md0 /dev/sda6 /dev/sdb6 /dev/sdc6 /dev/sdd6

And after that mounting the Array, which is formatted with an XFS filesystem, was also not a problem:

mkdir /mnt/raid
mount -t xfs /dev/md0 /mnt/raid

At this point it was only a matter of copying the data to another disk or system.

Conclusion

Most resources on the Web about how to recover broken Terastations only cover the older PPC-based systems, and it was a bit hard to find out which endianess my Terastation uses. After I found that missing piece of information it was pretty straight forward to recovery the data. Neither the RAID nor the XFS filesystem was damaged, only the Firmware was broken. In the end it took longer to find a suitable Linux image which was USB-Stick ready than to reassemble the Array.

Now that I have recovered the data the step next will be to reflash the firmware, hoping that it this will fix the problem so that I could sell the Terastation.

The Terastation has no grafical output and no serial-console (the RS-232 port only exists to attach a UPS) and the lesson I learned while troubleshooting the broken Terastation was that it is very hard to fix a problem, if you cannot see what the problem actually is, so most propably I will sell the Terastation as I do not want to go through this trouble again.

Veröffentlicht von Arne Brodowski am 22. Sept. 2009, 12:15 in , , , , , , , .

Verwandte Einträge

Kommentare