ZFS data recovery specialists
We can recover data pools from all ZFS-based file systems, from all levels of RAIDZ using single, double, and triple parity. Multiple and single pools with any number of Vdevs.
Failure modes
- Drive failure
- Resilver failure
- Controller failure
- Deleted pools
- Deleted Vdev
- Overwritten Vdev
RAID levels
- Non RAID
- RAIDz
- RAIDz1
- RAIDz2
- RAIDz3
- Multiple pools
What is ZFS
The ZFS file system is developed by Sun Microsystems and has built-in protection for data corruption and supports data compression. It also has built-in automatic repair features, snapshots, and other advanced file system management. However, when there are failures, the system will stop working. Querying using the console or terminal can help identify the failure.
If a pool is showing errors after a status command, for example;
# zpool status -v
pool: tank
state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
The status output shows there are physical problems with the drives, bad enough to take the pool offline. It may be something simple like faulty power or data connections, so it’s worth checking that first.
When hardware errors like the above occur, we do not recommend forcing the drives back online followed by resilvering, as it is unlikely to resolve the issues with failed drives. For example, if one drive went offline before the latest failure, forcing the failed drives back online can cause serious file system damage.
ZFS recovery example
A recent job we did was for a large organisation that had set up a new ZFS pool with RAIDZ1 by running a script to create the pool and numerous Vdevs. On completion, it was put it into service. Then, some users started complaining they could not access their data, they realised that the script had been run on the wrong pool and had removed several hundred Vdevs containing vital data. They took the pool offline, and tried rolling back. At this point, one of the drives failed, so they replaced that and resilvered the pool, only to find that another drive had failed some time before the current failure. A check of the backups revealed there were no recent ones available. Talk about bad luck!
So, they contacted us and sent in the drives. After making two sets of clones, we could see 1 pool and some Vdevs, but these were the new ones and had in total about 15 GB data already stored on them during the time the pool was online. We could see that the data was still present on the drives, so we set about modifying our software and scripts to tease the data out. This was done by identifying which Vdev each block belonged to. Using our scripts, we were able to extract the data and restore folder and file structures, mainly intact, and we managed to recover, in total, over 95% of the data.
The customer was very pleased with this result, as they were expecting nothing to be recovered at all. Extraction of the data was a long and tedious task, but well worth it for the success we had.