We recently had to debug a zfs related crash, and to recover the data, we had to roll back a few transaction groups. In addition to that we ran into a recent problem that prevented us from importing the pool read-only when there were cache devices attached to the pool. This post will get together a few pieces of information that hopefully will get you out of a potential mess.

Doing this instead of recovering from backup, allowed us no to throw away a whole day of work at the customer's site... Fortunately.

1. How to break the reboot boot loop (system crashes when importing a pool other than rpool)
  • Stop at grub and edit the boot environment entry's '$kernel' line ('e' key)
  • Add at the end: -m milestone=none (this will boot just enough of the system to be able to make changes, but will not automatically import any pools)
  • Log in as 'admin' with its password
  • You will be left in at a root prompt
  • Remove /etc/zfs/zpool.cache (this will prevent the os from attempting to load any other pools than rpool)
  • Reboot and the system will start with all pools exported
1.a. If this happens with the root pool, boot from CD/DVD/USB and continue with step 2.
2. How to import pool read-only
  • If you import the pool normally, it will crash the system, so your target is to import the pool read only
  • If the crash happens on any write to the pool, attempt a read-only import:

zpool import -o readonly=on tank

  • If the read-only import still crashes the pool, you may attempt to import the pool without any read cache devices:
    • First, read the exported pool structure: 'zpool import':

   pool: tank

    id: 13998658358300648211

 state: ONLINE

action: The pool can be imported using its name or numeric identifier.

config:

 

       tank                       ONLINE

         raidz2-0                 ONLINE

           c5t5000CCA01B90FCECd0  ONLINE

           c5t5000CCA01B91A8A4d0  ONLINE

           c5t5000CCA01C486130d0  ONLINE

           c5t5000CCA01B91A96Cd0  ONLINE

       cache

         c3t4d0

       logs

         mirror-1                 ONLINE

           c3t2d0                 ONLINE

           c3t3d0                 ONLINE

    • To convince ZFS to ignore cache device, you can use cfgadm to disconnect the cache device from the controller:

# cfgadm -a | grep c3t4d0

sata1/4::dsk/c3t4d0 disk connected configured ok

  • Unconfigure the SATA port:

 

# cfgadm -c unconfigure sata1/4

 

  • Unconfigure the device at: 

 

/devices/pci@0,0/pci8086,1c02@1f,2:4

 

  • This operation will suspend activity on the SATA device

Continue (yes/no)? yes

 

  • After disconnecting the device, retry importing the pool normally:

# zpool import tank 

  • If you still got the crash, try to import with write cache devices disconnected. Disconnect all log devices by using cfgadm unconfigure as before on the log disks: 

zpool import -m -o readonly=on tank

 

3. How to import pool read-only, rolling back a few transaction groups:
  • If you still get a crash by disconnecting all cache and log devices while importing read-only, you have the option of importing the pool by discarding the last few transaction groups.
  • For a first try, you will be able to just dump the last transaction group without knowing its txg number by using the undocumented 'zpool import' parameter -F:

 

# zpool import -o readonly=on -f -F tank

  • If this still crashes, determine the last transaction groups by using zdb:

root@storage01:~# zdb -ul /dev/rdsk/c5t5000CCA01B90FCECd0s0

--------------------------------------------

LABEL 0

--------------------------------------------

version: 5000

name: 'tank'

[ ... ]

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 13516623673009291289

path: '/dev/dsk/c5t5000CCA01B91A96Cd0s0'

devid: 'id1,sd@n5000cca01b91a96c/a'

phys_path: '/scsi_vhci/disk@g5000cca01b91a96c:a'

whole_disk: 1

create_txg: 4

features_for_read:

Uberblock[0]

magic = 0000000000bab10c

version = 5000

txg = 6267008

guid_sum = 8831168779967537219

timestamp = 1390925682 UTC = Tue Jan 28 17:14:42 2014

Uberblock[1]

magic = 0000000000bab10c

version = 5000

txg = 6267009

guid_sum = 8831168779967537219

timestamp = 1390925687 UTC = Tue Jan 28 17:14:47 2014

 

 

- This will read the disk labels from one of the disks. There are 4 labels, and they are (or should) be the same on all data disks from the pool. From the output you must pick the 'tag' numbers of the most recent transaction groups. The uberblock versions are reused, so the newest transaction group is not the uberblock 127 (pay attention at the timestamps).
- To determine the latest txg number, first determine the latest transaction group timestamp:
# zdb -ul /dev/rdsk/c5t5000CCA01B90FCECd0s0 | grep timestamp | awk '{print $3}' | sort -u | tail -n 1
1390926237
- Now dump txg information for this timestamp:
root@storage01:~# zdb -ul /dev/rdsk/c5t5000CCA01B90FCECd0s0 | grep -B5 'timestamp = 1390926237'
Uberblock[111]
magic = 0000000000bab10c
version = 5000
txg = 6267119
guid_sum = 8831168779967537219
timestamp = 1390926237 UTC = Tue Jan 28 17:23:57 2014
--
Uberblock[111]
magic = 0000000000bab10c
version = 5000
txg = 6267119
guid_sum = 8831168779967537219
timestamp = 1390926237 UTC = Tue Jan 28 17:23:57 2014
--
Uberblock[111]
magic = 0000000000bab10c
version = 5000
txg = 6267119
guid_sum = 8831168779967537219
timestamp = 1390926237 UTC = Tue Jan 28 17:23:57 2014
--
Uberblock[111]
magic = 0000000000bab10c
version = 5000
txg = 6267119
guid_sum = 8831168779967537219
timestamp = 1390926237 UTC = Tue Jan 28 17:23:57 2014
- What you can see from the output is that there's one tag in that second: 6267119. You can safely assume that the txg's before it are 6267118 and 6267117
- Phew. Now you can use an undocumented feature of 'zpool import' to import the pool read-only at any txg you want:
- zpool import -o readonly=on -f -T 6267117 tank

4. How to import pool read-only, rolling back a few transaction groups and not mounting any filesystems, so you can mount them one by one to see which generates the crash:
zpool import -N -o readonly=on -f tank
You can combine this option with rolling back one or more transaction groups as before.