'-b' argument for 'zfs send' gives error: ASSERT at module/nvpair/fnvpair.c:409:fnvlist_lookup_string()

[maanloper]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	24.10
Kernel Version	6.11.0-14-generic
Architecture	x86-64
OpenZFS Version	zfs-2.2.6-1ubuntu1

Describe the problem you're observing

When running the following command to restore a backupped filesystem:

ssh <hostname> zfs send -b -w -R "${backup_dataset}@${snapshot}" | zfs receive "${receive_pool}"

It always gives the following error:

ASSERT at module/nvpair/fnvpair.c:409:fnvlist_lookup_string()
nvlist_lookup_string(nvl, name, &rv) == 0 (0x2 == 0)
  PID: 43818     COMM: zfs
  TID: 43818     NAME: zfs
Call trace:
  /lib/x86_64-linux-gnu/libnvpair.so.3(libspl_assertf+0x159) [0x7d2f2602d859]
  /lib/x86_64-linux-gnu/libnvpair.so.3(fnvlist_lookup_string+0x68) [0x7d2f2602f768]
  /lib/x86_64-linux-gnu/libzfs.so.4(+0x41cc2) [0x7d2f2608bcc2]
  /lib/x86_64-linux-gnu/libzfs.so.4(zfs_receive+0xc3) [0x7d2f2608c353]
  zfs(+0x13044) [0x60ebac426044]
  zfs(+0x8328) [0x60ebac41b328]
  /lib/x86_64-linux-gnu/libc.so.6(+0x2a3b8) [0x7d2f25c2a3b8]
  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7d2f25c2a47b]
  zfs(+0x8545) [0x60ebac41b545]
/usr/local/zorra/zfs/restore-backup.sh: line 34: 43817 Done                    zfs send -b -w -R "${backup_dataset}@${snapshot}"
     43818 Aborted                 (core dumped) | zfs receive "${receive_pool}"

Note: the datasets are restored (including an child datasets and all snapshot up to the specified snapshot), it just gives this error when it is done transferring all snapshots/datasets.

Describe how to reproduce the problem

1. Create encrypted pool on serverA
2. Create unencrypted pool on serverB
3. Send/receive all datasets from serverA to B using
   zfs send -w -R poolA | ssh <hostname> zfs receive poolB/poolA (do not load key for any dataset on poolB)
4. Destroy any dataset(s) on poolA OR destroy poolA and recreate poolA
5. Restore a dataset from poolB to poolA using
   ssh <hostname> zfs send -b -w -R poolB/poolA/dataset | zfs receive poolB/poolA (note the -b option)
6. Error shows when done. To my knowledge, all (child)datasets and snapshots are correctly restored, it just throws the error.

Any ideas? Happy to run debugging commands.

[maanloper]

Nothing from any dev to at least point in the right direction?

[dcarosone]

I see this too. Current NixOS unstable, zfs 2.3.0, Linux 6.12.11

I have backups sent (encrypted, raw) to other pools. As part of #13483 (comment) I had to restore them, and hit this error.

[root@nixos:~]# zfs send -bRw cpool/znap/ilus/rpool/home@2025-02-07-030000  | zfs recv -F -vu -e rpool
...
received 49.8M stream in 0.51 seconds (98.6M/sec)
ASSERT at module/nvpair/fnvpair.c:409:fnvlist_lookup_string()
nvlist_lookup_string(nvl, name, &rv) == 0 (0x2 == 0)
  PID: 19679     COMM: zfs
  TID: 19679     NAME: zfs
Call trace:
/nix/store/9sx675iy7671jrmw185giq221jaqhp3y-zfs-user-2.3.0/lib/libzfs.so.6(libspl_backtrace+0x26) [0x7f6fb5e57926]
/nix/store/9sx675iy7671jrmw185giq221jaqhp3y-zfs-user-2.3.0/lib/libzfs.so.6(libspl_assertf+0x135) [0x7f6fb5e578c5]
/nix/store/9sx675iy7671jrmw185giq221jaqhp3y-zfs-user-2.3.0/lib/libnvpair.so.3(fnvlist_lookup_string+0x58) [0x7f6fb5603718]
/nix/store/9sx675iy7671jrmw185giq221jaqhp3y-zfs-user-2.3.0/lib/libzfs.so.6(+0x3fe1d) [0x7f6fb5e3be1d]
/nix/store/9sx675iy7671jrmw185giq221jaqhp3y-zfs-user-2.3.0/lib/libzfs.so.6(zfs_receive+0xb2) [0x7f6fb5e3e372]
zfs() [0x4105f2]
zfs() [0x4072ce]
/nix/store/h7zcxabfxa7v5xdna45y2hplj31ncf8a-glibc-2.40-36/lib/libc.so.6(+0x2a1fc) [0x7f6fb53af1fc]
/nix/store/h7zcxabfxa7v5xdna45y2hplj31ncf8a-glibc-2.40-36/lib/libc.so.6(__libc_start_main+0x89) [0x7f6fb53af2b9]
zfs() [0x4074a5]
Aborted (core dumped)

I suspect it might be related to the -b because although the dataset are restored, they don't have any non-inherited properties set at the destination.

[maanloper]

I can confirm the behavior only shows when using -b. I do use -p and -b flags simultaneously to make sure properties are set on the receiving end.