Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exabyte dedup_table_size reported on ZFS 2.3 without fast dedup. #17019

Open
IvanVolosyuk opened this issue Feb 2, 2025 · 2 comments
Open

Exabyte dedup_table_size reported on ZFS 2.3 without fast dedup. #17019

IvanVolosyuk opened this issue Feb 2, 2025 · 2 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@IvanVolosyuk
Copy link
Contributor

IvanVolosyuk commented Feb 2, 2025 •
edited
Loading

System information

Type Version/Name
Distribution Name Gentoo
Distribution Version Live
Kernel Version 6.12.12-gentoo
Architecture x86_64
OpenZFS Version 2.3.0
ECC RAM Yes

Describe the problem you're observing

I tried to make sense of new pool properties before migrating to fast dedup. The output puzzled me:

zpool list -o dedup_table_size net
DDTSIZE
  15.9E

zpool list -p -o dedup_table_size net
             DDTSIZE
18338641226299867136

Describe how to reproduce the problem

I didn't do anything special, I have updated to ZFS-2.3.0 from ZFS-2.2.7 and didn't do any feature upgrades (carefully considering migrating to fast dedup). I'm pretty sure ZFS 2.2.7 had reasonable value for size on disk.

  pool: net
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:43:16 with 0 errors on Fri Jan 17 04:43:16 2025
config:

        NAME                                       STATE     READ WRITE CKSUM
        net                                        ONLINE       0     0     0
          ata-ST10000VN0008-2PJ103_ZS5131ST-part4  ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 1920038, size 15.9E on disk, 306M in core

bucket              allocated                       referenced          
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    1.14M   99.6G   53.9G   54.3G    1.14M   99.6G   53.9G   54.3G
     2     485K   57.1G   54.4G   54.5G    1.04M    125G    119G    119G
     4     207K   24.6G   24.3G   24.3G     992K    117G    116G    116G
     8    7.90K   66.0M   35.1M   47.8M    82.5K    649M    348M    483M
    16    12.0K   38.3M   23.2M   49.7M     309K    947M    575M   1.25G
    32       84    426K    230K    464K    3.57K   18.8M   9.04M   19.3M
    64       23    149K     24K     92K    2.10K   10.2M   2.04M   8.39M
   128        7   3.50K   3.50K     28K    1.34K    686K    686K   5.36M
   256        4   5.50K   5.50K     16K    1.33K   1.59M   1.59M   5.30M
   512        3   1.50K   1.50K     12K    1.97K   1008K   1008K   7.88M
 Total    1.83M    181G    133G    133G    3.54M    344G    290G    292G

Looks like reporting is off for legacy dedup, which makes it kinda scary to upgrade the pool and try the new feature.

For comparison the output from zpool status -D net on ZFS 2.2.7.

  pool: net
 state: ONLINE
  scan: scrub repaired 0B in 00:43:16 with 0 errors on Fri Jan 17 04:43:16 2025
config:

	NAME                                       STATE     READ WRITE CKSUM
	net                                        ONLINE       0     0     0
	  ata-ST10000VN0008-2PJ103_ZS5131ST-part4  ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 1920038, size 283 on disk, 167 in core

bucket              allocated                       referenced          
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    1.14M   99.6G   53.9G   54.3G    1.14M   99.6G   53.9G   54.3G
     2     485K   57.1G   54.4G   54.5G    1.04M    125G    119G    119G
     4     207K   24.6G   24.3G   24.3G     992K    117G    116G    116G
     8    7.90K   66.0M   35.1M   47.8M    82.5K    649M    348M    483M
    16    12.0K   38.3M   23.2M   49.7M     309K    947M    575M   1.25G
    32       84    426K    230K    464K    3.57K   18.8M   9.04M   19.3M
    64       23    149K     24K     92K    2.10K   10.2M   2.04M   8.39M
   128        7   3.50K   3.50K     28K    1.34K    686K    686K   5.36M
   256        4   5.50K   5.50K     16K    1.33K   1.59M   1.59M   5.30M
   512        3   1.50K   1.50K     12K    1.97K   1008K   1008K   7.88M
 Total    1.83M    181G    133G    133G    3.54M    344G    290G    292G
@IvanVolosyuk IvanVolosyuk added the Type: Defect Incorrect behavior (e.g. crash, hang) label Feb 2, 2025
@IvanVolosyuk
Copy link
Contributor Author

I think the issues goes away after I do some writes using the same hash algorithm, but I'm not 100% sure if that what triggered the correct numbers to appear. It definitely stayed broken before that when I was writting new data using different dedup hash function.

@pjedrzejczak-catalogicsoftware
Copy link

I am experiencing the same issue on my test environments, here are the details. Let me know if you need more I ran the upgrade twice and both times I encountered this issue.

Pre-upgrade

Type Version/Name
Distribution Name AlmaLinux
Distribution Version 8.10 (Cerulean Leopard)
Kernel Version 4.18.0-553.8.1.el8_10.x86_64
Architecture x86_64
OpenZFS Version zfs-2.2.4-20240703.el8.x86_64
ECC RAM No 
zpool status -D vpool1

  pool: vpool1
 state: ONLINE
config:

        NAME                                      STATE     READ WRITE CKSUM
        vpool1                                    ONLINE       0     0     0
          scsi-36000c29dcd74a9ea30c6559724c2e5b1  ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 249386, size 267B on disk, 145B in core

bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1     224K   28.1G   28.1G   28.1G     224K   28.1G   28.1G   28.1G
    32    18.9K   2.37G   2.37G   2.37G     954K    119G    119G    119G
   512      200     25M     25M     25M     106K   13.2G   13.2G   13.2G
 Total     244K   30.4G   30.4G   30.4G    1.25M    161G    161G    161G
zdb -D vpool1

DDT-sha256-zap-duplicate: 19575 entries, size 244 on disk, 132 in core
DDT-sha256-zap-unique: 229811 entries, size 269 on disk, 146 in core

dedup = 5.27, compress = 1.00, copies = 1.00, dedup * compress / copies = 5.27

Post-upgrade, in my case there was also a kernel bump.

Type Version/Name
Kernel Version 4.18.0-553.34.1.el8_10.x86_64
OpenZFS Version zfs-2.3.0-20250120.el8.x86_64 
zpool status -D vpool1

  pool: vpool1
 state: ONLINE
config:

        NAME                                      STATE     READ WRITE CKSUM
        vpool1                                    ONLINE       0     0     0
          scsi-36000c29dcd74a9ea30c6559724c2e5b1  ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 249386, size 16.0E on disk, 34.5M in core

bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1     224K   28.1G   28.1G   28.1G     224K   28.1G   28.1G   28.1G
    32    18.9K   2.37G   2.37G   2.37G     954K    119G    119G    119G
   512      200     25M     25M     25M     106K   13.2G   13.2G   13.2G
 Total     244K   30.4G   30.4G   30.4G    1.25M    161G    161G    161G
zdb -D vpool1

DDT-sha256: version=0 [LEGACY]; flags=0x00; rootobj=1
DDT-sha256-zap-duplicate: dspace=4780032; mspace=2588672; entries=19575
DDT-sha256-zap-unique: dspace=61882368; mspace=33587200; entries=229811

dedup = 5.27, compress = 1.00, copies = 1.00, dedup * compress / copies = 5.27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@IvanVolosyuk @pjedrzejczak-catalogicsoftware