Xattr objects, which are created before upgrade to feature@project_quota, get account-ed in the wrong project id after the upgrade

[jsai20]

System information

Type	Version/Name
Distribution Name
Distribution Version
Kernel Version	5.10.194-1.el8
Architecture
OpenZFS Version	zfs-kmod-2.2.99-1.*.c98295eed

Describe the problem you're observing

zpool upgraded with 'feature@project_quota' needs re-layout of SA's to fix the SA_ZPL_PROJID at SA_PROJID_OFFSET (128). Its necessary for the correct accounting of object usage against its projid. Old object (created before upgrade) when gets a projid assigned, its SA gets re-layout via sa_add_projid(). If object has xattr dir, SA of xattr dir also gets re-layout. But SA re-layout of xattr objects inside a xattr dir is not done.

Describe how to reproduce the problem

Create zpool with 'feature@project_quota' disabled

$ sudo zpool create mytestpool -o feature@project_quota=disabled disk.1 
$ sudo zpool get feature@project_quota mytestpool
NAME        PROPERTY               VALUE                  SOURCE
mytestpool  feature@project_quota  disabled               local
$ sudo zfs create mytestpool/fs1

Create xattr objects

$ cd /mytestpool/fs1
$ sudo chmod 0777 ../fs1
$ mkdir dir1
$ ls -id dir1
2 dir1
$ attr -s passwd dir1 < /etc/passwd

 Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         2    1   128K    512      0     512    512  100.00  ZFS directory
                                               176   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
        dnode maxblkid: 0
        path    /dir1
        uid     1000
        gid     1000
        atime   Sun Jul  7 13:12:07 2024
        mtime   Sun Jul  7 13:12:07 2024
        ctime   Sun Jul  7 13:14:50 2024
        crtime  Sun Jul  7 13:12:07 2024
        gen     14
        mode    40700
        size    2
        parent  34
        links   2
        pflags  40800000044
        xattr   3
        microzap: 512 bytes, 0 entries

Indirect blocks:
               0 L0 EMBEDDED et=0 200L/1cP B=14

                segment [0000000000000000, 0000000000000200) size   512
                

  Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         3    1   128K    512      0     512    512  100.00  ZFS directory
                                               168   bonus  System attributes
        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED 
        dnode maxblkid: 0
        path    /dir1/<xattrdir>
        uid     1000
        gid     1000
        atime   Sun Jul  7 13:12:07 2024
        mtime   Sun Jul  7 13:14:50 2024
        ctime   Sun Jul  7 13:14:50 2024
        crtime  Sun Jul  7 13:12:07 2024
        gen     14
        mode    41777
        size    4
        parent  2
        links   2
        pflags  40800000145
        microzap: 512 bytes, 2 entries

                user.passwd = 128 (type: Regular File)
                security.selinux = 4 (type: Regular File)
Indirect blocks:
               0 L0 EMBEDDED et=0 200L/55P B=46

                segment [0000000000000000, 0000000000000200) size   512

Upgrade feature@project_quota:

$ sudo zpool get feature@project_quota mytestpool
NAME        PROPERTY               VALUE                  SOURCE
mytestpool  feature@project_quota  disabled               local

$ sudo zpool upgrade mytestpool
This system supports ZFS pool feature flags.

Enabled the following features on 'mytestpool':
  project_quota
$  sudo zpool get feature@project_quota mytestpool
NAME        PROPERTY               VALUE                  SOURCE
mytestpool  feature@project_quota  enabled                local
$cd ../
$ sudo zfs umount mytestpool/fs1
$ sudo zfs mount mytestpool/fs1
$ cd fs1
$  sudo zpool get feature@project_quota mytestpool
NAME        PROPERTY               VALUE                  SOURCE
mytestpool  feature@project_quota  active                 local

Configure project 111 on dir1

$ sudo zfs projectspace mytestpool/fs1
$
$ zfs project dir1
    0 - dir1
$
Set project id 111 on dir1
$ zfs project -p 111 -s -r dir1
$

Check usage of Project 111

$ sudo zfs projectspace mytestpool/fs1
TYPE     NAME   USED  QUOTA  OBJUSED  OBJQUOTA
Project  0     16.0E   none    16.0E      none
Project  3     2.50K   none        2      none
Project  111      1K   none        2      none

Project 111 should have a total of 4 objects account-ed (1 for dir1 ino 2, 1 for xattr dir ino 3 and 2 for xattr’s ino 128 and ino 4. But it's only accounting 2 in project 111 (1 for dir1 ino 2 and 1 for xattr dir ino 3). Xattr ino 128 and ino 4 are account-ed in the invalid projid 3.

Include any warning/errors/backtraces from the system logs