L2ARC with metadata and MFU only

[tkittich]

Describe the feature would like to see added to OpenZFS

Option to store only metadata (all metadata) and MFU (both metadata and data) in L2ARC

How will this feature improve OpenZFS?

This feature should be the default and would make a perfect L2ARC for home users. Most metadata and most frequently used data would eventually get stored in the L2ARC!! This would be the perfect tiered storage system.

Of course, this is currently possible by using metadata special vdev together with l2arc_mfuonly L2ARC. But a special vdev introduces another risk if damaged without enough redundancy. For RAIDZ2 of home users, it's not easy to have 3 nvme's as special vdev because most mainboards only have 2 nvme slots.

Additional context

Currently primarycache and secondarycache ZFS properties only allow all, none, and metadata. Perhaps this feature could be added as a new setting, i.e. metaandmfu, for primarycache and secondarycache .

[amotin]

I think we really need to rethink what information are we writing into L2ARC and how. That area got no significant love for decade probably. L2ARC from the beginning was made to write metadata and MFU first. But I think it may got affected by introduction of multilists in ARC. We may also recheck interaction of L2ARC with prefetch -- we may not want to write to L2ARC data that are successfully prefetched from the main pool. We could also redesign write boost there not based on ARC being warm, but on L2ARC being above certain fill percent, that combined with the above preference should result in writes of more usable data.

[AllKind]

Just as a side-note a method to make failures of special vdev non-fatal is in the works: #16185

[shodanshok]

I submitted #16402 which should enable what asked in this issue: L2 cache of all metadata but MFU data only. I used the already defined l2arc_mfuonly tunable: while using a dataset property would be great, it seems to me that the L2 ARC feed thread know nothing about properties (it only cares about buffer lists/sublist).