[Bug/Crosspost] [1.20.1]: Low Memory Search makes removing ingredients ridiculously slow

[carbonatek]

Steps to Reproduce the Bug

• Enable Low Memory Search option
• Use a mod which removes ingredients from the game. (In my case, Tinkers' Construct alpha on 1.20.1)
• See/feel that it takes a RIDICULOUS amount of time.

Expected Behavior

The mod removes ingredients in a sane amount of time.

Actual Behavior

The mod removes ingredients in a ridiculous amount of time (>=40 seconds).

Mod Pack URL (Optional)

No response

Mod Pack Version (Optional)

No response

Extra Notes (Optional)

If I disable low memory search, it takes about 425 milliseconds.

Crosspost requested, from:
SlimeKnights/TinkersConstruct#5290

Extra details which were not filled out here will be found over there.

latest.log

No response

[KnightMiner]

Worth noting, the problem would be solved in tinkers case if I had a method to skip adding the items from my creative tab in the first place. I used to be able to do this by not adding the items to search but now JEI adds all items regardless. Some sort of argument passed to my creative tab telling me it's JEI loading would let me skip those items entirely instead of removing them from JEI. Not sure what you can do without changing the signature, maybe a fake feature flag or extending one of the argument classes with something I can identify as JEI using reflection