roachtest: disable thp to see if it helps OOMS in costfuzz

[cucaroach]

See #90683

Release note: None

[cockroach-teamcity]

This change is 

[srosenberg]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @cucaroach and @smg260)

---

pkg/roachprod/install/cockroach.go line 421 at r1 (raw file):
From what I recall, jemalloc is pretty conservative wrt THP. When debugging another issue we used strace to trace allocations; only one MADV_HUGEPAGE was issued by jemalloc [1]. The more likely culprit is go's GC which according to [2] doesn't call MADV_NOHUGEPAGE to break up allocations,

The result of all this is that Linux may be keeping around huge pages transparently using up a lot more memory than intended, and reducing the effectiveness of the scavenger.

[1] #83790 (comment)
[2] golang/go#55328

[renatolabs]

I'd also suggest we only use this setting in the costfuzz roachtest instead of enabling it in all of them.

[srosenberg]

I'd also suggest we only use this setting in the costfuzz roachtest instead of enabling it in all of them.

Good catch, I didn't realize this was disabling it for everything; that's highly unadvisable.

[srosenberg]

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @cucaroach and @smg260)

[cucaroach]

I'd also suggest we only use this setting in the costfuzz roachtest instead of enabling it in all of them.

Good catch, I didn't realize this was disabling it for everything; that's highly unadvisable.

Why? Just curious, I can make it only take affect for costfuzz but my theory is that THP is likely always bad for jemalloc. Also because we use jemalloc for large block allocations I don't think THP buys us much performance wise. Note that setting MALLOC_CONF only affects jemalloc and doesn't affect the go heap. Making it specific to costfuzz can be done and if we want to start with that more limited in scope experiment thats fine with me. Maybe instead of this experiment what I should be doing is adding code to determine where the THP is kicking in. If we dumped the jemalloc arenas and the proc/smaps we could make that determination. I'll give that a go...

[srosenberg]

Why? Just curious, I can make it only take affect for costfuzz but my theory is that THP is likely always bad for jemalloc.

We don't know the side-effects of this change. This could potentially affect multiple roachtest benchmarks, causing a regression; other roachtests could also misbehave in unexpected ways. It'd be safer to test this out in an isolated manner first.

Also because we use jemalloc for large block allocations I don't think THP buys us much performance wise. Note that setting MALLOC_CONF only affects jemalloc and doesn't affect the go heap.

Right, the only consumer of this change is pebble's block cache.

Making it specific to costfuzz can be done and if we want to start with that more limited in scope experiment thats fine with me. Maybe instead of this experiment what I should be doing is adding code to determine where the THP is kicking in. If we dumped the jemalloc arenas and the proc/smaps we could make that determination. I'll give that a go...

Seems like a more fruitful next step. My hunch it isn't jemalloc. Kernel is using madvise by default (for THP). You could confirm/refute by running strace (example in [1]).