Add prefetching to x64 bulk vector implementations

[thecoop]

Prefetch the next vectors whilst processing one set of vectors for sqri7u, bulki8, and 2, 4-bit BBQ.
Remove AVX512 BBQ implementations, only using AVX2, as AVX512 is often slower than just the AVX2 impls

This causes a small performance drop for small number of vectors (-5%), but substantially faster for large numbers (+70%).

eg,

VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk              256    1024   EUCLIDEAN            NATIVE           128  thrpt    5  335315.141 ±  4260.120  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk              256    1024   EUCLIDEAN            NATIVE          1500  thrpt    5   27202.387 ±   305.012  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk              256    1024   EUCLIDEAN            NATIVE        130000  thrpt    5     535.276 ±    57.630  ops/s

becomes

VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk              256    1024   EUCLIDEAN            NATIVE           128  thrpt    5  328152.116 ±  2151.307  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk              256    1024   EUCLIDEAN            NATIVE          1500  thrpt    5   26192.751 ±   128.196  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk              256    1024   EUCLIDEAN            NATIVE        130000  thrpt    5     767.641 ±    25.418  ops/s

KnnIndexTester shows a 20% speed improvement for sqri7 searches, and 10-20% faster for BBQ 2/4-bit

[thecoop]

This bulk pattern seems to be template-able, but it needs:

• the 'inner' method to be a template param (here sqri7u_inner)
• the vectors tail method can be a template param (vec_sqri7u)
• the dimension tail. This is a lot less obvious, as it's in-line to the method, and there are several method variables updated by that bit of code. How could that be templated out sensibly?

[thecoop]

ALternatively, we could use classes, but then I'm not sure what that does to inline-ability and (potentially virtual) method calls

[ldematte]

We tried to stay avoid from classes, and especially virtual methods, up to now, and focus on what can be done with templating instead.
A template would probably be not easy on the eyes; as you point out, it would require 3 template functions.

And it would be good to look at the cost of virtual calls for bulk; the overhead for single distance/score made it not feasible, but maybe with bulk operations we are OK?

Let's split and try both; I can give a template-based version a go, you could try a class-based one.

[benwtrent]

I am not sure how to interpret the benchmarks.

How native scoring is actually used, we will score only bulks of N size at a time, and drop to native only for those bulk sizes. Then return to "java land", rinse and repeat.

This is true for diskbbq, hnsw, and flat (eventually).

Do the benchmarks indicate that there is only a performance gain if we do VERY large bulk sizes? Or do the benchmarks also take into account the switching between Java and Native every 16/32/64 vectors at a time?

[benwtrent]

Do the benchmarks indicate that there is only a performance gain if we do VERY large bulk sizes? Or do the benchmarks also take into account the switching between Java and Native every 16/32/64 vectors at a time?

Reading the benchmarks, they don't it just assumes we are bulk scoring every vector in the set, this is not how things actually work. I will see if I can iterate on adjusting this one benchmark, but the others do seem complicated to add this logic to.

[benwtrent]

So, I have a local branch that adds bulkScore parameter, I ran it against main (so, not this branch right now)

PR: #142480

Could we see how the prefetching works with this type of thing?

Benchmark                                                    (bulkSize)  (dims)   (function)  (implementation)  (numVectors)   Mode  Cnt      Score      Error  Units
VectorScorerInt7uBulkBenchmark.scoreMultipleRandom                   32    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  15568.442 ±  840.138  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandom                 1500    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  16323.343 ±  796.789  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk               32    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  25851.679 ± 2296.438  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleRandomBulk             1500    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  25972.461 ± 1751.552  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleSequential               32    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  17561.196 ±  957.424  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleSequential             1500    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  17510.794 ± 1528.157  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleSequentialBulk           32    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  26485.337 ±  380.044  ops/s
VectorScorerInt7uBulkBenchmark.scoreMultipleSequentialBulk         1500    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  26095.281 ±  701.095  ops/s
VectorScorerInt7uBulkBenchmark.scoreQueryMultipleRandom              32    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  15366.295 ± 1316.324  ops/s
VectorScorerInt7uBulkBenchmark.scoreQueryMultipleRandom            1500    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  15804.847 ±  149.808  ops/s
VectorScorerInt7uBulkBenchmark.scoreQueryMultipleRandomBulk          32    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  27118.706 ±  892.149  ops/s
VectorScorerInt7uBulkBenchmark.scoreQueryMultipleRandomBulk        1500    1024  DOT_PRODUCT            NATIVE          1500  thrpt    5  27325.290 ±  226.967  ops/s

[ldematte]

Base on what we found, it makes sense to concentrate on smaller bulk sizes.
We should keep in mind that prefetching has very little benefit when accessing data sequentially (bbq/diskbbq); Also, I'm not sure how we should reconcile it with the striped nature of 2 and 4 bit vectors -- I think we are missing something here, we need to prefetch different areas. Ssee my comment.

Prefetching should matter more for HNSW; the "MultipleRandomBulk" thing, where we use bulk but we have offsets (int[] nodes)

[thecoop]

The AVX512 BBQ implementations are slower than the AVX2 implementations, so I've just removed them so we always use AVX2 implementations.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[ldematte]

Looks good, some questions about possible alternative for prefetching and experimenting, if they make sense/if you haven't already tried them.
Also one consideration about the bbq stuff for AVX-512 -- looking at the code, I'm not surprised this is slower for dims < 1024.
However, I'm happy to re-introduce them with the proposed fixes in another PR.

[ldematte]

I'm really surprised this is slower, as it should be the optimal instruction to use. However, of course this path is taken only if we can fill the whole STRIDE_BYTES_LEN (512 bits), which is not a given! Dimensions < 512 will not benefit from this at all, and probably this makes sense only for 1024 or more.

[ldematte]

A way of making this better/on par with AVX2 without removing it (as I think this should still be faster for higher dimensions), is to use fall back to the AVX2 code path here -- or to _mm256_popcnt_epi64, _mm_popcnt_epi64, __builtin_popcountll progressivelty,

[thecoop]

I think it'll be best if we do AVX512 for D[124]Q4 implementations all at once, then we can ensure it's all done together. We also need to work out how to do the fallback (whether just copy-paste AVX2 code, or something more sophisticated) at the same time

[ldematte]

Agree to do AVX512 for D[124]Q4 implementations all at once.
LGTM