Add bulk off-heap scoring for float32 vectors

[ChrisHegarty]

This commit adds bulk off-heap scoring for float32 vectors.

The bulk scorer scores 4 vectors against the query vector at a time. The general idea is to structure things so that we can somewhat tackle memory latency allowing the CPU to do overlapping memory loads.

Initial results from the micro benchmarks shows good potential improvement. The benchmark creates a flat vector index with 128,000 float32 vectors with 1024 dimensions (~500MB). And times how long it takes to scores 20,000 random vectors against a query vector (lower times are better)

Benchmark                                            (size)  Mode  Cnt  Score   Error  Units
VectorScorerFloat32Benchmark.dotProductDefault         1024  avgt   15  8.505 ± 0.256  ms/op
VectorScorerFloat32Benchmark.dotProductNewBulkScore    1024  avgt   15  3.717 ± 0.158  ms/op
VectorScorerFloat32Benchmark.dotProductNewScorer       1024  avgt   15  7.287 ± 0.181  ms/op

Notes:

• Just dot product for now, but other distance functions can be added as a follow up.
• The bulk scorer just does 4 vectors at time, since the implementation in Lucene is more straightforward, but this could be adjusted.
• we seem to suffer pollution of the query vector type, so for now I just added two separate independent almost identical versions of the vector dot op.

[ChrisHegarty]

/cc @mccullocht

[benwtrent]

I switched the benchmark to be throughput per second. I get similar results, JMH indicates 2x+ improvement on macbook ARM with the vector ops side of things:

Benchmark                                            (size)   Mode  Cnt    Score    Error  Units
VectorScorerFloat32Benchmark.dotProductDefault         1024  thrpt   15  106.205 ± 10.434  ops/s
VectorScorerFloat32Benchmark.dotProductDefaultBulk     1024  thrpt   15  119.524 ±  1.094  ops/s
VectorScorerFloat32Benchmark.dotProductOptBulkScore    1024  thrpt   15  283.673 ±  2.391  ops/s
VectorScorerFloat32Benchmark.dotProductOptScorer       1024  thrpt   15  140.599 ±  0.449  ops/s

[benwtrent]

I left some minor comments. Seems like jars were updated by accident?

However, the change looks great!

[benwtrent]

Love it! Now to bulk score ALL THE THINGS!!!!

[rmuir]

I think when I tried my hand at a similar approach with panama I ran into
similar (neutral) results on graviton2 and really the only thing that
helped there was prefetching into cpu cache. Totally believe the results
are better elsewhere, it's just been a bit of a struggle to stand up the
tests on other machines given the resources I have at hand.

When benchmarking ARM, I really recommend to only worry about Graviton3 (ARM SVE).

IMO we should stop investing any time into 128-bit Neon (Macs, Graviton2, etc). I realize macs are convenient, but the sooner we drop vectorization support for this legacy instruction set, the better. It is not well supported by openjdk, so i see no path to success there.

[benwtrent]

Yep, Lucene nightlies show a nice improvement (variance is high, but its consistently better than before): https://benchmarks.mikemccandless.com/VectorSearch.html

Maybe 10+% better it seems.

[mikemccand]

Yep, Lucene nightlies show a nice improvement (variance is high, but its consistently better than before): https://benchmarks.mikemccandless.com/VectorSearch.html

Yay! I will add annotation for this, even though it is noisy. (Separately I would love to reduce this noise...)