Skip to content

Added Gating so VectorScorerInt4BenchmarkTests and VectorScorerInt4BulkBenchmarkTests Ignored on Java 21 #145060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
john-wagster wants to merge 2 commits into from
Closed

Added Gating so VectorScorerInt4BenchmarkTests and VectorScorerInt4BulkBenchmarkTests Ignored on Java 21 #145060

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
88aa050
added gating so the tests don't run for native and lucene unless on j…
john-wagster Mar 27, 2026
d73bb3b
[CI] Auto commit changes from spotless
Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 30 additions & 12 deletions ...c/test/java/org/elasticsearch/benchmark/vector/scorer/VectorScorerInt4BenchmarkTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

import java.util.Arrays;

import static org.elasticsearch.benchmark.vector.scorer.BenchmarkUtils.supportsHeapSegments;

public class VectorScorerInt4BenchmarkTests extends ESTestCase {
Comment on lines +22 to 24
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR title/description says these tests are "ignored on Java 21", but this change only conditionally skips creating lucene/native benches and assertions; the test methods still execute and will report as passing rather than skipped on Java 21. If the intent is to truly ignore/skip on Java 21, consider using an assumption (assumeTrue/assumeFalse) or a JUnit ignore/conditional mechanism so CI reflects the tests as skipped.

Copilot uses AI. Check for mistakes.

private final double delta = 1e-3;
Expand Down Expand Up @@ -48,17 +50,25 @@ public void testScores() throws Exception {
for (int i = 0; i < 100; i++) {
var data = new VectorScorerInt4Benchmark.VectorData(dims);
var scalar = createBench(VectorImplementation.SCALAR, data);
var lucene = createBench(VectorImplementation.LUCENE, data);
var nativeBench = createBench(VectorImplementation.NATIVE, data);
VectorScorerInt4Benchmark lucene = null;
VectorScorerInt4Benchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, data);
nativeBench = createBench(VectorImplementation.NATIVE, data);
}

try {
float expected = scalar.score();
assertEquals("LUCENE score", expected, lucene.score(), delta);
assertEquals("NATIVE score", expected, nativeBench.score(), delta);
if (supportsHeapSegments()) {
assertEquals("LUCENE score", expected, lucene.score(), delta);
assertEquals("NATIVE score", expected, nativeBench.score(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
Comment on lines 61 to +69
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are several if(supportsHeapSegments()) blocks without a space after if. This is inconsistent with the surrounding style in this file (e.g., if (supportsHeapSegments())) and may fail automated formatting/style checks. Please update to if (supportsHeapSegments()) consistently.

Copilot uses AI. Check for mistakes.
nativeBench.teardown();
}
}
}
}
Expand All @@ -67,17 +77,25 @@ public void testQueryScores() throws Exception {
for (int i = 0; i < 100; i++) {
var data = new VectorScorerInt4Benchmark.VectorData(dims);
var scalar = createBench(VectorImplementation.SCALAR, data);
var lucene = createBench(VectorImplementation.LUCENE, data);
var nativeBench = createBench(VectorImplementation.NATIVE, data);
VectorScorerInt4Benchmark lucene = null;
VectorScorerInt4Benchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, data);
nativeBench = createBench(VectorImplementation.NATIVE, data);
}

try {
float expected = scalar.scoreQuery();
assertEquals("LUCENE scoreQuery", expected, lucene.scoreQuery(), delta);
assertEquals("NATIVE scoreQuery", expected, nativeBench.scoreQuery(), delta);
if (supportsHeapSegments()) {
assertEquals("LUCENE scoreQuery", expected, lucene.scoreQuery(), delta);
assertEquals("NATIVE scoreQuery", expected, nativeBench.scoreQuery(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}
Expand Down
123 changes: 87 additions & 36 deletions ...st/java/org/elasticsearch/benchmark/vector/scorer/VectorScorerInt4BulkBenchmarkTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

import java.util.Arrays;

import static org.elasticsearch.benchmark.vector.scorer.BenchmarkUtils.supportsHeapSegments;

public class VectorScorerInt4BulkBenchmarkTests extends ESTestCase {

private final VectorSimilarityType function;
Expand Down Expand Up @@ -52,17 +54,25 @@ public void testSequential() throws Exception {
for (int i = 0; i < 100; i++) {
var vectorData = new VectorScorerInt4BulkBenchmark.VectorData(dims, 1000, 200);
var scalar = createBench(VectorImplementation.SCALAR, vectorData);
var lucene = createBench(VectorImplementation.LUCENE, vectorData);
var nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
VectorScorerInt4BulkBenchmark lucene = null;
VectorScorerInt4BulkBenchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, vectorData);
nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
}

try {
float[] expected = scalar.scoreMultipleSequential();
assertArrayEquals("LUCENE sequential", expected, lucene.scoreMultipleSequential(), delta);
assertArrayEquals("NATIVE sequential", expected, nativeBench.scoreMultipleSequential(), delta);
if (supportsHeapSegments()) {
assertArrayEquals("LUCENE sequential", expected, lucene.scoreMultipleSequential(), delta);
assertArrayEquals("NATIVE sequential", expected, nativeBench.scoreMultipleSequential(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}
Expand All @@ -71,17 +81,25 @@ public void testRandom() throws Exception {
for (int i = 0; i < 100; i++) {
var vectorData = new VectorScorerInt4BulkBenchmark.VectorData(dims, 1000, 200);
var scalar = createBench(VectorImplementation.SCALAR, vectorData);
var lucene = createBench(VectorImplementation.LUCENE, vectorData);
var nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
VectorScorerInt4BulkBenchmark lucene = null;
VectorScorerInt4BulkBenchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, vectorData);
nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
}

try {
float[] expected = scalar.scoreMultipleRandom();
assertArrayEquals("LUCENE random", expected, lucene.scoreMultipleRandom(), delta);
assertArrayEquals("NATIVE random", expected, nativeBench.scoreMultipleRandom(), delta);
if (supportsHeapSegments()) {
assertArrayEquals("LUCENE random", expected, lucene.scoreMultipleRandom(), delta);
assertArrayEquals("NATIVE random", expected, nativeBench.scoreMultipleRandom(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}
Expand All @@ -90,17 +108,25 @@ public void testQueryRandom() throws Exception {
for (int i = 0; i < 100; i++) {
var vectorData = new VectorScorerInt4BulkBenchmark.VectorData(dims, 1000, 200);
var scalar = createBench(VectorImplementation.SCALAR, vectorData);
var lucene = createBench(VectorImplementation.LUCENE, vectorData);
var nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
VectorScorerInt4BulkBenchmark lucene = null;
VectorScorerInt4BulkBenchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, vectorData);
nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
}

try {
float[] expected = scalar.scoreQueryMultipleRandom();
assertArrayEquals("LUCENE queryRandom", expected, lucene.scoreQueryMultipleRandom(), delta);
assertArrayEquals("NATIVE queryRandom", expected, nativeBench.scoreQueryMultipleRandom(), delta);
if (supportsHeapSegments()) {
assertArrayEquals("LUCENE queryRandom", expected, lucene.scoreQueryMultipleRandom(), delta);
assertArrayEquals("NATIVE queryRandom", expected, nativeBench.scoreQueryMultipleRandom(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}
Expand All @@ -109,17 +135,25 @@ public void testSequentialBulk() throws Exception {
for (int i = 0; i < 100; i++) {
var vectorData = new VectorScorerInt4BulkBenchmark.VectorData(dims, 1000, 200);
var scalar = createBench(VectorImplementation.SCALAR, vectorData);
var lucene = createBench(VectorImplementation.LUCENE, vectorData);
var nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
VectorScorerInt4BulkBenchmark lucene = null;
VectorScorerInt4BulkBenchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, vectorData);
nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
}

try {
float[] expected = scalar.scoreMultipleSequentialBulk();
assertArrayEquals("LUCENE sequentialBulk", expected, lucene.scoreMultipleSequentialBulk(), delta);
assertArrayEquals("NATIVE sequentialBulk", expected, nativeBench.scoreMultipleSequentialBulk(), delta);
if (supportsHeapSegments()) {
assertArrayEquals("LUCENE sequentialBulk", expected, lucene.scoreMultipleSequentialBulk(), delta);
assertArrayEquals("NATIVE sequentialBulk", expected, nativeBench.scoreMultipleSequentialBulk(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}
Expand All @@ -128,36 +162,53 @@ public void testRandomBulk() throws Exception {
for (int i = 0; i < 100; i++) {
var vectorData = new VectorScorerInt4BulkBenchmark.VectorData(dims, 1000, 200);
var scalar = createBench(VectorImplementation.SCALAR, vectorData);
var lucene = createBench(VectorImplementation.LUCENE, vectorData);
var nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
VectorScorerInt4BulkBenchmark lucene = null;
VectorScorerInt4BulkBenchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, vectorData);
nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
}

try {
float[] expected = scalar.scoreMultipleRandomBulk();
assertArrayEquals("LUCENE randomBulk", expected, lucene.scoreMultipleRandomBulk(), delta);
assertArrayEquals("NATIVE randomBulk", expected, nativeBench.scoreMultipleRandomBulk(), delta);
if (supportsHeapSegments()) {
assertArrayEquals("LUCENE randomBulk", expected, lucene.scoreMultipleRandomBulk(), delta);
assertArrayEquals("NATIVE randomBulk", expected, nativeBench.scoreMultipleRandomBulk(), delta);
}
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}

public void testQueryRandomBulk() throws Exception {

for (int i = 0; i < 100; i++) {
var vectorData = new VectorScorerInt4BulkBenchmark.VectorData(dims, 1000, 200);
var scalar = createBench(VectorImplementation.SCALAR, vectorData);
var lucene = createBench(VectorImplementation.LUCENE, vectorData);
var nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
VectorScorerInt4BulkBenchmark lucene = null;
VectorScorerInt4BulkBenchmark nativeBench = null;
if (supportsHeapSegments()) {
lucene = createBench(VectorImplementation.LUCENE, vectorData);
nativeBench = createBench(VectorImplementation.NATIVE, vectorData);
}

try {
float[] expected = scalar.scoreQueryMultipleRandomBulk();
assertArrayEquals("LUCENE queryRandomBulk", expected, lucene.scoreQueryMultipleRandomBulk(), delta);
assertArrayEquals("NATIVE queryRandomBulk", expected, nativeBench.scoreQueryMultipleRandomBulk(), delta);
if (supportsHeapSegments()) {
assertArrayEquals("LUCENE queryRandomBulk", expected, lucene.scoreQueryMultipleRandomBulk(), delta);
assertArrayEquals("NATIVE queryRandomBulk", expected, nativeBench.scoreQueryMultipleRandomBulk(), delta);
}
Comment on lines 188 to +205
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the query-scoring tests (e.g., testQueryRandomBulk()), the method still runs when supportsHeapSegments() is false but then skips all lucene/native assertions and setup. This doesn't mark the test as ignored/skipped on Java 21 and still does the scalar work inside the loop. Consider adding assumeTrue("Only test with heap segments", supportsHeapSegments()) at the start of query-scoring test methods (as done in other scorer benchmark tests) and removing the repeated per-iteration conditional checks.

Copilot uses AI. Check for mistakes.
} finally {
scalar.teardown();
lucene.teardown();
nativeBench.teardown();
if (supportsHeapSegments()) {
lucene.teardown();
nativeBench.teardown();
}
}
}
}
Expand Down
Loading