Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench

[nvzm123]

Summary

This PR adds a built-in pylucene backend to cuVS Bench for building and searching local Lucene vector indexes through PyLucene and the production codecs proposed in NVIDIA/cuvs-lucene#174.

The implementation targets the current PR #174 API at fda97b3d: Lucene101AcceleratedHNSWCodec for cuVS-assisted HNSW construction with Lucene HNSW search, and CuVS2510GPUSearchCodec for CAGRA construction and search. It follows cuVS Bench's backend/config-loader architecture and the current in-process result-export contract.

Backend behavior

• Registers pylucene as a built-in backend and configuration loader, with packaged pylucene_cuvs_hnsw and pylucene_cuvs_cagra configurations.
• Lazily initializes PyLucene's process-wide JVM with separately supplied cuvs-java and thin cuvs-lucene JARs, validates classpath inputs, configures the native-library search path, and resolves production codecs through Lucene SPI.
• Builds indexes through Lucene IndexWriter and searches through the public KnnFloatVectorQuery API.
• Supports deterministic configuration selection, custom overrides, dry runs, index reuse, forced rebuilds, and query batching.
• Returns the standard BuildResult and List[SearchResult] models, including throughput, per-batch latency, percentiles, recall inputs, and backend metadata.

The HNSW codec uses its production contract: it selects the cuVS writer when GPU support is available and intentionally falls back, with cuVS-Lucene's warning, to Lucene's CPU HNSW writer otherwise. The CAGRA codec requires GPU support and is validated fail-closed before its results are accepted.

Index integrity and lifecycle

The backend writes atomic, commit-bound provenance manifests for HNSW and CAGRA indexes. Reuse and search validate the codec, writer policy, expected dataset shape, and Lucene commit fingerprint before accepting an existing index.

CAGRA indexes receive additional structural and checksum validation through Lucene's codec APIs, including segment/field metadata, vector counts and dimensions, data-file coverage, codec headers and footers, deletions, and checksums. Missing, stale, truncated, corrupted, brute-force-fallback, or non-CAGRA vector segments are rejected before search.

Index paths and result identities are validated and contained beneath their configured roots. Failed new builds remove partial output without deleting unrelated or caller-untrusted paths, while preflight failures preserve existing indexes.

Result and CLI integration

The backend uses cuVS Bench's current direct CSV exporter for in-process results. The shared result handling added here:

• joins build and search data by logical index identity rather than physical paths;
• preserves latency percentiles and keeps algorithm/group/subset series and codec-specific index identities distinct;
• validates path components before creating artifacts;
• selectively removes stale search artifacts after a failed build; and
• exports any successful measurements before returning a non-zero CLI outcome for failed or empty Python-backend sweeps.

Supported inputs and current limits

• FLOAT32 datasets using Euclidean/L2 distance
• dimensions up to 4096 and at least two indexed vectors
• latency mode with one search thread
• query batching through --batch-size
• CAGRA searches with k <= 1024

The public Lucene query API does not expose backend-specific search tuning parameters. PyLucene's JVM is process-global, so JAR paths, native-library paths, and JVM arguments must be fixed before the first PyLucene run in a process.

Dependencies and documentation

PyLucene remains an external, source-built dependency. The documented environment uses JDK 22, PyLucene/Lucene 10.2.0, matching cuVS 26.10 native libraries and cuvs-java, and the standard thin cuVS-Lucene JAR. Fat cuVS-Lucene JARs are rejected because PyLucene supplies Lucene's classes.

The installation and usage guides cover dependency builds, artifact validation, runtime configuration, the HNSW fallback and CAGRA contracts, supported limits, index reuse, result semantics, and a complete manual smoke workflow. The dependency manifest also declares h5py, which the existing dataset-preparation commands import at runtime.

Compatibility remains provisional while PR #174 is under review. The live validation used its exact current head fda97b3d, PyLucene 10.2.0, and matching cuVS 26.10.0 native/Java artifacts.

Test coverage

The pytest suites cover registration and configuration, lazy imports, JVM/classpath/native validation, thin-JAR enforcement, SPI resolution, build/reuse/rebuild/cleanup behavior, query conversion and batching, result metrics, provenance mismatches, path containment, CAGRA segment traversal and corruption cases, direct CSV export, and CLI failure propagation.

The opt-in integration suite uses a downstream test-only Java codec adapter that delegates to the real production HNSW codec and wraps only fieldsWriter() diagnostics. Stock PyLucene instantiates it through Class.forName(...).newInstance(), allowing the test to prove the concrete writer selected without adding production APIs or SPI names to cuVS-Lucene. Live cases exercise both production codecs, real JVM/GPU build and search, index reuse, recall/distance checks, CAGRA fallback rejection, integrity failures, and a fresh-process CLI workflow.

Validation

• python -m pytest -q cuvs_bench/tests: 408 passed, 31 skipped
• Focused PyLucene suites with branch measurement enabled: 212 passed, 89% coverage for pylucene.py
• Opt-in real PyLucene/JVM/cuVS integration suite: 5 passed
• Current PR [FEA] expose python & C API for prefiltered brute force #174 direct PyLucene suite: 23 passed
• Current PR [FEA] expose python & C API for prefiltered brute force #174 Maven suite: 298 tests, 30 skipped, 0 failures or errors
• Pre-commit hooks for the complete changed-file set: passed
• fern/build_docs.sh check: 248 MDX files valid, 0 errors (2 pre-existing warnings)

Related work

• NVIDIA/cuvs#1907
• NVIDIA/cuvs-lucene#174

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.