Java Integration Tests Failing In CI

[tustvold]

Java Integration tests are failing with some strange error about not being able to find flight

https://github.com/apache/arrow-rs/actions/runs/9284625938/job/25547387572

 FAILED TEST: custom_metadata Rust producing,  Java consuming
<class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND -XX:-UsePerfData --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-17.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 39249 -j /tmp/arrow-integration-oo50s7yc/generated_custom_metadata.json
With output:
--------------
WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
Unknown error
Could not find flight. /tmp/arrow-integration-oo50s7yc/generated_custom_metadata.json

[tustvold]

@lidavidm perhaps you might know what is causing this, or know someone who might (I'm just going off the git history of the Java arrow impl) 😅

[lidavidm]

The upstream pipeline seems fine...

[lidavidm]

assertion failed: variadic_counts.is_empty()

@vibhatha seems like the recent changes did affect IPC? But in a way that only affects Rust?

[vibhatha]

Yes, I looked into the code,

arrow-rs/arrow-ipc/src/reader.rs

Line 590 in 7fe01bb

assert!(variadic_counts.is_empty());

here it checks for the variadic_count and it is only failing one branch of the code (else branch).

In Java, we added the missing spec attribute variadicBufferCounts. Maybe this caused an issue in the Rust integration tests?

[tustvold]

Why would variadicBufferCounts be populated for payloads that don't contain variadic data?

2024-05-29T11:06:56.3475547Z FAILED TEST: interval_mdn Java producing,  Rust consuming
2024-05-29T11:06:56.3477808Z <class 'RuntimeError'>: Command failed: /build/rust/debug/arrow-json-integration-test --integration --arrow=/tmp/tmpx06ebyg3/5302f58e_generated_interval_mdn.json_as_file --json=/tmp/arrow-integration-oo50s7yc/generated_interval_mdn.json --mode=VALIDATE
2024-05-29T11:06:56.3479652Z With output:
2024-05-29T11:06:56.3479985Z --------------
2024-05-29T11:06:56.3480494Z thread 'main' panicked at arrow-ipc/src/reader.rs:590:9:
2024-05-29T11:06:56.3481312Z assertion failed: variadic_counts.is_empty()
2024-05-29T11:06:56.3482053Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This possibly shouldn't be an assertion failure in Rust but a regular error, but the data Java is producing is malformed is it not?

[lidavidm]

Can't they just all be 0?

[tustvold]

The specification states that they should only contain entries for the ones that contain variadic buffers - https://github.com/apache/arrow-rs/blob/master/format/Message.fbs#L103

[lidavidm]

Hmm, Java is probably wrong then

[tustvold]

I've filled apache/arrow#41902

[vibhatha]

@tustvold we have marged a fix for the filed issue. Could you please check the CIs?

[tustvold]

All green now, thank you