[Dev][Archery][Integration] Reduce needless test matrix

[kou]

Describe the enhancement requested

Archery generates product of all testers:

arrow/dev/archery/archery/integration/runner.py

Lines 88 to 90 in 8556001

	for producer, consumer in itertools.product(
	filter(lambda t: t.PRODUCER, self.testers),
	filter(lambda t: t.CONSUMER, self.testers)):

If we enable C++, Java and Rust, we use the following patterns:

Producer	Consumer
C++	C++
C++	Java
C++	Rust
Java	C++
Java	Java
Java	Rust
Rust	C++
Rust	Java
Rust	Rust

In apache/arrow, the following patterns are redundant because they should be done in apache/arrow-rs:

Producer	Consumer
Rust	Rust

In apache/arror-rs, the following patterns are redundant because they should be done in apache/arrow:

Producer	Consumer
C++	C++
C++	Java
Java	C++
Java	Java

Can we remove redundant patterns?

Component(s)

Developer Tools, Integration

[kou]

@alamb @tustvold @paleolimbot @zeroshade @pitrou What do you think about this?

[pitrou]

No opinion from me. I suppose it will make the test a bit shorter, but not by much since it's still O(n**2) where n is the number of implementations.

[alamb]

Seems like a good idea to me. Thanks @kou

[zeroshade]

Also seems like a good idea to me too, thanks @kou

[paleolimbot]

I would also love this for the integration run in nanoarrow (i.e., I would rather only run the nanoarrow-related scenarios!).

[kou]

Thanks for sharing your opinions.
I'll implement this.

[alamb]

In apache/arror-rs, the following patterns are redundant because they should be done in apache/arrow:

Can you give me some hint about how to reduce the test matrix in arrow-rs to remove the redundancies?

Here is how the job is run

https://github.com/apache/arrow-rs/blob/889a90436ffd69e94975049b047c413c53b2dfe3/.github/workflows/integration.yml#L56-L67

Perhaps the idea is to wait for #44099 to merge and then add something equivalent to

: ${ARROW_INTEGRATION_TARGET_LANGUAGES:=cpp,csharp,go,java,js}

[kou]

Yes. The following change is needed with the current #44099 implementation:

diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
index 41edc1bb19..dcaf03c44a 100644
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -64,6 +64,7 @@ jobs:
       ARROW_INTEGRATION_GO: ON
       ARROW_INTEGRATION_JAVA: ON
       ARROW_INTEGRATION_JS: ON
+      ARCHERY_INTEGRATION_TARGET_LANGUAGES: "rust"
       ARCHERY_INTEGRATION_WITH_NANOARROW: "1"
       # https://github.com/apache/arrow/pull/38403/files#r1371281630
       ARCHERY_INTEGRATION_WITH_RUST: "1"

But this may be changed.

I'll open a PR in apache/arrow-rs after I complete this. Please wait for a while.

[alamb]

Thank you @kou

[kou]

Issue resolved by pull request 44099
#44099

[kou]

apache/arrow-rs#6406