GH-825: Add UUID canonical extension type #903

jhrotko · 2025-11-03T14:51:38Z

What's Changed

Add Uuid Extension type canonical implementation in Java. The proposed implementation was validated in Dremio engine

Closes #825

lidavidm

See previous PR. It is not sufficient to move the existing files, the "uuid" type used in tests was never meant to implement the canonical type. #824

jhrotko · 2025-11-07T13:59:16Z

@lidavidm could you take another look? Let me know if anything else is missing

bodduv

Thank you for adding the canonical type.

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java

vector/src/main/java/org/apache/arrow/vector/complex/impl/UuidWriterImpl.java

vector/src/main/java/org/apache/arrow/vector/FixedSizeExtensionType.java

vector/src/main/java/org/apache/arrow/vector/UuidVector.java

jhrotko · 2025-11-11T12:18:42Z

there are some integration tests failling:

  ################# FAILURES #################
  FAILED TEST: custom_metadata Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-json-integration-test -arrow /tmp/tmprky9nvsk/020d3b5c_generated_custom_metadata.json_as_file -json /tmp/arrow-integration-ypsmstn1/generated_custom_metadata.json -mode VALIDATE
  With output:
  --------------
  arrow-json: JSON schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  Arrow schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]
  arrow-json: schemas did not match
  
  --------------
  
  
  FAILED TEST: flight_sql Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=43857 -scenario flight_sql
  With output:
  --------------
  gc 1 @3.115s 0%: 0.078+0.59+0.008 ms clock, 0.31+0.37/0.34/0.60+0.032 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
  panic: expected: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
  
  --------------
  
  
  FAILED TEST: flight_sql:extension Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=35551 -scenario flight_sql:extension
  With output:
  --------------
  panic: expected: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
  
  --------------
  
  
  FAILED TEST: flight_sql Go producing,  Java consuming
  <class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 40707 -scenario flight_sql
  With output:
  --------------
  WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  Unknown error
  expected: '%!s(bool=true)', got: '%!s(bool=false)'
  
  --------------
  
  
  FAILED TEST: flight_sql:extension Go producing,  Java consuming
  <class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 32823 -scenario flight_sql:extension
  With output:
  --------------
  WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  Unknown error
  expected: '%!s(bool=true)', got: '%!s(bool=false)'
  
  --------------
  
  
  FAILED TEST: custom_metadata Java producing,  Go consuming
  <class 'RuntimeError'>: Go C Data Integration call failed: Schemas are different:
  - Json Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  - Imported Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]

It seems the metadata elements when it's a map the order is not maintained. I this a known issue? @lidavidm

lidavidm · 2025-11-11T13:04:32Z

It's apache/arrow-go#571

jhrotko · 2025-11-11T16:07:04Z

@lidavidm it was merged, should this be re-run?

jhrotko · 2025-11-12T08:22:51Z

@lidavidm anything else missing?

bodduv

Thank you for addressing the comments.

lidavidm

This has conflicts

lidavidm · 2025-11-19T10:59:25Z

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java

+  /** Registers the UuidType in the extension type registry. */
+  private static final AtomicBoolean registered = new AtomicBoolean(false);
+
+  static {
+    ExtensionTypeRegistry.register(INSTANCE);
+  }
+
+  /** Register the extension type so it can be used globally. */
+  public static void ensureRegistered() {
+    if (!registered.getAndSet(true)) {
+      // The values don't matter, we just need an instance
+      ExtensionTypeRegistry.register(UuidType.INSTANCE);
+    }
+  }


Isn't all of this redundant? If we have the static initializer, then ensureRegistered can just be a no-op. Or if we're going to use the atomic boolean, then we don't need the static initializer.

this is only used in tests similarly to Opaque extension type

There is no static initializer there. There is one here.

Thank you! I added the static implementation following your suggestion

github-actions bot added the breaking-change label Nov 3, 2025

This comment has been minimized.

Sign in to view

jhrotko force-pushed the GH-825 branch from 595d2ae to 4e8de67 Compare November 3, 2025 14:53

lidavidm requested changes Nov 3, 2025

View reviewed changes

jhrotko changed the title ~~GH-825: Move UuidType to arrow.vector from test~~ GH-825: Implement Uuid canonical type Nov 7, 2025

jhrotko changed the title ~~GH-825: Implement Uuid canonical type~~ GH-825: Add a canonical extension type UUID Nov 7, 2025

jhrotko force-pushed the GH-825 branch from a298920 to ebca624 Compare November 7, 2025 13:58

jhrotko marked this pull request as ready for review November 7, 2025 13:58

jhrotko requested review from jbonofre, laurentgo and wgtmac as code owners November 7, 2025 13:58

jhrotko changed the title ~~GH-825: Add a canonical extension type UUID~~ GH-825: Add UUID canonical extension type Nov 7, 2025

bodduv reviewed Nov 9, 2025

View reviewed changes

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java Show resolved Hide resolved

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java Show resolved Hide resolved

lidavidm reviewed Nov 10, 2025

View reviewed changes

lidavidm added enhancement PRs that add or improve features. and removed breaking-change labels Nov 11, 2025

github-actions bot added this to the 18.4.0 milestone Nov 11, 2025

bodduv approved these changes Nov 14, 2025

View reviewed changes

jhrotko requested a review from lidavidm November 14, 2025 14:13

lidavidm reviewed Nov 17, 2025

View reviewed changes

jhrotko added 4 commits November 17, 2025 13:00

apacheGH-825: Move UuidType to arrow.vector from test

334cd0b

Add canonical implementation

c088e10

Address comments

931408d

Remove FixedSizeExtensionType

da42227

jhrotko force-pushed the GH-825 branch from b04af47 to da42227 Compare November 17, 2025 13:02

jhrotko requested a review from lidavidm November 17, 2025 13:56

lidavidm reviewed Nov 19, 2025

View reviewed changes

remove ensureRegistered method from UuidType

9b7c583

GH-825: Add UUID canonical extension type #903

Are you sure you want to change the base?

GH-825: Add UUID canonical extension type #903

Uh oh!

Conversation

jhrotko commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's Changed

Uh oh!

This comment has been minimized.

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

jhrotko commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

lidavidm commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 12, 2025

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

lidavidm Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhrotko Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

lidavidm Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jhrotko Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jhrotko commented Nov 3, 2025 •

edited

Loading

jhrotko commented Nov 7, 2025 •

edited

Loading