Coral-Spark: Build awareness in Coral for coalsce_struct UDF

[autumnust]

This patch includes change to swap extract_union UDF with coalesce_struct UDF in Coral-Spark during Coral RelNode -> Spark RelNode.

In #192, we have changed the interpretation of union field in IR to a struct that aligns with Trino's after-explosion schema. The output struct which contains tag field and names each member field as fieldN is backward incompatible with the schema from extract_union UDF. This patch is need to bridge the schema difference between both.

---

Note that coalesce_struct UDF will be statically registered.

[wmoustafa]

Should the coalesce_struct UDF be part of Coral?

[wmoustafa]

Should both Spark and Trino be handled in the same PR?

[autumnust]

Should the coalesce_struct UDF be part of Coral?

Why is that if that is going to be a Hive UDF, or do you think it should be statically defined in Coral and being loaded into engines when being used ?

[autumnust]

Should both Spark and Trino be handled in the same PR?

Before the change, only Spark IR Transformer (during IR -> Spark RelNode) handles the extract_union UDF (by getting rid of it). Trino path simply cannot execute SQL statements with extract_union as part of it. Do we want to preserve that behavior or you believe we should also get rid of extract_union in the trino SQL whenever appearing ?

[wmoustafa]

Should the coalesce_struct UDF be part of Coral?

Why is that if that is going to be a Hive UDF, or do you think it should be statically defined in Coral and being loaded into engines when being used ?

I meant the GenericUDF definition, yes. But, I think it is okay if the function is not part of Coral.

[wmoustafa]

Should both Spark and Trino be handled in the same PR?

Before the change, only Spark IR Transformer (during IR -> Spark RelNode) handles the extract_union UDF (by getting rid of it). Trino path simply cannot execute SQL statements with extract_union as part of it. Do we want to preserve that behavior or you believe we should also get rid of extract_union in the trino SQL whenever appearing ?

Ok, makes sense. We can restrict extract_union usage to Spark.

[autumnust]

@wmoustafa Can you take a second look ? thanks !

[ljfgem]

Thanks @autumnust for this PR!

[autumnust]

Looks like the Github action is failing with weird reason:

> Task :coral-hive:compileTestJava FAILED
    ToRelConverterTestUtils.setup();
                           ^
  required: HiveConf
  found: no arguments
  reason: actual and formal argument lists differ in length
Note: /home/runner/work/coral/coral/coral-hive/src/test/java/com/linkedin/coral/hive/hive2rel/parsetree/ParseTreeBuilderTest.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 error

@ljfgem have you seen this ? this failure couldn't be reproduced locally and the error message doesn't seem to make sense since there's only one setup method without no arguments. Binding issue ?

[wmoustafa]

@ljfgem have you seen this ? this failure couldn't be reproduced locally and the error message doesn't seem to make sense since there's only one setup method without no arguments. Binding issue ?

You probably need to rebase. This method has changed in #198.

[ljfgem]

Thanks @autumnust for this patch!

[ljfgem]

LGTM, thanks
@wmoustafa @funcheetah PTAL if you get a chance.

[funcheetah]

Thanks @autumnust ! LGTM