Spark 3.5: Backport: Refactor Spark procedures to consistently use ProcedureInput for parameter handling. #14179
Conversation
…ocedureInput for parameter handling.
slfan1989
changed the title
| return ProcedureParameter.required(name, dataType); | ||
| } | ||
|
|
||
| protected static ProcedureParameter optionalInParameter(String name, DataType dataType) { | ||
| return ProcedureParameter.optional(name, dataType); | ||
| } |
There was a problem hiding this comment.
any reason why we didn't have this in 4.0 ? are we missing backport of some commit to 3.5
seems like not - 30ee7e8
can you please double confirm ?
There was a problem hiding this comment.
Thank you very much for reviewing the code! This is not a clean backport, since there is a PR (#13106) in the Spark 4.0 branch that cannot be directly backported to Spark 3.4/3.5. Because Spark 3.5 lacks some features compared to Spark 4.0, I made certain adjustments based on the current Spark 3.5 code.
There was a problem hiding this comment.
In Spark 3.5, we use org.apache.spark.sql.connector.iceberg.catalog.ProcedureParameter for parameter parsing, whereas in the Spark 4 branch, org.apache.spark.sql.connector.catalog.procedures.ProcedureParameter is used to achieve the same functionality. Personally, I believe the difference in functionality between the two is not significant.
|
@singhpk234 @nastra @huaxingao Thank you very much for helping to review the code! |
…ut for parameter handling. (apache#14179)
…ut for parameter handling. (apache#14179)
…ut for parameter handling. (apache#14179)
4 participants
Backport #13913
Compared to Spark 4.0, Spark 3.5 is missing PR #13106, which cannot be directly backported. Therefore, some code modifications are required.