-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-34499][SQL] Improve the catalyst expression dsl internal APIs #31612
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now "2018-03-18".date returns a date type attribute, so I updated this test to use a method.
|
@sarutak how about this? |
| } | ||
| implicit class DslAttr(attr: UnresolvedAttribute) extends ImplicitAttribute { | ||
| def s: String = attr.name | ||
| implicit class DslString(str: String) extends ImplicitAttribute { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not s: String ?
|
Hmm, it seems to break the usage of string literals to represent |
|
Kubernetes integration test starting |
|
Test build #135353 has finished for PR 31612 at commit
|
|
Kubernetes integration test status success |
|
It's internal so I think we can update the DSL API to the best shape we want. However, it does raise some issues after I looking into the test failures: Previously they all create expressions with string literal, and breaking them seems a much bigger impact than I thought. I tend to agree that we should create a new syntax for creating attributes from strings. |
What changes were proposed in this pull request?
The current expression dsl API heavily relies on Symbol, which is deprecated in Scala 3. However, the string support in the expression dsl API is pretty weak.
This PR improves the dsl API to let
"name".intreturn an int attribute, which is the same as'name.int. The same to other types.To do this, this PR removes the string literal implicit, which is not widely used, and adds a new string implicit that creates attributes, which is the same as the symbol implicit. As a consequence,
"abc" like ...now creates a Like expression with attributeabc, not string literal "abc". To use literal, this PR adds a new way:"abc".lit like ...Why are the changes needed?
To slowly retire Symbol.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Comment out
DslStringand compile, and see which places rely on string implicits. Also check failed tests.Existing tests