Add BLOCK_AND_POSITION calling convention to annotation framework#10961
Add BLOCK_AND_POSITION calling convention to annotation framework#10961gerlou merged 4 commits intoprestodb:masterfrom
Conversation
|
Change commit message to You don't need a separate PR for this. One PR in GitHub can have multiple commits. This PR is a precursor step of a larger PR you are going to put up. As a result, we'll have this commit as part of that PR, instead of creating a standalone PR for this. |
697a3c0 to
802ad6b
Compare
There was a problem hiding this comment.
Why Move @BlockPosition and @BlockIndex to SPI commit moves other classes around as well?
There was a problem hiding this comment.
@findepi : Separated into a different commit now.
There was a problem hiding this comment.
Is this code change directly related to moving BlockPosition and BlockIndex annotations to SPI?
8d5f43c to
119588c
Compare
|
A few high-level comments:
Let's work together offline on this rebase.
|
wenleix
left a comment
There was a problem hiding this comment.
Some quick comments after glancing. Let's first separate the commits.
There was a problem hiding this comment.
Move this fix to a separate commit (the third commit noted in #10961 (comment))
There was a problem hiding this comment.
nit: need to update this comment
There was a problem hiding this comment.
Originally, we thought Signature itself can distinguish the different PSI. Thus we only use Signature as the hash key.
Now since Signature itself is not enough to distinguish PSI, and argument native container + specialize native container type are required. The current implementation is incorrect since only one ParametricScalarImplementation will be put into the signature.
Lets put Signature, argument and return value native container type, type parameter specialization native container type into an inner class SpecializedSignature, and use that as key.
There was a problem hiding this comment.
assertEquals is for assertion in test only. Here, you can use checkArgument. In addition, you would want to provide an error messages. The audience of the error message is people who are implementing custom SQL functions through the annotation message.
There was a problem hiding this comment.
This if should not be in if (method.getParameterCount() > (i + 1)). Instead, you would want to move it out so that you check for this annotation regardless of whether a next argument exists. In addition, you would want to assert that 1) the next argument exists, 2) the next argument has @BlockIndex annotation.
There was a problem hiding this comment.
With these 3 boolean flags nullableArgument, hasNullFlag, and isBlockAndPositionNullConvention, there exists 8 possible combinations. However, only 4 of them is legal.
Previously, there were 4 possible combinations, but only 3 of them is legal. That was not ideal. However, It was not too hard to 1) conclude that the illegal one is impossible; 2) parser fails loudly when both @IsNull and @SqlNullable is present.
With the addition, reasoning about that just became pretty much intractable.
There are two alternatives ways to address this:
- This code is organized in such a way that consumption of these variables and production of these variables are clearly split. At the split, add assertion.
- Use an enum instead of 3 flags.
I highly recommend taking the 2nd alternative.
There was a problem hiding this comment.
Make your test contains all 3 of: generic, specialize, and exact. Ideally, there should be at least of specialized ones, and 2 exact ones.
|
@wenleix When you reference |
119588c to
03b003c
Compare
9fa7194 to
4c67825
Compare
4c67825 to
ec298ad
Compare
wenleix
left a comment
There was a problem hiding this comment.
The first two commits look good.
Move @BlockPosition and @BlockIndex to SPIRename ScalarImplementation to ParametricScalarImplementation
@haozhun : Do we also want to rename ScalarFunctionChoice to ScalarFunctionImplementationChoice ?
|
@findepi A function that implements |
| this.constructorDependencies = ImmutableList.copyOf(requireNonNull(constructorDependencies, "constructorDependencies is null")); | ||
| this.argumentNativeContainerTypes = ImmutableList.copyOf(requireNonNull(argumentNativeContainerTypes, "argumentNativeContainerTypes is null")); | ||
| this.specializedTypeParameters = ImmutableMap.copyOf(requireNonNull(specializedTypeParameters, "specializedTypeParameters is null")); | ||
| this.choices = choices; |
There was a problem hiding this comment.
nit: Guard these by requireNonNull, do this for returnContainerType and specializedSignature as well.
| return Optional.empty(); | ||
| } | ||
| } | ||
| Class<?> returnContainerType = getNullAwareReturnType(typeManager.getType(boundSignature.getReturnType()).getJavaType(), nullable); |
There was a problem hiding this comment.
nit: Remove unused method getNullAwareReturnType
| else { | ||
| if (argumentProperty.getArgumentType() != VALUE_TYPE) { | ||
| return Optional.empty(); | ||
| for (ParametricScalarImplementationChoice choice : choices) { |
There was a problem hiding this comment.
At high level, whether the ParametricScalarImplementation (PSI) suited for the specialization should be a decision at PSI level without consulting underlying ParametricScalarImplementationChoice (PSIC).
I understand currently some information is only in PSIC. Let's discuss offline about how to reorg fields between PSI / PSIC.
|
|
||
| this.methodHandle = getMethodHandle(method); | ||
|
|
||
| ParametricScalarImplementationChoice choice = new ParametricScalarImplementationChoice(nullable, argumentProperties, methodHandle, constructorMethodHandle, specializedTypeParameters, argumentNativeContainerTypes, numberOfBlockPositionArguments); |
There was a problem hiding this comment.
numberOfBlockPositionArguments should be calculated in the constructor instead of passed in by caller.
| private final List<ArgumentProperty> argumentProperties; | ||
| private final MethodHandle methodHandle; | ||
| private final Optional<MethodHandle> constructor; | ||
| private final Map<String, Class<?>> specializedTypeParameters; |
There was a problem hiding this comment.
specializedTypeParameters should be in ParametricScalarImplementation
| argumentNativeContainerTypes.add(type.nativeContainerType()); | ||
| } | ||
| else { | ||
| checkArgument(type.nativeContainerType().equals(Object.class), "Native container type for parameter needs to be an Object"); |
There was a problem hiding this comment.
I don't understand this check?
There was a problem hiding this comment.
I am also thinking, when parameterType from the method itself is Object, we should enforce nativeContainerType is annotated.
So we always have the Java type for the scalar function in PSIC.
cc @haozhun
2dba166 to
f2e53d3
Compare
93877c1 to
58c77e9
Compare
wenleix
left a comment
There was a problem hiding this comment.
Some quick comments.
Let's talk about the comments about nullConventionFlag part in person.
There was a problem hiding this comment.
nit: I squash the fixup commit to the wrong commit. This two lines should go to the first commit...
There was a problem hiding this comment.
nit: Cleanup these . Same for other debugging comments :)
There was a problem hiding this comment.
Also check
constructorDependenciesconstructor
There was a problem hiding this comment.
Since we have done unwrap for argumentNativeContainerTypes already, I think we can do
if (!(argumentNativeContainerTypes.get(i).get() == Object.class || argumentNativeContainerTypes.get(i).get() == argumentType)) {
...
}There was a problem hiding this comment.
This should never happen. Either remove it or make it to be an checkState statement.
There was a problem hiding this comment.
nit: I would put an numberOfBlockPositionArguments before the for loop
There was a problem hiding this comment.
It's really one-to-one mapping with NullConvention, let's not create this new enum.
There was a problem hiding this comment.
Use a local variable for nullConventionFlag here (instead of using the field)
There was a problem hiding this comment.
Here NullConventionFlag is indicating the return convention, which is not a natural usage.
2591ffe to
39a5fd3
Compare
|
Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours has expired. Before we can review or merge your code, we need you to email cla@fb.com with your details so we can update your status. |
wenleix
left a comment
There was a problem hiding this comment.
Generally looks good. I will take a look at tests later.
Remember to reorg changes to BlockIndex / BlockPosition to the first commit. (in ApproximateCountDistinctAggregation.java and TestAnnotationEngineForAggregates.java
@haozhun can you also start to review ? :)
| Class<?> argumentType = typeManager.getType(boundSignature.getArgumentTypes().get(i)).getJavaType(); | ||
| Class<?> argumentContainerType = getNullAwareContainerType(argumentType, argumentProperty.getNullConvention()); | ||
| if (!argumentNativeContainerTypes.get(i).isAssignableFrom(argumentContainerType)) { | ||
| if (!(argumentNativeContainerTypes.get(i).get().isAssignableFrom(argumentType))) { |
There was a problem hiding this comment.
How does this work when argumentNativeContainerTypes is Object.class and argumentType is primitive type? isAssignableFrom will return false.
Does that mean such case never exists?
There was a problem hiding this comment.
nit: remove the redundant brackets !(...) -> !....
@wenleix There is indeed no code that depends on it. TypeOfFunction is an interesting example that demonstrates the deficiency here.
Supporting it is not hard, but also nontrivial.
- Boxing will be necessary for primitives.
- Downcasting will be necessary for return types.
- Once the support is added, this place can be changed to
argumentNativeContainerType != Object.class && argumentNativeContainerType == argumentType(NullConventionis no longer available here.)
| public SpecializedSignature getSpecializedSignature() | ||
| { | ||
| return dependencies; | ||
| SpecializedSignature specializedSignature = new SpecializedSignature( |
There was a problem hiding this comment.
nit: You can create the SpecializedSignature and return it in one step.
return new SpecializedSignature(
signature,
argumentNativeContainerTypes,
specializedTypeParameters,
returnNativeContainerType);| // USE_NULL_FLAG or RETURN_NULL_ON_NULL | ||
| checkState(parameterType == Void.class || !Primitives.isWrapperType(parameterType), "Method [%s] has parameter expected to use USE_NULL_FLAG or RETURN_NULL_ON_NULL null convention. Parameter type is unexpected: %s", method, parameterType.getSimpleName()); | ||
|
|
||
| boolean useNullFlag = false; |
There was a problem hiding this comment.
@haozhun Any comments about this auxiliary boolean flag?
39a5fd3 to
4706596
Compare
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
|
A better commit message would be something like The original message did not provide a high level context. As a result, a reader wouldn't know what it is about. |
haozhun
left a comment
There was a problem hiding this comment.
Add BLOCK_AND_POSITION calling convention to annotation framework
There was a problem hiding this comment.
Don't remove this assertion:
checkCondition(methodHandleAndConstructor.isPresent(), FUNCTION_IMPLEMENTATION_ERROR, String.format("Exact implementation of %s do not match expected java types.", boundSignature.getName()));
There was a problem hiding this comment.
For all 3,
checkCondition(..., FUNCTION_IMPLEMENTATION_ERROR, "Some descriptive message");
| Class<?> argumentType = typeManager.getType(boundSignature.getArgumentTypes().get(i)).getJavaType(); | ||
| Class<?> argumentContainerType = getNullAwareContainerType(argumentType, argumentProperty.getNullConvention()); | ||
| if (!argumentNativeContainerTypes.get(i).isAssignableFrom(argumentContainerType)) { | ||
| if (!(argumentNativeContainerTypes.get(i).get().isAssignableFrom(argumentType))) { |
There was a problem hiding this comment.
nit: remove the redundant brackets !(...) -> !....
@wenleix There is indeed no code that depends on it. TypeOfFunction is an interesting example that demonstrates the deficiency here.
Supporting it is not hard, but also nontrivial.
- Boxing will be necessary for primitives.
- Downcasting will be necessary for return types.
- Once the support is added, this place can be changed to
argumentNativeContainerType != Object.class && argumentNativeContainerType == argumentType(NullConventionis no longer available here.)
There was a problem hiding this comment.
Remove. This is no longer used.
There was a problem hiding this comment.
Add comment that all choices are required to have the same constructor dependencies at this time. And point out that this is asserted in the constructor.
There was a problem hiding this comment.
Use a Builder if necessary. Making this class mutable is:
- surprising and unintuitive
- undermines your validations in the constructor.
There was a problem hiding this comment.
@SqlType can only contain an explicitly specified nativeContainerType when using @BlockPosition
There was a problem hiding this comment.
There is a bug here. The message is wrong.
The return type in PrimitiveParameterWithNullable is long. But that's correct because there is no @SqlNullable annotation on the method.
There was a problem hiding this comment.
I can't read this sentence. Please change the error message and this assertion to
A parameter with USE_NULL_FLAG or RETURN_NULL_ON_NULL convention must not use wrapper type. Found in method ...
There was a problem hiding this comment.
The assertions should have use
checkCondition(..., FUNCTION_IMPLEMENTATION_ERROR, ...)
Fix all in ScalarFromAnnotationsParser and ParametricScalarImplementation after this PR is merged.
5273ef1 to
eaf0399
Compare
There was a problem hiding this comment.
Implementations for the same function signature must have matching dependencies: <signature>
Ditto for the next two.
eaf0399 to
7c121d0
Compare
The computation of specializedTypeParameters was incorrect. Inference was only applied to argument types, but not the return type.
7c121d0 to
0284f25
Compare
Move @BlockPosition and @BlockIndex to SPI