Skip to content

Add BLOCK_AND_POSITION calling convention to annotation framework#10961

Merged
gerlou merged 4 commits intoprestodb:masterfrom
gerlou:annotation_block_position
Aug 2, 2018
Merged

Add BLOCK_AND_POSITION calling convention to annotation framework#10961
gerlou merged 4 commits intoprestodb:masterfrom
gerlou:annotation_block_position

Conversation

@gerlou
Copy link
Copy Markdown
Contributor

@gerlou gerlou commented Jul 2, 2018

Move @BlockPosition and @BlockIndex to SPI

@haozhun
Copy link
Copy Markdown
Contributor

haozhun commented Jul 2, 2018

Change commit message to Move @BlockPosition and @BlockIndex to SPI

You don't need a separate PR for this. One PR in GitHub can have multiple commits. This PR is a precursor step of a larger PR you are going to put up. As a result, we'll have this commit as part of that PR, instead of creating a standalone PR for this.

@gerlou gerlou force-pushed the annotation_block_position branch from 697a3c0 to 802ad6b Compare July 2, 2018 19:59
@gerlou gerlou changed the title Move @BlockPosition, @BlockIndex to SPI Move @BlockPosition and @BlockIndex to SPI Jul 2, 2018
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Move @BlockPosition and @BlockIndex to SPI commit moves other classes around as well?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@findepi : Separated into a different commit now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this code change directly related to moving BlockPosition and BlockIndex annotations to SPI?

@wenleix wenleix assigned wenleix and unassigned wenleix Jul 9, 2018
@gerlou gerlou force-pushed the annotation_block_position branch from 8d5f43c to 119588c Compare July 12, 2018 20:43
@wenleix
Copy link
Copy Markdown
Contributor

wenleix commented Jul 13, 2018

A few high-level comments:

  • The commits are a bit commingled and the commit message doesn't reflect what it does. For example, the first commit doesn't only move @BlockPosition and @BlockIndex to SPI, but also did some renaming and some support of block_and_position convention in ParmatericScalarImplementation.

    We should rebase the PR into the 5 following commits:

    1. Move @BlockPosition and @BlockIndex to SPI (a straightforward move)
      2. Rename SFC -> SFIC, SI -> PSI. (SFC: ScalarFunctionChoice, SFIC: ScalarFunctionImplementationChoice, SI: ScalarImplementation, PSI: ParametricScalarImplementation.
      3. Add type parameter specialization inference based on return native container type.
      4. Add return native container type into ParametricScalarImplementation
      5. Add BLOCK_AND_POSITION calling convention for annotation framework.

Let's work together offline on this rebase.

  • The PR title is not accurate.

  • return native container type should be added into ParametricScalarImplementation. Let's put them in a separate commit. This will resolve the test failures in Travis.

  • There are test failures in Travis.

[ERROR] testBlockPosition(com.facebook.presto.type.TestBlockAndPositionNullConvention)  Time elapsed: 0.018 s  <<< FAILURE!
java.lang.invoke.WrongMethodTypeException: cannot convert MethodHandle(Block,int)long to (Object)Object
	at com.facebook.presto.type.TestBlockAndPositionNullConvention.testBlockPosition(TestBlockAndPositionNullConvention.java:44)

[ERROR] testRound(com.facebook.presto.operator.scalar.TestMathFunctions)  Time elapsed: 1.072 s  <<< FAILURE!
com.facebook.presto.spi.PrestoException: Unsupported type parameters (BoundVariables{typeVariables={}, longVariables={p=1, s=0, rp=2}}) for round<rp:min(38, p + 1)>(decimal(p,s),integer):decimal(rp,s)
	at com.facebook.presto.operator.scalar.TestMathFunctions.testRound(TestMathFunctions.java:891)

Copy link
Copy Markdown
Contributor

@wenleix wenleix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some quick comments after glancing. Let's first separate the commits.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this fix to a separate commit (the third commit noted in #10961 (comment))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: need to update this comment

Copy link
Copy Markdown
Contributor

@wenleix wenleix Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally, we thought Signature itself can distinguish the different PSI. Thus we only use Signature as the hash key.

Now since Signature itself is not enough to distinguish PSI, and argument native container + specialize native container type are required. The current implementation is incorrect since only one ParametricScalarImplementation will be put into the signature.

Lets put Signature, argument and return value native container type, type parameter specialization native container type into an inner class SpecializedSignature, and use that as key.

Copy link
Copy Markdown
Contributor

@haozhun haozhun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a quick pass.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assertEquals is for assertion in test only. Here, you can use checkArgument. In addition, you would want to provide an error messages. The audience of the error message is people who are implementing custom SQL functions through the annotation message.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if should not be in if (method.getParameterCount() > (i + 1)). Instead, you would want to move it out so that you check for this annotation regardless of whether a next argument exists. In addition, you would want to assert that 1) the next argument exists, 2) the next argument has @BlockIndex annotation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these 3 boolean flags nullableArgument, hasNullFlag, and isBlockAndPositionNullConvention, there exists 8 possible combinations. However, only 4 of them is legal.

Previously, there were 4 possible combinations, but only 3 of them is legal. That was not ideal. However, It was not too hard to 1) conclude that the illegal one is impossible; 2) parser fails loudly when both @IsNull and @SqlNullable is present.

With the addition, reasoning about that just became pretty much intractable.

There are two alternatives ways to address this:

  1. This code is organized in such a way that consumption of these variables and production of these variables are clearly split. At the split, add assertion.
  2. Use an enum instead of 3 flags.

I highly recommend taking the 2nd alternative.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make your test contains all 3 of: generic, specialize, and exact. Ideally, there should be at least of specialized ones, and 2 exact ones.

@haozhun
Copy link
Copy Markdown
Contributor

haozhun commented Jul 16, 2018

@wenleix When you reference @x, make sure you always put it in a pair of "`". We've previously received complaint from the github user whose username is isnull.

@gerlou gerlou force-pushed the annotation_block_position branch from 119588c to 03b003c Compare July 19, 2018 17:51
@gerlou gerlou changed the title Move @BlockPosition and @BlockIndex to SPI Support @BlockPosition and @BlockIndex annotation in scalar function Jul 19, 2018
@gerlou gerlou force-pushed the annotation_block_position branch 6 times, most recently from 9fa7194 to 4c67825 Compare July 19, 2018 19:20
@gerlou gerlou changed the title Support @BlockPosition and @BlockIndex annotation in scalar function Add BLOCK_AND_POSITION calling convention to annotation framework Jul 19, 2018
@gerlou gerlou force-pushed the annotation_block_position branch from 4c67825 to ec298ad Compare July 19, 2018 22:53
Copy link
Copy Markdown
Contributor

@wenleix wenleix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first two commits look good.

  • Move @BlockPosition and @BlockIndex to SPI
  • Rename ScalarImplementation to ParametricScalarImplementation

@haozhun : Do we also want to rename ScalarFunctionChoice to ScalarFunctionImplementationChoice ?

@findepi
Copy link
Copy Markdown
Contributor

findepi commented Jul 20, 2018

@gerlou @haozhun @wenleix can you please describe to me the benefits of block and position calling convention?
This looks good at first, because we don't need to extract a value from a block before calling a scalar. However, this looks like preventing composition of scalars. Am I missing something?

@haozhun
Copy link
Copy Markdown
Contributor

haozhun commented Jul 20, 2018

@findepi A function that implements @BlockPosition convention is also required to implement the regular one. A function with the exact same signature (signature, native container type, type parameter specialization) can have multiple implementations now.

@findepi
Copy link
Copy Markdown
Contributor

findepi commented Jul 20, 2018

@haozhun that makes sense! i was fooled by "migrate..." PR (#11085) and i admit, i didn't investigate the code

Copy link
Copy Markdown
Contributor

@wenleix wenleix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments plus a major one in ParametricScalarImplementation.specialize. Let's discuss in person on this.

this.constructorDependencies = ImmutableList.copyOf(requireNonNull(constructorDependencies, "constructorDependencies is null"));
this.argumentNativeContainerTypes = ImmutableList.copyOf(requireNonNull(argumentNativeContainerTypes, "argumentNativeContainerTypes is null"));
this.specializedTypeParameters = ImmutableMap.copyOf(requireNonNull(specializedTypeParameters, "specializedTypeParameters is null"));
this.choices = choices;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Guard these by requireNonNull, do this for returnContainerType and specializedSignature as well.

return Optional.empty();
}
}
Class<?> returnContainerType = getNullAwareReturnType(typeManager.getType(boundSignature.getReturnType()).getJavaType(), nullable);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Remove unused method getNullAwareReturnType

else {
if (argumentProperty.getArgumentType() != VALUE_TYPE) {
return Optional.empty();
for (ParametricScalarImplementationChoice choice : choices) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At high level, whether the ParametricScalarImplementation (PSI) suited for the specialization should be a decision at PSI level without consulting underlying ParametricScalarImplementationChoice (PSIC).

I understand currently some information is only in PSIC. Let's discuss offline about how to reorg fields between PSI / PSIC.


this.methodHandle = getMethodHandle(method);

ParametricScalarImplementationChoice choice = new ParametricScalarImplementationChoice(nullable, argumentProperties, methodHandle, constructorMethodHandle, specializedTypeParameters, argumentNativeContainerTypes, numberOfBlockPositionArguments);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

numberOfBlockPositionArguments should be calculated in the constructor instead of passed in by caller.

private final List<ArgumentProperty> argumentProperties;
private final MethodHandle methodHandle;
private final Optional<MethodHandle> constructor;
private final Map<String, Class<?>> specializedTypeParameters;
Copy link
Copy Markdown
Contributor

@wenleix wenleix Jul 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specializedTypeParameters should be in ParametricScalarImplementation

argumentNativeContainerTypes.add(type.nativeContainerType());
}
else {
checkArgument(type.nativeContainerType().equals(Object.class), "Native container type for parameter needs to be an Object");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this check?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also thinking, when parameterType from the method itself is Object, we should enforce nativeContainerType is annotated.

So we always have the Java type for the scalar function in PSIC.

cc @haozhun

@gerlou gerlou force-pushed the annotation_block_position branch 2 times, most recently from 2dba166 to f2e53d3 Compare July 23, 2018 23:58
@gerlou gerlou force-pushed the annotation_block_position branch 2 times, most recently from 93877c1 to 58c77e9 Compare July 24, 2018 00:01
Copy link
Copy Markdown
Contributor

@wenleix wenleix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some quick comments.

Let's talk about the comments about nullConventionFlag part in person.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I squash the fixup commit to the wrong commit. This two lines should go to the first commit...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Cleanup these . Same for other debugging comments :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also check

  • constructorDependencies
  • constructor

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have done unwrap for argumentNativeContainerTypes already, I think we can do

    if (!(argumentNativeContainerTypes.get(i).get() == Object.class || argumentNativeContainerTypes.get(i).get() == argumentType)) {
    ...
    }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should never happen. Either remove it or make it to be an checkState statement.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would put an numberOfBlockPositionArguments before the for loop

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's really one-to-one mapping with NullConvention, let's not create this new enum.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a local variable for nullConventionFlag here (instead of using the field)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here NullConventionFlag is indicating the return convention, which is not a natural usage.

@gerlou gerlou force-pushed the annotation_block_position branch 4 times, most recently from 2591ffe to 39a5fd3 Compare July 25, 2018 00:09
@facebook-github-bot
Copy link
Copy Markdown
Collaborator

Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours has expired.

Before we can review or merge your code, we need you to email cla@fb.com with your details so we can update your status.

Copy link
Copy Markdown
Contributor

@wenleix wenleix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good. I will take a look at tests later.

Remember to reorg changes to BlockIndex / BlockPosition to the first commit. (in ApproximateCountDistinctAggregation.java and TestAnnotationEngineForAggregates.java

@haozhun can you also start to review ? :)

Class<?> argumentType = typeManager.getType(boundSignature.getArgumentTypes().get(i)).getJavaType();
Class<?> argumentContainerType = getNullAwareContainerType(argumentType, argumentProperty.getNullConvention());
if (!argumentNativeContainerTypes.get(i).isAssignableFrom(argumentContainerType)) {
if (!(argumentNativeContainerTypes.get(i).get().isAssignableFrom(argumentType))) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work when argumentNativeContainerTypes is Object.class and argumentType is primitive type? isAssignableFrom will return false.

Does that mean such case never exists?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove the redundant brackets !(...) -> !....

@wenleix There is indeed no code that depends on it. TypeOfFunction is an interesting example that demonstrates the deficiency here.

Supporting it is not hard, but also nontrivial.

  • Boxing will be necessary for primitives.
  • Downcasting will be necessary for return types.
  • Once the support is added, this place can be changed to argumentNativeContainerType != Object.class && argumentNativeContainerType == argumentType (NullConvention is no longer available here.)

public SpecializedSignature getSpecializedSignature()
{
return dependencies;
SpecializedSignature specializedSignature = new SpecializedSignature(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: You can create the SpecializedSignature and return it in one step.

return new SpecializedSignature(
                signature,
                argumentNativeContainerTypes,
                specializedTypeParameters,
                returnNativeContainerType);

// USE_NULL_FLAG or RETURN_NULL_ON_NULL
checkState(parameterType == Void.class || !Primitives.isWrapperType(parameterType), "Method [%s] has parameter expected to use USE_NULL_FLAG or RETURN_NULL_ON_NULL null convention. Parameter type is unexpected: %s", method, parameterType.getSimpleName());

boolean useNullFlag = false;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haozhun Any comments about this auxiliary boolean flag?

@gerlou gerlou force-pushed the annotation_block_position branch from 39a5fd3 to 4706596 Compare July 26, 2018 19:52
@facebook-github-bot
Copy link
Copy Markdown
Collaborator

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

@haozhun
Copy link
Copy Markdown
Contributor

haozhun commented Jul 27, 2018

Add type parameter specialization inference based on return type

A better commit message would be something like

Fix specializedTypeParameters in ParametricScalarImplementation

The computation of specializedTypeParameters was incorrect.
Inference was only applied to argument types, but not the return type.

The original message did not provide a high level context. As a result, a reader wouldn't know what it is about.

Copy link
Copy Markdown
Contributor

@haozhun haozhun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add BLOCK_AND_POSITION calling convention to annotation framework

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't remove this assertion:

checkCondition(methodHandleAndConstructor.isPresent(), FUNCTION_IMPLEMENTATION_ERROR, String.format("Exact implementation of %s do not match expected java types.", boundSignature.getName()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all 3,

checkCondition(..., FUNCTION_IMPLEMENTATION_ERROR, "Some descriptive message");

Class<?> argumentType = typeManager.getType(boundSignature.getArgumentTypes().get(i)).getJavaType();
Class<?> argumentContainerType = getNullAwareContainerType(argumentType, argumentProperty.getNullConvention());
if (!argumentNativeContainerTypes.get(i).isAssignableFrom(argumentContainerType)) {
if (!(argumentNativeContainerTypes.get(i).get().isAssignableFrom(argumentType))) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove the redundant brackets !(...) -> !....

@wenleix There is indeed no code that depends on it. TypeOfFunction is an interesting example that demonstrates the deficiency here.

Supporting it is not hard, but also nontrivial.

  • Boxing will be necessary for primitives.
  • Downcasting will be necessary for return types.
  • Once the support is added, this place can be changed to argumentNativeContainerType != Object.class && argumentNativeContainerType == argumentType (NullConvention is no longer available here.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove. This is no longer used.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add comment that all choices are required to have the same constructor dependencies at this time. And point out that this is asserted in the constructor.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a Builder if necessary. Making this class mutable is:

  1. surprising and unintuitive
  2. undermines your validations in the constructor.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SqlType can only contain an explicitly specified nativeContainerType when using @BlockPosition

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a bug here. The message is wrong.

The return type in PrimitiveParameterWithNullable is long. But that's correct because there is no @SqlNullable annotation on the method.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't read this sentence. Please change the error message and this assertion to

A parameter with USE_NULL_FLAG or RETURN_NULL_ON_NULL convention must not use wrapper type. Found in method ...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assertions should have use

checkCondition(..., FUNCTION_IMPLEMENTATION_ERROR, ...)

Fix all in ScalarFromAnnotationsParser and ParametricScalarImplementation after this PR is merged.

@gerlou gerlou force-pushed the annotation_block_position branch 6 times, most recently from 5273ef1 to eaf0399 Compare July 31, 2018 05:36
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementations for the same function signature must have matching dependencies: <signature>

Ditto for the next two.

@gerlou gerlou force-pushed the annotation_block_position branch from eaf0399 to 7c121d0 Compare August 1, 2018 16:58
@gerlou gerlou force-pushed the annotation_block_position branch from 7c121d0 to 0284f25 Compare August 2, 2018 18:23
@gerlou gerlou merged commit 0284f25 into prestodb:master Aug 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants