Skip to content

Conversation

@yuancu
Copy link
Collaborator

@yuancu yuancu commented Dec 25, 2025

Description

Status: blocked by #4892 , diff can be viewed from 42a7386

Lambda function is used in transform function:

## TRANSFORM
### Description
Usage: `transform(array, function)` transform the element of array one by one using lambda. The lambda function can accept one single input or two input. If the lambda accepts two argument, the second one is the index of element in array.
Argument type: array:ARRAY, function:LAMBDA
Return type: ARRAY
Example
```ppl
source=people
| eval array = array(1, -2, 3), result = transform(array, x -> x + 2)
| fields result
| head 1
```

Normally, the operand type of a function can be automatically inferred. Therefore, we did not need to define an SqlOperandTypeInference for most functions.However, it's different for lambda functions -- the system does not automatically know the parameter types of a lambda function. When defining a lambda function, its parameter type needs to be explicitly specified. Otherwise, the parameters will be deemed of type ANY per the constructor of SqlLambdaScope, blocking the subsequent type inference.

public SqlLambdaScope(
    SqlValidatorScope parent, SqlLambda lambdaExpr) {
  super(parent);
  this.lambdaExpr = lambdaExpr;

  // default parameter type is ANY
  final RelDataType any =
      validator.typeFactory.createTypeWithNullability(
          validator.typeFactory.createSqlType(SqlTypeName.ANY), true);
  parameterTypes = new HashMap<>();
  lambdaExpr.getParameters().forEach(param -> parameterTypes.put(param.toString(), any));
}

For the expressiontransform(array, (x, i) -> x + i) , x should be of array's element type, while i should be of integer type (it represents index). This PR passes on this semantic via a SqlOperandTypeInference definition.

TODOs:

  • Make lambda return type inferrable
  • Yaml test, Explain IT, integration test

Related Issues

Resolves #4972

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…checking

Signed-off-by: Yuanchun Shen <[email protected]>

# Conflicts:
#	core/src/main/java/org/opensearch/sql/executor/QueryService.java

# Conflicts:
#	core/src/main/java/org/opensearch/sql/executor/QueryService.java
- 2 more ITs passed in PPLBuiltinFunctionIT

Signed-off-by: Yuanchun Shen <[email protected]>

# Conflicts:
#	core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java
- this fix testRand, where desrialization of sarg does not restore its type
- todo: update the toRex in ExtendedRelJson to the align with the latest version

Signed-off-by: Yuanchun Shen <[email protected]>
…estamp; (time, timestamp) -> timestamp (1240/1599)

Signed-off-by: Yuanchun Shen <[email protected]>
- allow type cast
- rewrite call to sql compare to custom ip comapre

Signed-off-by: Yuanchun Shen <[email protected]>

# Conflicts:
#	core/src/main/java/org/opensearch/sql/expression/function/PPLBuiltinOperators.java
…d in mvindex's implementation (1580/2015)

Signed-off-by: Yuanchun Shen <[email protected]>
…lLibrary.SPARK from lookup table to reduce confusion (1717/2018)

Signed-off-by: Yuanchun Shen <[email protected]>
…y and isblank (1738/2018)

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>

Update big5 explain results

Signed-off-by: Yuanchun Shen <[email protected]>

Fix clickbench explain ITs

Signed-off-by: Yuanchun Shen <[email protected]>

Update 35 explain IT plans

Signed-off-by: Yuanchun Shen <[email protected]>

wip

Signed-off-by: Yuanchun Shen <[email protected]>

Fix pushdown calcite explain ITs

Signed-off-by: Yuanchun Shen <[email protected]>

Fix no pushdown ITs

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
…king is deferred to the validation phase against the finished logical plan

Signed-off-by: Yuanchun Shen <[email protected]>
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Dec 25, 2025

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (3)
  • WIP
  • DO NOT MERGE
  • DRAFT

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

@yuancu yuancu added enhancement New feature or request bugFix labels Dec 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugFix enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Make return type inferrable for TRANSFORM and other lambda functions

1 participant