Skip to content

Conversation

@LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Aug 18, 2025

Description

Filter expression with OR isnull(x) should be pushed down in Calcite engine
Example:

| where firstname = 'Amber' or isnull(gender)

Should be pushed down to

  "query": {
    "bool": {
      "should": [
        {
          "bool": {
            "must_not": [
              {
                "exists": {
                  "field": "gender",
                  "boost": 1
                }
              }
            ],
            "adjust_pure_negative": true,
            "boost": 1
          }
        },
        {
          "term": {
            "firstname.keyword": {
              "value": "Amber",
              "boost": 1
            }
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }

Related Issues

Resolves #4046

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin LantaoJin added bug Something isn't working pushdown pushdown related issues labels Aug 18, 2025
@LantaoJin LantaoJin marked this pull request as ready for review August 18, 2025 10:42
@LantaoJin LantaoJin added the calcite calcite migration releated label Aug 18, 2025
@LantaoJin LantaoJin changed the title Filter with isnull or other expression should be pushed down Filter expression with OR isnull(x) should be pushed down Aug 18, 2025
.anyMatch(
o ->
o.getKind() == SqlKind.OTHER
&& ((RexCall) o).getOperator().equals(SqlStdOperatorTable.IS_EMPTY));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we check other functions as well? e.g. LENGTH

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function LENGTH support accepting NULL as its arguments. No need to change.

@penghuo penghuo merged commit fce8383 into opensearch-project:main Aug 20, 2025
23 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4055-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fce8383d56444092eed01b01b889fc6e4be4e810
# Push it to GitHub
git push --set-upstream origin backport/backport-4055-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4055-to-2.19-dev.

LantaoJin added a commit to LantaoJin/search-plugins-sql that referenced this pull request Aug 21, 2025
…ch-project#4055)

* Filter with isnull or other expression should be pushded down

Signed-off-by: Lantao Jin <[email protected]>

* revert unexpected deletion

Signed-off-by: Lantao Jin <[email protected]>

* Fix conflicts

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
(cherry picked from commit fce8383)
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Aug 21, 2025
qianheng-aws pushed a commit that referenced this pull request Aug 22, 2025
…4098)

* Filter with isnull or other expression should be pushded down



* revert unexpected deletion



* Fix conflicts



---------


(cherry picked from commit fce8383)

Signed-off-by: Lantao Jin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev backport-failed backport-manually Filed a PR to backport manually. bug Something isn't working calcite calcite migration releated pushdown pushdown related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Pushdown not work with expression OR isnull(x)

3 participants