Support pagination in V2 engine, phase 1

[Yury-Fridlyand]

Supersedes #1483 and Bit-Quill#226

Description

https://github.com/Bit-Quill/opensearch-project-sql/blob/61767f2b200b2a7f8c8b2df32de209b3c30caa61/docs/dev/Pagination-v2.md

PHASE 1
Pagination supports queries like

select * from <index>;

For example:

curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"query": "SELECT * from calcs", "fetch_size": 5}'

Cursor request:

curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"cursor": "...." }'

All credits to @MaxKsyunz.

You can use attached script for testing as well. Command line: ./cursor_test.sh <table> <page size>. Requires jq. Rename it before use.
cursor_test.sh.txt

Scroll API usage doc:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html

Please, see team review and discussion in Bit-Quill#226

Pagination demo: https://user-images.githubusercontent.com/88679692/224208630-8d38d833-abf8-4035-8d15-d5fb4382deca.mp4

Features

1. Validating whether query could be paged
2. Processing paging requests (first page) - create paged index scan builder
3. Serialization physical plan tree
4. Deserialization physical plan tree - without ResourceMonitorPlan
5. Processing cursor requests (second and next pages) - skipping Planner and Implementor
   Last page is empty - this shows pagination is finished.
6. Calculating totalHits - total number of matched rows/docs (see details)

Design Diagrams

New code workflows are highlighted.
Design document with these graphs is also available in the docs.

First page

sequenceDiagram
    participant SQLService
    participant QueryPlanFactory
    participant ResponseListener
    participant ResponseFormatter
    participant CanPaginateVisitor
    participant QueryService
    participant Planner
    participant CreatePagingTableScanBuilder
    participant OpenSearchExecutionEngine
    participant PlanSerializer
    participant Physical Plan Tree
    participant OpenSearchPagedIndexScan
    participant OpenSearchScrollRequest

SQLService->>QueryPlanFactory:execute
  critical
  QueryPlanFactory->>CanPaginateVisitor:canConvertToCursor
    CanPaginateVisitor-->>QueryPlanFactory:true
  end
  QueryPlanFactory->>QueryService:execute
    QueryService->>Planner:optimize
      critical
      Planner->>CreatePagingTableScanBuilder:apply
        CreatePagingTableScanBuilder-->>QueryService:paged index scan
      end
    QueryService->>OpenSearchExecutionEngine:execute
      OpenSearchExecutionEngine-->OpenSearchExecutionEngine:iterate result set
      critical Serialization
      OpenSearchExecutionEngine->>PlanSerializer:convertToCursor
        PlanSerializer->>Physical Plan Tree:serialize
          Physical Plan Tree->>OpenSearchPagedIndexScan:serialize
            OpenSearchPagedIndexScan->>OpenSearchScrollRequest:toCursor
              OpenSearchScrollRequest-->>OpenSearchPagedIndexScan:scroll Id
            OpenSearchPagedIndexScan-->>PlanSerializer:Serialized Plan Tree
        PlanSerializer-->>OpenSearchExecutionEngine:cursor
      end
      critical
      OpenSearchExecutionEngine->>Physical Plan Tree:getTotalHits
        Physical Plan Tree-->>OpenSearchExecutionEngine:total hits
      end
      OpenSearchExecutionEngine-->>ResponseListener:QueryResponse
    ResponseListener->>ResponseFormatter:format with cursor

Loading

Second page

sequenceDiagram
    participant SQLService
    participant QueryPlanFactory
    participant ResponseListener
    participant ResponseFormatter
    participant QueryService
    participant OpenSearchExecutionEngine
    participant PlanSerializer
    participant Physical Plan Tree
    participant OpenSearchPagedIndexScan
    participant ContinuePageRequest

SQLService->>QueryPlanFactory:execute
  QueryPlanFactory->>QueryService:execute
    critical Deserialization
    QueryService->>PlanSerializer:convertToPlan
      PlanSerializer->>Physical Plan Tree:deserialize
        Physical Plan Tree->>OpenSearchPagedIndexScan:deserialize
          OpenSearchPagedIndexScan-->>PlanSerializer:resolve engine
        PlanSerializer->>OpenSearchPagedIndexScan:OpenSearchStorageEngine
          OpenSearchPagedIndexScan->>ContinuePageRequest:create new
            OpenSearchPagedIndexScan-->>PlanSerializer:deserialized plan tree
      PlanSerializer-->>QueryService:Physical plan tree
    end
    QueryService->>OpenSearchExecutionEngine:execute
      OpenSearchExecutionEngine-->OpenSearchExecutionEngine:iterate result set
      critical Serialization
      OpenSearchExecutionEngine->>PlanSerializer:convertToCursor
        PlanSerializer->>Physical Plan Tree:serialize
          Physical Plan Tree->>OpenSearchPagedIndexScan:serialize
            OpenSearchPagedIndexScan->>ContinuePageRequest:toCursor
              ContinuePageRequest-->>OpenSearchPagedIndexScan:scroll Id
            OpenSearchPagedIndexScan-->>PlanSerializer:Serialized Plan Tree
        PlanSerializer-->>OpenSearchExecutionEngine:cursor
      end
      critical
      OpenSearchExecutionEngine->>Physical Plan Tree:getTotalHits
        Physical Plan Tree-->>OpenSearchExecutionEngine:total hits
      end
      OpenSearchExecutionEngine-->>ResponseListener:QueryResponse
    ResponseListener->>ResponseFormatter:format with cursor

Loading

Legacy Engine Fallback

sequenceDiagram
    participant RestSQLQueryAction
    participant Legacy Engine
    participant SQLService
    participant QueryPlanFactory
    participant CanPaginateVisitor

RestSQLQueryAction->>SQLService:prepareRequest
  SQLService->>QueryPlanFactory:execute
    critical V2 support check
    QueryPlanFactory->>CanPaginateVisitor:createContinuePaginatedPlan
      CanPaginateVisitor-->>QueryPlanFactory:false
    QueryPlanFactory-->>RestSQLQueryAction:UnsupportedCursorRequestException
    end
      RestSQLQueryAction->>Legacy Engine:accept

Loading

Further improvements are coming

1. Support queries with LIMIT, WHERE, HAVING and ORDER BY and queries without FROM into pagination. in #253 - Phase 2
2. Rework ComparisonIT to use pagination. in #254 - Phase 2

Issues Resolved

#656

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Yury-Fridlyand]

I copied unresolved questions/issues from Bit-Quill#226

[dai-chen]

I'm also confused about why we need to maintain totalHits in each physical operator.

total hits == number of rows matching the search criteria

Do you mean total hits = matched #rows in each page or totally? If the latter, why it becomes 0 finally?

[penghuo]

This class just a holder, and seems do nothing?

[Yury-Fridlyand]

Nice finding. Yes, it is. It was simplified so it became just a holder.

[anirudha]

Lets we tag this for 2.8 and discuss the requested changes. we are a few hours away from code freeze as this is a huge PR with critical changes; lets move this to the next release train

The base branch was changed.

[MaxKsyunz]

@Yury-Fridlyand thank you so much for the diagrams!

Could you add method returns and activations to objects where appropriate? They make it easier to visualize the call stack.