Skip to content

[Doc] Enable doctest with Calcite #4379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Swiddis merged 13 commits into from
Sep 29, 2025
Merged

[Doc] Enable doctest with Calcite #4379

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
316c697
Enable Calcite by default and implicit fallback the unsupported commands
LantaoJin Sep 24, 2025
fa8284d
Update docs
LantaoJin Sep 25, 2025
09122c2
doctest update
LantaoJin Sep 25, 2025
6dbc293
Merge remote-tracking branch 'upstream/main' into issues/4314_test
LantaoJin Sep 25, 2025
854f311
Enable doctest with Calcite
LantaoJin Sep 25, 2025
8aab53c
Merge remote-tracking branch 'upstream/main' into issues/4314_test
LantaoJin Sep 26, 2025
2901483
Merge remote-tracking branch 'upstream/main' into issues/4314_test
LantaoJin Sep 28, 2025
6fa4426
revert useless code
LantaoJin Sep 28, 2025
891bd32
Merge remote-tracking branch 'upstream/main' into issues/4314_test
LantaoJin Sep 29, 2025
4c3e510
remove empty file
LantaoJin Sep 29, 2025
cf9a12b
revert the math.rst
LantaoJin Sep 29, 2025
cf9debe
not stable in doctest
LantaoJin Sep 29, 2025
8d629ce
Merge remote-tracking branch 'upstream/main' into issues/4314_test
LantaoJin Sep 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 28 additions & 29 deletions docs/category.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,35 +7,9 @@
"user/admin/settings.rst"
],
"ppl_cli": [
"user/ppl/cmd/ad.rst",
"user/ppl/cmd/dedup.rst",
"user/ppl/cmd/describe.rst",
"user/ppl/cmd/showdatasources.rst",
"user/ppl/cmd/eval.rst",
"user/ppl/cmd/fillnull.rst",
"user/ppl/cmd/grok.rst",
"user/ppl/cmd/head.rst",
"user/ppl/cmd/parse.rst",
"user/ppl/cmd/patterns.rst",
"user/ppl/cmd/rare.rst",
"user/ppl/cmd/search.rst",
"user/ppl/cmd/sort.rst",
"user/ppl/cmd/syntax.rst",
"user/ppl/cmd/trendline.rst",
"user/ppl/cmd/top.rst",
"user/ppl/cmd/where.rst",
"user/ppl/cmd/join.rst",
"user/ppl/cmd/lookup.rst",
"user/ppl/cmd/subquery.rst",
"user/ppl/general/identifiers.rst",
"user/ppl/general/datatypes.rst",
"user/ppl/functions/datetime.rst",
"user/ppl/functions/expressions.rst",
"user/ppl/functions/ip.rst",
"user/ppl/functions/json.rst",
"user/ppl/functions/math.rst",
"user/ppl/functions/relevance.rst",
"user/ppl/functions/string.rst"
"user/ppl/cmd/patterns.rst"
Comment on lines 10 to +12
Copy link
Member Author

@LantaoJin LantaoJin Sep 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the #4380 closed. but grok, parse, patterns are still not able to 100% match the result of doctest in v2. @songkant-aws can you fix it or link an issue. Or it just needs to update the doc.

Copy link
Collaborator

@penghuo penghuo Sep 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reopen #4380

],
"sql_cli": [
"user/dql/expressions.rst",
Expand All @@ -52,16 +26,41 @@
"user/dql/metadata.rst"
],
"ppl_cli_calcite": [
"user/ppl/cmd/ad.rst",
"user/ppl/cmd/append.rst",
"user/ppl/cmd/bin.rst",
"user/ppl/functions/condition.rst",
"user/ppl/cmd/dedup.rst",
"user/ppl/cmd/describe.rst",
"user/ppl/cmd/eventstats.rst",
"user/ppl/cmd/eval.rst",
"user/ppl/cmd/fields.rst",
"user/ppl/cmd/fillnull.rst",
"user/ppl/cmd/head.rst",
"user/ppl/cmd/join.rst",
"user/ppl/cmd/lookup.rst",
"user/ppl/cmd/rare.rst",
"user/ppl/cmd/regex.rst",
"user/ppl/cmd/rename.rst",
"user/ppl/cmd/rex.rst",
"user/ppl/cmd/search.rst",
"user/ppl/cmd/showdatasources.rst",
"user/ppl/cmd/sort.rst",
"user/ppl/cmd/stats.rst",
"user/ppl/cmd/subquery.rst",
"user/ppl/cmd/syntax.rst",
"user/ppl/cmd/timechart.rst",
"user/ppl/cmd/search.rst"
"user/ppl/cmd/top.rst",
"user/ppl/cmd/trendline.rst",
"user/ppl/cmd/where.rst",
"user/ppl/functions/condition.rst",
"user/ppl/functions/datetime.rst",
"user/ppl/functions/expressions.rst",
"user/ppl/functions/ip.rst",
"user/ppl/functions/json.rst",
"user/ppl/functions/math.rst",
"user/ppl/functions/relevance.rst",
"user/ppl/functions/string.rst",
"user/ppl/general/datatypes.rst",
"user/ppl/general/identifiers.rst"
]
}
10 changes: 5 additions & 5 deletions docs/user/ppl/cmd/dedup.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The example show dedup the document with gender field.

PPL query::

os> source=accounts | dedup gender | fields account_number, gender;
os> source=accounts | dedup gender | fields account_number, gender | sort account_number;
fetched rows / total rows = 2/2
+----------------+--------+
| account_number | gender |
Expand All @@ -48,7 +48,7 @@ The example show dedup the document with gender field keep 2 duplication.

PPL query::

os> source=accounts | dedup 2 gender | fields account_number, gender;
os> source=accounts | dedup 2 gender | fields account_number, gender | sort account_number;
fetched rows / total rows = 3/3
+----------------+--------+
| account_number | gender |
Expand All @@ -65,7 +65,7 @@ The example show dedup the document by keep null value field.

PPL query::

os> source=accounts | dedup email keepempty=true | fields account_number, email;
os> source=accounts | dedup email keepempty=true | fields account_number, email | sort account_number;
fetched rows / total rows = 4/4
+----------------+-----------------------+
| account_number | email |
Expand All @@ -81,7 +81,7 @@ The example show dedup the document by ignore the empty value field.

PPL query::

os> source=accounts | dedup email | fields account_number, email;
os> source=accounts | dedup email | fields account_number, email | sort account_number;
fetched rows / total rows = 3/3
+----------------+-----------------------+
| account_number | email |
Expand All @@ -99,7 +99,7 @@ The example show dedup the consecutive document.

PPL query::

os> source=accounts | dedup gender consecutive=true | fields account_number, gender;
os> source=accounts | dedup gender consecutive=true | fields account_number, gender | sort account_number;
fetched rows / total rows = 3/3
+----------------+--------+
| account_number | gender |
Expand Down
46 changes: 14 additions & 32 deletions docs/user/ppl/cmd/join.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Example 1: Two indices join

PPL query::

PPL> source = state_country | inner join left=a right=b ON a.name = b.name occupation | stats avg(salary) by span(age, 10) as age_span, b.country;
os> source = state_country | inner join left=a right=b ON a.name = b.name occupation | stats avg(salary) by span(age, 10) as age_span, b.country;
fetched rows / total rows = 5/5
+-------------+----------+-----------+
| avg(salary) | age_span | b.country |
Expand All @@ -116,17 +116,8 @@ Example 2: Join with subsearch

PPL query::

PPL> source = state_country as a
| where country = 'USA' OR country = 'England'
| left join ON a.name = b.name [
source = occupation
| where salary > 0
| fields name, country, salary
| sort salary
| head 3
] as b
| stats avg(salary) by span(age, 10) as age_span, b.country;
fetched rows / total rows = 5/5
PPL> source = state_country as a | where country = 'USA' OR country = 'England' | left join ON a.name = b.name [ source = occupation | where salary > 0 | fields name, country, salary | sort salary | head 3 ] as b | stats avg(salary) by span(age, 10) as age_span, b.country;
Copy link
Member Author

@LantaoJin LantaoJin Sep 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why this query failed in linux/doctest only. I cannot reproduce in local and IT. Will keep the PPL> to avoid failure.

    ppl_cmd.process("source = state_country as a | where country = 'USA' OR country = 'England' | left join ON a.name = b.name [ source = occupation | where salary > 0 | fields name, country, salary | sort salary | head 3 ] as b | stats avg(salary) by span(age, 10) as age_span, b.country")Expected:
    fetched rows / total rows = 3/3
    +-------------+----------+-----------+
    | avg(salary) | age_span | b.country |
    |-------------+----------+-----------|
    | null        | 40       | null      |
    | 70000.0     | 30       | USA       |
    | 100000.0    | 70       | England   |
    +-------------+----------+-----------+
Got:
    fetched rows / total rows = 2/2
    +-------------+----------+-----------+
    | avg(salary) | age_span | b.country |
    |-------------+----------+-----------|
    | 70000.0     | 30       | USA       |
    | 100000.0    | 70       | England   |
    +-------------+----------+-----------+
----------------------------------------------------------------------

fetched rows / total rows = 3/3
+-------------+----------+-----------+
| avg(salary) | age_span | b.country |
|-------------+----------+-----------|
Expand All @@ -140,32 +131,23 @@ Example 3: Join with field list

PPL query::

PPL> source = state_country
| where country = 'USA' OR country = 'England'
| join type=left overwrite=true name [
source = occupation
| where salary > 0
| fields name, country, salary
| sort salary
| head 3
]
| stats avg(salary) by span(age, 10) as age_span, country;
fetched rows / total rows = 5/5
+-------------+----------+-----------+
| avg(salary) | age_span | country |
|-------------+----------+-----------|
| null | 40 | null |
| 70000.0 | 30 | USA |
| 100000.0 | 70 | England |
+-------------+----------+-----------+
PPL> source = state_country | where country = 'USA' OR country = 'England' | join type=left overwrite=true name [ source = occupation | where salary > 0 | fields name, country, salary | sort salary | head 3 ] | stats avg(salary) by span(age, 10) as age_span, country;
fetched rows / total rows = 3/3
+-------------+----------+---------+
| avg(salary) | age_span | country |
|-------------+----------+---------|
| null | 40 | null |
| 70000.0 | 30 | USA |
| 100000.0 | 70 | England |
+-------------+----------+---------+

Example 4: Join with options
============================

PPL query::

PPL> source = state_country | join type=inner overwrite=false max=1 name occupation | stats avg(salary) by span(age, 10) as age_span, country;
fetched rows / total rows = 5/5
os> source = state_country | join type=inner overwrite=false max=1 name occupation | stats avg(salary) by span(age, 10) as age_span, country;
fetched rows / total rows = 4/4
+-------------+----------+---------+
| avg(salary) | age_span | country |
|-------------+----------+---------|
Expand Down
6 changes: 3 additions & 3 deletions docs/user/ppl/cmd/rare.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Syntax
============
rare <field-list> [by-clause]

rare <field-list> [rare-options] [by-clause] ``(available from 3.1.0+)``
rare [rare-options] <field-list> [by-clause] ``(available from 3.1.0+)``

* field-list: mandatory. comma-delimited list of field names.
* by-clause: optional. one or more fields to group the results by.
Expand All @@ -35,7 +35,7 @@ The example finds least common gender of all the accounts.

PPL query::

os> source=accounts | rare gender;
os> source=accounts | rare showcount=false gender;
fetched rows / total rows = 2/2
+--------+
| gender |
Expand All @@ -52,7 +52,7 @@ The example finds least common age of all the accounts group by gender.

PPL query::

os> source=accounts | rare age by gender;
os> source=accounts | rare showcount=false age by gender;
fetched rows / total rows = 4/4
+--------+-----+
| gender | age |
Expand Down
Loading
Loading