Update big5 ppl queries #708
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,7 +31,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | sort + `@timestamp` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -40,7 +40,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | sort + `@timestamp` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -76,7 +76,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where `@timestamp` >= '2023-01-02 00:00:00' and `@timestamp` < '2023-01-02 10:00:00' | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}count() by `process.name`, `cloud.region`, `aws.cloudwatch.log_stream` | sort - `process.name`, + `cloud.region`, + `aws.cloudwatch.log_stream` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -85,7 +85,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where `@timestamp` >= '2023-01-02 00:00:00' and `@timestamp` < '2023-01-02 10:00:00' | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}count() by `process.name`, `cloud.region` | sort - `process.name`, + `cloud.region` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -112,7 +112,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | sort - `@timestamp` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -121,7 +121,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | sort - `@timestamp` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -148,7 +148,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | where `@timestamp` >= '2023-01-01 00:00:00' and `@timestamp` < '2023-01-03 00:00:00' | head 10" | ||
|
Collaborator
There was a problem hiding this comment. If I understand this correctly, if the version is equal or greater than 3.3.0 then use process.name, else use "`process.name`" ? Would this cause existing customer queries to break when they upgrade from lower versions to 3.3.0 and above or is it backward compatible?
Contributor
Author
There was a problem hiding this comment.
opensearch-project/sql#4152 corrected usage of match function
Collaborator
There was a problem hiding this comment. AFAIK we are not supposed to have breaking changes in minor version updates.
Member
There was a problem hiding this comment. Yes breaking changes is only in major release, especially API. cc: @getsaurabh02 into the conversation here, and whether we should restore the original behavior, possibly in a patch or in 3.4.0. Thanks. There was a problem hiding this comment. @peterzhuamazon @noCharger |
||
| } | ||
| }, | ||
| { | ||
|
|
@@ -157,7 +157,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}count() as country by `aws.cloudwatch.log_stream` | sort - country | head 50" | ||
|
Collaborator
There was a problem hiding this comment. is this a new param introduced in 3.3.0?
Contributor
Author
There was a problem hiding this comment. yes, as mentioned in description section |
||
| } | ||
| }, | ||
| { | ||
|
|
@@ -166,7 +166,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}count() as station by `aws.cloudwatch.log_stream` | sort - station | head 500" | ||
|
Collaborator
There was a problem hiding this comment. Same as comment above about spacing. |
||
| } | ||
| }, | ||
| { | ||
|
|
@@ -175,7 +175,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where `@timestamp` >= '2023-01-05 00:00:00' and `@timestamp` < '2023-01-05 05:00:00' | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}count() by `process.name`, `cloud.region` | sort - `count()` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -184,7 +184,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where query_string(['message'], 'monkey jackal bear') and `@timestamp` >= '2023-01-03 00:00:00' and `@timestamp` < '2023-01-03 10:00:00' | sort + `@timestamp` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -193,7 +193,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where query_string(['message'], 'monkey jackal bear') and `@timestamp` >= '2023-01-03 00:00:00' and `@timestamp` < '2023-01-03 10:00:00' | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -202,7 +202,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}| where query_string(['message'], 'monkey jackal bear'){% else %}query_string(['message'], 'monkey jackal bear'){% endif %} | head 10" | ||
noCharger marked this conversation as resolved.
Show resolved
Hide resolved
Collaborator
There was a problem hiding this comment. wold queries still work in 3.3.0 without the where clause?
Contributor
Author
There was a problem hiding this comment. no, it's also introduced from opensearch-project/sql#4152
Member
There was a problem hiding this comment. It’s not a breaking change — we never intended to support complex The earlier support for complex conditions was unintentional, resulting from the use of @noCharger Could you please reword the description? This is essentially a rectification of the benchmark workload queries. |
||
| } | ||
| }, | ||
| { | ||
|
|
@@ -211,7 +211,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | eval range_bucket = case(`metrics.size` < -10, 'range_1', `metrics.size` >= -10 and `metrics.size` < 10, 'range_2', `metrics.size` >= 10 and `metrics.size` < 100, 'range_3', `metrics.size` >= 100 and `metrics.size` < 1000, 'range_4', `metrics.size` >= 1000 and `metrics.size` < 2000, 'range_5', `metrics.size` >= 2000, 'range_6') | {% if distribution_version.split('.') | map('int') | list >= '3.4.0'.split('.') | map('int') | list %}bin `@timestamp` bins=10 | stats min(`metrics.tmin`) as tmin, avg(`metrics.size`) as tavg, max(`metrics.size`) as tmax by range_bucket, `@timestamp`{% else %}stats min(`metrics.tmin`) as tmin, avg(`metrics.size`) as tavg, max(`metrics.size`) as tmax by range_bucket, span(`@timestamp`, 1h) as auto_span | sort + range_bucket, + auto_span{% endif %}" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -220,7 +220,52 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | eval range_bucket = case(`metrics.size` < -10, 'range_1', `metrics.size` >= -10 and `metrics.size` < 10, 'range_2', `metrics.size` >= 10 and `metrics.size` < 100, 'range_3', `metrics.size` >= 100 and `metrics.size` < 1000, 'range_4', `metrics.size` >= 1000 and `metrics.size` < 2000, 'range_5', `metrics.size` >= 2000, 'range_6') | {% if distribution_version.split('.') | map('int') | list >= '3.4.0'.split('.') | map('int') | list %}bin `@timestamp` bins=20 | stats count() by range_bucket, `@timestamp`{% else %}stats count() by range_bucket, span(`@timestamp`, 1h) as auto_span | sort + range_bucket, + auto_span{% endif %}" | ||
| } | ||
| }, | ||
| { | ||
| "name": "ppl-cardinality-agg-high", | ||
| "operation-type": "raw-request", | ||
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}dc(`agent.name`)" | ||
| } | ||
| }, | ||
| { | ||
| "name": "ppl-cardinality-agg-high-2", | ||
| "operation-type": "raw-request", | ||
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}dc(`event.id`)" | ||
| } | ||
| }, | ||
| { | ||
| "name": "ppl-cardinality-agg-low", | ||
| "operation-type": "raw-request", | ||
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | stats {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}bucket_nullable = false {% endif %}dc(`cloud.region`)" | ||
| } | ||
| }, | ||
| { | ||
| "name": "ppl-range-agg-1", | ||
| "operation-type": "raw-request", | ||
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | eval range_bucket = case(`metrics.size` < -10, 'range_1', `metrics.size` >= -10 and `metrics.size` < 10, 'range_2', `metrics.size` >= 10 and `metrics.size` < 100, 'range_3', `metrics.size` >= 100 and `metrics.size` < 1000, 'range_4', `metrics.size` >= 1000 and `metrics.size` < 2000, 'range_5', `metrics.size` >= 2000, 'range_6') | stats count() by range_bucket" | ||
| } | ||
| }, | ||
| { | ||
| "name": "ppl-range-agg-2", | ||
| "operation-type": "raw-request", | ||
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | eval range_bucket = case(`metrics.size` < 100, 'range_1', `metrics.size` >= 100 and `metrics.size` < 1000, 'range_2', `metrics.size` >= 1000 and `metrics.size` < 2000, 'range_3', `metrics.size` >= 2000, 'range_4') | stats count() by range_bucket" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -301,7 +346,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | sort + `meta.file` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -310,7 +355,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}process.name=kernel{% else %}match(`process.name`, 'kernel'){% endif %} | sort + `meta.file` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -319,7 +364,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}log.file.path=\"/var/log/messages/solarshark\"{% else %}match(`log.file.path`, '/var/log/messages/solarshark'){% endif %} | sort + `metrics.size` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -337,7 +382,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} {% if distribution_version.split('.') | map('int') | list >= '3.3.0'.split('.') | map('int') | list %}log.file.path=\"/var/log/messages/solarshark\"{% else %}match(`log.file.path`, '/var/log/messages/solarshark'){% endif %} | sort - `metrics.size` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -355,7 +400,7 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where `@timestamp` >= '2023-01-01 00:00:00' and `@timestamp` < '2023-01-03 00:00:00' | stats count() by `aws.cloudwatch.log_stream`, `process.name` | head 10" | ||
| } | ||
| }, | ||
| { | ||
|
|
@@ -364,6 +409,6 @@ | |
| "path": "/_plugins/_ppl", | ||
| "method": "POST", | ||
| "body": { | ||
| "query": "source = {{index_name | default('big5')}} | where `@timestamp` >= '2023-01-01 00:00:00' and `@timestamp` < '2023-01-03 00:00:00' | stats count() by `process.name`, `aws.cloudwatch.log_stream` | head 10" | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| /* | ||
| { | ||
| "name": "cardinality-agg-high", | ||
| "operation-type": "search", | ||
| "index": "{{index_name | default('big5')}}", | ||
| "body": { | ||
| "size": 0, | ||
| "aggs": { | ||
| "agent": { | ||
| "cardinality": { | ||
| "field": "agent.name" | ||
| {% if distribution_version.split('.') | map('int') | list >= "2.19.1".split('.') | map('int') | list and distribution_version.split('.') | map('int') | list < "6.0.0".split('.') | map('int') | list %} | ||
| , "execution_hint": "ordinals" | ||
| {% endif %} | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| */ | ||
| source = big5 | ||
| | stats dc(`agent.name`) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| /* | ||
| { | ||
| "name": "cardinality-agg-high-2", | ||
| "operation-type": "search", | ||
| "index": "{{index_name | default('big5')}}", | ||
| "request-timeout": 1800, | ||
| "body": { | ||
| "size": 0, | ||
| "aggs": { | ||
| "agent": { | ||
| "cardinality": { | ||
| "field": "event.id", | ||
| "execution_hint":"ordinals" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| */ | ||
| source = big5 | ||
| | stats dc(`event.id`) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| /* | ||
| { | ||
| "name": "cardinality-agg-low", | ||
| "operation-type": "search", | ||
| "index": "{{index_name | default('big5')}}", | ||
| "body": { | ||
| "size": 0, | ||
| "aggs": { | ||
| "region": { | ||
| "cardinality": { | ||
| "field": "cloud.region" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| */ | ||
| source = big5 | ||
| | stats dc(`cloud.region`) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| /* | ||
| { | ||
| "name": "range-agg-1", | ||
| "operation-type": "search", | ||
| "index": "{{index_name | default('big5')}}", | ||
| "body": { | ||
| "size": 0, | ||
| "aggs": { | ||
| "tmax": { | ||
| "range": { | ||
| "field": "metrics.size", | ||
| "ranges": [ | ||
| { | ||
| "to": -10 | ||
| }, | ||
| { | ||
| "from": -10, | ||
| "to": 10 | ||
| }, | ||
| { | ||
| "from": 10, | ||
| "to": 100 | ||
| }, | ||
| { | ||
| "from": 100, | ||
| "to": 1000 | ||
| }, | ||
| { | ||
| "from": 1000, | ||
| "to": 2000 | ||
| }, | ||
| { | ||
| "from": 2000 | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| */ | ||
| source = big5 | ||
| | eval range_bucket = case( | ||
| `metrics.size` < -10, 'range_1', | ||
| `metrics.size` >= -10 and `metrics.size` < 10, 'range_2', | ||
| `metrics.size` >= 10 and `metrics.size` < 100, 'range_3', | ||
| `metrics.size` >= 100 and `metrics.size` < 1000, 'range_4', | ||
| `metrics.size` >= 1000 and `metrics.size` < 2000, 'range_5', | ||
| `metrics.size` >= 2000, 'range_6') | ||
| | stats count() by range_bucket |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| /* | ||
| { | ||
| "name": "range-agg-2", | ||
| "operation-type": "search", | ||
| "index": "{{index_name | default('big5')}}", | ||
| "body": { | ||
| "size": 0, | ||
| "aggs": { | ||
| "tmax": { | ||
| "range": { | ||
| "field": "metrics.size", | ||
| "ranges": [ | ||
| { | ||
| "to": 100 | ||
| }, | ||
| { | ||
| "from": 100, | ||
| "to": 1000 | ||
| }, | ||
| { | ||
| "from": 1000, | ||
| "to": 2000 | ||
| }, | ||
| { | ||
| "from": 2000 | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| */ | ||
| source = big5 | ||
| | eval range_bucket = case( | ||
| `metrics.size` < 100, 'range_1', | ||
| `metrics.size` >= 100 and `metrics.size` < 1000, 'range_2', | ||
| `metrics.size` >= 1000 and `metrics.size` < 2000, 'range_3', | ||
| `metrics.size` >= 2000, 'range_4') | ||
| | stats count() by range_bucket |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this need a space between
}andc?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, the current format is correct and functional.