Skip to content

Enhance the ingest node simulate verbose output #60433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Aug 4, 2020
Merged

Enhance the ingest node simulate verbose output #60433

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a8a9d87
Per processor description for verbose simulate
jakelandis Jun 16, 2020
6078c45
wip
jakelandis Jun 17, 2020
aacbf21
Merge branch 'master' into show_conditional
jakelandis Jul 21, 2020
c5498e0
Merge remote-tracking branch 'upstream/master' into show_conditional
jakelandis Jul 21, 2020
12979f6
tests pass
jakelandis Jul 29, 2020
0e991b4
Merge branch 'master' into show_conditional
jakelandis Jul 29, 2020
cf599ee
serialization fixes
jakelandis Jul 29, 2020
fbfc566
test status
jakelandis Jul 29, 2020
4441ff3
minor clean up
jakelandis Jul 29, 2020
5a58161
minor testing
jakelandis Jul 29, 2020
db354e4
fix showing of pipeline processor
jakelandis Jul 29, 2020
c97e5ab
precomit
jakelandis Jul 29, 2020
e1a65e5
handle non-existant pipeline
jakelandis Jul 29, 2020
2b1f3da
fix existing yaml tests
jakelandis Jul 30, 2020
79bb3fb
add no pipeline test
jakelandis Jul 30, 2020
a829e38
add a bit of coverage
jakelandis Jul 30, 2020
374b7ae
Merge remote-tracking branch 'upstream/master' into show_conditional
jakelandis Jul 30, 2020
cb3d4a0
fix doc test
jakelandis Jul 30, 2020
5c26a6b
use local variable
jakelandis Aug 4, 2020
dc93ec3
Merge remote-tracking branch 'upstream/master' into show_conditional
jakelandis Aug 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 82 additions & 9 deletions modules/ingest-common/src/yamlRestTest/resources/rest-api-spec/test/ingest/90_simulate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,11 +546,15 @@ teardown:
- match: { docs.0.processor_results.0.tag: "setstatus-1" }
- match: { docs.0.processor_results.0.doc._source.field1: "123.42 400 <foo>" }
- match: { docs.0.processor_results.0.doc._source.status: 200 }
- match: { docs.0.processor_results.0.status: "success" }
- match: { docs.0.processor_results.0.processor_type: "set" }
- match: { docs.0.processor_results.1.tag: "rename-1" }
- match: { docs.0.processor_results.1.ignored_error.error.type: "illegal_argument_exception" }
- match: { docs.0.processor_results.1.ignored_error.error.reason: "field [foofield] doesn't exist" }
- match: { docs.0.processor_results.1.doc._source.field1: "123.42 400 <foo>" }
- match: { docs.0.processor_results.1.doc._source.status: 200 }
- match: { docs.0.processor_results.1.status: "error_ignored" }
- match: { docs.0.processor_results.1.processor_type: "rename" }

---
"Test verbose simulate with ignore_failure and no exception thrown":
Expand Down Expand Up @@ -591,11 +595,16 @@ teardown:
}
- length: { docs: 1 }
- length: { docs.0.processor_results: 2 }
- length: { docs.0.processor_results.0: 4 }
- match: { docs.0.processor_results.0.tag: "setstatus-1" }
- match: { docs.0.processor_results.0.status: "success" }
- match: { docs.0.processor_results.0.processor_type: "set" }
- match: { docs.0.processor_results.0.doc._source.field1: "123.42 400 <foo>" }
- match: { docs.0.processor_results.0.doc._source.status: 200 }
- length: { docs.0.processor_results.1: 2 }
- length: { docs.0.processor_results.1: 4 }
Copy link
Contributor Author

@jakelandis jakelandis Jul 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because we now always return status and processor_type

- match: { docs.0.processor_results.1.tag: "rename-1" }
- match: { docs.0.processor_results.1.status: "success" }
- match: { docs.0.processor_results.1.processor_type: "rename" }
- match: { docs.0.processor_results.1.doc._source.new_status: 200 }

---
Expand Down Expand Up @@ -710,7 +719,8 @@ teardown:
{
"set": {
"field": "pipeline0",
"value": true
"value": true,
"description" : "first_set"
}
},
{
Expand All @@ -731,16 +741,25 @@ teardown:
]
}
- length: { docs: 1 }
- length: { docs.0.processor_results: 3 }
- length: { docs.0.processor_results: 5 }
Copy link
Contributor Author

@jakelandis jakelandis Jul 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because we now show the pipeline processor in the results

- match: { docs.0.processor_results.0.doc._source.pipeline0: true }
- match: { docs.0.processor_results.0.status: "success" }
- match: { docs.0.processor_results.0.processor_type: "set" }
- match: { docs.0.processor_results.0.description: "first_set" }
- is_false: docs.0.processor_results.0.doc._source.pipeline1
- is_false: docs.0.processor_results.0.doc._source.pipeline2
- match: { docs.0.processor_results.1.doc._source.pipeline0: true }
- match: { docs.0.processor_results.1.doc._source.pipeline1: true }
- is_false: docs.0.processor_results.1.doc._source.pipeline2
- match: { docs.0.processor_results.1.doc: null }
- match: { docs.0.processor_results.1.status: "success" }
- match: { docs.0.processor_results.1.processor_type: "pipeline" }
- match: { docs.0.processor_results.2.doc._source.pipeline0: true }
- match: { docs.0.processor_results.2.doc._source.pipeline1: true }
- match: { docs.0.processor_results.2.doc._source.pipeline2: true }
- is_false: docs.0.processor_results.2.doc._source.pipeline2
- match: { docs.0.processor_results.3.doc: null }
- match: { docs.0.processor_results.3.status: "success" }
- match: { docs.0.processor_results.3.processor_type: "pipeline" }
- match: { docs.0.processor_results.4.doc._source.pipeline0: true }
- match: { docs.0.processor_results.4.doc._source.pipeline1: true }
- match: { docs.0.processor_results.4.doc._source.pipeline2: true }

---
"Test verbose simulate with true conditional and on failure":
Expand Down Expand Up @@ -801,19 +820,27 @@ teardown:
- length: { docs.0.processor_results: 4 }
- match: { docs.0.processor_results.0.tag: "gunna_fail" }
- match: { docs.0.processor_results.0.error.reason: "field [foo1] doesn't exist" }
- match: { docs.0.processor_results.0.status: "error" }
- match: { docs.0.processor_results.0.processor_type: "rename" }
- match: { docs.0.processor_results.1.tag: "failed1" }
- match: { docs.0.processor_results.1.doc._source.failed1: "failed1" }
- match: { docs.0.processor_results.1.doc._ingest.on_failure_processor_tag: "gunna_fail" }
- match: { docs.0.processor_results.1.status: "success" }
- match: { docs.0.processor_results.1.processor_type: "set" }
- match: { docs.0.processor_results.2.tag: "gunna_fail_again" }
- match: { docs.0.processor_results.2.error.reason: "field [foo2] doesn't exist" }
- match: { docs.0.processor_results.2.status: "error" }
- match: { docs.0.processor_results.2.processor_type: "rename" }
- match: { docs.0.processor_results.3.tag: "failed2" }
- match: { docs.0.processor_results.3.doc._source.failed1: "failed1" }
- match: { docs.0.processor_results.3.doc._source.failed2: "failed2" }
- match: { docs.0.processor_results.3.doc._ingest.on_failure_processor_tag: "gunna_fail_again" }
- match: { docs.0.processor_results.3.status: "success" }
- match: { docs.0.processor_results.3.processor_type: "set" }


---
"Test simulate with provided pipeline definition with tag and description in processors":
"Test simulate with pipeline with conditional and skipped and dropped":
- do:
ingest.simulate:
verbose: true
Expand All @@ -829,6 +856,16 @@ teardown:
"field" : "field2",
"value" : "_value"
}
},
{
"drop" : {
"if": "false"
}
},
{
"drop" : {
"if": "true"
}
}
]
},
Expand All @@ -843,7 +880,43 @@ teardown:
]
}
- length: { docs: 1 }
- length: { docs.0.processor_results: 1 }
- length: { docs.0.processor_results: 3 }
- match: { docs.0.processor_results.0.doc._source.field2: "_value" }
- match: { docs.0.processor_results.0.description: "processor_description" }
- match: { docs.0.processor_results.0.tag: "processor_tag" }
- match: { docs.0.processor_results.0.status: "success" }
- match: { docs.0.processor_results.0.processor_type: "set" }
- match: { docs.0.processor_results.1.status: "skipped" }
- match: { docs.0.processor_results.1.processor_type: "drop" }
- match: { docs.0.processor_results.1.if.condition: "false" }
- match: { docs.0.processor_results.1.if.result: false }
- match: { docs.0.processor_results.2.status: "dropped" }
- match: { docs.0.processor_results.2.processor_type: "drop" }
- match: { docs.0.processor_results.2.if.condition: "true" }
- match: { docs.0.processor_results.2.if.result: true }
---
"Test simulate with provided pipeline that does not exist":
- do:
catch: bad_request
ingest.simulate:
verbose: true
body: >
{
"pipeline": {
"description": "_description",
"processors": [
{
"pipeline": {
"name": "____pipeline_doesnot_exist___"
}
}
]
},
"docs": [
{
"_source": {}
}
]
}
- match: { error.root_cause.0.type: "illegal_argument_exception" }
- match: { error.root_cause.0.reason: "Pipeline processor configured for non-existent pipeline [____pipeline_doesnot_exist___]" }
Loading