[Reporting] Fix incorrect number of hits being exported from CSV #112406

jloleysens · 2021-09-16T11:33:20Z

Summary

The following was being reported for subsequent runs of a large CSV export on CI. TL;DR, the CSV row count was random, but always below the expected 4675 total.

Data

Run scan outputs:
------------------
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 171
   │ proc [kibana] results 0
------------------
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 171
   │ proc [kibana] results 0
------------------
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 171
   │ proc [kibana] results 0
-------------------
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 146
   │ proc [kibana] results 0
-------------------
   │ proc [kibana] searchBody {
   │ proc [kibana]   "fields": [
   │ proc [kibana]     {
   │ proc [kibana]       "field": "*",
   │ proc [kibana]       "include_unmapped": "true"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "customer_birth_date",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "order_date",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "products.created_on",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     }
   │ proc [kibana]   ],
   │ proc [kibana]   "sort": [
   │ proc [kibana]     {
   │ proc [kibana]       "order_date": {
   │ proc [kibana]         "order": "desc",
   │ proc [kibana]         "unmapped_type": "boolean"
   │ proc [kibana]       }
   │ proc [kibana]     }
   │ proc [kibana]   ],
   │ proc [kibana]   "track_total_hits": true,
   │ proc [kibana]   "script_fields": {},
   │ proc [kibana]   "stored_fields": [
   │ proc [kibana]     "*"
   │ proc [kibana]   ],
   │ proc [kibana]   "runtime_mappings": {},
   │ proc [kibana]   "_source": false,
   │ proc [kibana]   "query": {
   │ proc [kibana]     "bool": {
   │ proc [kibana]       "must": [],
   │ proc [kibana]       "filter": [
   │ proc [kibana]         {
   │ proc [kibana]           "range": {
   │ proc [kibana]             "order_date": {
   │ proc [kibana]               "format": "strict_date_optional_time",
   │ proc [kibana]               "gte": "2019-04-27T23:56:51.374Z",
   │ proc [kibana]               "lte": "2019-08-23T16:18:51.821Z"
   │ proc [kibana]             }
   │ proc [kibana]           }
   │ proc [kibana]         }
   │ proc [kibana]       ],
   │ proc [kibana]       "should": [],
   │ proc [kibana]       "must_not": []
   │ proc [kibana]     }
   │ proc [kibana]   }
   │ proc [kibana] }
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 147
   │ proc [kibana] results 0
   │ proc [kibana] this.csvRowCount 4647
-------------------
   │ proc [kibana] searchBody {
   │ proc [kibana]   "fields": [
   │ proc [kibana]     {
   │ proc [kibana]       "field": "*",
   │ proc [kibana]       "include_unmapped": "true"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "customer_birth_date",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "order_date",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "products.created_on",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     }
   │ proc [kibana]   ],
   │ proc [kibana]   "sort": [
   │ proc [kibana]     {
   │ proc [kibana]       "order_date": {
   │ proc [kibana]         "order": "desc",
   │ proc [kibana]         "unmapped_type": "boolean"
   │ proc [kibana]       }
   │ proc [kibana]     }
   │ proc [kibana]   ],
   │ proc [kibana]   "track_total_hits": true,
   │ proc [kibana]   "script_fields": {},
   │ proc [kibana]   "stored_fields": [
   │ proc [kibana]     "*"
   │ proc [kibana]   ],
   │ proc [kibana]   "runtime_mappings": {},
   │ proc [kibana]   "_source": false,
   │ proc [kibana]   "query": {
   │ proc [kibana]     "bool": {
   │ proc [kibana]       "must": [],
   │ proc [kibana]       "filter": [
   │ proc [kibana]         {
   │ proc [kibana]           "range": {
   │ proc [kibana]             "order_date": {
   │ proc [kibana]               "format": "strict_date_optional_time",
   │ proc [kibana]               "gte": "2019-04-27T23:56:51.374Z",
   │ proc [kibana]               "lte": "2019-08-23T16:18:51.821Z"
   │ proc [kibana]             }
   │ proc [kibana]           }
   │ proc [kibana]         }
   │ proc [kibana]       ],
   │ proc [kibana]       "should": [],
   │ proc [kibana]       "must_not": []
   │ proc [kibana]     }
   │ proc [kibana]   }
   │ proc [kibana] }
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 139
   │ proc [kibana] results 0
----------------
   │ proc [kibana] searchBody {
   │ proc [kibana]   "fields": [
   │ proc [kibana]     {
   │ proc [kibana]       "field": "*",
   │ proc [kibana]       "include_unmapped": "true"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "customer_birth_date",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "order_date",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     },
   │ proc [kibana]     {
   │ proc [kibana]       "field": "products.created_on",
   │ proc [kibana]       "format": "strict_date_optional_time"
   │ proc [kibana]     }
   │ proc [kibana]   ],
   │ proc [kibana]   "sort": [
   │ proc [kibana]     {
   │ proc [kibana]       "order_date": {
   │ proc [kibana]         "order": "desc",
   │ proc [kibana]         "unmapped_type": "boolean"
   │ proc [kibana]       }
   │ proc [kibana]     }
   │ proc [kibana]   ],
   │ proc [kibana]   "track_total_hits": true,
   │ proc [kibana]   "script_fields": {},
   │ proc [kibana]   "stored_fields": [
   │ proc [kibana]     "*"
   │ proc [kibana]   ],
   │ proc [kibana]   "runtime_mappings": {},
   │ proc [kibana]   "_source": false,
   │ proc [kibana]   "query": {
   │ proc [kibana]     "bool": {
   │ proc [kibana]       "must": [],
   │ proc [kibana]       "filter": [
   │ proc [kibana]         {
   │ proc [kibana]           "range": {
   │ proc [kibana]             "order_date": {
   │ proc [kibana]               "format": "strict_date_optional_time",
   │ proc [kibana]               "gte": "2019-04-27T23:56:51.374Z",
   │ proc [kibana]               "lte": "2019-08-23T16:18:51.821Z"
   │ proc [kibana]             }
   │ proc [kibana]           }
   │ proc [kibana]         }
   │ proc [kibana]       ],
   │ proc [kibana]       "should": [],
   │ proc [kibana]       "must_not": []
   │ proc [kibana]     }
   │ proc [kibana]   }
   │ proc [kibana] }
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 42
   │ proc [kibana] results 0
   │ proc [kibana] this.csvRowCount 454

It appears that this was reproducible only with using the _scroll endpoint. After switching to using point in time per the recommendation in the docs, we are getting consistent CSV row counts again:

   │ proc [kibana] total hits 4675
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 500
   │ proc [kibana] results 175
   │ proc [kibana] this.csvRowCount 4675

The docs indicate that scroll should not be used to span more than 10000 docs, but in this case we were spanning less than half that. We should do an analysis to determine how far back this was introduced as it is likely a result of something in ES changing (still investigating).

How to test locally

(this can be automated by running the functional test generates a report from a new search with data: default)

Set up ES with the data archive from x-pack/test/functional/es_archives/reporting/ecommerce
Start Kibana
Issue the following request:

curl 'http://localhost:5620/api/reporting/generate/csv_searchsource' \
  -H 'Connection: keep-alive' \
  -H 'sec-ch-ua: "Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"' \
  -H 'Content-Type: application/json' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36' \
  -H 'kbn-version: 7.16.0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'Accept: */*' \
  -H 'Origin: http://localhost:5620' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: http://localhost:5620/app/discover?_t=1631716621016' \
  -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
  -u elastic:changeme \
  --data-raw $'{"jobParams":"(browserTimezone:UTC,columns:\u0021(),objectType:search,searchSource:(fields:\u0021((field:\'*\',include_unmapped:true)),filter:\u0021((meta:(field:order_date,index:\'5193f870-d861-11e9-a311-0fa548c5f953\',params:()),range:(order_date:(format:strict_date_optional_time,gte:\'2019-04-04T23:56:51.374Z\',lte:\'2019-08-29T16:18:51.821Z\')))),index:\'5193f870-d861-11e9-a311-0fa548c5f953\',parent:(filter:\u0021(),index:\'5193f870-d861-11e9-a311-0fa548c5f953\',query:(language:kuery,query:\'\')),sort:\u0021((order_date:desc)),trackTotalHits:\u0021t),title:\'Discover search [2021-09-15T14:48:11.140+00:00]\',version:\'7.16.0\')"}' \
  --compressed

Release note

Does this need a public release note?

Checklist

Unit or functional tests were updated or added to match the most common scenarios

jloleysens · 2021-09-16T14:29:09Z

@elasticmachine merge upstream

tsullivan · 2021-09-20T19:03:51Z

After switching to using point in time per the recommendation in the docs, we are getting consistent CSV row counts again:

I looked back in the PR branch that added the current CSV export implementation: #88303. In the first commit / original implementation, the plan was to use PIT instead of the ES _scroll API. Unfortunately that plan was scrapped when we had test failures for exporting non-timebased data and unsorted data. I think we should come up with a plan to identify when PIT should be used to export the CSV. My guess is that 99% of the time, using PIT is the "right way" to do it.

jloleysens · 2021-09-21T12:24:41Z

@elasticmachine merge upstream

jloleysens · 2021-09-22T07:42:09Z

@elasticmachine merge upstream

kibanamachine · 2021-09-22T10:06:20Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💛 Build #154545 was flaky 34ff860
💔 Build #154174 failed 7db7614
💔 Build #153913 failed 14582f9
💔 Build #153874 failed 4c09af6
💔 Build #153804 failed f202622

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

WIP, switched CSV export to use PIT instead of scroll

8d4c581

jloleysens added release_note:fix zDeprecated Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead v8.0.0 Team:AppServices v7.16.0 v7.15.1 labels Sep 16, 2021

jloleysens requested a review from tsullivan September 16, 2021 11:33

remove .skip

f202622

Merge branch '7.x' into reporting/csv-export-fix

198b8f0

jloleysens added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:fix labels Sep 16, 2021

jloleysens added 4 commits September 16, 2021 16:40

clean up the generate csv file

e059f08

re-add username to get query

4c09af6

set ignore_unavailable to undefined

14582f9

revert back to using _scroll, unskip the CSV export tests

7db7614

Merge branch '7.x' into reporting/csv-export-fix

34ff860

Merge branch '7.x' into reporting/csv-export-fix

a9fc818

jloleysens closed this Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Reporting] Fix incorrect number of hits being exported from CSV #112406

[Reporting] Fix incorrect number of hits being exported from CSV #112406

Uh oh!

jloleysens commented Sep 16, 2021 •

edited

Loading

Uh oh!

jloleysens commented Sep 16, 2021

Uh oh!

tsullivan commented Sep 20, 2021

Uh oh!

jloleysens commented Sep 21, 2021

Uh oh!

jloleysens commented Sep 22, 2021

Uh oh!

kibanamachine commented Sep 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Reporting] Fix incorrect number of hits being exported from CSV #112406

[Reporting] Fix incorrect number of hits being exported from CSV #112406

Uh oh!

Conversation

jloleysens commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test locally

Release note

Checklist

Uh oh!

jloleysens commented Sep 16, 2021

Uh oh!

tsullivan commented Sep 20, 2021

Uh oh!

jloleysens commented Sep 21, 2021

Uh oh!

jloleysens commented Sep 22, 2021

Uh oh!

kibanamachine commented Sep 22, 2021

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jloleysens commented Sep 16, 2021 •

edited

Loading