[ML] Switch file data visualizer to use Papa Parse for CSV parsing #24329
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peteharverson
added
review
non-issue
peteharverson
requested review from
alvarezmelissa87,
jgowdyelastic and
walterra
Contributor
|
Pinging @elastic/ml-ui |
Contributor
💚 Build Succeeded |
jgowdyelastic
added a commit
that referenced
this pull request
Oct 23, 2018
* [ML] File datavisualizer initial commit (#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (#23500) * [ML] Changes to timestamp formats (#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (#23763) * [ML] Adding license check to datavisualizer landing page (#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (#24387) * [ML] Adding ability to override various settings when importing data (#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test
jgowdyelastic
added a commit
to jgowdyelastic/kibana
that referenced
this pull request
Oct 23, 2018
* [ML] File datavisualizer initial commit (elastic#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (elastic#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (elastic#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (elastic#23500) * [ML] Changes to timestamp formats (elastic#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (elastic#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (elastic#23763) * [ML] Adding license check to datavisualizer landing page (elastic#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (elastic#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (elastic#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (elastic#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (elastic#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (elastic#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (elastic#24387) * [ML] Adding ability to override various settings when importing data (elastic#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test
jgowdyelastic
added a commit
that referenced
this pull request
Oct 24, 2018
* [ML] File datavisualizer initial commit (#22828) * [ML] File datavisualizer initial commit * removing mocked data and adding initial stats * adding card styling to fields * Revert "". accidentally added with no commit message This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6. * adding date type to timestamp field * renaming FileStats to FieldsStats * code clean up * changes based on review * changes to error handling * [ML] Adding file datavisualizer overrides (#23194) * [ML] Adding file datavisualizer overrides * improvements to overrides * removing comment * small refactor * removing accidentally added file * updates based on review * fixing broken test * adding missing grok pattern override * fixing test * [ML] Refactoring override option lists (#23424) * [ML] Refactoring override option lists * moving lists out of class * updating test snapshot * [ML] Fixing field editing (#23500) * [ML] Changes to timestamp formats (#23498) * [ML] Changes to timestamp formats * updating test snapshot * [ML] Allow Datavisualizer use on basic license (#23748) * [ML] Allow ML use on basic license * removing timeout change * adding permission checks * updating tests * removing unnecessary checks * [ML] Adds new page for choosing file or index based data visualizer (#23763) * [ML] Adding license check to datavisualizer landing page (#23809) * [ML] Adding license check to datavisualizer landing page * removing comments * updating redirect to landing page * [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042) * [ML] Initial work for delimited file upload * adding results links cards * adding nav menu * removing accidental debugger * initial work for importing semi structured text * using ingest pipeline for import * adding json importer and better error reporting * better progress steps * time range added to results links * first import only creates index and pipeline * adding status constants * using status constants * adding explanation comment * updating yarn.lock * changes based on review * fixing space * fixing space again, stort it out git * removing oversized background container causing constant scrollbar * [ML] Adding basic license check when loading privileges (#24173) * [ML] Adding basic license check * missing import * [ML] Adds an About panel to the file data visualizer landing page (#24260) * [ML] Adds an About panel to the file data visualizer landing page * [ML] Remove unnecessary style from file data visualizer scss * [ML] Adding better error reporting for reading and importing data (#24269) * [ML] Adding better error reporting for reading and importing data * changes to endpoint errors * displaying errors * step logic refactor * removing log statements * [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329) * [ML] Fixes layout of Data Visualizer selector page for IE (#24387) * [ML] Adding ability to override various settings when importing data (#24346) * [ML] Adding ability to override various settings when importing data * second commit with most of the outstanding code * improving index pattern name validation * better index pattern matching * adding comments * adding empty index pattern check * changes based on review * fixing test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Switches the library used for parsing CSV files during the 'import data' stage from csv-parse, which was not supported on IE and caused the ML plugin to fail to load on IE, to Papa Parse, which is already being used by Canvas.
Performance of Papa Parse was around 5 times faster compared to csv-parse in tests on files up to 90MB in size.
Checklist
All checklist items N/A apart from: