[Lens] CSV Export for Lens by dej611 · Pull Request #83430 · elastic/kibana

dej611 · 2020-11-16T14:51:02Z

Summary

First implementation of the CSV export feature for Lens.

Implement a first draft of the datatableToCsv function
- First implementation based on the inspector one
- Should support multiple layers (naming logic)
- Should support formatters from the Datatable field hints
- Should live integrated as data plugin + download share plugin
- Add an option for string export to make the function testable?
Hook to the onData$ callback and pass the activeData to the app, which in turns shows the button
- Display a button to download the data (show/hide logic)
- Make the button download the data
- Hook to settings to define separators/other options

Note: the embeddable action will be implemented in a subsequent PR.

Fixes part of #74509

Open questions:

at the moment, in case of multiple layers, each csv file will have a postfix with an index value (i.e. filename-N.csv). In [Lens] CSV export #74509 the suggestion was to use the postfix -Layer N (=> filename-Layer N.csv), which is a valid option. Is the - Layer N postfix the final decision on that? ( Change in later on will be easy )

Possible enhancement discussion (out of scope for this PR, but worth discussing):

A possible improvement on this side, as proposed in the linked issue, would be to have layers with names to use as postfix.
Multiple csv download is quite tricky, perhaps a possible future enhancement could be to download a zip file rather than multiple files?
Define a UI to make the User choose between formatted or raw values?

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

dej611 · 2020-11-17T11:26:30Z

Made the Download CSV button stay on the top nav all the time with enable/disable styling: on bootstrap I saw it blinking with the previous behaviour, so I moved to this other option which is less "noisy".
cc @MichaelMarcialis

…export

elasticmachine · 2020-11-18T17:35:42Z

Pinging @elastic/kibana-app (Team:KibanaApp)

dej611 · 2020-11-19T10:28:05Z

@elasticmachine merge upstream

ppisljar · 2020-11-19T13:48:11Z

src/plugins/data/public/exports/export_csv.tsx

+ *
+ * @returns undefined (download) - Record\<string, string\> (only for testing)
+ */
+export function exportAsCSVs(


lets change this function so it doesn't download the file but just returns csv as a string, this way we can move the function to common and make it usable from server.

the download aspect of it should be moved to some other place (possibly share plugin) so that can be used for other things as well, as that download function would now just take filename and filecontent in.

You proposing splitting this into two exportCSVs + download plugins?

yes, two functions as converting to csv can be common and downloading can be used for other things beside csv as well.

…to feature/lens/csv-export

dej611 · 2020-11-23T09:04:54Z

@elasticmachine merge upstream

ppisljar · 2020-11-23T13:08:19Z

src/plugins/data/public/exports/export_csv.tsx

+  raw?: boolean;
+}
+
+function buildCSV(


lets rename this one to datatableToCSV and export it

ppisljar · 2020-11-23T13:11:00Z

src/plugins/data/public/exports/export_csv.tsx

+ *
+ * @returns A dictionary of files to download: the key is the filename and the value the CSV string
+ */
+export function exportAsCSVs(


i am wondering if this function should be kept inside lens for now. how often will we want to convert multiple datatables at once ? also i am wondering if the reference to filename here should be omited but rather just use the datatable name ? @lukeelmers whats your opinion ?

Mind that the DataTable type has no name property. The key:string passed for the record may be just internal ids (for instance in Lens it represents the layer id).

As we move down the path of creating these various utility functions for datatables, I think it's important to have some consistency in how you interact with them. For any datatable utility function, I'd expect an interface similar to the following:

type SomeFn = (table: Datatable, options?: Record<string, whatever>) => Promise<Something> | Something;

how often will we want to convert multiple datatables at once ?

For the sake of consistency, I would vote to make this function operate on a single table just like our other datatable helpers.

i am wondering if the reference to filename here should be omited but rather just use the datatable name

I think the concept of a filename is only here because the file download functionality was extracted from this function originally. IMHO it would make more sense to remove it and handle it when calling the download helper from the share plugin.

This would make the interface something like:

exportAsCsv(table: Datatable, options: CSVOptions) => string;

Then Lens would handle calling it multiple times (for multiple tables), assigning a filename, and passing the output to the download helper.

To me this feels like the best way to make these utilities as decoupled as possible, and keeps the lens-specific stuff in lens. (And of course we can always revisit down the road if some of the lens logic is needed for a bunch of other apps)

flash1293

Tested in Firefox and works well for me. It would be super cool to have a functional test for that but I'm not sure whether that's possible with our current setup.

flash1293 · 2020-11-23T11:27:25Z

x-pack/plugins/lens/public/persistence/saved_object_store.ts

      state?: unknown;
    };
    filters: PersistableFilter[];
+    activeData?: TableInspectorAdapter;


Seems like this is a leftover

…to feature/lens/csv-export

lukeelmers · 2020-11-23T15:53:17Z

src/plugins/share/public/index.ts

 import { SharePlugin } from './plugin';

 export { KibanaURL } from './kibana_url';
+export * from './lib/download_as';


nit: We prefer to explicitly export named items from the top-level public/index.ts (instead of using export *)... that way we avoid accidentally leaking internal items into the public contract later.

lukeelmers · 2020-11-23T17:03:05Z

src/plugins/data/public/exports/export_csv.tsx

+ *
+ * @returns A dictionary of files to download: the key is the filename and the value the CSV string
+ */
+export function exportAsCSVs(


As we move down the path of creating these various utility functions for datatables, I think it's important to have some consistency in how you interact with them. For any datatable utility function, I'd expect an interface similar to the following:

type SomeFn = (table: Datatable, options?: Record<string, whatever>) => Promise<Something> | Something;

how often will we want to convert multiple datatables at once ?

For the sake of consistency, I would vote to make this function operate on a single table just like our other datatable helpers.

i am wondering if the reference to filename here should be omited but rather just use the datatable name

I think the concept of a filename is only here because the file download functionality was extracted from this function originally. IMHO it would make more sense to remove it and handle it when calling the download helper from the share plugin.

This would make the interface something like:

exportAsCsv(table: Datatable, options: CSVOptions) => string;

Then Lens would handle calling it multiple times (for multiple tables), assigning a filename, and passing the output to the download helper.

To me this feels like the best way to make these utilities as decoupled as possible, and keeps the lens-specific stuff in lens. (And of course we can always revisit down the road if some of the lens logic is needed for a bunch of other apps)

lukeelmers · 2020-11-23T17:05:22Z

src/plugins/data/public/exports/index.ts

+ * under the License.
+ */
+
+export * from './export_csv';


If we make my proposed changes to exportAsCsv, then we could move this all to the common directory. (Maybe common/utils?) and export it from both public/index.ts and server/index.ts

lukeelmers · 2020-11-23T17:06:06Z

src/plugins/data/public/index.ts

+ * Exporters (CSV)
+ */
+
+export * from './exports';


Same comment re: using named exports from top-level public/index.ts and server/index.ts

…export

lukeelmers

Thanks for the edits @dej611! Code changes here LGTM pending green build. Let's wait to make sure @ppisljar has one last look at it before it merges.

ppisljar

code LGTM

dej611 · 2020-11-24T09:18:41Z

It would be super cool to have a functional test for that but I'm not sure whether that's possible with our current setup.

@flash1293 added basic functional test :)

kibanamachine · 2020-11-24T11:32:18Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c3941a1

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`data`	601	603	+2
`share`	39	47	+8
total			+10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lens`	935.0KB	936.6KB	+1.6KB

Distributable file count

id	before	after	diff
`default`	43058	43060	+2
`oss`	22583	22585	+2

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	974.3KB	976.0KB	+1.7KB
`lens`	52.4KB	52.6KB	+232.0B
`share`	70.4KB	81.2KB	+10.8KB
total			+12.7KB

History

💚 Build #89761 succeeded 80eac9f
💚 Build #89679 succeeded 084a22f
💔 Build #89657 failed 1fbdeef
💚 Build #89520 succeeded 3da4fab
💚 Build #89330 succeeded 4674a83

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2020-11-26T11:44:39Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 83430 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-11-27T11:54:38Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 83430 or prevent reminders by adding the backport:skip label.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/plugins/data/server/server.api.md

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

dej611 added 6 commits November 16, 2020 11:15

✨ First draft of csv export in Lens

078e554

⚗️ Create action stub

5d789d6

🚚 + ✅ Move export csv to data plugins + basic testing

965bb69

💄 Enable/disable rather than show/hide button

456d14a

🐛 Fix tests

8f917b1

💬 Fix i18n checks + :packages: Update API docs

851b98f

dej611 added 2 commits November 17, 2020 13:23

🔥 removed embeddable action for now

08d75d0

Merge remote-tracking branch 'upstream/master' into feature/lens/csv-…

12ac1e0

…export

dej611 added Feature:Lens release_note:enhancement v7.11.0 v8.0.0 labels Nov 17, 2020

dej611 added 5 commits November 18, 2020 13:11

🐛 Forgotten type

88eefc6

Merge remote-tracking branch 'upstream/master' into feature/lens/csv-…

ce68a93

…export

💄 Move the cancel button on the right

88c2a8d

🐛 Avoid to save the activeData content

27678f4

🐛 Fix tests

52f606f

dej611 mentioned this pull request Nov 18, 2020

[Embeddable] Export CSV action for Lens embeddables in dashboard #83654

Merged

15 tasks

dej611 added the Team:Visualizations Team label for Lens, elastic-charts, Graph, legacy editors (TSVB, Visualize, Timelion) t// label Nov 18, 2020

dej611 marked this pull request as ready for review November 18, 2020 17:35

dej611 requested a review from a team November 18, 2020 17:35

dej611 requested a review from a team as a code owner November 18, 2020 17:35

Merge branch 'master' into feature/lens/csv-export

237664e

ppisljar reviewed Nov 19, 2020

View reviewed changes

dej611 added 3 commits November 19, 2020 18:07

✨ split the current plugin in two distinct areas

4de7ae8

Merge branch 'feature/lens/csv-export' of github.com:dej611/kibana in…

6e88319

…to feature/lens/csv-export

📝 Update API documentation

f851e69

Merge branch 'master' into feature/lens/csv-export

3da4fab

ppisljar reviewed Nov 23, 2020

View reviewed changes

flash1293 approved these changes Nov 23, 2020

View reviewed changes

dej611 added 3 commits November 23, 2020 16:26

👌 Integrate feedback

4cada29

Merge branch 'feature/lens/csv-export' of github.com:dej611/kibana in…

1fbdeef

…to feature/lens/csv-export

📝 Update API doc

084a22f

lukeelmers requested changes Nov 23, 2020

View reviewed changes

dej611 added 4 commits November 23, 2020 18:28

Merge remote-tracking branch 'upstream/master' into feature/lens/csv-…

cb173b0

…export

♻️ Refactor export plugin as per feedback received

ca868b3

📝 Update API doc

0e7770c

👌 use named exports

80eac9f

lukeelmers approved these changes Nov 23, 2020

View reviewed changes

ppisljar approved these changes Nov 24, 2020

View reviewed changes

✅ Add basic functional test

c3941a1

dej611 merged commit 423888c into elastic:master Nov 24, 2020

dej611 deleted the feature/lens/csv-export branch November 24, 2020 11:37

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 26, 2020

dej611 added a commit to dej611/kibana that referenced this pull request Nov 30, 2020

[Lens] CSV Export for Lens (elastic#83430)

89ab95b

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/plugins/data/server/server.api.md

dej611 mentioned this pull request Nov 30, 2020

[7.x] [Lens] CSV Export for Lens (#83430) #84474

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 30, 2020

dej611 added a commit that referenced this pull request Nov 30, 2020

[7.x] [Lens] CSV Export for Lens (#83430) (#84474)

a5974ac

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

timroes mentioned this pull request Dec 2, 2020

[Data table] use shared CSV export mechanism #84759

Closed

flash1293 mentioned this pull request Dec 3, 2020

[Lens] CSV export #74509

Closed

rajvi-patel-22 mentioned this pull request Feb 14, 2023

[Lens Migration] Migrate visualizations to lens for ActiveMQ, Spring Boot and WebSphere Application Server elastic/integrations#5250

Closed

9 tasks

Conversation

dej611 commented Nov 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Open questions:

Possible enhancement discussion (out of scope for this PR, but worth discussing):

Checklist

Uh oh!

dej611 commented Nov 17, 2020

Uh oh!

elasticmachine commented Nov 18, 2020

Uh oh!

dej611 commented Nov 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dej611 commented Nov 23, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flash1293 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lukeelmers left a comment

Choose a reason for hiding this comment

Uh oh!

ppisljar left a comment

Choose a reason for hiding this comment

Uh oh!

dej611 commented Nov 24, 2020

Uh oh!

kibanamachine commented Nov 24, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Distributable file count

Page load bundle

History

Uh oh!

kibanamachine commented Nov 26, 2020

Uh oh!

kibanamachine commented Nov 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dej611 commented Nov 16, 2020 •

edited

Loading