feat: add ECharts BoxPlot chart

[villebro]

SUMMARY

Replace the BoxPlot chart with an ECharts implementation (The bulk of work has been done in apache-superset/superset-ui#801 ). Features of the new viz plugin:

• Full feature parity with current D3 based BoxPlot plugin
• Chart now refreshes when dashboard filters are updated. Closes Box plot does not filter in dashboard  #9130
• Adds possibility to do non-temporal boxplots. Closes What should I do if I do not need time series in the box plot chart? #10468, closes BoxPlot - multiple issues #2222
• Groupby is now optional.
• Use midpoint interpolation for quartile calculations to replicate behaviour of the boxplot function in R. See ?boxplot.stats:

The two ‘hinges’ are versions of the first and third quartile, i.e., close to quantile(x, c(1,3)/4). The hinges equal the quartiles for odd n (where n <- length(x)) and differ for even n. Whereas the quartiles only equal observations for n %% 4 == 1 (n = 1 mod 4), the hinges do so additionally for n %% 4 == 2 (n = 2 mod 4), and are in the middle of two observations otherwise.

Also introduces a new dedicated post processing operation for boxplot to make it easier for viz plugin developers to tap into this functionality.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

World's Bank Data example dashboard after:

World's Bank Data example dashboard before:

Control panels:

TEST PLAN

CI + new tests

ADDITIONAL INFORMATION

• Has associated issue: [SIP-50] Proposal for using ECharts as our main charting library #10418
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

[codecov-io]

Codecov Report

Merging #11199 (e60d19d) into master (77dff0e) will increase coverage by 0.99%.
The diff coverage is 82.89%.

@@            Coverage Diff             @@
##           master   #11199      +/-   ##
==========================================
+ Coverage   65.61%   66.61%   +0.99%     
==========================================
  Files         873      874       +1     
  Lines       42315    42319       +4     
  Branches     3971     3972       +1     
==========================================
+ Hits        27765    28190     +425     
+ Misses      14433    14031     -402     
+ Partials      117       98      -19     

Flag	Coverage Δ
cypress	55.24% <87.50%> (+5.60%)	⬆️
javascript	62.85% <77.77%> (+<0.01%)	⬆️
python	62.03% <82.19%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...-frontend/src/visualizations/presets/MainPreset.js	100.00% <ø> (ø)
superset/utils/dashboard_import_export.py	0.00% <ø> (-60.72%)	⬇️
superset/viz.py	59.33% <ø> (+1.60%)	⬆️
superset/cli.py	40.39% <12.50%> (+0.39%)	⬆️
superset/views/core.py	74.32% <66.66%> (+0.02%)	⬆️
superset/dashboards/commands/importers/v0.py	80.92% <80.92%> (ø)
superset/utils/pandas_postprocessing.py	78.60% <95.45%> (+3.12%)	⬆️
superset-frontend/src/components/Icon/index.tsx	100.00% <100.00%> (ø)
...tend/src/explore/components/ExploreChartHeader.jsx	83.78% <100.00%> (ø)
...rc/explore/components/controls/AnnotationLayer.jsx	51.17% <100.00%> (+9.85%)	⬆️
... and 80 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77dff0e...c39f18a. Read the comment docs.

[villebro]

For some reason this test started flaking out on the legacy plugins, despite those being unchanged. Although I xited this test, I updated it in a way which IMO should work.

[bkyryliuk]

overall everything looks good! let me test it on staging once conflicts are resolved

[bkyryliuk]

nice cleanup!

[bkyryliuk]

there was a bug with 3rd quantile, will test it out once conflicts are resolved

[villebro]

Good catch, let me take a look

[bkyryliuk]

I meant prior to your PR, will test now if that is still an issue

[bkyryliuk]

Actually it is still a problem

Data for the bucket:

[f] 1000-2499 | 7251270
[f] 1000-2499 | 1820590.6666666667
[f] 1000-2499 | 1187242.142857143
[f] 1000-2499 | 1151020
e.g. look like outlier was not excluded for quartile calculations

[villebro]

I was able to reproduce. Interesting edge case with so few observations, I'll replicate with some other commonly used stats package to see what it would give for a similar case.

[bkyryliuk]

nit: Old naming seems a bit more intuitive and cleaner,

Old:

New:

[villebro]

nit: Old naming seems a bit more intuitive and cleaner,

I'll make a follow-up PR to use the same wording as in the previous version.

[villebro]

For reference Bogdan's sample data in Superset and results for same dataset from two different online boxplot calculators:
Superset:

http://www.alcula.com/calculators/statistics/box-plot/

http://shiny.chemgrid.org/boxplotr/:

[villebro]

It seems most stats packages are using midpoint interpolation for quartiles. Results with new interpolation parameters seem to reproduce the BoxPlotR results (I'm guessing this is THE benchmark for boxplots):

[villebro]

@ktmud (pinging as I noticed you had introduced this test recently) I had to disable this test, as apparently the function annotation layer that was added to the ECharts Timeseries viz loads mathjs despite not being used here. The relevant code is here (is called by transformProps.ts): https://github.com/apache-superset/superset-ui/blob/master/plugins/plugin-chart-echarts/src/utils/annotation.ts . Is there something that can be done differently to avoid having mathjs load here?

[ktmud]

You chart plugin seems to be using a different version of mathjs, which is probably incompatible with what's used in AnnotationLayer control. Do you mind changing them to the same version and see if the test still fails?

[villebro]

Ok, good to know, I'll do that 👍

[villebro]

@ktmud I've now upgraded all mathjs deps to 8.0.1, and still failing the test. I propose disabling it for now so we can merge this.

[ktmud]

The test is failing because of this import chain: @superset-ui/plugin-chart-echarts -> Timeseries -> transformProps -> utils/annotation -> mathjs. Unlike loadChart, transformProps is not loaded asynchronously, so Webpack could not defer the loading of mathjs, which means it will be loaded for all first-visits on almost all pages, undoing part of the optimization we did in #10837 .

The test is there to make sure this optimization still works. It didn't break because it was flaky. It caught the breaking changes it is supposed to catch.

It's OK to merge this PR as is for now, but I'd recommend finding ways to move the mathjs logic to the chart renderer so that we could continue benefiting from code split and asynchronous loading.

In general I think transformProps should be as lightweight as possible and preferably only return scalars. It should only be about applying defaults and consolidating parameter names/shapes. Most complex logics should be dealt with either by the chart renderer itself or at another data-processing layer on top of the chart. This way it's easier to reuse your chart at other places outside of Superset charting plugins.

[villebro]

Thanks for the context. I'll move the mathjs (and any other similar) rendering logic out of transformProps and re-enable the test in a subsequent bump.

[junlincc]

@villebro i think it make sense to expand box plot to include candlesticks chart and whisker chart two variations since they share the same set of control, just like you did for time series. what do you think for post 1.0 hardening?

[villebro]

@junlincc hmm, not a bad idea, as candlesticks just use pick slightly different data points from within the group (when we do that it might make sense to rename the boxplot post transformation operation to something more generic). However, candlesticks tend to have a temporal x-axis, so we need to add proper support for that first.