docs: new dataset source, from vega_dataset to altair.dataset

[mattijn]

Updated test expectations and example files to work with the new vega-datasets source. Fixed field name references from underscores to spaces (eg., IMDB_Rating → IMDB Rating) and updated row count expectations in transformed data tests.

[jonmmease]

ValueError: DataFusion error: Object Store error: Generic HTTP error: Header

Is that then end of the error? Was expecting the nature of the header error to follow.

[mattijn]

I was able to make a minimal working example reproducing the error with only Vega and VegaFusion:

Open the Chart in the Vega Editor

import json
import vegafusion as vf

vega_spec = json.loads("""
{
  "$schema": "https://vega.github.io/schema/vega/v6.json",
  "data": [
    {
      "name": "source_0",
      "url": "https://cdn.jsdelivr.net/npm/vega-datasets@v3.2.0/data/co2-concentration.csv",
      "format": {
        "type": "csv",
        "parse": {
          "Date": "date"
        }
      }
    }
  ]
}
""")

# This will raise the HTTP error
datasets, warnings = vf.runtime.pre_transform_datasets(vega_spec, ["source_0"])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[14], line 24
      4 vega_spec = json.loads("""
      5 {
      6   "$schema": "https://vega.github.io/schema/vega/v6.json",
   (...)
     20 
     21 """)
     23 # This will raise the HTTP error
---> 24 datasets, warnings = vf.runtime.pre_transform_datasets(vega_spec, ["source_0"])

File ~/miniconda3/envs/stable/lib/python3.12/site-packages/vegafusion/runtime.py:535, in VegaFusionRuntime.pre_transform_datasets(self, spec, datasets, local_tz, default_input_tz, row_limit, inline_datasets, trim_unused_columns, dataset_format)
    527 # Serialize inline datasets
    528 inline_arrow_dataset = self._import_inline_datasets(
    529     inline_datasets,
    530     inline_dataset_usage=get_inline_column_usage(spec)
    531     if trim_unused_columns
    532     else None,
    533 )
--> 535 values, warnings = self.runtime.pre_transform_datasets(
    536     spec,
    537     pre_tx_vars,
    538     local_tz=local_tz,
    539     default_input_tz=default_input_tz,
    540     row_limit=row_limit,
    541     inline_datasets=inline_arrow_dataset,
    542 )
    544 def normalize_timezones(
    545     dfs: list[nw.DataFrame[IntoFrameT] | nw.LazyFrame[IntoFrameT]],
    546 ) -> list[DataFrameLike]:
    547     # Convert to `local_tz` (or, set to UTC and then convert if starting
    548     # from time-zone-naive data), then extract the native DataFrame to return.
    549     processed_datasets = []

ValueError: DataFusion error: Object Store error: Generic HTTP error: Header

[mattijn]

Finally all tests are passing! @joelostblom would you be willing to do a review?

[dsmedia]

@mattijn Looks like in 577ed87 you had to remove iris.json because it's in the legacy altair-viz/vega_datasets but not in vega/vega-datasets. Do I understand that right?

If so, would it help you if I added iris.json to vega/vega-datasets? For future development plans, I'd like to have a clean match between the datasets pulled in by example-gallery examples from all the vega visualization libraries and the vega-datasets repo. So this would be beneficial even if you decide not to undo 577ed87.

[mattijn]

That dataset was removed from vega/vega-datasets a few years ago, in this PR vega/vega-datasets#187.

[joelostblom]

Thanks for all the work you have done on getting the sample data integrating into altair @mattijn ! I will aim to have a look at this in more detail during next week. One question I have now already is how we are releasing and announcing the changes to the sample data import convention.

My understanding is that we are not breaking anything from vega-datasets and that old code will still work as before #3848. So this is probably released with a minor version bump? Still, the fact that the column names have changed in the examples might be confusing without an explicit note somewhere. Maybe it is enough to clearly spell this out in the release notes and we don't need to also note it in the documentation? Although when we added the new method-based syntax for channel options, we included a note in the docs like so:

Do you think we should include a similar callout somewhere in the docs (maybe in the "Specifying Data" page) to emphasize the move to altair.datasets?

[mattijn]

On the main branch, we moved to vegalite version 6 already, so I think we can bump this as part of a major release. I like the suggestion to have a note in the specifying data section, "With the release of Altair 6 etc"

[mattijn]

Add a note in this commit b1c648f at the end of the first paragraph in this section: https://altair-viz.github.io/gallery/index.html, where the source of the datasets in the examples are discussed.

[joelostblom]

Looks great overall! Thanks for all your work getting this updated! I think it is really convenient that the same import convention is kept while having the data in the altair package directly.

I clicked through the entire user guides and the examples, and noticed a few places where the charts were not rendering, which I have commented on inline. I noticed a couple of things not related to this PR too which I will fix separately.

Add a note in this commit b1c648f at the end of the first paragraph in this section: https://altair-viz.github.io/gallery/index.html, where the source of the datasets in the examples are discussed.

Looks good!

address all feedback from @joelostblom by the following commit: aa5d353

[mattijn]

Thanks @joelostblom for the review! Addressed all of them. Had a thorough check as well, and all charts are rendering now, as far as I can see. Merging this PR since all tests are green on CI 🥳