Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Croissant support 🥐 #10341

Closed
pdurbin opened this issue Feb 26, 2024 · 1 comment · Fixed by #10533
Closed

Croissant support 🥐 #10341

pdurbin opened this issue Feb 26, 2024 · 1 comment · Fixed by #10533
Labels
Feature: Metadata NIH CAFE Issues related to and/or funded by the NIH CAFE project Size: 50 A percentage of a sprint. 35 hours.
Milestone
6.4

Comments

@pdurbin
Copy link
Member

pdurbin commented Feb 26, 2024 •
edited
Loading

We plan to implement support for the Croissant format (video).

The code is implemented as an external exporter:

Originally, we planned to use use https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/UTY03A about earthquakes to experiment with the format. This dataset also appears in Kaggle at https://www.kaggle.com/datasets/gustavobmgm/earthquakes-for-ml-prediction . We planned to create the dataset using our dataverse_json.json (native JSON) format. Our initial target, provided by the Kaggle team exported from their dataset on the Kaggle side, looks like kaggle-1.0.json available from https://www.kaggle.com/datasets/gustavobmgm/earthquakes-for-ml-prediction . However, that dataset on the Dataverse side is just a CSV with no additional metadata. Instead we have switched to a dataset about cars provided by Stata as an example. See gdcc/dataverse-exporters#4 and I've deployed my code so far to https://dev3.dataverse.org/dataset.xhtml?persistentId=doi:10.5072/FK2/DZRHUP

Kaggle has advised that we keep an eye on these.

For additional context of how the Croissant project plans to index our Croissant-enabled datasets:

Other related Croissant issues:

Discussion:
@pdurbin pdurbin added Feature: Metadata Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) labels Feb 26, 2024
@pdurbin pdurbin self-assigned this Feb 26, 2024
@pdurbin pdurbin moved this to In Progress 💻 in IQSS Dataverse Project Feb 26, 2024
@pdurbin pdurbin added the NIH CAFE Issues related to and/or funded by the NIH CAFE project label Feb 26, 2024
pdurbin added a commit to gdcc/dataverse-exporters that referenced this issue Feb 26, 2024
@pdurbin
stub out Croissant exporter
e8b564f 
See IQSS/dataverse#10341
@pdurbin pdurbin mentioned this issue Feb 26, 2024
Closed
pdurbin added a commit to IQSS/dataverse-sample-data that referenced this issue Feb 27, 2024
@pdurbin
earthquake data, useful for croissant
758dde4 
from https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/UTY03A
and https://www.kaggle.com/datasets/gustavobmgm/earthquakes-for-ml-prediction

See also IQSS/dataverse#10341
@pdurbin pdurbin mentioned this issue Feb 27, 2024
Merged
@jp-tosca jp-tosca mentioned this issue Mar 15, 2024
Merged
@cmbz cmbz mentioned this issue Mar 29, 2024
Open
12 tasks
@pdurbin pdurbin mentioned this issue Apr 23, 2024
Open
pdurbin added a commit that referenced this issue Apr 26, 2024
@pdurbin
add docs for Croissant, tweak exporter docs #10341
9505141
@pdurbin pdurbin mentioned this issue Apr 26, 2024
Merged
pdurbin added a commit that referenced this issue Apr 29, 2024
@pdurbin
typo #10341
844a08e
pdurbin added a commit that referenced this issue Apr 30, 2024
@pdurbin
link to Croissant JARs, suggest dir for exporters #10341
87c0f58
pdurbin added a commit that referenced this issue Apr 30, 2024
@pdurbin
how to publish snapshots to maven central #10341
f625375
pdurbin added a commit that referenced this issue Apr 30, 2024
@pdurbin
note need to git pull after mvn release:prepare #10341
a84c58e
pdurbin added a commit that referenced this issue May 1, 2024
@pdurbin
update groupId #10341
1c6e7b0
@pdurbin pdurbin mentioned this issue May 6, 2024
Open
@cmbz cmbz added Size: 50 A percentage of a sprint. 35 hours. and removed Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) labels May 8, 2024
pdurbin added a commit that referenced this issue May 21, 2024
@pdurbin
secrets at org level #10341
7f51e19
pdurbin added a commit that referenced this issue May 21, 2024
@pdurbin
Merge branch 'develop' into 10341-croissant #10341
f7e039f
pdurbin added a commit that referenced this issue May 21, 2024
@pdurbin
typo #10341
e83c1b7
pdurbin added a commit that referenced this issue May 28, 2024
@pdurbin
Merge branch 'develop' into 10341-croissant #10341
eaefabe
pdurbin added a commit that referenced this issue May 28, 2024
@pdurbin
for croissant, link to new dedicated git repo and release #10341
d6c0fe1
pdurbin added a commit that referenced this issue May 28, 2024
@pdurbin
update release note #10341
a8e0edf
@pdurbin
Copy link
Member Author

pdurbin commented May 28, 2024

I'm removing this issue from the board now that I've removed "draft" from both of these PRs:

@pdurbin pdurbin removed their assignment May 28, 2024
pdurbin added a commit that referenced this issue May 28, 2024
@pdurbin
add debug exporter, remove most refs to dataverse-exporters #10341
4af01e0
pdurbin added a commit that referenced this issue May 29, 2024
@pdurbin
remove MyJSON, the last reference to dataverse-exporters #10341
c7a7057
pdurbin added a commit that referenced this issue Jun 4, 2024
@pdurbin
Merge branch 'develop' into 10341-croissant #10341
30f560e
pdurbin added a commit that referenced this issue Jul 9, 2024
@pdurbin
Merge branch 'develop' into 10341-croissant #10341
10f03b5
pdurbin added a commit that referenced this issue Jul 9, 2024
@pdurbin
remove inventory of exporters, link to README instead #10341
b5e7401
@pdurbin pdurbin added this to the 6.4 milestone Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature: Metadata NIH CAFE Issues related to and/or funded by the NIH CAFE project Size: 50 A percentage of a sprint. 35 hours.
Projects
None yet
Milestone
6.4
Development

Successfully merging a pull request may close this issue.

make Croissant support official, update exporter docs IQSS/dataverse
2 participants
@pdurbin @cmbz