Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web API Endpoint to programmatically upload biom for a study #2320

Closed
mwang87 opened this issue Oct 2, 2017 · 9 comments
Closed

Web API Endpoint to programmatically upload biom for a study #2320

mwang87 opened this issue Oct 2, 2017 · 9 comments

Comments

@mwang87
Copy link

mwang87 commented Oct 2, 2017

Web API to upload a biom file (or provide a URL for qiita to pull down the file) would be useful for a service like GNPS to export biom file for metabolomics analysis to be analyzed easily in Qiita directly and easily for GNPS users.

@antgonza antgonza closed this as completed Oct 2, 2017
@antgonza antgonza reopened this Oct 2, 2017
@antgonza
Copy link
Member

antgonza commented Oct 2, 2017

Sorry, fat fingers. I think the main problem is that to be able to add a biom we first need a study with a sample info file and prep info file; you can add a biom to a prep. So perhaps adding we can allow study, sample and prep info creation via the API or some way figure out how to transfer that info back and forth.

Now, this doesn't actually solve the issue with GNPS as it doesn't have sample information (not sure about prep) so I think if we add the raw data to Qiita, then send the info for processing to GNPS, then send back the biom, will be the best way forward.

@mwang87, what do you think?

@ElDeveloper
Copy link
Member

@antgonza, I think those points are covered by the REST API endpoints that @wasade and I put together a few months ago. See here and here for the full list of endpoints.

The main problem is that these are not exposed to users outside of Qiita, exposing this might take some work.

@antgonza
Copy link
Member

antgonza commented Oct 3, 2017

Right, thanks. We could start with only internal and go from there ...

@mwang87
Copy link
Author

mwang87 commented Oct 3, 2017

Cool, that makes sense. Are internal APIs just not exposed by blocking ports or could we just have a credentialing process?

@josenavas
Copy link
Contributor

I just want to drop my 2 cents here. As per previous discussions with @mwang87 and others we agreed that creating a plugin would be the best way of connecting the 2 systems. Following this route, no new development in Qiita should be done as the plugin infrastructure should be complete.

@ElDeveloper
Copy link
Member

ElDeveloper commented Oct 3, 2017 via email

@ElDeveloper
Copy link
Member

ElDeveloper commented Oct 3, 2017 via email

@mwang87
Copy link
Author

mwang87 commented Oct 3, 2017

@josenavas Yes, I think it would be great if users are starting at Qiita with also sequencing data. I do also think there are a lot of people that come from the mass spectrometry side that would like to have access to the statistical tools available at Qiita that are not familiar with the platform but are familiar with GNPS. Currently, we automatically generate a subset of emperor features, but I want to avoid recreating all the analysis features that are already implemented in Qiita.

@antgonza
Copy link
Member

antgonza commented Oct 3, 2017

plugin will be the best. Figuring out how to "request" sample/prep info from GNPS users, will be the best. As you know, without metadata analysis are impossible. BTW we have this: https://github.com/qiita-spots/qtp-mass-spec

@adswafford adswafford added this to the Pre-publication milestone Oct 4, 2017
@antgonza antgonza modified the milestones: Nov 2018 Update, Jan 2019 Dec 7, 2018
@antgonza antgonza modified the milestones: Jan 2019, Mar 2019 Jan 29, 2019
antgonza added a commit to antgonza/qiita that referenced this issue Sep 4, 2020
ElDeveloper pushed a commit that referenced this issue Sep 9, 2020
* fix #2320

* adding prints to debug

* children -> 1

* APIArtifactHandlerTest -> APIArtifactHandlerTests

* configure_biom

* qdb.util.activate_or_update_plugins

* improving code

* almost there

* add values.template

* fix filepaths

* filepaths -> files

* fixing errors

* add prep.artifact insertion

* addressing @ElDeveloper comments
ElDeveloper added a commit that referenced this issue Nov 9, 2020
* Version 092020 (#3034)

* inject study_type in EBI and improvements to current automatic processing pipeline (#3023)

* inject study_type in ebi and improvements to current automatic proecssing pipeline

* addressing @ElDeveloper comments

* some general fixes/additions for next release (#3026)

* some general fixes/additions for next release

* adding test for not None job.release_validator_job

* fix #2839

* fix #2868 (#3028)

* fix #2868

* 2nd round

* fix errors

* more changes

* fix errors

* fix ProcessingJobTest

* fix PY_PATCH

* add missing TRN.add

* encapsulated_query -> perform_as_transaction

* fix #3022 (#3030)

* fix #3022

* adding tests

* fix #2320 (#3031)

* fix #2320

* adding prints to debug

* children -> 1

* APIArtifactHandlerTest -> APIArtifactHandlerTests

* configure_biom

* qdb.util.activate_or_update_plugins

* improving code

* almost there

* add values.template

* fix filepaths

* filepaths -> files

* fixing errors

* add prep.artifact insertion

* addressing @ElDeveloper comments

* fix artifact_definition active command

* != -> ==

* Added three tutorial sections to the Qiita documentation (#3032)

* Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab.

* Update redbiom.rst

* Update redbiom.rst

* Update redbiom.rst

* Further updates to redbiom.rst and the Stats tutorial.

* update redbiom.rst

* Finished proof-reading

* Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data

* added a new introduction, with links to the three sections

* Added figures to stats tutorial and contexts explanation

* Added figures to stats tutorial and contexts explanation

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Antonio Gonzalez <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* 092020 (#3033)

* 092020

* connect artifact with job

* rm INSERT qiita.artifact_processing_job

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* fix #3036

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>
ElDeveloper added a commit that referenced this issue Nov 9, 2020
…3040)

* Version 092020 (#3034)

* inject study_type in EBI and improvements to current automatic processing pipeline (#3023)

* inject study_type in ebi and improvements to current automatic proecssing pipeline

* addressing @ElDeveloper comments

* some general fixes/additions for next release (#3026)

* some general fixes/additions for next release

* adding test for not None job.release_validator_job

* fix #2839

* fix #2868 (#3028)

* fix #2868

* 2nd round

* fix errors

* more changes

* fix errors

* fix ProcessingJobTest

* fix PY_PATCH

* add missing TRN.add

* encapsulated_query -> perform_as_transaction

* fix #3022 (#3030)

* fix #3022

* adding tests

* fix #2320 (#3031)

* fix #2320

* adding prints to debug

* children -> 1

* APIArtifactHandlerTest -> APIArtifactHandlerTests

* configure_biom

* qdb.util.activate_or_update_plugins

* improving code

* almost there

* add values.template

* fix filepaths

* filepaths -> files

* fixing errors

* add prep.artifact insertion

* addressing @ElDeveloper comments

* fix artifact_definition active command

* != -> ==

* Added three tutorial sections to the Qiita documentation (#3032)

* Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab.

* Update redbiom.rst

* Update redbiom.rst

* Update redbiom.rst

* Further updates to redbiom.rst and the Stats tutorial.

* update redbiom.rst

* Finished proof-reading

* Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data

* added a new introduction, with links to the three sections

* Added figures to stats tutorial and contexts explanation

* Added figures to stats tutorial and contexts explanation

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Antonio Gonzalez <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* 092020 (#3033)

* 092020

* connect artifact with job

* rm INSERT qiita.artifact_processing_job

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* adding job submit ENVIRONMENT and minor improvement to runWorkflow

* fix test

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>
ElDeveloper added a commit that referenced this issue Nov 9, 2020
* Version 092020 (#3034)

* inject study_type in EBI and improvements to current automatic processing pipeline (#3023)

* inject study_type in ebi and improvements to current automatic proecssing pipeline

* addressing @ElDeveloper comments

* some general fixes/additions for next release (#3026)

* some general fixes/additions for next release

* adding test for not None job.release_validator_job

* fix #2839

* fix #2868 (#3028)

* fix #2868

* 2nd round

* fix errors

* more changes

* fix errors

* fix ProcessingJobTest

* fix PY_PATCH

* add missing TRN.add

* encapsulated_query -> perform_as_transaction

* fix #3022 (#3030)

* fix #3022

* adding tests

* fix #2320 (#3031)

* fix #2320

* adding prints to debug

* children -> 1

* APIArtifactHandlerTest -> APIArtifactHandlerTests

* configure_biom

* qdb.util.activate_or_update_plugins

* improving code

* almost there

* add values.template

* fix filepaths

* filepaths -> files

* fixing errors

* add prep.artifact insertion

* addressing @ElDeveloper comments

* fix artifact_definition active command

* != -> ==

* Added three tutorial sections to the Qiita documentation (#3032)

* Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab.

* Update redbiom.rst

* Update redbiom.rst

* Update redbiom.rst

* Further updates to redbiom.rst and the Stats tutorial.

* update redbiom.rst

* Finished proof-reading

* Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data

* added a new introduction, with links to the three sections

* Added figures to stats tutorial and contexts explanation

* Added figures to stats tutorial and contexts explanation

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Antonio Gonzalez <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* 092020 (#3033)

* 092020

* connect artifact with job

* rm INSERT qiita.artifact_processing_job

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* fix #2920

* fix test

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>
ElDeveloper added a commit that referenced this issue Nov 10, 2020
* Version 092020 (#3034)

* inject study_type in EBI and improvements to current automatic processing pipeline (#3023)

* inject study_type in ebi and improvements to current automatic proecssing pipeline

* addressing @ElDeveloper comments

* some general fixes/additions for next release (#3026)

* some general fixes/additions for next release

* adding test for not None job.release_validator_job

* fix #2839

* fix #2868 (#3028)

* fix #2868

* 2nd round

* fix errors

* more changes

* fix errors

* fix ProcessingJobTest

* fix PY_PATCH

* add missing TRN.add

* encapsulated_query -> perform_as_transaction

* fix #3022 (#3030)

* fix #3022

* adding tests

* fix #2320 (#3031)

* fix #2320

* adding prints to debug

* children -> 1

* APIArtifactHandlerTest -> APIArtifactHandlerTests

* configure_biom

* qdb.util.activate_or_update_plugins

* improving code

* almost there

* add values.template

* fix filepaths

* filepaths -> files

* fixing errors

* add prep.artifact insertion

* addressing @ElDeveloper comments

* fix artifact_definition active command

* != -> ==

* Added three tutorial sections to the Qiita documentation (#3032)

* Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab.

* Update redbiom.rst

* Update redbiom.rst

* Update redbiom.rst

* Further updates to redbiom.rst and the Stats tutorial.

* update redbiom.rst

* Finished proof-reading

* Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data

* added a new introduction, with links to the three sections

* Added figures to stats tutorial and contexts explanation

* Added figures to stats tutorial and contexts explanation

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Antonio Gonzalez <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* 092020 (#3033)

* 092020

* connect artifact with job

* rm INSERT qiita.artifact_processing_job

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* rm create_qiime_mapping_file

* fixing some tests

* fixing more tests

* fix even more tests

* rm npt.assert_warns

* qiime-map -> sample-file

* update not_merged_samples.txt

* adding @ElDeveloper changes

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>
ElDeveloper added a commit that referenced this issue Nov 17, 2020
* Version 092020 (#3034)

* inject study_type in EBI and improvements to current automatic processing pipeline (#3023)

* inject study_type in ebi and improvements to current automatic proecssing pipeline

* addressing @ElDeveloper comments

* some general fixes/additions for next release (#3026)

* some general fixes/additions for next release

* adding test for not None job.release_validator_job

* fix #2839

* fix #2868 (#3028)

* fix #2868

* 2nd round

* fix errors

* more changes

* fix errors

* fix ProcessingJobTest

* fix PY_PATCH

* add missing TRN.add

* encapsulated_query -> perform_as_transaction

* fix #3022 (#3030)

* fix #3022

* adding tests

* fix #2320 (#3031)

* fix #2320

* adding prints to debug

* children -> 1

* APIArtifactHandlerTest -> APIArtifactHandlerTests

* configure_biom

* qdb.util.activate_or_update_plugins

* improving code

* almost there

* add values.template

* fix filepaths

* filepaths -> files

* fixing errors

* add prep.artifact insertion

* addressing @ElDeveloper comments

* fix artifact_definition active command

* != -> ==

* Added three tutorial sections to the Qiita documentation (#3032)

* Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab.

* Update redbiom.rst

* Update redbiom.rst

* Update redbiom.rst

* Further updates to redbiom.rst and the Stats tutorial.

* update redbiom.rst

* Finished proof-reading

* Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data

* added a new introduction, with links to the three sections

* Added figures to stats tutorial and contexts explanation

* Added figures to stats tutorial and contexts explanation

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Antonio Gonzalez <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* 092020 (#3033)

* 092020

* connect artifact with job

* rm INSERT qiita.artifact_processing_job

* Apply suggestions from code review [skip ci]

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

* adding qiita.study autoloaded column

* cleaning tests

* fix redbiom test

* avoiding clog submissions

* Apply suggestions from code review

Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>

Co-authored-by: Daniel McDonald <[email protected]>
Co-authored-by: Mirte Kuijpers <[email protected]>
Co-authored-by: Yoshiki Vázquez Baeza <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants