This repository was archived by the owner on Aug 4, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 7
[PF-308] Add GCS bucket and bigquery data references #212
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
9773494
initial work, still broken
zloery f8e7cd0
fix name and run spotless
jaycarlton ea5233b
progress
jaycarlton de2f9ba
add basic reference validation. still needs testing and cleanup
zloery 8e912c0
use shared UIDs for cloud resources, tests
zloery 2a1ef38
move controller translation to util, rename createCow methods
zloery 0d74bc6
merge from dev
zloery 0226124
formatting
zloery 9235d98
update other crl versions
zloery 9bdadf3
review comments
zloery fc7d63a
add logging for invalid references
zloery 950fdfd
remove commented out yaml
zloery 4a2f918
merge dev changes
zloery 2e5ac95
add bucket/bq name validation, return values in errors
zloery a2f3c49
remove typeMap
zloery c2a9dd9
merge dev changes
zloery b22c01a
major refactoring of endpoints to per-reference type
zloery eaae8be
add controller annotations
zloery 6b17d85
updated endpoint locations to remove ambiguity
zloery 84afee2
make endpoints plural
zloery f792853
remove per-type delete endpoints
zloery 5ebcc69
merge changes from dev
zloery 26bfe9a
factor out common ReferenceController, proper description wiring
zloery c40e94a
logging formatting
zloery 7937bf1
formatting, fix swagger-ui interaction with remote yaml
zloery 51c06c0
merge dev changes
zloery ac1f476
bump version, responses to comments
zloery File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,7 +15,7 @@ sourceCompatibility = JavaVersion.VERSION_11 | |
|
|
||
| allprojects { | ||
| group = "bio.terra" | ||
| version = "0.11.0-SNAPSHOT" | ||
| version = "0.12.0-SNAPSHOT" | ||
| ext { | ||
| artifactGroup = "${group}.workspace" | ||
| swaggerOutputDir = "${buildDir}/swagger-code" | ||
|
|
@@ -36,6 +36,7 @@ allprojects { | |
| entry "opencensus-impl" | ||
| } | ||
|
|
||
| dependency group: "bio.terra.cloud-resource-lib", name: "cloud-resource-schema", version: "0.4.0" | ||
| dependency group: "io.swagger.core.v3", name: "swagger-annotations", version: "2.1.6" | ||
| dependency group: "io.swagger.codegen.v3", name: "swagger-codegen-cli", version: "3.0.23" | ||
| } | ||
|
|
@@ -69,9 +70,11 @@ dependencies { | |
| implementation group: "bio.terra", name: "datarepo-client", version: "1.0.1-SNAPSHOT" | ||
| implementation group: "bio.terra", name: "stairway", version: "0.0.36-SNAPSHOT" | ||
| implementation group: "bio.terra", name: "terra-common-lib", version: "0.0.13-SNAPSHOT" | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-billing', version: '0.0.4-SNAPSHOT' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-cloudresourcemanager', version: '0.0.8-SNAPSHOT' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-serviceusage', version: '0.0.4-SNAPSHOT' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-billing', version: '0.4.0' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-bigquery', version: '0.4.0' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-cloudresourcemanager', version: '0.4.0' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-serviceusage', version: '0.4.0' | ||
| implementation group: 'bio.terra.cloud-resource-lib', name: 'google-storage', version: '0.4.0' | ||
| implementation group: "org.broadinstitute.dsde.workbench", name: "sam-client_2.12", version: "0.1-61135c7" | ||
| implementation group: "bio.terra", name: "terra-resource-buffer-client", version: "0.4.3-SNAPSHOT" | ||
|
|
||
|
|
@@ -125,6 +128,20 @@ if (hasProperty("buildScan")) { | |
| } | ||
| } | ||
|
|
||
| // CRL schema extraction | ||
| configurations { | ||
| cloudResourceSchema { transitive = false } | ||
| } | ||
| dependencies { | ||
| cloudResourceSchema group: 'bio.terra.cloud-resource-lib', name: 'cloud-resource-schema' | ||
| } | ||
| task unzipCloudResourceSchema(type: Copy) { | ||
| from zipTree(configurations.cloudResourceSchema.singleFile).matching { | ||
| include 'cloud_resources_uid.yaml' | ||
| } | ||
| into "${resourceDir}/api/common" | ||
| } | ||
|
|
||
| // Defs | ||
|
|
||
| def generatedResourceDir = "${resourceDir}/generated" | ||
|
|
@@ -230,9 +247,11 @@ jib { | |
| } | ||
| } | ||
|
|
||
| // Internal dependencies | ||
|
|
||
| sourceSets.main.java.srcDir "${swaggerOutputDir}/src/main/java" | ||
| compileJava.dependsOn swaggerSources.server.code, spotlessApply, setGitVersion | ||
|
|
||
| // Internal dependencies | ||
| // Note The Open API schema depends on an external library - cloud-resource-schema, so need to unzip it first. | ||
| swaggerSources.server.code.dependsOn tasks.unzipCloudResourceSchema | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what about the swagger client tasks?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This took longer to investigate and fix than I want to admit, but you're right that swagger client tasks didn't depend on this. Should be fixed now |
||
| swaggerSources.server.ui.dependsOn tasks.unzipCloudResourceSchema | ||
| compileJava.dependsOn swaggerSources.server.code, spotlessApply, setGitVersion | ||
| clean.dependsOn cleanSetGitVersion | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
111 changes: 111 additions & 0 deletions
111
src/main/java/bio/terra/workspace/app/controller/DataRepoReferenceController.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| package bio.terra.workspace.app.controller; | ||
|
|
||
| import bio.terra.workspace.app.controller.common.ReferenceController; | ||
| import bio.terra.workspace.common.utils.ControllerTranslationUtils; | ||
| import bio.terra.workspace.generated.controller.DataRepoReferenceApi; | ||
| import bio.terra.workspace.generated.model.CreateDataRepoSnapshotReferenceRequestBody; | ||
| import bio.terra.workspace.generated.model.DataRepoSnapshotReference; | ||
| import bio.terra.workspace.generated.model.UpdateDataReferenceRequestBody; | ||
| import bio.terra.workspace.service.datareference.DataReferenceService; | ||
| import bio.terra.workspace.service.datareference.model.DataReference; | ||
| import bio.terra.workspace.service.datareference.model.DataReferenceType; | ||
| import bio.terra.workspace.service.datareference.model.SnapshotReference; | ||
| import bio.terra.workspace.service.datareference.utils.DataReferenceValidationUtils; | ||
| import bio.terra.workspace.service.iam.AuthenticatedUserRequest; | ||
| import bio.terra.workspace.service.iam.AuthenticatedUserRequestFactory; | ||
| import java.util.UUID; | ||
| import javax.servlet.http.HttpServletRequest; | ||
| import javax.validation.Valid; | ||
| import org.springframework.beans.factory.annotation.Autowired; | ||
| import org.springframework.http.HttpStatus; | ||
| import org.springframework.http.ResponseEntity; | ||
| import org.springframework.stereotype.Controller; | ||
| import org.springframework.web.bind.annotation.PathVariable; | ||
| import org.springframework.web.bind.annotation.RequestBody; | ||
|
|
||
| @Controller | ||
| public class DataRepoReferenceController extends ReferenceController | ||
| implements DataRepoReferenceApi { | ||
|
|
||
| private final DataReferenceValidationUtils dataReferenceValidation; | ||
| private final DataReferenceService dataReferenceService; | ||
| private final AuthenticatedUserRequestFactory authenticatedUserRequestFactory; | ||
| private final HttpServletRequest request; | ||
|
|
||
| @Autowired | ||
| public DataRepoReferenceController( | ||
| DataReferenceValidationUtils dataReferenceValidation, | ||
| DataReferenceService dataReferenceService, | ||
| AuthenticatedUserRequestFactory authenticatedUserRequestFactory, | ||
| HttpServletRequest request) { | ||
| this.dataReferenceValidation = dataReferenceValidation; | ||
| this.dataReferenceService = dataReferenceService; | ||
| this.authenticatedUserRequestFactory = authenticatedUserRequestFactory; | ||
| this.request = request; | ||
| } | ||
|
|
||
| private AuthenticatedUserRequest getAuthenticatedInfo() { | ||
| return authenticatedUserRequestFactory.from(request); | ||
| } | ||
|
|
||
| @Override | ||
| public ResponseEntity<DataRepoSnapshotReference> createDataRepoSnapshotReference( | ||
| UUID id, @Valid CreateDataRepoSnapshotReferenceRequestBody body) { | ||
| SnapshotReference referenceObject = | ||
| SnapshotReference.create( | ||
| body.getSnapshot().getInstanceName(), body.getSnapshot().getSnapshot()); | ||
| DataReference reference = | ||
| createDataReference( | ||
| id, | ||
| body.getMetadata(), | ||
| DataReferenceType.DATA_REPO_SNAPSHOT, | ||
| referenceObject, | ||
| getAuthenticatedInfo(), | ||
| dataReferenceValidation, | ||
| dataReferenceService); | ||
|
|
||
| DataRepoSnapshotReference response = | ||
| new DataRepoSnapshotReference() | ||
| .metadata(ControllerTranslationUtils.metadataFromDataReference(reference)) | ||
| .snapshot(((SnapshotReference) reference.referenceObject()).toApiModel()); | ||
| return new ResponseEntity<>(response, HttpStatus.OK); | ||
| } | ||
|
|
||
| @Override | ||
| public ResponseEntity<DataRepoSnapshotReference> getDataRepoSnapshotReference( | ||
| UUID id, UUID referenceId) { | ||
| DataReference ref = getReference(id, referenceId, getAuthenticatedInfo(), dataReferenceService); | ||
|
|
||
| DataRepoSnapshotReference response = | ||
| new DataRepoSnapshotReference() | ||
| .snapshot(((SnapshotReference) ref.referenceObject()).toApiModel()) | ||
| .metadata(ControllerTranslationUtils.metadataFromDataReference(ref)); | ||
| return new ResponseEntity<>(response, HttpStatus.OK); | ||
| } | ||
|
|
||
| @Override | ||
| public ResponseEntity<DataRepoSnapshotReference> getDataRepoSnapshotReferenceByName( | ||
| UUID id, String name) { | ||
| DataReference ref = | ||
| getReferenceByName( | ||
| id, | ||
| DataReferenceType.DATA_REPO_SNAPSHOT, | ||
| name, | ||
| getAuthenticatedInfo(), | ||
| dataReferenceService); | ||
| DataRepoSnapshotReference response = | ||
| new DataRepoSnapshotReference() | ||
| .snapshot(((SnapshotReference) ref.referenceObject()).toApiModel()) | ||
| .metadata(ControllerTranslationUtils.metadataFromDataReference(ref)); | ||
| return new ResponseEntity<>(response, HttpStatus.OK); | ||
| } | ||
|
|
||
| @Override | ||
| public ResponseEntity<Void> updateDataRepoSnapshotReference( | ||
| @PathVariable("id") UUID id, | ||
| @PathVariable("referenceId") UUID referenceId, | ||
| @RequestBody UpdateDataReferenceRequestBody body) { | ||
| updateReference(id, referenceId, body, getAuthenticatedInfo(), dataReferenceService); | ||
| return new ResponseEntity<>(HttpStatus.NO_CONTENT); | ||
| } | ||
| } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also from #221 this needs to include |
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to unpack this into the source tree?
Is an alternative to copy our resources into the build tree?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice to deploy our yaml file into a build dir so we could use relative (non-upward) paths.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I unpacked it into the source tree so that our swagger-ui (also in the source dir) can parse references to it. otherwise, those schemas won't show up in swagger-ui, which breaks example values.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can leave it for now. I'll make a ticket.
Another approach might be to move all of the parts into a build subdirectory and do the codegen from there.
I'll make a ticket.