Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: resume standardise validate TDE-710 #444

Merged
merged 14 commits into from
Apr 18, 2023

Conversation

paulfouquet
Copy link
Collaborator

@paulfouquet paulfouquet commented Apr 12, 2023

Allows standardise_validate.py to be resumed using the same target (output directory) after a failure in order to avoid re-process the entire source.

Changes

  • ⚠️ Takes a target directory/s3 path as argument where the output files (TIFFS and STAC metadata .json) will be saved
  • verify that the target exists to enter in the "resume" mode
  • if "resume" mode, both STAC and TIFF files existance are check:
    • if STAC exists, the TIFF is considered to exists too and the source file will be ignored (not processed)
    • if STAC does not exists but TIFF exists, the TIFF will be validated (Non Visual QA) and its STAC metadata created

NOTE: Draft PR to modify the standardising Argo Workflow: linz/topo-workflows#123

Tests

Test Argo Workflows: https://argo.linzaccess.com/workflows/argo/test-imagery-standardising-resume-gt8wr?tab=workflow

scripts/files/fs.py Outdated Show resolved Hide resolved
@paulfouquet paulfouquet changed the title feat: resume standardise validate TDE-710 feat!: resume standardise validate TDE-710 Apr 13, 2023
@paulfouquet paulfouquet marked this pull request as ready for review April 13, 2023 22:40
@paulfouquet paulfouquet requested a review from a team as a code owner April 13, 2023 22:40
@kodiakhq kodiakhq bot merged commit b4022a5 into master Apr 18, 2023
@kodiakhq kodiakhq bot deleted the feat/resume-standardise-validate-tde-710 branch April 18, 2023 23:49
@github-actions github-actions bot mentioned this pull request Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants