[CUMULUS-3368]: [User Contribution] Introducing allFilesPresent metadata field for Cumulus Collections. Fixes Issue #3193 #3492
Conversation
botanical
changed the title
| @@ -48,6 +48,11 @@ Users/clients that do not make use of these endpoints will not be impacted. | |||
| - Added optional `maxDownloadTime` field to `provider` schema | |||
| - Added `max_download_time` column to PostgreSQL `providers` table | |||
| - Updated `@cumulus/ingest/lock` to check expired locks based on `provider.maxDownloadTime` | |||
| - **CUMULUS-3368** | |||
| - Added `allFilesPresent` field for `collection.meta` that defines whether or not all files | |||
There was a problem hiding this comment.
Since allFilesPresent is used in our task, suggest to add it to https://github.com/nasa/cumulus/blob/master/packages/api/lib/schemas.js#L211
There was a problem hiding this comment.
Also, please add the following to collection and granule mapping:
https://github.com/nasa/cumulus/blob/master/packages/es-client/config/mappings.json
"meta": {
"type": "object",
"dynamic": false
},
There is a ticket CUMULUS-3417 about mapping issue, and it can happen for collection record as well
| @@ -48,6 +48,10 @@ A list of buckets with types that will be used to assign bucket targets based on | |||
|
|
|||
| A Cumulus [collection](https://github.com/nasa/cumulus/blob/master/packages/api/lib/schemas.js) object. Used to define granule file groupings and granule metadata for discovered files. The collection object utilizes the collection type key to generate types in the output object on discovery. | |||
|
|
|||
| ##### Collection Meta | |||
|
|
|||
| If the collection is configured with `collection.meta.allFilesPresent` set to `true`, the task will remove granules's missing files. Otherwise, the default behavior ignores missing files in granules. | |||
There was a problem hiding this comment.
| If the collection is configured with `collection.meta.allFilesPresent` set to `true`, the task will remove granules's missing files. Otherwise, the default behavior ignores missing files in granules. | |
| If the collection is configured with `collection.meta.allFilesPresent` set to `true`, the task will remove granules with all the files. Otherwise, by default, task continue to process granules without all the files. |
remove granules's missing files is confusing
There was a problem hiding this comment.
The wording above is still confusing. Do you mean something like the following?
If the collection is configured with
collection.meta.allFilesPresentset totrue, the task will include only granules where all files specified in thecollection.fileslist are present, thus excluding granules where at least one file is absent. Otherwise, by default (i.e.,collection.meta.allFilesPresentdefaults tofalse), the task includes every granule, whether or not all of its files are present.
| * @returns {Boolean} - returns true if all file in collection config are present for granuleId | ||
| */ | ||
| const checkGranuleHasAllFiles = (config, filesByGranuleId, granuleId) => { | ||
| const granuleHasAllFiles = filesByGranuleId[granuleId].length >= config.collection.files.length; |
There was a problem hiding this comment.
If collection.files[].regex matchs multiple granule files, how do we make sure each regex has at least one granule file? The assertion is weak.
There was a problem hiding this comment.
A granule can have 10 files but only 5 collection.files configuration.
tasks/discover-granules/index.js
Outdated
| @@ -314,6 +370,16 @@ const discoverGranules = async ({ config }) => { | |||
| concurrency: get(config, 'concurrency', 3), | |||
| }); | |||
|
|
|||
| let allFilesPresent = false; | |||
There was a problem hiding this comment.
Suggest to move line 373-376 to handleGranulesWithoutAllFilesPresent function
There was a problem hiding this comment.
Summary: Summary of changes
Addresses CUMULUS-3368: Introducing allFilesPresent metadata field for Cumulus Collections.
Changes
Change allows Cumulus Collections to enforce that all files specified in Collections list to be present before ingesting Granules.
PR Checklist