Update the queue-granules task to fetch collection configs for each granule from S3 #284

marchuffnagle · 2018-03-30T12:30:42Z

Summary: Update the queue-granules task to fetch collection configs for each granule from S3

Addresses CUMULUS-450: Restore functionality to get correct collection for IngestGranule

Changes

Updated the queue-granules task to fetch collection configs for each granule from S3 instead of relying on the single collection config specified in the event config.
Updated the parse-pdr task to use the correct granuleIdExtraction in cases where the granules in a PDR don't all belong to the same collection.
Removed the deprecated findTmpTestDataDirectory function from@cumulus/common/test-utils.
Updated the sftp mixin to export sftpMixin instead of using a default export. This matches the mixins for the other protocols.
Refactored the parsePdr() function.

Test Plan

Things that should succeed before merging.

Unit tests
Have LPDAAC run a workflow using these updates
Update CHANGELOG
Run ./bin/eslint-ratchet and verify that eslint errors have not increased
- Commit .eslint-ratchet-high-water-mark if the score has improved

This makes it match the other protocol mixins.

laurenfrederick · 2018-03-30T13:01:54Z

CHANGELOG.md

@@ -6,6 +6,23 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

 ## [Unreleased]

+### Changed
+- The **queue-granules** task no longer takes a "collection" in its config.


Add the ticket number to these notes please. Also this would be a breaking change, correct?

Maybe we should put all these changes in bullets under one line item and call out what changes will need to be made to the workflow configuration to support this.

laurenfrederick · 2018-03-30T13:04:04Z

packages/ingest/parse-pdr.js

+  // get all the file specs in each group
+  const specs = fileGroup.objects('FILE_SPEC');
+  // FIXME This is a very generic error
+  if (specs.length === 0) throw new Error();


Probably should at least put a message in the error?

And test the error somehow

laurenfrederick · 2018-03-30T13:05:41Z

packages/ingest/parse-pdr.js


-    if (!dataType) throw new PDRParsingError('DATA_TYPE is missing');
+  const files = specs.map(parseSpec.bind(null, pdrName));
+  const granuleId = extractGranuleId(files[0].name, granuleIdExtraction);


Is it possible files would be empty here? Is that something we should check for?

laurenfrederick · 2018-03-30T13:06:05Z

packages/ingest/parse-pdr.js


-  // Get all the file groups
-  const fileGroups = parsed.objects('FILE_GROUP');
+function granuleFromFileGroup(fileGroup, granuleIdExtraction, pdrName) {


This function needs docs.

laurenfrederick · 2018-03-30T13:07:48Z

packages/ingest/sftp.js

 const Crypto = require('./crypto').DefaultProvider;
+const log = require('@cumulus/common/log');


Shouldn't this be const { log } = require('@cumulus/common')'?

laurenfrederick · 2018-03-30T13:10:35Z

tasks/queue-granules/index.js

+  return [dataType, JSON.parse(collectionConfig)];
+}
+
+async function fetchCollectionConfigs(stackName, Bucket, dataTypes) { // eslint-disable-line require-jsdoc, max-len


Document this function

laurenfrederick · 2018-03-30T13:11:50Z

tasks/queue-granules/tests/index.js

    recursivelyDeleteS3Bucket(t.context.templateBucket),
    sqs().deleteQueue({ QueueUrl: t.context.event.config.queueUrl }).promise()
  ]);
 });

+function uploadCollectionConfig(testContext, dataType, collectionConfig) {


laurenfrederick · 2018-03-30T13:25:47Z

packages/ingest/parse-pdr.js

-  // Get all the file groups
-  const fileGroups = parsed.objects('FILE_GROUP');
+function granuleFromFileGroup(fileGroup, granuleIdExtraction, pdrName) {
+  if (!fileGroup.get('DATA_TYPE')) throw new PDRParsingError('DATA_TYPE is missing');


Can we test this error?

jennyhliu

The workflow needs to be updated.
https://github.com/cumulus-nasa/cumulus-integration-tests/blob/master/workflows.yml
Also, in the integration test, it would be better to have a PDR which contains granules from different datatypes.

laurenfrederick · 2018-04-02T12:39:52Z

packages/ingest/parse-pdr.js


-    dataType = dataType.value;
+  const files = specs.map(parseSpec.bind(null, pdrName));
+  const granuleId = extractGranuleId(files[0].name, granuleIdExtraction);


Should we throw an error on empty files?

I've created CUMULUS-507 to handle that.

…are present in PDR

marchuffnagle · 2018-04-05T15:17:49Z

packages/api/models/collections.js

-    await S3.put(process.env.internal, key, JSON.stringify(item));
+    const collectionConfigStore =
+      new CollectionConfigStore(process.env.internal, process.env.stackName);
+    await collectionConfigStore.put(item.name, item);


We now have one central piece of code that manages the storage and retrieval of collection configs in S3.

Marc Huffnagle added 7 commits March 29, 2018 12:42

Refactored parse-pdr

54a9a1a

Update sftp mixin to remove default export

0b91ef4

This makes it match the other protocol mixins.

Remove unused findTmpTestDataDirectory function from test-utils

ac3504a

Add a simple parse-pdr test

1363fed

Merge branch 'master' into CUMULUS-450

cca6efc

CUMULUS-450 Fix incorrect collection configs in queue-granules

8994bad

Updated changelog

70c75fb

marchuffnagle requested review from scisco, sethvincent, kkelly51, abarciauskas-bgse, laurenfrederick and jennyhliu and removed request for scisco March 30, 2018 12:47

marchuffnagle changed the title ~~WIP CUMULUS-450~~ CUMULUS-450 Mar 30, 2018

marchuffnagle changed the title ~~CUMULUS-450~~ Update the queue-granules task to fetch collection configs for each granule from S3 Mar 30, 2018

Ratchet-down eslint score

a56240e

laurenfrederick suggested changes Mar 30, 2018

View reviewed changes

Marc Huffnagle added 3 commits March 30, 2018 10:31

Updates based on PR feedback

fdbd682

Added a reminder to document the granuleFromFileGroup function

e4d9842

Ratchet-down eslint score

ab21eb8

jennyhliu reviewed Mar 30, 2018

View reviewed changes

laurenfrederick reviewed Apr 2, 2018

View reviewed changes

laurenfrederick approved these changes Apr 2, 2018

View reviewed changes

Marc Huffnagle added 5 commits April 2, 2018 16:00

Create 1.2.1-alpha.1 release of ingest and queue-granules

267681a

Merge branch 'master' of https://github.com/cumulus-nasa/cumulus

2e68f00

Merge branch 'master' into CUMULUS-450

731c205

ParsePdr task uses correct granuleExtractionId if multiple dataTypes …

d49a7ad

…are present in PDR

Merge branch 'master' into CUMULUS-450

86a9180

marchuffnagle commented Apr 5, 2018

View reviewed changes

Marc Huffnagle added 14 commits April 5, 2018 11:30

Add comments to collection-config-store

74449e9

Revert index reorganization

5aa0aaa

Add comments to collection-config-store tests

ef39309

Revert import order changes

23c3cd3

Updated changelog

27b9844

Revert PDR import order changes

299a4ea

Revert SFTP import order changes

d758e44

Revert queue-granules array order

bcb3798

Update eslint ratchet score

b0c67d5

Fix e2e tests

d599326

Updates to support cumulus integration tests

1ef5f0b

Updates to get integration tests working

7e90148

Merge branch 'master' into CUMULUS-450

ac8d9e1

Merge branch 'master' into CUMULUS-450

8dcaecd

marchuffnagle merged commit a11e4fd into master Apr 9, 2018

marchuffnagle deleted the CUMULUS-450 branch April 9, 2018 19:57

botanical mentioned this pull request Jun 30, 2023

[Snyk] Security upgrade tough-cookie from 4.0.0 to 4.1.3 #3420

Closed

botanical mentioned this pull request May 20, 2024

[Snyk] Upgrade tough-cookie from 4.0.0 to 4.1.4 #3661

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the queue-granules task to fetch collection configs for each granule from S3 #284

Update the queue-granules task to fetch collection configs for each granule from S3 #284

marchuffnagle commented Mar 30, 2018 •

edited

Loading

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

laurenfrederick Mar 30, 2018

jennyhliu left a comment

laurenfrederick Apr 2, 2018

marchuffnagle Apr 2, 2018

marchuffnagle Apr 5, 2018

		const Crypto = require('./crypto').DefaultProvider;
		const log = require('@cumulus/common/log');

Update the queue-granules task to fetch collection configs for each granule from S3 #284

Update the queue-granules task to fetch collection configs for each granule from S3 #284

Conversation

marchuffnagle commented Mar 30, 2018 • edited Loading

Changes

Test Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennyhliu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marchuffnagle commented Mar 30, 2018 •

edited

Loading