CUMULUS-359 kinesis consumer users sf-scheduler #262
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…' into abarciauskas-bgse_CUMULUS-359-kinesis-consumer-uses-sfscheduler
…uskas-bgse_CUMULUS-359-kinesis-consumer-uses-sfscheduler
| collection: { name: 'test-collection' }, | ||
| payload: { test: 'test payload' } | ||
| }; | ||
| test('the queue receives a correctly formatted workflow message', async(t) => { |
There was a problem hiding this comment.
Since we removed queueWorkflowMessage this is being repurposed to test enqueueGranuleIngestMessage
laurenfrederick
approved these changes
Mar 19, 2018
| )); | ||
| )) | ||
| .catch((err) => { | ||
| console.log('Caught error in process record:'); |
There was a problem hiding this comment.
Instead of console.log, this should use the logger i.e. log.error
packages/api/lambdas/sf-scheduler.js
Outdated
| * @param {function} cb lambda callback | ||
| * @param {Object} event - lambda input message | ||
| * @param {Object} context - lambda context | ||
| * @param {function} cb - lambda callback |
There was a problem hiding this comment.
Can you add a returns to this one please?
CHANGELOG.md
Outdated
| - Pass encrypted password to the ApiGranule Lambda function [CUMULUS-424] | ||
| - `@cumulus/api`: `kinesis-consumer.js` uses `sf-scheduler.js#schedule` instead of placing a message directly on the `startSF` SQS queue. This is a fix for [CUMULUS-359](https://bugs.earthdata.nasa.gov/browse/CUMULUS-359) because `sf-scheduler.js#schedule` looks up the provider and collection data in DynamoDB and adds it to the `meta` object of the enqueued message payload. |
There was a problem hiding this comment.
This should be in the unreleased section
scisco
deleted the
abarciauskas-bgse_CUMULUS-359-kinesis-consumer-uses-sfscheduler
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Kinesis consumer users sf-scheduler
Kinesis Consumer's invocation of Rule.buildPayload uses the values of provider and collection in the the kinesis rule. The kinesis rule values for collection and provider are in the payload sent to sfScheduler#schedule. So the provider (the string id) and collection object (name and version object), that are stored in the rule are used to look up the provider and collection objects stored in dynamo. The full dynamodb data is what is set as the cumulus message's
$.meta.providerand$.meta.collection. This is important because downstream tasks, such as sync granule, require the provider and collection data.Addresses CUMULUS-359
Detailed Changes
packages/api/lambdasTest Plan
Things that should succeed before merging.
Created a stream, provider, collection and kinesis rule for HelloWorldWorkflow. Important that the kinesis rule has correct provider and collection information. Put a record on the stream for the collection. Verified hello world workflow had started and the payload includes the collection and provider data in the meta and the entire kinesis record in the payload.
./bin/eslint-ratchetand verify that eslint errors have not increased.eslint-ratchet-high-water-markif the score has improved