CUMULUS-2846: Fix rules PUT logic #2733

markdboyd · 2022-02-24T17:43:50Z

Summary: Summary of changes

Addresses CUMULUS-2846: Fix rules PUT logic

Changes

Separate rule logic to update rule triggers from logic to update record in DynamoDB
Update localstack
Update data-migration1 to support force re-migrating rules from DynamoDB to PG

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

…mo/ES & add unit tests for updating SNS rule

* use createAt for reconciliation report time range * update stack selection * compare cumulus granule with orca * add createBackupReconciliationReport * update initial report format * add ORCASearchCatalogQueue * add unit test for backup-reconciliation-report * update unit test * rearrange unit test * add unit test * add more test granules to unit test * add ORCA Backup report type * add orca reconciliation report integration test to spec * add and fix unit tests * fix unit test with regex * change report structure * refactor * revert back to updatedAt range for report * v4.0.0 is not available * update from PR feedback * fix integration test * remove startTimestamp for GNF report spec * fix changelog for rds2 branch * remove duplicate entries from changelog

packages/api/tests/models/rules/test-rules-model.js

packages/api/tests/models/rules/test-sns-rules.js

packages/db/src/models/base.ts

packages/db/src/translate/rules.ts

botanical · 2022-02-25T21:54:04Z

example/spec/parallel/kinesisTests/LambdaEventSourceSpec.js

@@ -149,7 +155,7 @@ describe('When adding multiple rules that share a kinesis event stream', () => {
          recordIdentifier,
          workflowArn,
          maxWaitForSFExistSecs,
-          2
+          1


Why did this value change?

This test is really only expecting 1 workflow to be started for the one enabled rule, but the value of 2 meant that the helper kept looking for another workflow until the timeout for waitForAllTestSfForRecord expired. So it was making the test take longer for no reason.

Oh 😯 Thank you for fixing this and for the explanation!

CHANGELOG.md

botanical

Pulled these changes and deployed the data-migration1 lambda and was able to verify that it will re-sync correctly. I left a couple comments, some around test cleanup, some of them are 🐜 nits. Great work here! ⭐

botanical

I re-reviewed and it looks good to me. Thanks for addressing my feedback!

* Addressing lodash vulnerability CVE-2021-23337 (#2730) * updating lodash vulnerability * Updated changelog Authored-by: Katherine Kelly <[email protected]> * Update changelog with recent maintenance branch releases (#2732) * Release v10.1.0 (#2731) merge back to master (#2734) * Release v10.1.0 (#2731) * Release v10.1.0 * Update Cumulus package API documentation * v10.1.0 Cumulus Documentation * update orca version to v4.0.1 * add the missing fix note to release * Adds logs surrounding the preparation and launch of asynchronous operations (#2726) * Adds logging surrounding the preparation and launch of asynchronous operations. * sets new docker image as default * Put proper grammar into your logs. * Also spell your logs correctly. * Update the actual async_operation_image_version I was confusing that with teh async_operation_image. * more fixing of changelog * really fix all grammar. * CUMULUS-2846: Fix rules PUT logic (#2733) * fix unit tests * update rule PUT endpoint logic to ensure consistent record in PG/Dynamo/ES & add unit tests for updating SNS rule * fix unit test * add unit test for SQS rule * add unit test for PUT for SQS rules * add unit tests * fix unit test * fix revert logic for PUT endpoint and add unit tests * add unit test for rules PG model * update rule.upsert() to accept fieldsToDelete parameter * CUMULUS-2703:Implement ORCA Reconciliation Reports (#2683) * use createAt for reconciliation report time range * update stack selection * compare cumulus granule with orca * add createBackupReconciliationReport * update initial report format * add ORCASearchCatalogQueue * add unit test for backup-reconciliation-report * update unit test * rearrange unit test * add unit test * add more test granules to unit test * add ORCA Backup report type * add orca reconciliation report integration test to spec * add and fix unit tests * fix unit test with regex * change report structure * refactor * revert back to updatedAt range for report * v4.0.0 is not available * update from PR feedback * fix integration test * remove startTimestamp for GNF report spec * fix changelog for rds2 branch * remove duplicate entries from changelog * add translateApiRuleToPostgresRuleRaw helper and unit test * fix PUT endpoint logic and update unit test * export new helper * update data-migration1 lambda to allow overriding rules in PG * modify rule update logic & update unit tests * remove unused code * clean up code * remove unnecessary code * add unit test * update unit tests * update CHANGELOG * remove deletion from rules model * add unit test for updateRuleTrigger(0 * update unit tests * refactoring test * delete mocking code * update localstack to 0.14.0 * fix unit test for SNS triggers * update unit tests * add unit tests for deleting SNS triggers * fixing unit tests * refactor unit tests * fix unit tests * fix unit tests * fix unit tests * update changelog * update detection of unit test bootstrapping * add dummy lambda zip * fix unit tests * fix unit tests * update logic for rule SNS permission ID * fix int test * revert ORCA changes * revert changes * revert changes * revert changes * fix unresolved merge conflicts * fix CHANGELOG * updates * update base model * fix unit tests * update unit tests * fix unit tests * revert schema change * fix unit test * fix unit test * fix unit test * fix unit tests * update tests * fix tests * fix test for local mocking * fix tests * fix unit tests * update test to use API * fix integration test helper * downgrade localstack to 0.11.5 * update unit tests * fix rules migration to overwrite and delete values * remove duplicate lines * remove unnecessary code * fix integration test * address feedback * address feedback * update test names * cleanup SNS topics * cleanup SNS topics * update CHANGELOG * fix integration test cleanup * fix unit tests Co-authored-by: jennyhliu <[email protected]> * Port CUMULUS-2845 to master (#2736) * Add environment variables to unit test * Update changelog, add createRuleTrigger before rulesModel.create calls * Add missing createRuleTrigger calls * update unit tests * update CHANGLEOG * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix CHANGELOG * revert rules model change * update unit test * fix unit test * fix unit tests * add unit tests Co-authored-by: Jennifer Tran <[email protected]> * CUMULUS-2854: Update queue-granules to correctly set granule createdAt stamp (#2863) Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * Update Cumulus version and package API documentation * Cut new version of Cumulus documentation * Update unreleased compare link in CHANGELOG.md * Release-10.1.1 - Address PR review comments * Add personal CI stack to Bamboo config * Update config.yml for personal CI stack * Update example/config.yml for personal sandbox stack Co-authored-by: kkelly51 <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Mark Boyd <[email protected]> Co-authored-by: Jennifer Tran <[email protected]>

* Release v10.1.0 (#2731) * Release v10.1.0 * Update Cumulus package API documentation * v10.1.0 Cumulus Documentation * update orca version to v4.0.1 * add the missing fix note to release * Release v10.1.1 (#2880) * Addressing lodash vulnerability CVE-2021-23337 (#2730) * updating lodash vulnerability * Updated changelog Authored-by: Katherine Kelly <[email protected]> * Update changelog with recent maintenance branch releases (#2732) * Release v10.1.0 (#2731) merge back to master (#2734) * Release v10.1.0 (#2731) * Release v10.1.0 * Update Cumulus package API documentation * v10.1.0 Cumulus Documentation * update orca version to v4.0.1 * add the missing fix note to release * Adds logs surrounding the preparation and launch of asynchronous operations (#2726) * Adds logging surrounding the preparation and launch of asynchronous operations. * sets new docker image as default * Put proper grammar into your logs. * Also spell your logs correctly. * Update the actual async_operation_image_version I was confusing that with teh async_operation_image. * more fixing of changelog * really fix all grammar. * CUMULUS-2846: Fix rules PUT logic (#2733) * fix unit tests * update rule PUT endpoint logic to ensure consistent record in PG/Dynamo/ES & add unit tests for updating SNS rule * fix unit test * add unit test for SQS rule * add unit test for PUT for SQS rules * add unit tests * fix unit test * fix revert logic for PUT endpoint and add unit tests * add unit test for rules PG model * update rule.upsert() to accept fieldsToDelete parameter * CUMULUS-2703:Implement ORCA Reconciliation Reports (#2683) * use createAt for reconciliation report time range * update stack selection * compare cumulus granule with orca * add createBackupReconciliationReport * update initial report format * add ORCASearchCatalogQueue * add unit test for backup-reconciliation-report * update unit test * rearrange unit test * add unit test * add more test granules to unit test * add ORCA Backup report type * add orca reconciliation report integration test to spec * add and fix unit tests * fix unit test with regex * change report structure * refactor * revert back to updatedAt range for report * v4.0.0 is not available * update from PR feedback * fix integration test * remove startTimestamp for GNF report spec * fix changelog for rds2 branch * remove duplicate entries from changelog * add translateApiRuleToPostgresRuleRaw helper and unit test * fix PUT endpoint logic and update unit test * export new helper * update data-migration1 lambda to allow overriding rules in PG * modify rule update logic & update unit tests * remove unused code * clean up code * remove unnecessary code * add unit test * update unit tests * update CHANGELOG * remove deletion from rules model * add unit test for updateRuleTrigger(0 * update unit tests * refactoring test * delete mocking code * update localstack to 0.14.0 * fix unit test for SNS triggers * update unit tests * add unit tests for deleting SNS triggers * fixing unit tests * refactor unit tests * fix unit tests * fix unit tests * fix unit tests * update changelog * update detection of unit test bootstrapping * add dummy lambda zip * fix unit tests * fix unit tests * update logic for rule SNS permission ID * fix int test * revert ORCA changes * revert changes * revert changes * revert changes * fix unresolved merge conflicts * fix CHANGELOG * updates * update base model * fix unit tests * update unit tests * fix unit tests * revert schema change * fix unit test * fix unit test * fix unit test * fix unit tests * update tests * fix tests * fix test for local mocking * fix tests * fix unit tests * update test to use API * fix integration test helper * downgrade localstack to 0.11.5 * update unit tests * fix rules migration to overwrite and delete values * remove duplicate lines * remove unnecessary code * fix integration test * address feedback * address feedback * update test names * cleanup SNS topics * cleanup SNS topics * update CHANGELOG * fix integration test cleanup * fix unit tests Co-authored-by: jennyhliu <[email protected]> * Port CUMULUS-2845 to master (#2736) * Add environment variables to unit test * Update changelog, add createRuleTrigger before rulesModel.create calls * Add missing createRuleTrigger calls * update unit tests * update CHANGLEOG * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix CHANGELOG * revert rules model change * update unit test * fix unit test * fix unit tests * add unit tests Co-authored-by: Jennifer Tran <[email protected]> * CUMULUS-2854: Update queue-granules to correctly set granule createdAt stamp (#2863) Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * Update Cumulus version and package API documentation * Cut new version of Cumulus documentation * Update unreleased compare link in CHANGELOG.md * Release-10.1.1 - Address PR review comments * Add personal CI stack to Bamboo config * Update config.yml for personal CI stack * Update example/config.yml for personal sandbox stack Co-authored-by: kkelly51 <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Mark Boyd <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> * Address PR review comment skip-integration-tests Co-authored-by: jennyhliu <[email protected]> Co-authored-by: kkelly51 <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Mark Boyd <[email protected]> Co-authored-by: Jennifer Tran <[email protected]>

* v10.0.0-beta.0 for orca testing (#2588) * improve waitForObjectExists logging * Fix lint * revert version bumps * Update process-s3-dead-letter-archive to unpack SQS events * Fix typing * Fix lint * Fix units * WIP resolve missing status/refactor * Update docker-compose-local to use dockerhub for httpd:alpine * Refactor/remove redundant case * Remove redundant tests, fix broken units * Update CHANGELOG/cleanup excess files * Need a deep copy of repo to compare changelog against main's (#2681) * Need a deep copy of repo to compare changelog against main's * updates changelog * Release 10.0.0 into 10.0.x (#2688) * Upgrade package versions to v10.0.0 * removes "only" from test-async_operations.js * adds website docs at v10.0.0 * Update all cumulus 9.9.0 packages to 10.0.0 * Mhs/cumulus bugfix/removes only from async test (#2691) * removes errant .only() from unit tests. * Update Deployment documentation with explicit note on finding old versions. * lint 🔔 * Mhs/increase timeout integration test failure (#2682) * Increase the time that getOnlineResourcesWithRetries waits. It was configured with a backoff of { retries: 60, maxTimeout: 1000 } This leads to 60 attempts in 60 seconds It is now { retries: 60, maxTimeout: 5000, factor: 1.05 } This leads to 60 attempts in 215.064 seconds (3.584 min.) I verified that the test did complete correctly and the CMR did not reflect the updated granule in the specified time. * update changelog * Update API lambda handlers to pull env vars from SM on startup * Update CHANGELOG * Update packages/api/tests/lambdas/test-process-s3-dead-letter-archive.js Co-authored-by: Mark Boyd <[email protected]> * Respond to PR feedback * Update packages/message/tests/test-StepFunctions.js Co-authored-by: Mark Boyd <[email protected]> * Update packages/message/tests/test-StepFunctions.js Co-authored-by: Mark Boyd <[email protected]> * Cumulus 2844: Fix migration lambda (#2696) * refactor deployment code * fix deployment code * update CHANGELOG * Update typescript to be more explicit * fixup * Add explicit message rejection on missing/expected Cumulus Message keys * Update CHANGELOG * Add CHANGELOG for 10.0.1 (#2698) * add CHANGELOG for 10.0.1 * update CHANGELOG * update CHANGELOG * Revert "Add explicit message rejection on missing/expected Cumulus Message keys" This reverts commit b0c39d9. * Respond to PR feedback/split api/app tests * Cleanup * Merge in user contribution #2693: "ab/add events permissions to steprole policy" (#2700) * Add events actions to step_policy for the step aws_am_role * Update CHANGELOG.md Co-authored-by: Aimee Barciauskas <[email protected]> * Throw error on secret_id missing * Address PR feedback * Update packages/api/app/index.js Co-authored-by: Mark Boyd <[email protected]> * Address PR feedback * Add units/update units per PR feedback * Update API env setup to use test flags * fixup * Use common modules 🔔 * Update packages/api/tests/app/test-app-env-vars-throws-on-missing-secret.js Co-authored-by: Mark Boyd <[email protected]> * Update packages/api/tests/app/test-app-env-var-throws-on-invalid-secret.js Co-authored-by: Mark Boyd <[email protected]> * Fix logic issue 🔔 * Remove unneeded context key * Fix units * cumulus 2833/update schema with correct titles (#2709) * Update provider titles to be what the Dashboard should display. * add changelog * Mhs/cumulus 2833/build bug fix (#2704) * clean up correctly * hacky way to ensure local build is complete after a bootstrap the package.sh build does not include dev dependencies, so after building the lambda zip file, run build again to build locally. * Here is a verison that moves the dev install into the package script * Pull database table names from s3, add logging * Update packages/api/app/index.js Co-authored-by: Jennifer Tran <[email protected]> * Remove unneeded logging * Update 2706 to use secrets manager * CUMULUS-2775: Default RDS cluster improvements (#2692) * update RDS cluster config to support custom DB paramter group * add connection logging * update logging * add isKnexDebugEnabled() and unit tests * update CHANGELOG * update CHANGELOG * remove cloudwatch log exports from RDS cluster definition * decrease minimum duration of queries for logging to 150 milliseconds * fix unit test * upgrade PG version to 10.14 * remove default parameters to min log query duration * Update CHANGELOG * Update unit tests * Lint * Merge CUMULUS-2840 (#2702) * db migration for files gcid index * changelog * fix lint issue * add default sort for file records * modify database model search logic * remove model sorting & fix unit tests * remove unit test Co-authored-by: Menno <[email protected]> * Adds clean scripts to packages and tasks that didn't have them. (#2715) * Update bootstrap/clean commands to drop/cleanup a .bootstrapSha file (#2713) * Mhs/cumulus 2492/handle trailing underscores in collection short name (#2723) * Refactor to use lastIndexOf * Add tests to show trailing underscores are correctly handled. * Remove split call in db/src/translate/executions.js * replaces fixture collectionIds with call to constructCollectionId() * removes deconstructCollectionId clone from api/lib/utils.js There was a second version of deconstructCollectionId that was being used imported from api/lib/utils.js This replaces all of those imports with an import to the canonical @cumulus/message/Collections version. * Adds spice by replacing some ___ with ____ in tests. Also calls constructCollectionId instead of assuming the separator. * Removes string created collectionIds with constructCollectionId calls. Also randoms the collectionIds * Removes strings created collectionIds with constructCollectionId calls. In test-normalizeEvent.js * Removes strings created collectionIds with constructCollectionId calls. cleans up test granule. * Various updates to replace string collectionIds with a function call. Mostly just replacing x___y with a call to constructCollectionId(x,y) Removed a few buildCollectionId functions with the canonical from @cumulus/message/Collections * Better words on Changelog * CUMULUS-2703:Implement ORCA Reconciliation Reports (#2683) * use createAt for reconciliation report time range * update stack selection * compare cumulus granule with orca * add createBackupReconciliationReport * update initial report format * add ORCASearchCatalogQueue * add unit test for backup-reconciliation-report * update unit test * rearrange unit test * add unit test * add more test granules to unit test * add ORCA Backup report type * add orca reconciliation report integration test to spec * add and fix unit tests * fix unit test with regex * change report structure * refactor * revert back to updatedAt range for report * v4.0.0 is not available * update from PR feedback * fix integration test * remove startTimestamp for GNF report spec * fix changelog for rds2 branch * remove duplicate entries from changelog * Port API auth config forward from #2724 (#2727) * Addressing lodash vulnerability CVE-2021-23337 (#2730) * updating lodash vulnerability * Updated changelog Authored-by: Katherine Kelly <[email protected]> * Update changelog with recent maintenance branch releases (#2732) * Release v10.1.0 (#2731) merge back to master (#2734) * Release v10.1.0 (#2731) * Release v10.1.0 * Update Cumulus package API documentation * v10.1.0 Cumulus Documentation * update orca version to v4.0.1 * add the missing fix note to release * Adds logs surrounding the preparation and launch of asynchronous operations (#2726) * Adds logging surrounding the preparation and launch of asynchronous operations. * sets new docker image as default * Put proper grammar into your logs. * Also spell your logs correctly. * Update the actual async_operation_image_version I was confusing that with teh async_operation_image. * more fixing of changelog * really fix all grammar. * CUMULUS-2846: Fix rules PUT logic (#2733) * fix unit tests * update rule PUT endpoint logic to ensure consistent record in PG/Dynamo/ES & add unit tests for updating SNS rule * fix unit test * add unit test for SQS rule * add unit test for PUT for SQS rules * add unit tests * fix unit test * fix revert logic for PUT endpoint and add unit tests * add unit test for rules PG model * update rule.upsert() to accept fieldsToDelete parameter * CUMULUS-2703:Implement ORCA Reconciliation Reports (#2683) * use createAt for reconciliation report time range * update stack selection * compare cumulus granule with orca * add createBackupReconciliationReport * update initial report format * add ORCASearchCatalogQueue * add unit test for backup-reconciliation-report * update unit test * rearrange unit test * add unit test * add more test granules to unit test * add ORCA Backup report type * add orca reconciliation report integration test to spec * add and fix unit tests * fix unit test with regex * change report structure * refactor * revert back to updatedAt range for report * v4.0.0 is not available * update from PR feedback * fix integration test * remove startTimestamp for GNF report spec * fix changelog for rds2 branch * remove duplicate entries from changelog * add translateApiRuleToPostgresRuleRaw helper and unit test * fix PUT endpoint logic and update unit test * export new helper * update data-migration1 lambda to allow overriding rules in PG * modify rule update logic & update unit tests * remove unused code * clean up code * remove unnecessary code * add unit test * update unit tests * update CHANGELOG * remove deletion from rules model * add unit test for updateRuleTrigger(0 * update unit tests * refactoring test * delete mocking code * update localstack to 0.14.0 * fix unit test for SNS triggers * update unit tests * add unit tests for deleting SNS triggers * fixing unit tests * refactor unit tests * fix unit tests * fix unit tests * fix unit tests * update changelog * update detection of unit test bootstrapping * add dummy lambda zip * fix unit tests * fix unit tests * update logic for rule SNS permission ID * fix int test * revert ORCA changes * revert changes * revert changes * revert changes * fix unresolved merge conflicts * fix CHANGELOG * updates * update base model * fix unit tests * update unit tests * fix unit tests * revert schema change * fix unit test * fix unit test * fix unit test * fix unit tests * update tests * fix tests * fix test for local mocking * fix tests * fix unit tests * update test to use API * fix integration test helper * downgrade localstack to 0.11.5 * update unit tests * fix rules migration to overwrite and delete values * remove duplicate lines * remove unnecessary code * fix integration test * address feedback * address feedback * update test names * cleanup SNS topics * cleanup SNS topics * update CHANGELOG * fix integration test cleanup * fix unit tests Co-authored-by: jennyhliu <[email protected]> * Port CUMULUS-2845 to master (#2736) * Add environment variables to unit test * Update changelog, add createRuleTrigger before rulesModel.create calls * Add missing createRuleTrigger calls * update unit tests * update CHANGLEOG * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix CHANGELOG * revert rules model change * update unit test * fix unit test * fix unit tests * add unit tests Co-authored-by: Jennifer Tran <[email protected]> * CUMULUS-2854: Update queue-granules to correctly set granule createdAt stamp (#2863) Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * Add knex package to async-operations * Remove unneeded test * Fix api.tf * Sort files in unit test * Fix unit test * Add variable to api.tf * PUT endpoints - Add unit tests to ensure fields can be deleted (#2872) * add PUT collections endpoint unit test * update unit tests * update unit tests * Fix local api and get execution response (#2877) * Updated localAPI to set additional env variable, and fixed GET /executions/status response * Update changelog, upgrade cumuluss/async-operation to version 40 * skip-integration-tests - Update fake provider host template (#2879) * Forward port of translatePostgresProviderToApiProvider to phase2 (#2882) * Forward port of translatePostgresProviderToApiProvider to phase2 Found in phase3 work. Moving fix to phase2 * Fix CHANGELOG * Update es-client package version ecs/async-operation uses, update image version to 41 * Apply suggestions from code review Co-authored-by: Mark Boyd <[email protected]> * Fix snsRuleSpec * Fix lint * Release v10.1.1 (#2880) merge back to master (#2884) * Release v10.1.0 (#2731) * Release v10.1.0 * Update Cumulus package API documentation * v10.1.0 Cumulus Documentation * update orca version to v4.0.1 * add the missing fix note to release * Release v10.1.1 (#2880) * Addressing lodash vulnerability CVE-2021-23337 (#2730) * updating lodash vulnerability * Updated changelog Authored-by: Katherine Kelly <[email protected]> * Update changelog with recent maintenance branch releases (#2732) * Release v10.1.0 (#2731) merge back to master (#2734) * Release v10.1.0 (#2731) * Release v10.1.0 * Update Cumulus package API documentation * v10.1.0 Cumulus Documentation * update orca version to v4.0.1 * add the missing fix note to release * Adds logs surrounding the preparation and launch of asynchronous operations (#2726) * Adds logging surrounding the preparation and launch of asynchronous operations. * sets new docker image as default * Put proper grammar into your logs. * Also spell your logs correctly. * Update the actual async_operation_image_version I was confusing that with teh async_operation_image. * more fixing of changelog * really fix all grammar. * CUMULUS-2846: Fix rules PUT logic (#2733) * fix unit tests * update rule PUT endpoint logic to ensure consistent record in PG/Dynamo/ES & add unit tests for updating SNS rule * fix unit test * add unit test for SQS rule * add unit test for PUT for SQS rules * add unit tests * fix unit test * fix revert logic for PUT endpoint and add unit tests * add unit test for rules PG model * update rule.upsert() to accept fieldsToDelete parameter * CUMULUS-2703:Implement ORCA Reconciliation Reports (#2683) * use createAt for reconciliation report time range * update stack selection * compare cumulus granule with orca * add createBackupReconciliationReport * update initial report format * add ORCASearchCatalogQueue * add unit test for backup-reconciliation-report * update unit test * rearrange unit test * add unit test * add more test granules to unit test * add ORCA Backup report type * add orca reconciliation report integration test to spec * add and fix unit tests * fix unit test with regex * change report structure * refactor * revert back to updatedAt range for report * v4.0.0 is not available * update from PR feedback * fix integration test * remove startTimestamp for GNF report spec * fix changelog for rds2 branch * remove duplicate entries from changelog * add translateApiRuleToPostgresRuleRaw helper and unit test * fix PUT endpoint logic and update unit test * export new helper * update data-migration1 lambda to allow overriding rules in PG * modify rule update logic & update unit tests * remove unused code * clean up code * remove unnecessary code * add unit test * update unit tests * update CHANGELOG * remove deletion from rules model * add unit test for updateRuleTrigger(0 * update unit tests * refactoring test * delete mocking code * update localstack to 0.14.0 * fix unit test for SNS triggers * update unit tests * add unit tests for deleting SNS triggers * fixing unit tests * refactor unit tests * fix unit tests * fix unit tests * fix unit tests * update changelog * update detection of unit test bootstrapping * add dummy lambda zip * fix unit tests * fix unit tests * update logic for rule SNS permission ID * fix int test * revert ORCA changes * revert changes * revert changes * revert changes * fix unresolved merge conflicts * fix CHANGELOG * updates * update base model * fix unit tests * update unit tests * fix unit tests * revert schema change * fix unit test * fix unit test * fix unit test * fix unit tests * update tests * fix tests * fix test for local mocking * fix tests * fix unit tests * update test to use API * fix integration test helper * downgrade localstack to 0.11.5 * update unit tests * fix rules migration to overwrite and delete values * remove duplicate lines * remove unnecessary code * fix integration test * address feedback * address feedback * update test names * cleanup SNS topics * cleanup SNS topics * update CHANGELOG * fix integration test cleanup * fix unit tests Co-authored-by: jennyhliu <[email protected]> * Port CUMULUS-2845 to master (#2736) * Add environment variables to unit test * Update changelog, add createRuleTrigger before rulesModel.create calls * Add missing createRuleTrigger calls * update unit tests * update CHANGLEOG * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix unit tests * fix CHANGELOG * revert rules model change * update unit test * fix unit test * fix unit tests * add unit tests Co-authored-by: Jennifer Tran <[email protected]> * CUMULUS-2854: Update queue-granules to correctly set granule createdAt stamp (#2863) Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * Update Cumulus version and package API documentation * Cut new version of Cumulus documentation * Update unreleased compare link in CHANGELOG.md * Release-10.1.1 - Address PR review comments * Add personal CI stack to Bamboo config * Update config.yml for personal CI stack * Update example/config.yml for personal sandbox stack Co-authored-by: kkelly51 <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Mark Boyd <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> * Address PR review comment skip-integration-tests Co-authored-by: jennyhliu <[email protected]> Co-authored-by: kkelly51 <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Mark Boyd <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> * [CUMULUS-2841] forward port of : add integration test (#2729) (#2870) * added iam:PassRole to ingest step policy (#2888) (#2889) Co-authored-by: Yen, David (398B-Affiliate) <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: yenes56 <[email protected]> Co-authored-by: Yen, David (398B-Affiliate) <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * Update postgres migration lambda timeout/make timeout configurable (#2887) * [RDS phase 2] CUMULUS-2846: Update unit tests (#2725) * refactor rules model and update unit tests * fix unit tests * update rule PUT endpoint logic to ensure consistent record in PG/Dynamo/ES & add unit tests for updating SNS rule * update unit test * fix unit test * add unit test for SQS rule * add unit test for PUT for SQS rules * add unit tests * fix unit test * fix revert logic for PUT endpoint and add unit tests * add unit test for rules PG model * update rule.upsert() to accept fieldsToDelete parameter * add translateApiRuleToPostgresRuleRaw helper and unit test * fix PUT endpoint logic and update unit test * export new helper * fix unit tests * update data-migration1 lambda to allow overriding rules in PG * remove test.only * modify rule update logic & update unit tests * remove unused code * clean up code * remove unnecessary code * add unit test * update unit tests * update CHANGELOG * remove deletion from rules model * add unit test for updateRuleTrigger(0 * update unit tests * refactoring test * delete mocking code * update localstack to 0.14.0 * fix unit test for SNS triggers * update unit tests * add unit tests for deleting SNS triggers * fixing unit tests * refactor unit tests * fix unit tests * fix unit tests * fix unit tests * update changelog * update detection of unit test bootstrapping * add dummy lambda zip * fix unit tests * fix unit tests * update logic for rule SNS permission ID * fix int test * update SNS permission handling & add unit test * move SNS add permissions logic * add unit tests for getSnsTriggerPermissionId * fix CHANGELOG * remove comment * fix unit tests * remove duplicate unit tests * update CHANGELOG * move tests * move tests * move tests * add logging * update test timeout * remove unnecessary linting script * revert bucket name chagne * remove unnecessary mocking * fix bad test logic * fix unit test * Remove unneeded dependencies * Address initial lint failure * Fix rule unit test * Fix/update unit test * Update integration test to be p2 compliant * fixup * Fixup * Update spec to utilize API instead of Dynamo model method * Minor refactor * Update example/spec/serial/ingestPdrWithNodeNameFtpSpec.js Co-authored-by: Jennifer Tran <[email protected]> * Update example/spec/serial/ingestPdrWithNodeNameFtpSpec.js Co-authored-by: Jennifer Tran <[email protected]> * Update p2 branch config/versions * Update api/ecs/async-operations package to latest core version * Update db package to use executions table for write constraint This commit updates the granule upsert logic to evaluate running/queued status writes for existance of an execution and it's status from the executions table instead of the linking table. This is due to the linking record not being written before the granule causing an out-of-order write edge case when the running *and* completed granule are being written at the same tiem. * Fix unit/remediate bug * Fix horrific lint failure 🔔 * Fixup * Update unit to use completed execution * Update comments/logging/test fixtures Updated the above to better represent the actual state of the system * Fix audit? * Fix migration unit test * Update spacing * Fix unit * Remove inadvertant update * Fix typo 🔔 * Refactor/remove negation on running status query * Update packages/db/tests/models/test-granule-model.js Co-authored-by: Jennifer Tran <[email protected]> * Allow 1067259/dev dependency "findings": [ { "version": "1.2.5", "paths": [ "coveralls>minimist", "@babel/core>json5>minimist", "copy-webpack-plugin>loader-utils>json5>minimist", "nyc>istanbul-lib-instrument>@babel/core>json5>minimist", "lerna>@lerna/bootstrap>@lerna/command>@lerna/child-process>strong-log-transformer>minimist", "ava>update-notifier>latest-version>package-json>registry-auth-token>rc>minimist", "lerna>@lerna/publish>@lerna/version>@lerna/conventional-commits>conventional-changelog-core>conventional-changelog-writer>handlebars>minimist", "lerna>@lerna/add>@lerna/bootstrap>@lerna/filter-options>@lerna/collect-updates>@lerna/describe-ref>@lerna/child-process>strong-log-transformer>minimist" ] } ], "metadata": null, "vulnerable_versions": "<=1.2.5", "module_name": "minimist", "severity": "high", "github_advisory_id": "GHSA-xvch-5gv4-984h", "cves": [ "CVE-2021-44906" ], "access": "public", "patched_versions": "<0.0.0", "cvss": { "score": 0, "vectorString": null }, "updated": "2022-03-18T23:13:41.000Z", "recommendation": "None", "cwe": [ "CWE-1321" ], "found_by": null, "deleted": null, "id": 1067259, "references": "- https://nvd.nist.gov/vuln/detail/CVE-2021-44906\n- https://github.com/substack/minimist/issues/164\n- https://github.com/substack/minimist/blob/master/index.js#L69\n- https://snyk.io/vuln/SNYK-JS-MINIMIST-559764\n- https://stackoverflow.com/questions/8588563/adding-custom-properties-to-a-function/20278068#20278068\n- GHSA-xvch-5gv4-984h", "created": "2022-03-21T19:00:43.693Z", "reported_by": null, "title": "Prototype Pollution in minimist", "npm_advisory_id": null, "overview": "Minimist <=1.2.5 is vulnerable to Prototype Pollution via file index.js, function setKey() (lines 69-95).", "url": "GHSA-xvch-5gv4-984h" } }, * Fix api client possible status codes * Update post processing spec to filter all executions * Update api-client unit tests * Address GHSA-cfm4-qjh2-4765 * Update CHANGELOG * Explicitly set node-forge requirement * Update whitelist with non-snyk vuln advisories * Fixup * Update post processing spec to correctly use endpoint limit * Remove useless check based on group feedback * Fix validation logic * Mhs/cumulus 2877/granule timestamps bugfix (#2898) * Fix API client, update test * CUMULUS-2860:Add optional config skipMetadataValidation to hyrax-metadata-updates task (#2892) * Release 10.1.2 version up/changelog -> main branch (#2895) * CUMULUS-1 Updates from snyk #2868 #2867 #2866 #2865 #2864 (#2881) * CUMULUS-2864: update ingestGranule not to perform separate validation request (#2897) * CUMULUS-2864:ingest granule does not perform separate validation request * fix unit test * remove .only from unit test, fix audit * remove the specific config from integration test * Merge Release 11.0.0 back to main branch (#2905) (#2908) Co-authored-by: Jennifer Tran <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> * Clean-up around test-rules.js with duplicate line and update-provider.js to remove validation against dynamo for a recently added test * CUMULUS-2849: Update dynamo to SDK V3 (#2902) * fix memoizer logic * update unit tests * refactor * fix bad merge * update CHANGLEOG * update Dynamo clients from v2 to v3 * fix bad merge * update base Dynamo model code * fix migration 2 code * update GranulesFileCache for compat with v3 & update unit tests * fix async operation and unit tests * fix state file package * fix dynamo client options for v3 * fix unit tests * fix more code and tests for v3 compat * add test for Semaphores lib * update errors handler * update tsconfig for sdk v3 compat * update tsconfig for sdk v3 compat * fix tsconfig * fix data-migration2 code for v3 compat * refactor * fix tsconfig * update CHANGELOG * work on fixing update logic * refactor base dynamo model update() function for compat with aws sdk v3 * add unit tests * fix update() logic and add unit test * fix code and unit tests * fix unit test * fix unit tests * remove underscore from allowlist * update version of simple-git * fix stack-locking script * fix stack locking script * fix typo * fix lock stack script * fix audit * fix credential handling for localstack * add unit tests for services running in localstack * fix audit issue * remove unnecessary change * update audit list * update audit allowlist * fix code for compat with v3 * update unit tests * fix TS issue * remove unnecessary test changes * update TS * change variable name * update dynamodbstreams client * update JSdocs * fix async operation code for v3 compat * publish alpha version of aws-client and update async operation ECS code * update async operation image version * update package references * add additinoal logging * minor refactor * Add documentation on choosing and configuring RDS (#2914) Co-authored-by: Jennifer Tran <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> * Updates to rules enpoints tests and fixes for collections tests * Update README for building & pushing async operation Docker image (#2916) Co-authored-by: Jennifer Tran <[email protected]> * CUMULUS-2787: Adjustment to rules post endpoint to check for pg exists rather than dynamo * CUMULUS-2703-2: Added cumulusFilesCount and orcaFilesCount to ORCA Backup reconciliation report (#2913) * CUMULUS-2703-2: Added `cumulusFilesCount` and `orcaFilesCount` to `ORCA Backup` reconciliation report * update orca report spec * update orca report unit tests * updates for test-rules.js. one test commented out temporarily. * Used testUtils.js -> createSqsQueues helper method to create sqs queues needed for sqs rule tests to pass * temporarily commenting out to get an opportunity to see what integration tests need to be looked at * replacing temporarily commented out unit tests with temporarily "skip"d unit tests * CUMULUS-2905 - Update migration lambda to take override/files only options (#2918) Co-authored-by: Nate Pauzenga <[email protected]> * [CUMULUS-2870] Update terraform to redeploy ECS service when lambda is modified (#2912) * WIP add force_new_deployment to aws_ecs_service * WIP set force_new_deployment to true for aws_ecs_service * Add data source for python processing worfklow, update image value * Add data source for python reference worfklow, update image value * Remove image digest additions * Update hello world terraform and leverage last_modified output, update changelog, remove old force_new_deployment addition * Fix changelog, remove old code * Add newline * Update changelog, update all tasks in ingest module to output last_modified_date * Fix discover_granules_task last_modified_date output * update ecs task cookbook Co-authored-by: Jennifer Tran <[email protected]> Co-authored-by: Jenny Liu <[email protected]> Co-authored-by: jennyhliu <[email protected]> * Updates to kinesis involved rules tests to ensure a valid Arn pattern is provided * updates to test-updateRuleResources.js to address updates to kinesis calls * fixes for merge issues in snsRuleSpec.js * adding minimist and node-fetch to audit-ci.json * Release 11.1.0 (#2926) (#2927) * updated package versions to 11.1.0 * updates to changelog * update to cumulus api documentation * updating documentation to v11.1.0 Co-authored-by: Anthony Ortega <[email protected]> Co-authored-by: Anthony Ortega <[email protected]> * removed unused variable * removing unecessary items from allowlist * 🐳 IP Sprint - Enable orca workflow/tests for everyone (#2662) * remove optional ORCA config * adding key_name to jao-ci-tf stack * CUMULUS-2920 -- Update Core npm build version to 8.6.4 (#2928) * CUMULUS-2925 -- Fix CI (#2936) * CUMULUS-2924 -- Update database proivision lambda acquire timeout default to 240 sec (#2934) * Continue exporting original translate rule function to support serve.js Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Menno <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Mark Boyd <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Matt Savoie <[email protected]> Co-authored-by: Aimee Barciauskas <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> Co-authored-by: Jennifer Tran <[email protected]> Co-authored-by: kkelly51 <[email protected]> Co-authored-by: Vanh <[email protected]> Co-authored-by: yenes56 <[email protected]> Co-authored-by: Yen, David (398B-Affiliate) <[email protected]> Co-authored-by: Anthony Ortega <[email protected]> Co-authored-by: Nate Pauzenga <[email protected]> Co-authored-by: Jenny Liu <[email protected]>

markdboyd and others added 30 commits February 23, 2022 17:16

fix unit tests

ab95ae3

update rule PUT endpoint logic to ensure consistent record in PG/Dyna…

5365c59

…mo/ES & add unit tests for updating SNS rule

fix unit test

14882c0

add unit test for SQS rule

bd6da37

add unit test for PUT for SQS rules

4e51be9

add unit tests

e302895

fix unit test

e9b11e6

fix revert logic for PUT endpoint and add unit tests

941c345

add unit test for rules PG model

0705820

update rule.upsert() to accept fieldsToDelete parameter

4428aca

add translateApiRuleToPostgresRuleRaw helper and unit test

a592214

fix PUT endpoint logic and update unit test

73fb721

export new helper

9c95557

update data-migration1 lambda to allow overriding rules in PG

e73e25c

modify rule update logic & update unit tests

a1f97b7

remove unused code

49ff19a

clean up code

6528b2d

remove unnecessary code

fa41e43

add unit test

baa2aea

update unit tests

eecb568

update CHANGELOG

34fb5f1

remove deletion from rules model

b697d27

add unit test for updateRuleTrigger(0

deeaba6

update unit tests

2c6a89a

refactoring test

f7c7ef4

delete mocking code

72288f8

update localstack to 0.14.0

aa203c8

fix unit test for SNS triggers

e60ea51

update unit tests

0e781a8