-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUMULUS-3243:Updated granule delete logic #3338
Conversation
@@ -20,7 +20,7 @@ cp ./*.py ./dist/ | |||
|
|||
cd ./dist || exit 1 | |||
|
|||
find . -type f -print0 | xargs -0 node ../../../../bin/zip.js lambda.zip $(ls | grep -v lambda.zip) | |||
node ../../../../bin/zip.js lambda.zip $(ls | grep -v lambda.zip) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change is already in master branch
@@ -482,6 +483,19 @@ class Granule extends Manager { | |||
} | |||
return executionDescription; | |||
} | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When granule is not in dynamoDB, unpublish has error:
2023-04-18T15:46:33.756Z 489c96c9-fec7-411c-84b4-eac0df058c41 ERROR
{ "error": { "name": "RecordDoesNotExist", "message": "No record found for {"granuleId":"MOD09GQ.A7614271.S4rfYb.006.4748017559580"} in jl-tf-GranulesTable", "stack": [ "RecordDoesNotExist: No record found for {"granuleId":"MOD09GQ.A7614271.S4rfYb.006.4748017559580"} in jl-tf-GranulesTable", " at Object.get (/var/task/webpack:/src/DynamoDb.ts:63:9)", " at processTicksAndRejections (node:internal/process/task_queues:96:5)", " at Granule.get (/var/task/webpack:/models/base.js:228:12)", " at Granule.get (/var/task/webpack:/models/granules.js:88:29)", " at Granule.update (/var/task/webpack:/models/base.js:372:25)", " at /var/task/webpack:/lib/granule-remove-from-cmr.js:95:29" ] }, "level": "error", "message": "No record found for {"granuleId":"MOD09GQ.A7614271.S4rfYb.006.4748017559580"} in jl-tf-GranulesTable", "sender": "@cumulus/api", "timestamp": "2023-04-18T15:46:33.756Z" }
@@ -145,7 +147,7 @@ const deleteGranuleAndFiles = async (params: { | |||
deletedFiles: files, | |||
}; | |||
} catch (error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When granule is not in dynamoDB, delete has error:
{ "error": { "name": "TypeError", "message": "Cannot read properties of undefined (reading 'files')", "stack": [ "TypeError: Cannot read properties of undefined (reading 'files')", " at deleteGranuleAndFiles (/var/task/webpack:/granule-delete.ts:148:119)", " at processTicksAndRejections (node:internal/process/task_queues:96:5)", " at del (/var/task/webpack:/endpoints/granules.js:551:27)" ] }, "level": "error", "message": "Cannot read properties of undefined (reading 'files')", "sender": "@cumulus/api", "timestamp": "2023-04-18T20:35:10.277Z" }
@@ -29,15 +28,18 @@ const models = require('../models'); | |||
const _removeGranuleFromCmr = async (granule, collectionId) => { | |||
log.info(`granules.removeGranuleFromCmrByGranule granule_id: ${granule.granule_id}, colletion_id: ${collectionId}`); | |||
if (!granule.published || !granule.cmr_link) { | |||
throw new GranuleNotPublished(`Granule ${granule.granule_id} in Collection ${collectionId} is not published to CMR, so cannot be removed from CMR`); | |||
log.warn(`Granule ${granule.granule_id} in Collection ${collectionId} is not published to CMR, so cannot be removed from CMR`); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🙌
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good
* CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB * add unit tests
* CUMULUS-3243:Updated granule delete logic (#3338) * CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB * add unit tests * delete files not in master * update CHANGELOG * remove dynamodb from test
* [CUMULUS-3226] Remove refs to async operations dynamo table (#3331) * Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role * CUMULUS-3053: removing references to dynamodb from docs (#3322) * CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (merge to master) (#3350) (#3352) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341) * Jk rel16/cumulus 3079 (#3357) * Update endpoints/granules :granuleName parameter to :granuleId (#3333) * Move PUT logic to PATCH. Implement error for PUT endpoint * Add put units/refactor * Update API client to use PATCH protocol cumulus's api is moving all existing PUT requests to PATCH. This commit updates the api-client to use those, adds 'PATCH' to typings and updates the test fixtures for the thin unit tests. * Fix broken API unit * Update CHANGELOG * Fix naming/minor refactor * Fix Dynamodb granule.files default bug * Fix elasticsearch default array write value * Update/add PUT endpoint and relevant tests * Update CHANGELOG * Update unit test to take advantage of fixed unit helper * Update API version * Add api-client method to use updated PUT endpoint * Add fixed annotations to CHANGELOG * Update API version unit * Minor comment refactor * Add PUT API spec test * Fix incorrect test label * Add missing unit tests * Add removed unit tests * Jk/cumulus 3072 add header check (#3257) * Add API version compliance checks to app middleware * Update middleware/add test defaults/fix typing * Fix broken unit tests * cleanup * Add middleware units * Add endpoint tests to validate middleware * Fix unit test * Remove unneeded type exports * Remove unneeded test * Remove async * Minor formatting fix/changeset reduction * Add missing ts-check * Update CHANGELOG * Fix bad import * Update 'version' header to 'Cumulus-API-Version' * Respond to PR feedback * validateApiCompliance -> requireApiVersion * Update packages/api/lib/request.js Co-authored-by: Marc <[email protected]> * Fix isMinVersionApi expression * Update granule units/fix prior test fixtures * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * Fix spacing * Add api-client required headers * Update header restriction with new name, fix api-package header inclusions * Fix broken unit * Bring in fix from #3270 * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * isNumber -> Number.isFinite --------- Co-authored-by: Marc <[email protected]> * Update PUT to use collectionId endpoint *only* * Update CHANGELOG * Fix api-client to call right PUT endpoint * Remove unneeded log output * Update tests, add granuleId parameter updates * Fix bad logic assertion * Update endpoints/granules :granuleName parameter to :granuleId * Fix lint 🔔 * Fix lint * Fix merged unit test * Fix merged comments --------- Co-authored-by: Marc <[email protected]> * Reorder/updated CHANGELOG * Minor P3 release edits --------- Co-authored-by: Marc <[email protected]> * Cumulus 3120 bugfix (#3364) * [CUMULUS-3226] Remove refs to async operations dynamo table (#3331) * Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role * Update endpoints/granules :granuleName parameter to :granuleId (#3333) * Move PUT logic to PATCH. Implement error for PUT endpoint * Add put units/refactor * Update API client to use PATCH protocol cumulus's api is moving all existing PUT requests to PATCH. This commit updates the api-client to use those, adds 'PATCH' to typings and updates the test fixtures for the thin unit tests. * Fix broken API unit * Update CHANGELOG * Fix naming/minor refactor * Fix Dynamodb granule.files default bug * Fix elasticsearch default array write value * Update/add PUT endpoint and relevant tests * Update CHANGELOG * Update unit test to take advantage of fixed unit helper * Update API version * Add api-client method to use updated PUT endpoint * Add fixed annotations to CHANGELOG * Update API version unit * Minor comment refactor * Add PUT API spec test * Fix incorrect test label * Add missing unit tests * Add removed unit tests * Jk/cumulus 3072 add header check (#3257) * Add API version compliance checks to app middleware * Update middleware/add test defaults/fix typing * Fix broken unit tests * cleanup * Add middleware units * Add endpoint tests to validate middleware * Fix unit test * Remove unneeded type exports * Remove unneeded test * Remove async * Minor formatting fix/changeset reduction * Add missing ts-check * Update CHANGELOG * Fix bad import * Update 'version' header to 'Cumulus-API-Version' * Respond to PR feedback * validateApiCompliance -> requireApiVersion * Update packages/api/lib/request.js Co-authored-by: Marc <[email protected]> * Fix isMinVersionApi expression * Update granule units/fix prior test fixtures * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * Fix spacing * Add api-client required headers * Update header restriction with new name, fix api-package header inclusions * Fix broken unit * Bring in fix from #3270 * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * isNumber -> Number.isFinite --------- Co-authored-by: Marc <[email protected]> * Update PUT to use collectionId endpoint *only* * Update CHANGELOG * Fix api-client to call right PUT endpoint * Remove unneeded log output * Update tests, add granuleId parameter updates * Fix bad logic assertion * Update endpoints/granules :granuleName parameter to :granuleId * Fix lint 🔔 * Fix lint * Fix merged unit test * Fix merged comments --------- Co-authored-by: Marc <[email protected]> * CUMULUS-3053: removing references to dynamodb from docs (#3322) * CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (merge to master) (#3350) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341) * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * fixing CHANGELOG.md * removing docs + updating changelog * changing doc link --------- Co-authored-by: Nate Pauzenga <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Marc <[email protected]> Co-authored-by: nasamoduyebo <[email protected]> Co-authored-by: jennyhliu <[email protected]> * CUMULUS-3243:Updated granule delete logic (#3338) (#3366) (#3367) * CUMULUS-3243:Updated granule delete logic (#3338) * CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB * add unit tests * delete files not in master * update CHANGELOG * remove dynamodb from test * Release 16.0.0 (#3376) * update docs * Version up to v16.0.0 * Minor edits/update release doc * Add PI release version notes * Minor CL edit * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Address PR feedback --------- Co-authored-by: Nate Pauzenga <[email protected]> * [CUMULUS-3172] Update data integrity/migration docs (#3387) * update docs * update docs * add new doc to sidebars * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> --------- Co-authored-by: Jonathan Kovarik <[email protected]> * Fix lint issues (#3392) * Jk/cumulus 3307 (#3394) * Backport CUMULUS-3307/PR #3386 * Fixup * Re-issue v16 docs (#3400) * CUMULUS-3223: Fix failed granule stuck in queued (#3373) (#3402) * CUMULUS-3223:Fix failed granule stuck in queued * skip sqsQueueExists test * update getGranuleTemporalInfo * update test match schema * update getGranuleTemporalInfo * remove extra await * remove skip sqsQueueExists * update `@cumulus/cumulus-message-adapter-js` to `2.0.5` * update sfEventSqsToDbRecords to return partial batch failure * fix typo * handle process error seperately, multiple message test * update test to process multiple messages * Jk/cumulus 3315 (#3407) * CUMULUS-3135 - Update integration test scripts to fail on test timeout (#3401) * Update integration test scripts to fail on test timeout * Fixup * Fixup * Update script interpreter for test runs * Fix script * Fixup * Fixup * Update timeout pass/fail conditional * Jk/cumulus 3135 fix integration tests (#3403) * Update integration test scripts to fail on test timeout * Fixup * Fixup * Update script interpreter for test runs * Fix script * Fixup * Fixup * Update api/client and integration test usage of it to fix test failures * Fix formatting/lint/etc * Update test/minor fix --------- Co-authored-by: etcart <[email protected]> --------- Co-authored-by: etcart <[email protected]> * v16.0.1 alpha release for testing (#3409) * bump to 16.0.1-alpha.0 * resolve conflicts --------- Co-authored-by: Jonathan Kovarik <[email protected]> * Update serve.js to match main * bump version * Add missing method for erasing PG tables (#3419) * add missing method for erasing PG tables * Remove duplicate declaration * Remove duplicate declaration * Take serve.js and serveUtils.js from main * Update schema endpoint to utilize P3 api/lib schema --------- Co-authored-by: Jonathan Kovarik <[email protected]> * Release 16.0.0 (#3428) * version bump * update changelog version * changelog fixup * changelog fixup * update cumulus versions in new lambda * bump cumulus versions --------- Co-authored-by: nasamoduyebo <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Marc <[email protected]> Co-authored-by: Naga Nages <[email protected]> Co-authored-by: etcart <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]>
* release v15.0.0 (#3303) * fix @aws-sdk/[email protected] error * Added a new send-pan task - more to be added to it * Added send-pan task in TF * Fix linter warnings * Fix PVL dependency issue In `cumulus/packages/pvl` the exports were located in `t.js` and `index.js` was empty. This was causing a dependency failure in `cumulus/packages/api/models/pdrs.js`, which uses `require('@cumulus/pvl')`. At the same time, `cumulus/tasks/parse-pdr/index.js` had been configured to use `require('@cumulus/pvl/t')` in order to get around the empty PVL `index.js` file. This commit moves `t.js` to `index.js` and updates the `require` in the parse-pdr task. * Cleanup ESLint warnings * Add support for generating and writing the PAN locally * Update `got` package version Downgrade the `got` package to 11.8.6. The 12.x version was converted to native ESM and no longer supports CommonJS `require`. * Updated to make pan->pdr replacement case insensitive * Removed unused uploader methods * Fixed pdr replacement so we replace only instances of .pdr * Updated send_pan_test to include validation of PAN message * Updated several references to `../t` in pvl tests * Added details about the send-pan-task to the CHANGELOG * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header * add integration test * update version to 15.0.1 * update changelog * Release 15.0.2 (#3349) * backport PR * finalizing docs and changing remaining groups * 15.1.0 release * doc updates * PR feedback + testing changes * Revert "doc updates" This reverts commit 5037012. * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * changing version number * small fixes * small fix --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3243:Updated granule delete logic (#3338) * CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB * add unit tests * update to v15.0.3 * update changelog * pvl move t.js to index.js * fix version * fix package version * remove tasks/send-pan/package-lock.json * fix lint * add pdrHelpers * add unit test code coverage * add s3 pan support refactor * add unit test refactor * refactor * add SendPan task to ParsePdr workflow * add schema move pdr to input * default pans location * fix lint and integration test * fix lint fix orca lambda version * fix unit test coverage * convert to typescript * convert to typescript * fix test coverage value * fix typing, refactor * fix unit test * fix unit test * pr feedback --------- Co-authored-by: Mike Dorfman <[email protected]> Co-authored-by: Jim Cole <[email protected]> Co-authored-by: colecu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Naga Nages <[email protected]>
Summary: Summary of changes
Addresses CUMULUS-3243: Unable to delete granule from Postgres when it is missing in DynamoDB
Changes
PR Checklist
Test
Unit tests are added. I also tested manually:
curl -X POST
https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "forceRemoveFromCmr": true}'