Release 16 with master #3432

npauzenga · 2023-07-13T17:11:04Z

Summary: Summary of changes

Addresses CUMULUS-XX: Develop amazing new feature

Changes

Detailed list or prose of changes
...

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

* Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role

[CUMULUS-3226] Remove refs to async operations dynamo table to release branch (#3331)

* CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors

…horization header (merge to master) (#3350) (#3352) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341)

* Update endpoints/granules :granuleName parameter to :granuleId (#3333) * Move PUT logic to PATCH. Implement error for PUT endpoint * Add put units/refactor * Update API client to use PATCH protocol cumulus's api is moving all existing PUT requests to PATCH. This commit updates the api-client to use those, adds 'PATCH' to typings and updates the test fixtures for the thin unit tests. * Fix broken API unit * Update CHANGELOG * Fix naming/minor refactor * Fix Dynamodb granule.files default bug * Fix elasticsearch default array write value * Update/add PUT endpoint and relevant tests * Update CHANGELOG * Update unit test to take advantage of fixed unit helper * Update API version * Add api-client method to use updated PUT endpoint * Add fixed annotations to CHANGELOG * Update API version unit * Minor comment refactor * Add PUT API spec test * Fix incorrect test label * Add missing unit tests * Add removed unit tests * Jk/cumulus 3072 add header check (#3257) * Add API version compliance checks to app middleware * Update middleware/add test defaults/fix typing * Fix broken unit tests * cleanup * Add middleware units * Add endpoint tests to validate middleware * Fix unit test * Remove unneeded type exports * Remove unneeded test * Remove async * Minor formatting fix/changeset reduction * Add missing ts-check * Update CHANGELOG * Fix bad import * Update 'version' header to 'Cumulus-API-Version' * Respond to PR feedback * validateApiCompliance -> requireApiVersion * Update packages/api/lib/request.js Co-authored-by: Marc <[email protected]> * Fix isMinVersionApi expression * Update granule units/fix prior test fixtures * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * Fix spacing * Add api-client required headers * Update header restriction with new name, fix api-package header inclusions * Fix broken unit * Bring in fix from #3270 * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * isNumber -> Number.isFinite --------- Co-authored-by: Marc <[email protected]> * Update PUT to use collectionId endpoint *only* * Update CHANGELOG * Fix api-client to call right PUT endpoint * Remove unneeded log output * Update tests, add granuleId parameter updates * Fix bad logic assertion * Update endpoints/granules :granuleName parameter to :granuleId * Fix lint 🔔 * Fix lint * Fix merged unit test * Fix merged comments --------- Co-authored-by: Marc <[email protected]> * Reorder/updated CHANGELOG * Minor P3 release edits --------- Co-authored-by: Marc <[email protected]>

* [CUMULUS-3226] Remove refs to async operations dynamo table (#3331) * Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role * Update endpoints/granules :granuleName parameter to :granuleId (#3333) * Move PUT logic to PATCH. Implement error for PUT endpoint * Add put units/refactor * Update API client to use PATCH protocol cumulus's api is moving all existing PUT requests to PATCH. This commit updates the api-client to use those, adds 'PATCH' to typings and updates the test fixtures for the thin unit tests. * Fix broken API unit * Update CHANGELOG * Fix naming/minor refactor * Fix Dynamodb granule.files default bug * Fix elasticsearch default array write value * Update/add PUT endpoint and relevant tests * Update CHANGELOG * Update unit test to take advantage of fixed unit helper * Update API version * Add api-client method to use updated PUT endpoint * Add fixed annotations to CHANGELOG * Update API version unit * Minor comment refactor * Add PUT API spec test * Fix incorrect test label * Add missing unit tests * Add removed unit tests * Jk/cumulus 3072 add header check (#3257) * Add API version compliance checks to app middleware * Update middleware/add test defaults/fix typing * Fix broken unit tests * cleanup * Add middleware units * Add endpoint tests to validate middleware * Fix unit test * Remove unneeded type exports * Remove unneeded test * Remove async * Minor formatting fix/changeset reduction * Add missing ts-check * Update CHANGELOG * Fix bad import * Update 'version' header to 'Cumulus-API-Version' * Respond to PR feedback * validateApiCompliance -> requireApiVersion * Update packages/api/lib/request.js Co-authored-by: Marc <[email protected]> * Fix isMinVersionApi expression * Update granule units/fix prior test fixtures * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * Fix spacing * Add api-client required headers * Update header restriction with new name, fix api-package header inclusions * Fix broken unit * Bring in fix from #3270 * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * isNumber -> Number.isFinite --------- Co-authored-by: Marc <[email protected]> * Update PUT to use collectionId endpoint *only* * Update CHANGELOG * Fix api-client to call right PUT endpoint * Remove unneeded log output * Update tests, add granuleId parameter updates * Fix bad logic assertion * Update endpoints/granules :granuleName parameter to :granuleId * Fix lint 🔔 * Fix lint * Fix merged unit test * Fix merged comments --------- Co-authored-by: Marc <[email protected]> * CUMULUS-3053: removing references to dynamodb from docs (#3322) * CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (merge to master) (#3350) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341) * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * fixing CHANGELOG.md * removing docs + updating changelog * changing doc link --------- Co-authored-by: Nate Pauzenga <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Marc <[email protected]> Co-authored-by: nasamoduyebo <[email protected]> Co-authored-by: jennyhliu <[email protected]>

* CUMULUS-3243:Updated granule delete logic (#3338) * CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB * add unit tests * delete files not in master * update CHANGELOG * remove dynamodb from test

* update docs * Version up to v16.0.0 * Minor edits/update release doc * Add PI release version notes * Minor CL edit * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Address PR feedback --------- Co-authored-by: Nate Pauzenga <[email protected]>

* update docs * update docs * add new doc to sidebars * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> --------- Co-authored-by: Jonathan Kovarik <[email protected]>

* Backport CUMULUS-3307/PR #3386 * Fixup

* CUMULUS-3223:Fix failed granule stuck in queued * skip sqsQueueExists test * update getGranuleTemporalInfo * update test match schema * update getGranuleTemporalInfo * remove extra await * remove skip sqsQueueExists * update `@cumulus/cumulus-message-adapter-js` to `2.0.5` * update sfEventSqsToDbRecords to return partial batch failure * fix typo * handle process error seperately, multiple message test * update test to process multiple messages

* CUMULUS-3135 - Update integration test scripts to fail on test timeout (#3401) * Update integration test scripts to fail on test timeout * Fixup * Fixup * Update script interpreter for test runs * Fix script * Fixup * Fixup * Update timeout pass/fail conditional * Jk/cumulus 3135 fix integration tests (#3403) * Update integration test scripts to fail on test timeout * Fixup * Fixup * Update script interpreter for test runs * Fix script * Fixup * Fixup * Update api/client and integration test usage of it to fix test failures * Fix formatting/lint/etc * Update test/minor fix --------- Co-authored-by: etcart <[email protected]> --------- Co-authored-by: etcart <[email protected]>

* bump to 16.0.1-alpha.0 * resolve conflicts --------- Co-authored-by: Jonathan Kovarik <[email protected]>

* add missing method for erasing PG tables * Remove duplicate declaration * Remove duplicate declaration * Take serve.js and serveUtils.js from main * Update schema endpoint to utilize P3 api/lib schema --------- Co-authored-by: Jonathan Kovarik <[email protected]>

* version bump * update changelog version

Jkovarik · 2023-07-14T15:58:07Z

CHANGELOG.md

+When updating directly to v16 from prior releases older that V15, please make sure to
+read through all prior release notes.
+
+Notable migration concerns since the last PI release version (11.1.x):


Wasn't the last PI release version v 14?

Did we maybe mean "minimum supported version"?

Jkovarik · 2023-07-14T15:58:25Z

CHANGELOG.md

+- [v13.1.0] - Postgres update to add `files_granules_cumulus_id_index` to the
+  `files` table may require manual steps depending on load.
+
+#### RDS Phase 3 migration notes


This doesn't look right, shouldn't this have

#### RDS Phase 3 migration notes This release includes updates that remove existing DynamoDB tables as part of release deployment process. This release *cannot* be properly rolled back in production as redeploying a prior version of Cumulus will not recover the associated Dynamo tables. Please read the full change log for RDS Phase 3 and consult the [RDS Phase 3 update documentation](https://nasa.github.io/cumulus/docs/next/upgrade-notes/upgrade-rds-phase-3-release) #### API Endpoint Versioning As part of the work on CUMULUS-3072, we have added a required header for the granule PUT/PATCH endpoints -- to ensure that older clients/utilities do not unexpectedly make destructive use of those endpoints, a validation check of a header value against supported versions has been implemented. Moving forward, if a breaking change is made to an existing endpoint that requires user updates, as part of that update we will set the current version of the core API and require a header that confirms the client is compatible with the version required or greater. In this instance, the granule PUT/PATCH endpoints will require a `Cumulus-API-Version` value of at least `2`. ```bash curl --request PUT https://example.com/granules/granuleId.A19990103.006.1000\ --header 'Cumulus-API-Version': '2'\ --header 'Authorization: Bearer ReplaceWithToken'\ --data ...

Users/clients that do not make use of these endpoints will not be impacted.

RDS Phase 3

It looks like it's in the branch, but not in this merge diff. 🤔

Yeah, this was a mess. Check out the latest diff. I think everything is in the right spot.

CHANGELOG.md

Jkovarik

Looks good to me, thanks for updating the changelog

etcart · 2023-07-14T16:38:16Z

packages/api/endpoints/schemas.js

we should add something like
/** @deprecated use accessToken in /api/lib/schemas.js */
in api/models/schemas.js as we move away from using it (I realize it was deprecated before this PR)

Release 16 with master (nasa#3432)

npauzenga and others added 20 commits April 19, 2023 15:56

[CUMULUS-3226] Remove refs to async operations dynamo table (#3331)

5b57c6c

* Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role

Merge pull request #3340 from nasa/feature/CUMULUS-3226-to-16

51583ab

[CUMULUS-3226] Remove refs to async operations dynamo table to release branch (#3331)

CUMULUS-3053: removing references to dynamodb from docs (#3322)

086409d

* CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors

CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer aut…

4b47985

…horization header (merge to master) (#3350) (#3352) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341)

Fix lint issues (#3392)

d030d0b

Jk/cumulus 3307 (#3394)

0dee3c0

* Backport CUMULUS-3307/PR #3386 * Fixup

Re-issue v16 docs (#3400)

3176bd4

v16.0.1 alpha release for testing (#3409)

1c0e43b

* bump to 16.0.1-alpha.0 * resolve conflicts --------- Co-authored-by: Jonathan Kovarik <[email protected]>

Update serve.js to match main

81e3572

bump version

215833a

Release 16.0.0 (#3428)

f73d94f

* version bump * update changelog version

Merge branch 'master' into release-16-with-master

29de08d

npauzenga added the work in progress label Jul 13, 2023

Merge branch 'master' into release-16-with-master

0635244

npauzenga added needs review and removed work in progress labels Jul 14, 2023

Jkovarik added in review and removed needs review labels Jul 14, 2023

Jkovarik reviewed Jul 14, 2023

View reviewed changes

npauzenga added 2 commits July 14, 2023 12:09

changelog fixup

def9298

changelog fixup

2b6b9f8

Jkovarik reviewed Jul 14, 2023

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

Jkovarik approved these changes Jul 14, 2023

View reviewed changes

etcart reviewed Jul 14, 2023

View reviewed changes

npauzenga added 2 commits July 14, 2023 12:58

update cumulus versions in new lambda

96a8bf1

bump cumulus versions

74264db

npauzenga merged commit 49f3b08 into master Jul 14, 2023

npauzenga deleted the release-16-with-master branch July 14, 2023 19:32

1proprogrammerchant added a commit to 1proprogrammerchant/cumulus that referenced this pull request Jul 15, 2023

Merge pull request #2 from nasa/master

6e7bb04

Release 16 with master (nasa#3432)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 16 with master #3432

Release 16 with master #3432

npauzenga commented Jul 13, 2023

Jkovarik Jul 14, 2023

npauzenga Jul 14, 2023

Jkovarik Jul 14, 2023 •

edited

Loading

Jkovarik Jul 14, 2023

npauzenga Jul 14, 2023

Jkovarik left a comment

etcart Jul 14, 2023

Release 16 with master #3432

Release 16 with master #3432

Conversation

npauzenga commented Jul 13, 2023

Changes

PR Checklist

Jkovarik Jul 14, 2023

Choose a reason for hiding this comment

npauzenga Jul 14, 2023

Choose a reason for hiding this comment

Jkovarik Jul 14, 2023 • edited Loading

Choose a reason for hiding this comment

RDS Phase 3

Jkovarik Jul 14, 2023

Choose a reason for hiding this comment

npauzenga Jul 14, 2023

Choose a reason for hiding this comment

Jkovarik left a comment

Choose a reason for hiding this comment

etcart Jul 14, 2023

Choose a reason for hiding this comment

Jkovarik Jul 14, 2023 •

edited

Loading