[CUMULUS-3172] Update data integrity/migration docs #3387
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Also note that this is targeting the |
Jkovarik
reviewed
May 23, 2023
Jkovarik
reviewed
May 23, 2023
Jkovarik
reviewed
May 23, 2023
Jkovarik
reviewed
May 23, 2023
Jkovarik
requested changes
May 23, 2023
Co-authored-by: Jonathan Kovarik <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
Co-authored-by: Jonathan Kovarik <[email protected]>
| 1. Re-review the issues described above | ||
| - GHRC was able to scope the affected granules to specific collections, which makes the recovery process much easier. This may not be an option for all DAACs. | ||
|
|
||
| 2. If you have not ingested granules or performed partial granule updates on affected Cumulus versions (questions 1 and 2 on the survey), no action is required. You may update to the latest version of Cumulus. |
There was a problem hiding this comment.
I was going to comment above when you referenced the versions that had the fix for issue 1 that Eddie thought it would make more sense to tell them which releases were affected that they may have been doing ingest on. But since you reference the survey questions here (which listed affected versions), I think that works.
There was a problem hiding this comment.
Yeah I actually did consider more detail here around the impacted versions in the issue descriptions. I'm hesitant because I don't think these docs are really intended to introduce the issues and provide guidance for someone that hasn't been tracking the conversation. Referring back to earlier steps (like the survey) reinforces that I think.
npauzenga
added a commit
that referenced
this pull request
May 25, 2023
* update docs * update docs * add new doc to sidebars * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> --------- Co-authored-by: Jonathan Kovarik <[email protected]>
npauzenga
added a commit
that referenced
this pull request
Jul 14, 2023
* [CUMULUS-3226] Remove refs to async operations dynamo table (#3331) * Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role * CUMULUS-3053: removing references to dynamodb from docs (#3322) * CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (merge to master) (#3350) (#3352) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341) * Jk rel16/cumulus 3079 (#3357) * Update endpoints/granules :granuleName parameter to :granuleId (#3333) * Move PUT logic to PATCH. Implement error for PUT endpoint * Add put units/refactor * Update API client to use PATCH protocol cumulus's api is moving all existing PUT requests to PATCH. This commit updates the api-client to use those, adds 'PATCH' to typings and updates the test fixtures for the thin unit tests. * Fix broken API unit * Update CHANGELOG * Fix naming/minor refactor * Fix Dynamodb granule.files default bug * Fix elasticsearch default array write value * Update/add PUT endpoint and relevant tests * Update CHANGELOG * Update unit test to take advantage of fixed unit helper * Update API version * Add api-client method to use updated PUT endpoint * Add fixed annotations to CHANGELOG * Update API version unit * Minor comment refactor * Add PUT API spec test * Fix incorrect test label * Add missing unit tests * Add removed unit tests * Jk/cumulus 3072 add header check (#3257) * Add API version compliance checks to app middleware * Update middleware/add test defaults/fix typing * Fix broken unit tests * cleanup * Add middleware units * Add endpoint tests to validate middleware * Fix unit test * Remove unneeded type exports * Remove unneeded test * Remove async * Minor formatting fix/changeset reduction * Add missing ts-check * Update CHANGELOG * Fix bad import * Update 'version' header to 'Cumulus-API-Version' * Respond to PR feedback * validateApiCompliance -> requireApiVersion * Update packages/api/lib/request.js Co-authored-by: Marc <[email protected]> * Fix isMinVersionApi expression * Update granule units/fix prior test fixtures * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * Fix spacing * Add api-client required headers * Update header restriction with new name, fix api-package header inclusions * Fix broken unit * Bring in fix from #3270 * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * isNumber -> Number.isFinite --------- Co-authored-by: Marc <[email protected]> * Update PUT to use collectionId endpoint *only* * Update CHANGELOG * Fix api-client to call right PUT endpoint * Remove unneeded log output * Update tests, add granuleId parameter updates * Fix bad logic assertion * Update endpoints/granules :granuleName parameter to :granuleId * Fix lint 🔔 * Fix lint * Fix merged unit test * Fix merged comments --------- Co-authored-by: Marc <[email protected]> * Reorder/updated CHANGELOG * Minor P3 release edits --------- Co-authored-by: Marc <[email protected]> * Cumulus 3120 bugfix (#3364) * [CUMULUS-3226] Remove refs to async operations dynamo table (#3331) * Remove refs to async operations dynamo table * add changelog entry * Remove additional async operation table refs * Update process_dead_letter_archive role * Update endpoints/granules :granuleName parameter to :granuleId (#3333) * Move PUT logic to PATCH. Implement error for PUT endpoint * Add put units/refactor * Update API client to use PATCH protocol cumulus's api is moving all existing PUT requests to PATCH. This commit updates the api-client to use those, adds 'PATCH' to typings and updates the test fixtures for the thin unit tests. * Fix broken API unit * Update CHANGELOG * Fix naming/minor refactor * Fix Dynamodb granule.files default bug * Fix elasticsearch default array write value * Update/add PUT endpoint and relevant tests * Update CHANGELOG * Update unit test to take advantage of fixed unit helper * Update API version * Add api-client method to use updated PUT endpoint * Add fixed annotations to CHANGELOG * Update API version unit * Minor comment refactor * Add PUT API spec test * Fix incorrect test label * Add missing unit tests * Add removed unit tests * Jk/cumulus 3072 add header check (#3257) * Add API version compliance checks to app middleware * Update middleware/add test defaults/fix typing * Fix broken unit tests * cleanup * Add middleware units * Add endpoint tests to validate middleware * Fix unit test * Remove unneeded type exports * Remove unneeded test * Remove async * Minor formatting fix/changeset reduction * Add missing ts-check * Update CHANGELOG * Fix bad import * Update 'version' header to 'Cumulus-API-Version' * Respond to PR feedback * validateApiCompliance -> requireApiVersion * Update packages/api/lib/request.js Co-authored-by: Marc <[email protected]> * Fix isMinVersionApi expression * Update granule units/fix prior test fixtures * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * Fix spacing * Add api-client required headers * Update header restriction with new name, fix api-package header inclusions * Fix broken unit * Bring in fix from #3270 * Update packages/api/app/middleware.js Co-authored-by: Marc <[email protected]> * isNumber -> Number.isFinite --------- Co-authored-by: Marc <[email protected]> * Update PUT to use collectionId endpoint *only* * Update CHANGELOG * Fix api-client to call right PUT endpoint * Remove unneeded log output * Update tests, add granuleId parameter updates * Fix bad logic assertion * Update endpoints/granules :granuleName parameter to :granuleId * Fix lint 🔔 * Fix lint * Fix merged unit test * Fix merged comments --------- Co-authored-by: Marc <[email protected]> * CUMULUS-3053: removing references to dynamodb from docs (#3322) * CUMULUS-3053: removing references to dynamodb from docs * resolving lint errors * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (merge to master) (#3350) * CUMULUS-3285: Updated isAuthBearTokenRequest to handle non-Bearer authorization header (#3341) * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * CUMULUS-3121/3120 (#3360) * CUMULUS-3121/3120 v15.1.0 backport (#3346) * backport PR * finalizing docs and changing remaining groups * PR feedback + testing changes * PR feedback * reverting change * Revert "update changelog" This reverts commit ae4627c. * reverting changes * PR feedback * removing EgressLambda from doc * Update xml2js 0.4.22->0.5 strict (#3330) (#3339) * Update xml2js 0.4.22->0.5 strict * Address GHSA-776f-qx25-q3cc/update allow list * Update CHANGELOG * Update package pins to 0.5.0 for xmljs Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback * PR feedback * PR feedback * fixing documentation linting * PR feedback * PR feedback * PR feedback * adding variables to tf-modules/workflow * PR feedback * PR feedback * reverting previous change --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * PR feedback (docs+changelog) * fixing docs * undoing commit * PR feedback --------- Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> * fixing CHANGELOG.md * removing docs + updating changelog * changing doc link --------- Co-authored-by: Nate Pauzenga <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Marc <[email protected]> Co-authored-by: nasamoduyebo <[email protected]> Co-authored-by: jennyhliu <[email protected]> * CUMULUS-3243:Updated granule delete logic (#3338) (#3366) (#3367) * CUMULUS-3243:Updated granule delete logic (#3338) * CUMULUS-3243:Updated granule delete logic to delete granule which is not in DynamoDB * add unit tests * delete files not in master * update CHANGELOG * remove dynamodb from test * Release 16.0.0 (#3376) * update docs * Version up to v16.0.0 * Minor edits/update release doc * Add PI release version notes * Minor CL edit * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Update CHANGELOG.md Co-authored-by: Nate Pauzenga <[email protected]> * Address PR feedback --------- Co-authored-by: Nate Pauzenga <[email protected]> * [CUMULUS-3172] Update data integrity/migration docs (#3387) * update docs * update docs * add new doc to sidebars * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> * Update docs/upgrade-notes/rds-phase-3-data-migration-guidance.md Co-authored-by: Jonathan Kovarik <[email protected]> --------- Co-authored-by: Jonathan Kovarik <[email protected]> * Fix lint issues (#3392) * Jk/cumulus 3307 (#3394) * Backport CUMULUS-3307/PR #3386 * Fixup * Re-issue v16 docs (#3400) * CUMULUS-3223: Fix failed granule stuck in queued (#3373) (#3402) * CUMULUS-3223:Fix failed granule stuck in queued * skip sqsQueueExists test * update getGranuleTemporalInfo * update test match schema * update getGranuleTemporalInfo * remove extra await * remove skip sqsQueueExists * update `@cumulus/cumulus-message-adapter-js` to `2.0.5` * update sfEventSqsToDbRecords to return partial batch failure * fix typo * handle process error seperately, multiple message test * update test to process multiple messages * Jk/cumulus 3315 (#3407) * CUMULUS-3135 - Update integration test scripts to fail on test timeout (#3401) * Update integration test scripts to fail on test timeout * Fixup * Fixup * Update script interpreter for test runs * Fix script * Fixup * Fixup * Update timeout pass/fail conditional * Jk/cumulus 3135 fix integration tests (#3403) * Update integration test scripts to fail on test timeout * Fixup * Fixup * Update script interpreter for test runs * Fix script * Fixup * Fixup * Update api/client and integration test usage of it to fix test failures * Fix formatting/lint/etc * Update test/minor fix --------- Co-authored-by: etcart <[email protected]> --------- Co-authored-by: etcart <[email protected]> * v16.0.1 alpha release for testing (#3409) * bump to 16.0.1-alpha.0 * resolve conflicts --------- Co-authored-by: Jonathan Kovarik <[email protected]> * Update serve.js to match main * bump version * Add missing method for erasing PG tables (#3419) * add missing method for erasing PG tables * Remove duplicate declaration * Remove duplicate declaration * Take serve.js and serveUtils.js from main * Update schema endpoint to utilize P3 api/lib schema --------- Co-authored-by: Jonathan Kovarik <[email protected]> * Release 16.0.0 (#3428) * version bump * update changelog version * changelog fixup * changelog fixup * update cumulus versions in new lambda * bump cumulus versions --------- Co-authored-by: nasamoduyebo <[email protected]> Co-authored-by: jennyhliu <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]> Co-authored-by: Marc <[email protected]> Co-authored-by: Naga Nages <[email protected]> Co-authored-by: etcart <[email protected]> Co-authored-by: Jonathan Kovarik <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the combination of the granule write issue descriptions, the recommended actions for updating to version 16, and additional context about the data-migration2 Lambda.