nasa · npauzenga · Jul 14, 2023 · Apr 19, 2023 · Apr 20, 2023 · Apr 21, 2023
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
+
+## [v16.0.0] 2023-05-09
+
+### MIGRATION notes
+
+#### PI release version
+
+When updating directly to v16 from prior releases older that V15, please make sure to
+read through all prior release notes.
+
+Notable migration concerns since the last PI release version (11.1.x):
+
+- [v14.1.0] - Postgres compatibility update to Aurora PostgreSQL 11.13.
+- [v13.1.0] - Postgres update to add `files_granules_cumulus_id_index` to the
+  `files` table may require manual steps depending on load.
+
+#### RDS Phase 3 migration notes
 - **CUMULUS-3188**
   - Updated QueueGranules to support queueing granules that meet the required API granule schema.
 
@@ -167,7 +184,6 @@ Users/clients that do not make use of these endpoints will not be impacted.
     - Add new endpoints to update and delete granules by collectionId as well as
       granuleId
 
-
 #### Removed
 
 - **CUMULUS-2994**
@@ -226,6 +242,11 @@ Users/clients that do not make use of these endpoints will not be impacted.
   - Added support for sha512 as checksumType for LZARDs backup task.
 
 ### Changed
+
+- **CUMULUS-3315**
+  - Updated `@cumulus/api-client/granules.bulkOperation` to remove `ids`
+    parameter in favor of `granules` parameter, in the form of a
+    `@cumulus/types/ApiGranule` that requires the following keys: `[granuleId, collectionId]`
 - **CUMULUS-3307**
   - Pinned cumulus dependency on `pg` to `v8.10.x`
 - **CUMULUS-3279**
@@ -310,6 +331,50 @@ Users/clients that do not make use of these endpoints will not be impacted.
 
 ## Fixed
 
+- **CUMULUS-3285**
+  - Updated `api/lib/distribution.js isAuthBearTokenRequest` to handle non-Bearer authorization header
+
+### Fixed
+
+- **CUMULUS-3315**
+  - Update CI scripts to use shell logic/GNU timeout to bound test timeouts
+    instead of NPM `parallel` package, as timeouts were not resulting in
+    integration test failure
+- **CUMULUS-3223**
+  - Update `@cumulus/cmrjs/cmr-utils.getGranuleTemporalInfo` to handle the error when the cmr file s3url is not available
+  - Update `sfEventSqsToDbRecords` lambda to return [partial batch failure](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting),
+    and only reprocess messages when cumulus message can't be retrieved from the execution events.
+  - Update `@cumulus/cumulus-message-adapter-js` to `2.0.5` for all cumulus tasks
+
+## [v15.0.3] 2023-04-28
+
+### Fixed
+
+- **CUMULUS-3243**
+  - Updated granule delete logic to delete granule which is not in DynamoDB
+  - Updated granule unpublish logic to handle granule which is not in DynamoDB and/or CMR
+
+## [v15.0.2] 2023-04-25
+
+### Fixed
+
+- **CUMULUS-3120**
+  - Fixed a bug by adding in `default_log_retention_periods` and `cloudwatch_log_retention_periods`
+  to Cumulus modules so they can be used during deployment for configuring cloudwatch retention periods, for more information check here: [retention document](https://nasa.github.io/cumulus/docs/configuration/cloudwatch-retention)
+  - Updated cloudwatch retention documentation to reflect the bugfix changes
+
+## [v15.0.1] 2023-04-20
+
+### Changed
+
+- **CUMULUS-3279**
+  - Updated core dependencies on `xml2js` to `v0.5.0`
+  - Forcibly updated downstream dependency for `xml2js` in `saml2-js` to
+    `v0.5.0`
+  - Added audit-ci CVE override until July 1 to allow for Core package releases
+
+## Fixed
+
 - **CUMULUS-3285**
   - Updated `api/lib/distribution.js isAuthBearTokenRequest` to handle non-Bearer authorization header
 
@@ -7215,7 +7280,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[unreleased]: https://github.com/nasa/cumulus/compare/v15.0.3...HEAD
+[unreleased]: https://github.com/nasa/cumulus/compare/v16.0.0...HEAD
+[v16.0.0]: https://github.com/nasa/cumulus/compare/v15.0.3...v16.0.0
 [v15.0.3]: https://github.com/nasa/cumulus/compare/v15.0.2...v15.0.3
 [v15.0.2]: https://github.com/nasa/cumulus/compare/v15.0.1...v15.0.2
 [v15.0.1]: https://github.com/nasa/cumulus/compare/v15.0.0...v15.0.1

diff --git a/docs/upgrade-notes/rds-phase-3-data-migration-guidance.md b/docs/upgrade-notes/rds-phase-3-data-migration-guidance.md
@@ -0,0 +1,87 @@
+---
+id: rds-phase-3-data-migration-guidance
+title: Data Integrity & Migration Guidance (RDS Phase 3 Upgrade)
+hide_title: false
+---
+
+A few issues were identied as part of the RDS Phase 2 release. These issues could impact Granule data-integrity and are described below along with recommended actions and guidance going forward.
+
+## Issue Descriptions
+
+### Issue 1
+
+[Relevant ticket: CUMULUS-3019](https://bugs.earthdata.nasa.gov/browse/CUMULUS-3019)
+
+Ingesting granules will delete unrelated files from the Files Postgres table. This is due to an issue in our logic to remove excess files when writing granules and fixed in Cumulus versions 13.2.1, 12.0.2, 11.1.5
+
+With this bug we believe the data in Dynamo is the most reliable and Postgres is out-of-sync.
+
+### Issue 2
+
+[Relevant ticket: CUMULUS-3024](https://bugs.earthdata.nasa.gov/browse/CUMULUS-3024)
+
+Updating an existing granule either via API or Workflow could result in datastores becoming out-of-sync if a partial granule record is provided. Our update logic operates differently in Postgres and Dynamo/Elastic. If a partial object is provided in an update payload the Postgres record will delete/nullify fields not present in the payload. Dynamo/Elastic will retain existing values and not delete/nullify.
+
+With this bug it’s possible that either Dynamo or PG could be the source of truth. It’s likely that it’s still Dynamo.
+
+### Issue 3
+
+[Relevant ticket: CUMULUS-3024](https://bugs.earthdata.nasa.gov/browse/CUMULUS-3024)
+
+Updating an existing granule with an empty files array in the update payload results in datastores becoming out-of-sync. If an empty array is provided, existing files in Dynamo and Elastic will be removed. Existing files in Postgres will be retained.
+
+With this bug Postgres is the source of truth. Files are retained in PG and incorrectly removed in Dynamo/Elastic.
+
+### Issue 4
+
+[Relevant ticket: CUMULUS-3017](https://bugs.earthdata.nasa.gov/browse/CUMULUS-3017)
+
+Updating/putting a granule via framework writes that duplicates a granuleId but has a different collection results in overwrite of the DynamoDB granule but a *new* granule record for Postgres.  This *intended* post RDS transition, however should not be happening now.
+
+With this bug we believe Dynamo is the source of truth, and ‘excess’ older granules will be left in postgres.     This should be detectable with tooling/query to detect duplicate granuleIds in the granules table.
+
+### Issue 5
+
+[Relevant ticket: CUMULUS-3024](https://bugs.earthdata.nasa.gov/browse/CUMULUS-3024)
+
+This is a sub-issue of issue 2 above - due to the way we assign a PDR name to a record, if the `pdr` field is missing from the final payload for a granule as part of a workflow message write, the final granule record will not link the PDR to the granule properly in postgres, however the dynamo record *will* have the linked PDR.       This *can* happen in situations where the granule is written prior to completion with the PDR in the payload, but then downstream only the granule object is included, particularly in multi-workflow ingest scenarios and/or bulk update situations.
+
+## Immediate Actions
+
+1. Re-review the issues described above
+    - GHRC was able to scope the affected granules to specific collections, which makes the recovery process much easier. This may not be an option for all DAACs.
+
+2. If you have not ingested granules or performed partial granule updates on affected Cumulus versions (questions 1 and 2 on the survey), no action is required. You may update to the latest version of Cumulus.
+
+3. One option to ensure your Postgres data matches Dynamo is running the data-migration lambda (see below for instructions) before updating to the latest Cumulus version if both of the following are true:
+    - you have ingested granules using an affected Cumulus version
+    - your DAAC has not had any operations that updated an existing granule with an empty files array (granule.files = [])
+
+4. A second option for DAACs that have ingested data using an affected Cumulus version is to use your DAAC’s recovery tools or reingest the affected granules. This is likely the most certain method for ensuring Postgres contains the correct data but may be infeasible depending on the size of data holdings, etc..
+
+## Guidance Going Forward
+
+1. Before updating to Cumulus version 16.x and beyond, take a snapshot of your DynamoDB instance. The v16 update removes the DynamoDB tables. This snapshot would be for use in unexpected data recovery scenarios only.
+
+2. Cumulus recommends that you establish and follow a database backup/disaster recovery protocol for your RDS database, which should include periodic backups.  The frequency will depend on each DAAC’s database architecture, comfort level, datastore size, and time available. [Relevant AWS Docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html)
+
+3. Invest future development effort in data validation/integrity tools and procedures. Each DAAC has different requirements here. Each DAAC should maintain procedures for validating their Cumulus datastore against their holdings.
+
+## Running a Granule Migration
+
+[Instructions for running the data-migration operation to sync Granules from DynamoDB to PostgreSQL](./upgrade-rds.md#5-run-the-second-data-migration)
+
+The data-migration2 Lambda (which is invoked asynchronously using `${PREFIX}-postgres-migration-async-operation)` uses Cumulus' Granule upsert logic to write granules from DynamoDB to PostgreSQL. This is particularly notable because granules with a running or queued status will only migrate a subset of their fields:
+
+- status
+- timestamp
+- updated_at
+- created_at
+
+It is recommended that users ensure their granules are in a final state (`running`, `completed`) before running this data migration. If there are Granules with an incomplete status, it may impact the data migration.
+
+For example, if a Granule in the running status is updated by a workflow or API call (containing an updated status) and fails, that granule will have the original running status, not the intended/updated status. Failed Granule writes/updates should be evaluated and resolved prior to this data migration.
+
+Cumulus provides the Cumulus Dead Letter Archive which is populated by the Dead Letter Queue for the sfEventSqsToDbRecords Lambda, which is responsible for Cumulus message writes to PostgreSQL. This may not catch all write failures depending on where the failure happened and workflow configuration but may be a useful tool.
+
+If a Granule record is correct except for the status, Cumulus provides an API to update specific granule fields.
diff --git a/docs/upgrade-notes/upgrade_rds_phase_3_release.md b/docs/upgrade-notes/upgrade_rds_phase_3_release.md
@@ -34,9 +34,9 @@ In addition to the above requirements, we suggest users:
   and other efforts included in the outcome from CUMULUS-3035/CUMULUS-3071.
 
 - Halt all ingest prior to performing the version upgrade.
-- Run load testing/functional testing
+- Run load testing/functional testing.
 
-  While the majority of the modifications for release 16 are related to DynamoDB removal, we always encourage user engineering teams ensure compatibility at scale with their deployment's engineering configuration prior to promotion to a production environment to ensure a smooth upgrade.
+  While the majority of the modifications for release 16 are related to DynamoDB removal, we always encourage user engineering teams ensure compatibility at scale with their deployment's configuration prior to promotion to a production environment to ensure a smooth upgrade.
 
 ## Upgrade procedure
 
@@ -151,6 +151,6 @@ In addition to the above requirements, we suggest users:
   module.cumulus.module.postgres_migration_async_operation.aws_security_group.postgres_migration_async_operation[0]
   ```
 
-  Because the AWS resources associated with these security groups can take some time to be properly updated (in testing this was 20-35 minutes), these deletions may cause the deployment to take some time.   If for some unexpected reason this takes longer than expected this causes the update to time out, you should be able to continue the deployment by re-running terraform to completion.
+  Because the AWS resources associated with these security groups can take some time to be properly updated (in testing this was 20-35 minutes), these deletions may cause the deployment to take some time.   If for some unexpected reason this takes longer than expected and this causes the update to time out, you should be able to continue the deployment by re-running terraform to completion.
 
   Users may also opt to attempt to reassign the affected Network Interfaces from the Security Group/deleting the security group manually if this situation occurs and the deployment time is not desirable.
diff --git a/example/lambdas/asyncOperations/package.json b/example/lambdas/asyncOperations/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-async-operations",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "AsyncOperations Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/ftpPopulateTestLambda/package.json b/example/lambdas/ftpPopulateTestLambda/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/ftp-populate-test-lambda",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "FTP Population Utility Lambda",
   "main": "index.js",
   "private": true,
@@ -19,12 +19,12 @@
     "access": "private"
   },
   "dependencies": {
-    "@cumulus/api": "15.0.0",
-    "@cumulus/api-client": "15.0.0",
-    "@cumulus/common": "15.0.0",
-    "@cumulus/integration-tests": "15.0.0",
-    "@cumulus/logger": "15.0.0",
-    "@cumulus/test-data": "15.0.0",
+    "@cumulus/api": "16.0.0",
+    "@cumulus/api-client": "16.0.0",
+    "@cumulus/common": "16.0.0",
+    "@cumulus/integration-tests": "16.0.0",
+    "@cumulus/logger": "16.0.0",
+    "@cumulus/test-data": "16.0.0",
     "aws-sdk": "^2.585.0",
     "fs-extra": "^9.0.0",
     "jsftp": "https://github.com/jkovarik/jsftp.git#add_288",

diff --git a/example/lambdas/lzardsClientTest/package.json b/example/lambdas/lzardsClientTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-lzards-api-lambda",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "LZARDS API Client Test Lambda",
   "private": true,
   "engines": {
@@ -20,7 +20,7 @@
   "author": "Cumulus Authors",
   "license": "Apache-2.0",
   "dependencies": {
-    "@cumulus/logger": "15.0.0",
-    "@cumulus/lzards-api-client": "15.0.0"
+    "@cumulus/logger": "16.0.0",
+    "@cumulus/lzards-api-client": "16.0.0"
   }
 }
diff --git a/example/lambdas/python-processing/package.json b/example/lambdas/python-processing/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-process-activity",
   "private": true,
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-activity/package.json b/example/lambdas/python-reference-activity/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-activity",
   "private": true,
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

diff --git a/example/lambdas/python-reference-task/package.json b/example/lambdas/python-reference-task/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-task",
   "private": true,
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "Python reference task",
   "main": "index.js",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-task",

diff --git a/example/lambdas/s3AccessTest/package.json b/example/lambdas/s3AccessTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-s3-access",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "S3 Access Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/snsS3Test/package.json b/example/lambdas/snsS3Test/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-sns-s3",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "SNS to S3 Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/lambdas/versionUpTest/package.json b/example/lambdas/versionUpTest/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-version-up",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "Version Up Test Lambda",
   "main": "index.js",
   "private": true,

diff --git a/example/package.json b/example/package.json
@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/cumulus-integration-tests",
-  "version": "15.0.0",
+  "version": "16.0.0",
   "description": "Cumulus Integration Test Deployment",
   "private": true,
   "main": "index.js",
@@ -44,32 +44,32 @@
     ]
   },
   "dependencies": {
-    "@cumulus/api": "15.0.0",
-    "@cumulus/api-client": "15.0.0",
-    "@cumulus/async-operations": "15.0.0",
-    "@cumulus/aws-client": "15.0.0",
-    "@cumulus/checksum": "15.0.0",
-    "@cumulus/cmr-client": "15.0.0",
-    "@cumulus/cmrjs": "15.0.0",
-    "@cumulus/common": "15.0.0",
-    "@cumulus/discover-granules": "15.0.0",
-    "@cumulus/discover-pdrs": "15.0.0",
-    "@cumulus/files-to-granules": "15.0.0",
-    "@cumulus/hello-world": "15.0.0",
-    "@cumulus/ingest": "15.0.0",
-    "@cumulus/integration-tests": "15.0.0",
-    "@cumulus/message": "15.0.0",
-    "@cumulus/move-granules": "15.0.0",
-    "@cumulus/parse-pdr": "15.0.0",
-    "@cumulus/pdr-status-check": "15.0.0",
-    "@cumulus/post-to-cmr": "15.0.0",
-    "@cumulus/queue-granules": "15.0.0",
-    "@cumulus/queue-pdrs": "15.0.0",
-    "@cumulus/sf-sqs-report": "15.0.0",
-    "@cumulus/sync-granule": "15.0.0",
-    "@cumulus/test-processing": "15.0.0"
+    "@cumulus/api": "16.0.0",
+    "@cumulus/api-client": "16.0.0",
+    "@cumulus/async-operations": "16.0.0",
+    "@cumulus/aws-client": "16.0.0",
+    "@cumulus/checksum": "16.0.0",
+    "@cumulus/cmr-client": "16.0.0",
+    "@cumulus/cmrjs": "16.0.0",
+    "@cumulus/common": "16.0.0",
+    "@cumulus/discover-granules": "16.0.0",
+    "@cumulus/discover-pdrs": "16.0.0",
+    "@cumulus/files-to-granules": "16.0.0",
+    "@cumulus/hello-world": "16.0.0",
+    "@cumulus/ingest": "16.0.0",
+    "@cumulus/integration-tests": "16.0.0",
+    "@cumulus/message": "16.0.0",
+    "@cumulus/move-granules": "16.0.0",
+    "@cumulus/parse-pdr": "16.0.0",
+    "@cumulus/pdr-status-check": "16.0.0",
+    "@cumulus/post-to-cmr": "16.0.0",
+    "@cumulus/queue-granules": "16.0.0",
+    "@cumulus/queue-pdrs": "16.0.0",
+    "@cumulus/sf-sqs-report": "16.0.0",
+    "@cumulus/sync-granule": "16.0.0",
+    "@cumulus/test-processing": "16.0.0"
   },
   "devDependencies": {
-    "@cumulus/test-data": "15.0.0"
+    "@cumulus/test-data": "16.0.0"
   }
 }