Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release fix conflicts (PR to master) #270

Merged
merged 7 commits into from
Mar 20, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .eslint-ratchet-high-water-mark
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1577
1576
17 changes: 11 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,25 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## [Unreleased]

## [v1.2.0] - 2018-03-20

### Fixed
- Update vulnerable npm packages [CUMULUS-425]
- `@cumulus/api`: `kinesis-consumer.js` uses `sf-scheduler.js#schedule` instead of placing a message directly on the `startSF` SQS queue. This is a fix for [CUMULUS-359](https://bugs.earthdata.nasa.gov/browse/CUMULUS-359) because `sf-scheduler.js#schedule` looks up the provider and collection data in DynamoDB and adds it to the `meta` object of the enqueued message payload.
- `@cumulus/api`: `kinesis-consumer.js` catches and logs errors instead of doing an error callback. Before this change, `kinesis-consumer` was failing to process new records when an existing record caused an error because it would call back with an error and stop processing additional records. It keeps trying to process the record causing the error because it's "position" in the stream is unchanged. Catching and logging the errors is part 1 of the fix. Proposed part 2 is to enqueue the error and the message on a "dead-letter" queue so it can be processed later ([CUMULUS-413](https://bugs.earthdata.nasa.gov/browse/CUMULUS-413)).
- **CUMULUS-260: "PDR page on dashboard only shows zeros."** The PDR stats in LPDAAC are all 0s, even if the dashboard has been fixed to retrieve the correct fields. The current version of pdr-status-check has a few issues.
- pdr is not included in the input/output schema. It's available from the input event. So the pdr status and stats are not updated when the ParsePdr workflow is complete. Adding the pdr to the input/output of the task will fix this.
- pdr-status-check doesn't update pdr stats which prevent the real time pdr progress from showing up in the dashboard. To solve this, added lambda function sf-sns-report which is copied from @cumulus/api/lambdas/sf-sns-broadcast with modification, sf-sns-report can be used to report step function status anywhere inside a step function. So add step sf-sns-report after each pdr-status-check, we will get the PDR status progress at real time.
- It's possible an execution is still in the queue and doesn't exist in sfn yet. Added code to handle 'ExecutionDoesNotExist' error when checking the execution status.
- Fixed `aws.cloudwatchevents()` typo in `packages/ingest/aws.js`. This typo was the root cause of the error: `Error: Could not process scheduled_ingest, Error: : aws.cloudwatchevents is not a constructor` seen when trying to update a rule.


### Removed

- `@cumulus/ingest/aws`: Remove queueWorkflowMessage which is no longer being used by `@cumulus/api`'s `kinesis-consumer.js`.

## [v1.1.4] - 2018-03-15

### Added
- added flag `useList` to parse-pdr [CUMULUS-404]

Expand All @@ -41,11 +50,6 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
- `@cumulus/deployment` deploys DynamoDB streams for the Collections, Providers and Rules tables as well as a new lambda function called `dbIndexer`. The `dbIndexer` lambda has an event source mapping which listens to each of the DynamoDB streams. The dbIndexer lambda receives events referencing operations on the DynamoDB table and updates the elasticsearch cluster accordingly.
- The `@cumulus/api` endpoints for collections, providers and rules _only_ query DynamoDB, with the exception of LIST endpoints and the collections' GET endpoint.

- **CUMULUS-260: "PDR page on dashboard only shows zeros."** The PDR stats in LPDAAC are all 0s, even if the dashboard has been fixed to retrieve the correct fields. The current version of pdr-status-check has a few issues.
- pdr is not included in the input/output schema. It's available from the input event. So the pdr status and stats are not updated when the ParsePdr workflow is complete. Adding the pdr to the input/output of the task will fix this.
- pdr-status-check doesn't update pdr stats which prevent the real time pdr progress from showing up in the dashboard. To solve this, added lambda function sf-sns-report which is copied from @cumulus/api/lambdas/sf-sns-broadcast with modification, sf-sns-report can be used to report step function status anywhere inside a step function. So add step sf-sns-report after each pdr-status-check, we will get the PDR status progress at real time.
- It's possible an execution is still in the queue and doesn't exist in sfn yet. Added code to handle 'ExecutionDoesNotExist' error when checking the execution status.

### Updated
- Broke up `kes.override.js` of @cumulus/deployment to multiple modules and moved to a new location
- Expanded @cumulus/deployment test coverage
Expand Down Expand Up @@ -95,7 +99,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## [v1.0.0] - 2018-02-23

[Unreleased]: https://github.com/cumulus-nasa/cumulus/compare/v1.1.4...HEAD
[Unreleased]: https://github.com/cumulus-nasa/cumulus/compare/v1.2.0...HEAD
[v1.2.0]: https://github.com/cumulus-nasa/cumulus/compare/v1.1.4...v1.2.0
[v1.1.4]: https://github.com/cumulus-nasa/cumulus/compare/v1.1.3...v1.1.4
[v1.1.3]: https://github.com/cumulus-nasa/cumulus/compare/v1.1.2...v1.1.3
[v1.1.2]: https://github.com/cumulus-nasa/cumulus/compare/v1.1.1...v1.1.2
Expand Down
4 changes: 2 additions & 2 deletions cumulus/services/sfn-scheduler/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/sfn-scheduler",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "Runs ingest timers as configured in /config/products.json",
"main": "index.js",
"keywords": [
Expand All @@ -17,7 +17,7 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"lodash": "^4.17.4",
"uuid": "^3.0.1"
}
Expand Down
2 changes: 1 addition & 1 deletion cumulus/services/sfn-throttler/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "aws-step-function-throttling-service",
"version": "1.0.0",
"version": "1.2.0",
"private": true,
"description": "A service to throttle concurrent executions of AWS Step Functions.",
"main": "index.js",
Expand Down
6 changes: 3 additions & 3 deletions cumulus/tasks/copy-idx-from-s3-to-efs/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cumulus/copy-idx-from-s3-to-efs",
"version": "1.1.3",
"version": "1.2.0",
"description": "Task to copy idx file from S3 to EFS for OnEarth 2.0.",
"main": "index.js",
"private": true,
Expand Down Expand Up @@ -35,8 +35,8 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/test-data": "^1.1.3"
"@cumulus/common": "^1.2.0",
"@cumulus/test-data": "^1.2.0"
},
"devDependencies": {
"@ava/babel-preset-stage-4": "^1.1.0",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/delete-ingest-tracking-data/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cumulus/delete-ingest-tracking-data",
"version": "1.1.3",
"version": "1.2.0",
"description": "Task to delete ingest tracking data when ingest has completed.",
"main": "index.js",
"private": true,
Expand Down Expand Up @@ -36,7 +36,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3"
"@cumulus/common": "^1.2.0"
},
"devDependencies": {
"@ava/babel-preset-stage-4": "^1.1.0",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/delete-pdr-ftp/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/delete-pdr-ftp",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "SIPS handler PDR discovery task",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -35,7 +35,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/delete-pdr-s3/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/delete-pdr-s3",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "SIPS handler PDR discovery task",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -39,7 +39,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/discover-cmr-granules/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/discover-cmr-granules",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "Discovers granules from the CMR",
"main": "index.js",
"keywords": [
Expand All @@ -19,7 +19,7 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"moment": "^2.21.0",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1"
Expand Down
8 changes: 4 additions & 4 deletions cumulus/tasks/discover-granules/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cumulus/discover-granules",
"version": "1.1.4",
"version": "1.2.0",
"description": "Discover Granules in FTP/HTTP/SFTP endpoints",
"main": "index.js",
"directories": {
Expand Down Expand Up @@ -37,10 +37,10 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.4",
"@cumulus/test-data": "^1.1.3",
"@cumulus/ingest": "^1.2.0",
"@cumulus/test-data": "^1.2.0",
"babel-core": "^6.25.0",
"babel-loader": "^6.2.4",
"babel-polyfill": "^6.23.0",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/discover-http-tiles/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/discover-http-tiles",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "Crawls an HTTP endpoint to discover tiled imagery",
"main": "index.js",
"keywords": [
Expand All @@ -14,7 +14,7 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"async": "^2.0.0",
"lodash": "^4.15.0",
"simplecrawler": "git+https://github.com/cgiffard/node-simplecrawler.git#193e506c39164ddf045c7c6c502e1a015d85a290"
Expand Down
6 changes: 3 additions & 3 deletions cumulus/tasks/discover-pdr/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/discover-pdr",
"private": true,
"version": "1.1.4",
"version": "1.2.0",
"description": "SIPS handler PDR discovery task",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -36,8 +36,8 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/ingest": "^1.1.4",
"@cumulus/common": "^1.2.0",
"@cumulus/ingest": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
8 changes: 4 additions & 4 deletions cumulus/tasks/discover-pdrs/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cumulus/discover-pdrs",
"version": "1.1.4",
"version": "1.2.0",
"description": "Discover PDRs in FTP and HTTP endpoints",
"main": "index.js",
"directories": {
Expand Down Expand Up @@ -36,10 +36,10 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.4",
"@cumulus/test-data": "^1.1.3",
"@cumulus/ingest": "^1.2.0",
"@cumulus/test-data": "^1.2.0",
"babel-core": "^6.25.0",
"babel-loader": "^6.2.4",
"babel-polyfill": "^6.23.0",
Expand Down
8 changes: 4 additions & 4 deletions cumulus/tasks/discover-s3-granules/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cumulus/discover-s3-granules",
"version": "1.1.4",
"version": "1.2.0",
"description": "Discover granules from an S3 bucket",
"main": "index.js",
"directories": {
Expand Down Expand Up @@ -37,10 +37,10 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.4",
"@cumulus/test-data": "^1.1.3",
"@cumulus/ingest": "^1.2.0",
"@cumulus/test-data": "^1.2.0",
"babel-core": "^6.25.0",
"babel-loader": "^6.2.4",
"babel-polyfill": "^6.23.0",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/download-activity-mock/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/dowload-activity-mock",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "Mock for provider gateway",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -35,7 +35,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/filter-payload/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/filter-payload",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "Task to filter the payload from one task to the next",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -35,7 +35,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3"
"@cumulus/common": "^1.2.0"
},
"devDependencies": {
"@ava/babel-preset-stage-4": "^1.1.0",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/generate-mrf/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/generate-mrf",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "Generates MRFs for ingested tiles",
"main": "index.js",
"keywords": [
Expand All @@ -14,7 +14,7 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"async": "^2.0.0",
"lodash": "^4.13.1",
"mustache": "^2.2.1",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/generate-pan/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/generate-pan",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "SIPS handler PAN generation/upload task",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -35,7 +35,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
8 changes: 4 additions & 4 deletions cumulus/tasks/generate-pdr-file-list/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/generate-pdr-file-list",
"private": true,
"version": "1.1.4",
"version": "1.2.0",
"description": "SIPS handler PDR processing task",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -35,9 +35,9 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/ingest": "^1.1.4",
"@cumulus/pvl": "^1.0.0",
"@cumulus/common": "^1.2.0",
"@cumulus/ingest": "^1.2.0",
"@cumulus/pvl": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
4 changes: 2 additions & 2 deletions cumulus/tasks/generate-pdrd/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@cumulus/generate-pdrd",
"private": true,
"version": "1.1.3",
"version": "1.2.0",
"description": "SIPS handler generating a PDRD when PDR validation fails",
"main": "index.js",
"keywords": [
Expand Down Expand Up @@ -35,7 +35,7 @@
]
},
"dependencies": {
"@cumulus/common": "^1.1.3",
"@cumulus/common": "^1.2.0",
"ftp": "^0.3.10",
"node-fetch": "^1.6.1",
"parse-duration": "^0.1.1",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/hello-world/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@cumulus/hello-world",
"version": "1.1.3",
"version": "1.2.0",
"description": "Example task",
"main": "index.js",
"directories": {
Expand Down
Loading