Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

break up kes.override.js - CUMULUS-371-part-1 #237

Merged
merged 13 commits into from
Mar 8, 2018
Merged

break up kes.override.js - CUMULUS-371-part-1 #237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
db25cad
reorganize kes override to make it more reusable for integration testing
Mar 6, 2018
78c05ff
add tests for deployment adapter
Mar 6, 2018
9291dce
update caching on circleci
Mar 7, 2018
9790864
add fs-extra as deployment dependency
Mar 7, 2018
e47d380
break caches to multiple keys
Mar 7, 2018
bfeafe1
break up tasks cache to multiple caches
Mar 7, 2018
b9676d1
restore every cache key individually
Mar 7, 2018
249dcca
improve nested objects handling
Mar 8, 2018
fdb99e9
Merge branch 'master' into CUMULUS-371
Mar 8, 2018
795e82e
use message adapter 1.0.1
Mar 8, 2018
1a4d766
small docs fixes
sethvincent Mar 8, 2018
920cd34
update changelog
sethvincent Mar 8, 2018
ae36e46
Merge branch 'CUMULUS-371' of github.com:cumulus-nasa/cumulus into CU…
sethvincent Mar 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 81 additions & 18 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,37 @@ jobs:

- restore_cache:
keys:
- dependencies-{{ checksum "lerna.json" }}-{{ checksum "package.json" }}
- dependencies-
- core-{{ checksum "lerna.json" }}-{{ checksum "package.json" }}
- core-

- run:
name: core installation
command: |
yarn install

# generate global hash
./node_modules/.bin/lerna exec --concurrency 1 -- sha1sum package.json | awk '{print $1}' '' >> /home/circleci/project/global-hash

- save_cache:
paths:
- ./node_modules
key: core-{{ checksum "lerna.json" }}-{{ checksum "package.json" }}

- restore_cache:
keys:
- packages-{{ checksum "global-hash" }}

- restore_cache:
keys:
- cumulus-tasks-{{ checksum "global-hash" }}

- restore_cache:
keys:
- gitc-tasks-{{ checksum "global-hash" }}

- restore_cache:
keys:
- services-{{ checksum "global-hash" }}

- run:
name: Installing Dependencies
Expand All @@ -32,32 +61,66 @@ jobs:
sudo bash /usr/sbin/sftp.sh user:password
sudo cp -r /home/circleci/project/packages/test-data/* /home/user/

yarn install
yarn bootstrap-no-build

- save_cache:
key: packages-{{ checksum "global-hash" }}
paths:
- ./packages/deployment/node_modules
- ./packages/pvl/node_modules
- ./packages/test-data/node_modules
- ./packages/api/node_modules
- ./packages/common/node_modules
- ./packages/cmrjs/node_modules
- ./packages/ingest/node_modules
- ./packages/integration-tests/node_modules
- ./packages/task-debug/node_modules

- save_cache:
key: cumulus-tasks-{{ checksum "global-hash" }}
paths:
- ./node_modules
- ./cumulus/tasks/discover-pdr/node_modules
- ./cumulus/tasks/discover-pdrs/node_modules
- ./cumulus/tasks/discover-granules/node_modules
- ./cumulus/tasks/discover-s3-granules/node_modules
- ./cumulus/tasks/hello-world/node_modules
- ./cumulus/tasks/discover-granules/node_modules
- ./cumulus/tasks/discover-pdrs/node_modules
- ./cumulus/tasks/parse-pdr/node_modules
- ./cumulus/tasks/pdr-status-check/node_modules
- ./cumulus/tasks/post-to-cmr/node_modules
- ./cumulus/tasks/pdr-status-check/node_modules
- ./cumulus/tasks/queue-granules/node_modules
- ./cumulus/tasks/queue-pdrs/node_modules
- ./cumulus/tasks/run-gdal/node_modules
- ./cumulus/tasks/sync-granule/node_modules
- ./cumulus/gibs-ops-api/node_modules
- ./cumulus/packages/api/node_modules
- ./cumulus/packages/cmrjs/node_modules
- ./cumulus/packages/common/node_modules
- ./cumulus/packages/pvl/node_modules
- ./cumulus/packages/ingest/node_modules
- ./cumulus/packages/deployment/node_modules
key: dependencies-{{ checksum "lerna.json" }}-{{ checksum "package.json" }}
- ./cumulus/tasks/discover-s3-granules/node_modules

- save_cache:
key: gitc-tasks-{{ checksum "global-hash" }}
paths:
- ./cumulus/tasks/copy-idx-from-s3-to-efs/node_modules
- ./cumulus/tasks/delete-ingest-tracking-data/node_modules
- ./cumulus/tasks/delete-pdr-ftp/node_modules
- ./cumulus/tasks/delete-pdr-s3/node_modules
- ./cumulus/tasks/discover-cmr-granules/node_modules
- ./cumulus/tasks/discover-http-tiles/node_modules
- ./cumulus/tasks/dowload-activity-mock/node_modules
- ./cumulus/tasks/filter-payload/node_modules
- ./cumulus/tasks/generate-mrf/node_modules
- ./cumulus/tasks/generate-pan/node_modules
- ./cumulus/tasks/generate-pdrd/node_modules
- ./cumulus/tasks/run-gdal/node_modules
- ./cumulus/tasks/sync-http-urls/node_modules
- ./cumulus/tasks/sync-wms/node_modules
- ./cumulus/tasks/tee/node_modules
- ./cumulus/tasks/trigger-ingest/node_modules
- ./cumulus/tasks/trigger-mrf-gen/node_modules
- ./cumulus/tasks/trigger-process-pdrs/node_modules
- ./cumulus/tasks/validate-archives/node_modules
- ./cumulus/tasks/discover-pdr/node_modules
- ./cumulus/tasks/generate-pdr-file-list/node_modules
- ./cumulus/tasks/validate-pdr/node_modules

- save_cache:
key: services-{{ checksum "global-hash" }}
paths:
- ./cumulus/services/sfn-scheduler/node_modules
- ./cumulus/services/sfn-throttler/node_modules

- run:
name: Running Test
Expand Down
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,5 @@ packaged-*
.vscode
*-debug.log
profiles.clj
kes.js
*.map
.DS_Store
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## [Unreleased]

### Updated
- Broke up `kes.override.js` of @cumulus/deployment to multiple modules and moved to a new location
- Expanded @cumulus/deployment test coverage
- all tasks were updated to use cumulus-message-adapter-js 1.0.1

## [v1.1.1] - 2018-03-08

### Removed
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/discover-granules/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/discover-pdrs/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/discover-s3-granules/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/hello-world/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
"author": "Cumulus Authors",
"license": "Apache-2.0",
"dependencies": {
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"babel-core": "^6.25.0",
"babel-loader": "^6.2.4",
"babel-plugin-transform-async-to-generator": "^6.24.1",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/parse-pdr/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
},
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/pdr-status-check/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
},
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/post-to-cmr/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"dependencies": {
"@cumulus/cmrjs": "^1.1.0",
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/queue-granules/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"babel-core": "^6.25.0",
"babel-loader": "^6.2.4",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/queue-pdrs/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
"license": "Apache-2.0",
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"babel-core": "^6.25.0",
"babel-loader": "^6.2.4",
Expand Down
2 changes: 1 addition & 1 deletion cumulus/tasks/sync-granule/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
},
"dependencies": {
"@cumulus/common": "^1.1.0",
"@cumulus/cumulus-message-adapter-js": "~0.0.1-beta.3",
"@cumulus/cumulus-message-adapter-js": "^1.0.1",
"@cumulus/ingest": "^1.1.1",
"@cumulus/test-data": "^1.1.0",
"babel-core": "^6.25.0",
Expand Down
1 change: 1 addition & 0 deletions packages/deployment/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
app/*.js
Loading