feature/es-phase-2 #3785
Draft
feature/es-phase-2 #3785
+7,136
−5,934
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sic queries (#3637) * CUMULUS-3692:Granule list endpoint for basic postgres query
…er by field value (#3656) * CUMULUS-3692:Granule list endpoint for basic postgres query * refactor * refactor * typing * add changelog entry * skip search_after * skip searchafter unit tests * add granule list test * rename * refactor * build query parameters * update comment * add field-mapping * update jsdoc * use type over interface,add log * update test description * build term/terms * buildDbQueryParameters * add unit test no terms search * add doc * rename * add unit test * add fields test * add more unit tests * support error.Error search * fix lint * rename functions * ignore files * add convert query unit tests * add all types * add unit test for fieldmapping types fix timestamp * update timestamp test * add multiple term field test * ignore execution in granule list record
…se psql (#3659) * first commit on new branch * CHANGELOG change * small fix * PR feedback * adding jsdoc + fixing spelling/grammar
#3660) * add range query support
* first committ * CHANGELOG change * fixing sortQueryMethod * simplifying code * PR feedback * merge conflicts + improving code * small jsdoc fix * PR feedback * PR feedback * PR feedback * fixing test * PR feedback * PR feedback
#3674) * add methods to convert terms,not,exists
…ic (#3681) * reopening PR * PR feedback * small test fix * small PR feedbacks * adding new tests from match queries * PR feedback/formatting * temporary reversion to list endpoint for reconreport tests * reverting changes * adding logging * more logging * more logging * removing logging + commenting reconrep test temp * commenting out failing createReconReport spec * removing comment * reverting changes to reconReport test * reverting previous change * adding ts-check * PR feedback * PR feedback * adding in test * PR feedback fix * PR feedback
…ncludeStats (#3688) * first commit * CHANGELOG * fixing small things * changes + fixes * PR feedback * splitting queries separately * PR feedback * PR feedback * PR feedback
…ollections (#3693) * CUMULUS-3639:Add support to db/CollectionSearch to retrieve active collections * add test active collections * add ts-check * update /collections/active unit test * test snyk * fix field mapping * parallel search and fix urlPath * add cumulus-lp stack * add limit 1 to subquery
#3684) * first commit * adding changes * storing changes * updating progress * linting + small fixes * small fix * changing timestamp to string in tests * fixing timestamp * commenting out tests failing in CI but not locally * saving changes * collection support * adding async_ops support * changing endpoint + tests * fixing test * uncommenting tests + adding var * commenting out tests failing in CI but not locally * adding parentArn support + changing tests * added parentArn support + fixing tests * small endpoint test fix * Pr feedback + code improvements * small CHANGELOG fix * PR feedback * PR feedback + linting * PR feedback * PR feedback * fixing test * fixing execution tests after removing asyncCumulusOPId from mapping * PR feedback * removed includeFullRecord from search classes * PR feedback * PR feedback * reverting change
…e related es queries to query postgres (#3727) * removing granules * fixing lint * fixing test * small change * adding back in some deleted things * removing more * lint fix * removing tests * skipping execution search-by-granules tests * skipping execution tests * removing tests * more removing * adding in deleted test * removing more * adding back in needed code * removing ES_HOST, query, and index from bulk_ops * fixing bulk_ops tests * adding back in ELK stack refs * changing reconreports test to skip/adding back in getGranulesByPayload * PR feedback * adding back in skipped tests * CHANGELOG * PR feedback * PR feedback' . ; * PR feedback + syntax check * adding back sort to write-granules test
…ts (#3723) * CUMULUS-3240:Remove ElasticSearch dependency from Executions endpoints * update test-executions * remove es dependencies for execution * update changelog * fix lint and warning * address PR feedback * remove esClient from createExecutionRecords
* estimate table row count * add more indexes * use count * * fix test * fix test * fix lint and add file index * add more index * fix lint * fix lint update test * execution asyncOperationId is optional * defautl stats last day * fix granule patchByGranuleId put logic back * vaccum tables * update changelog * remove sql script to another pr * update changelog * update active collection query * Revert "update active collection query" This reverts commit 88024c2.
…UT, and DEL endpoints (#3746) * first commit * CHANGELOG change * PR feedback * PR feedback
* CUMULUS-3792:Add table indexes to improve search performance
…ostgres (#3771) * Update PR mentioned endpoints to not write to ES * Update POST units to not use ES * Fix/manage broken tests * Update package/async-operations to not use ES * Remove ES update from async-operations image * Update async image to use v53, which includes removal of ES code * Fix broken ecs package tests/logic * Minor refactor * Update CHANGELOG * Remove additional unneeded test * Fix CHANGELOG * Modify unit ratchet to match after removal of code Code was removed, not refactored - this change did not degrade/improve units so much as change the denominator of lines/methods tested * Fix integration test/make it not rely on list endpoint * Refactor/PR feedback * Update example/spec/serial/AsyncOperationRunnerSuccessfulLambdaSpec.js Co-authored-by: jennyhliu <[email protected]> * Remove unneeded transaction * Remove unneeded type assertion/error handling * Remove transaction * Address PR comments --------- Co-authored-by: jennyhliu <[email protected]>
…3768) * CUMULUS-3235: Updated asyncOperations api endpoint to query postgres * id::text * revert uuid for term search update unit tests * update unit test * refact basic search * raw-parameter-binding * deconstruct response * deconstruct fields * whereRaw parameter binding * remove extra knex parameter * fix unit test conflict
* Update PR mentioned endpoints to not write to ES * Update POST units to not use ES * Fix/manage broken tests * Update package/async-operations to not use ES * Remove ES update from async-operations image * Update async image to use v53, which includes removal of ES code * Fix broken ecs package tests/logic * Minor refactor * Update CHANGELOG * Remove additional unneeded test * Fix CHANGELOG * Modify unit ratchet to match after removal of code Code was removed, not refactored - this change did not degrade/improve units so much as change the denominator of lines/methods tested * Fix integration test/make it not rely on list endpoint * Refactor/PR feedback * Update example/spec/serial/AsyncOperationRunnerSuccessfulLambdaSpec.js Co-authored-by: jennyhliu <[email protected]> * Remove unneeded transaction * Remove unneeded type assertion/error handling * Remove transaction * Address PR comments * Remove ES calls from DEL/GET PDR endpoints * Update CHANGELOG * Remove unneede deps * Fix bad merge * Fix lint/etc --------- Co-authored-by: jennyhliu <[email protected]>
…arch dependency and query Postgres instead (#3764) * first commit * fixing CHANGELOG entry * PR feedback * PR feedback
…s, and translate files
CUMULUS-3837: Create ReconciliationReportsTable in RDS and add indexes
This reverts commit 1a7a104.
…tgres (#3797) * CUMULUS-3833: Migrate ReconciliationReports to RDS * Update ReconciliationReportsMigration lambda * update lambda readme * correct names * update reconciliation-report-migration * revert migration-helper-async-operation no need async * update lambda * fix terraform * update test coverage number * remove unused variable * add used variable back * update package description
…uth (#3794) * Update ORCA report to only read from postgres database * Parameterize getGranulesByApiPropertiesQuery * Fix unit bugs * Fix unexplained lint error :( * Fix granule translation for report output * Fix/improve spec tests * Update typings for orca-backup-reconciliation-report * Pre-review tag minor fixes * Remove configuration TODO, in favor of PR comment * Update typings * Remove premature mods * Update CHANGELOG * Fix recon report spec * Minor typedef refactor * Fix units * Re-order CHANGELOG * Fix merge lint issue
* Update ORCA report to only read from postgres database * Parameterize getGranulesByApiPropertiesQuery * Fix unit bugs * Fix unexplained lint error :( * Fix granule translation for report output * Fix/improve spec tests * Update typings for orca-backup-reconciliation-report * Pre-review tag minor fixes * Remove configuration TODO, in favor of PR comment * Update typings * Remove premature mods * Update CHANGELOG * Fix recon report spec * Minor typedef refactor * Fix units * WIP - fixup * re-enable disabled test, remove unused code * Dedupe knex objects/queues * Fix linting 🔔 * Update db package typing * Update cmr-client typings * Update test with bad test parameters/setup * Add typing fixes/etc to cmr-utils js * Add basic recon report typings * Fix annotations for normalizeEvent * Minor lint fix * Additional typing fixes * Fix knex typings * Remove remaining ES refs, tidy up * Remove shouldAggregateGranulesForCollections test * Fix/update unit tests broken due to normalizeEvent mods * Updated test-collections to create database per test, use common fixtures * Add tests, refactor collection lib queries * Re-order CHANGELOG * Fix merge mangle * Updating default configuration to modify RDS cert configuration * Revert "Updating default configuration to modify RDS cert configuration" This reverts commit 7e77b2d. * re-activate integration tests, add collation fix to granule sorting * Add FileReport typings * Misc typing fixes/updates * Update ORCA typings * Minor comment update * Remove TODO * Types refactor * Add types file * Update DB package to allow for limit:null/unbounded queries * Minor typing refactor * Fix param arguments * Add collation options * Minor comment update * Refactor collection query to use @cumulus/db/search * Remove debugging log output * Remove type-bound error conditional * Minor typing update * Add missing unit tests * Add CHANGELOG updates * Split defs out into seperate file * Fix unneeded newline/lint * Remove excess typedef * Fix bad param passing, update typing in createReconcilationReportForBucket * Fix docstring typing * Seperate typedefs into seperate file * Update error types * Fix typings * Fix unit test error type mistake
* Remove internal/ES specific recon report functions * Jk/cumulus 3806 2 (#3811) * Update ORCA report to only read from postgres database * Parameterize getGranulesByApiPropertiesQuery * Fix unit bugs * Fix unexplained lint error :( * Fix granule translation for report output * Fix/improve spec tests * Update typings for orca-backup-reconciliation-report * Pre-review tag minor fixes * Remove configuration TODO, in favor of PR comment * Update typings * Remove premature mods * Update CHANGELOG * Fix recon report spec * Minor typedef refactor * Fix units * WIP - fixup * re-enable disabled test, remove unused code * Dedupe knex objects/queues * Fix linting 🔔 * Update db package typing * Update cmr-client typings * Update test with bad test parameters/setup * Add typing fixes/etc to cmr-utils js * Add basic recon report typings * Fix annotations for normalizeEvent * Minor lint fix * Additional typing fixes * Fix knex typings * Remove remaining ES refs, tidy up * Remove shouldAggregateGranulesForCollections test * Fix/update unit tests broken due to normalizeEvent mods * Updated test-collections to create database per test, use common fixtures * Add tests, refactor collection lib queries * Re-order CHANGELOG * Fix merge mangle * Updating default configuration to modify RDS cert configuration * Revert "Updating default configuration to modify RDS cert configuration" This reverts commit 7e77b2d. * re-activate integration tests, add collation fix to granule sorting * Add FileReport typings * Misc typing fixes/updates * Update ORCA typings * Minor comment update * Remove TODO * Types refactor * Add types file * Update DB package to allow for limit:null/unbounded queries * Minor typing refactor * Fix param arguments * Add collation options * Minor comment update * Refactor collection query to use @cumulus/db/search * Remove debugging log output * Remove type-bound error conditional * Minor typing update * Add missing unit tests * Add CHANGELOG updates * Split defs out into seperate file * Fix unneeded newline/lint * Remove excess typedef * Fix bad param passing, update typing in createReconcilationReportForBucket * Fix docstring typing * Seperate typedefs into seperate file * Update error types * Fix typings * Fix unit test error type mistake * Remove internal recon report units
…stgres (#3829) * first pass at making this work, tests fail * CUMULUS-3718: make recon reports list endpoint work, add recon reports to stats endpoint * make db/search/ReconciliationReportSearch work * make api list endpoint for recon reports work using the fixed search * fix/clean test file for recon reports endpoints - list test works, remove ES references * add recon reports to stats endpoint * add tests for recon reports stats endpoint * add test for recon reports aggregate in test-StatsSearch * CUMULUS-3718: update changelog * CUMULUS-3718: fix lint errors * CUMULUS-3718: edits for PR comments, add field mapping and test, add test file for ReconciliationReportSearch * CUMULUS-3718: fix lint issues * CUMULUS-3718: remove commented out test that won't work, modifymethods using map * CUMULUS-3718: fix more lint issues, from modifying methods using map --------- Co-authored-by: etcart <[email protected]>
…ords to ElasticSearch (#3854)
* files in GranuleSearch * changelog * changelog linting * changelo g in the right place * files and inclusion in tests * add changelog * linter updates * integration test bits turned back on with this functionality * revert unneeded change ot outflow from translation * fixes to execution handling, maybe too clunky * remove leftove reference comment * clean up unnecessary join and unused function * allow for flexible request of file by granule cumulus ids * making sure typing is right * memory optimizations in execution handling * linting * add unit tests and a discovered fix * linter cleanup * fix some docstrings in executionSearch * check that limit is with respect to granules and that files are translated to api format * linter error * onlu include files and executions if fullrecord requested * adding integration tests for includeFullRecord * need to request full record in recon report int test * linter errors * add changelog line about includeFullRecord * review grammar and style nits * clean up bare granule distinction * add breaking change entry to changelog * linter fix * changelog breaking change about getExecutionArnByGranuleCumulusId
…ction search (#3861) * CUMULUS-3859: Remove elasticsearch from api server * Updated CollectionSearch to filter additional granule fields for active collections * fix intermittent test error * granuleId field * refactor * update changelog * update octokit/graphql to fix cross-spawn * addAsyncOperations to serveUtils * clean variables * refactor
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Summary of changes
Addresses CUMULUS-3225: Replace ElasticSearch (Phase 2)
Changes
PR Checklist