enhancement: refactor idb module #1334

shiqizng · 2022-11-18T20:09:28Z

remove go-algorand dependency from idb module. This PR focuses on remove go-algorand/transaction. There is a followup issue #1356 to remove the other dependencies.

* Local Ledger (#1011) * integrate block processor * Local Ledger Deployment (#1013) * add simple local ledger migration * add deleted opts * fast catchup (#1023) * add fast catchup * Localledger merge (#1036) * return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> * recovery scenario (#1024) * handle ledger recovery scenario * refactor create genesis block (#1026) * refactor create genesis block * Adds Local Ledger Readme (#1035) * Adds Local Ledger Readme Resolves #4109 Starts Readme docs * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Removed troubleshooting section Co-authored-by: Will Winder <[email protected]> * update ledger file path and migration (#1042) * LocalLedger Refactoring + Catchpoint Service (#1049) Part 1 cleanup genesis file access. put node catchup into a function that can be swapped out with the catchup service. pass the indexer logger into the block processor. move open ledger into a util function, and move the initial state util function into a new ledger util file. add initial catchupservice implementation. move ledger init from daemon.go to constructor. Merge multiple read genesis functions. Part 2 Merge local_ledger migration package into blockprocessor. Rename Migration to Initialize Use logger in catchup service catchup Part 3 Update submodule and use NewWrappedLogger. Make util.CreateInitState private * build: merge develop into localledger/integration (#1062) * Ledger init status (#1058) * Generate an error if the catchpoint is not valid for initialization. (#1075) * Use main logger in handler and fetcher. (#1077) * Switch from fullNode catchup to catchpoint catchup service. (#1076) * Refactor daemon, add more tests (#1039) Refactors daemon cmd into separate, testable pieces. * Merge develop into localledger/integration (#1083) * Misc Local Ledger cleanup (#1086) * Update processor/blockprocessor/initialize.go Co-authored-by: Zeph Grunschlag <[email protected]> * commit * fix function call args * RFC-0001: Rfc 0001 impl (#1069) Adds an Exporter interface and a noop exporter implementation with factory methods for construction * Fix test errors * Add/fix tests * Add postgresql_exporter tests * Update config loading * Change BlockExportData to pointers * Move and rename ExportData * Add Empty func to BlockData * Add comment Co-authored-by: shiqizng <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Updates for changes to our plugin design based on feedback in RFCs * Merges in submodule update for test failure fixes

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Update READEME * Add initial Conduit.md * Add example exporter, exporter README.md

* importer interface | algod importer implementation * added new tests | modified importer interface * exported mockAlgodServer from test * updated algodImporter plugin * Update importers/algod/algod_importer.go Co-authored-by: Eric Warehime <[email protected]> Co-authored-by: Eric Warehime <[email protected]>

* Added Processor Added processor functionality * pr updates * Pr * PR comments * Reduce proto checks * Linting * Pr * PR comments * PR comments

* Use config to control noop exporter round start

* Initial files * Initial Conduit Binary Resolves #1165 Creates an initial framework for conduit binary production. Introduces a data directory flag as well as init and shutdown (basic) functionality. * linting * PR comments * init via importer * Passing test * e2e test passing * Modify makefile so that algorand-indexer is built * Pr changes * Log level changes * Modify bindflagset for conduit * Makefile changes * Makefile * PR updates

* Add waits/retries to algod importer

* conduit pipeline run loop implementation

* Adds unit tests Resolves #1192 Adds unit tests to the config directory * Extra assert * PR comments * Wait for round increase * Added tests * context fix

Merge indexer/develop

* Adds cpu profile and pid flag Resolves #1172 Adds the options for cpu profiling and pid flags as well as unit tests * Update conduit/pipeline.go Co-authored-by: shiqizng <[email protected]> * linting * Added pid file function * rebase Co-authored-by: shiqizng <[email protected]>

* Convert indexer daemon to conduit

* adding file exporter

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <[email protected]> Co-authored-by: Will Winder <[email protected]> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: algolucky <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: John Lee <[email protected]> Co-authored-by: Or Aharonee <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: algolucky <[email protected]>

* Initial Filter Processor Resolves #1100 Adds dynamic filter processor and initial tests * Nits * Update processors/filterprocessor/filter_processor.go Co-authored-by: Eric Warehime <[email protected]> * Update processors/filterprocessor/filter_processor_test.go Co-authored-by: Eric Warehime <[email protected]> * Update processors/filterprocessor/filter_processor_test.go Co-authored-by: Eric Warehime <[email protected]> * PR comments * Pr comments * PR updates * PR updates * Pr updates * PR updates * PR updates * PR updates * Changed some to any and const to exact * Test failure fix * Added more unit tests * Config comments Co-authored-by: Eric Warehime <[email protected]>

* Add metrics reporting to conduit pipeline

* Changed tag format * Updated generation code * Added filtering of numerical values * PR comments * imports

* Add panic recovers Resolves #263 Adds recovery functions in main loop of conduit as well as in a go-routine in the start function of the pipeline * helper function

Resolves #1245 Explicitly spell out empty data directory as well as condensed error messages when running conduit

idb/postgres/postgres.go

idb/postgres/postgres_integration_test.go

types/types.go

winder

I didn't mark all of the todo -> TODO nits.

Overall, these changes look good and I think we should merge it into develop ASAP to avoid falling back into another long-running feature branch.

I would approve but have a couple questions:

Do you have a sense about how risky these changes are?
Would a full import test + validation would be useful to catch bugs?

shiqizng · 2022-12-01T17:01:42Z

I think the biggest risk is in ConvertValidatedBlock(vb ledgercore.ValidatedBlock)(types.ValidatedBlock, error). I hope no data is lost between base64 msgpack encoding/decoding. running a full import test and validation would be a good idea. passing those tests should make us feel more comfortable about these changes.

idb/postgres/postgres_factory.go

Eric-Warehime

Changes LGTM pending your manual e2e testing.

shiqizng · 2022-12-05T18:45:47Z

validation completed for betanet with no errors

Number of errors: [0 / 24419]
Skipped (account-limit): 0
Skipped (missing-account): 0
Retry count: 0

Eric-Warehime and others added 30 commits July 21, 2022 08:18

Add all PluginTypes (#1136)

e2c50a0

Conduit: Rfc updates (#1139)

2c945b4

* Updates for changes to our plugin design based on feedback in RFCs * Merges in submodule update for test failure fixes

docs: Plugin docs (#1150)

f2b422f

* Update READEME * Add initial Conduit.md * Add example exporter, exporter README.md

Create RFC for processor (#1140)

3d2720c

* Added Processor Added processor functionality * pr updates * Pr * PR comments * Reduce proto checks * Linting * Pr * PR comments * PR comments

Use config to control noop exporter round start (#1176)

34986db

* Use config to control noop exporter round start

Set pgsql exporter round from db on init (#1177)

959c576

Add waits/retries to algod importer (#1178)

a0b6ecc

* Add waits/retries to algod importer

conduit: pipeline run loop implementation (#1183)

9cc3d36

* conduit pipeline run loop implementation

Adds unit tests (#1202)

adbae7e

* Adds unit tests Resolves #1192 Adds unit tests to the config directory * Extra assert * PR comments * Wait for round increase * Added tests * context fix

succeed close calls when init fails (#1204)

bf8b441

silence usage data on error (#1205)

13aaba6

conduit: Conduit merge upstream (#1206)

d0046cb

Merge indexer/develop

conduit: Convert daemon to conduit pipeline (#1208)

88fca5c

* Convert indexer daemon to conduit

file exporter (#1213)

e8fa891

* adding file exporter

Conduit: e2e tests (#1221)

f93bd51

Add wait-for-block to block-gen (#1229)

756be9b

Conduit: prometheus metrics (#1230)

7cf43e8

* Add metrics reporting to conduit pipeline

Add filter_processor to all imports for processors (#1233)

a2e7bdc

File Exporter: save metadata more often (#1232)

ce54ff3

Conduit: init subcommand (#1235)

c809d35

Changed tag format (#1238)

3bc8f52

* Changed tag format * Updated generation code * Added filtering of numerical values * PR comments * imports

Add panic recovers (#1251)

a6b7473

* Add panic recovers Resolves #263 Adds recovery functions in main loop of conduit as well as in a go-routine in the start function of the pipeline * helper function

Condensed Duplicate logger output (#1256)

eaab3a1

Resolves #1245 Explicitly spell out empty data directory as well as condensed error messages when running conduit