New Feature: Adds Data Directory Support #1012

AlgoStephenAkiki · 2022-06-01T20:26:09Z

Resolves #997

This PR does the following:

Updates the go-algorand submodule hash to point to rel/beta
Moves the cpu profiling file, pid file and indexer configuration file
to be options of only the daemon sub-command
Changes os.Exit() to be a panic with a special handler. This is so
that defer's are handled instead of being ignored.
Detects auto-loading configuration files in the data directory and
issues errors if equivalent command line arguments are supplied.
Updates the README with instructions on how to use the auto-loading
configuration files and the data directory.

AlgoStephenAkiki · 2022-06-01T20:30:05Z

@shiqizng Please review to minimize merge conflicts with your branch.
@Eric-Warehime Please review to check that this still works with M1 Mac
@winder General review

Eric-Warehime · 2022-06-01T21:14:16Z

@Eric-Warehime Please review to check that this still works with M1 Mac

Yeah, it builds fine on my M1.

winder

Requesting some small changes, and had some comments / questions.

winder · 2022-06-01T23:13:28Z

README.md

@@ -195,6 +195,7 @@ Settings can be provided from the command line, a configuration file, or an envi
 | default-balances-limit        |         | default-balances-limit        | INDEXER_DEFAULT_BALANCES_LIMIT        |
 | max-applications-limit        |         | max-applications-limit        | INDEXER_MAX_APPLICATIONS_LIMIT        |
 | default-applications-limit    |         | default-applications-limit    | INDEXER_DEFAULT_APPLICATIONS_LIMIT    |
+| data-dir                      | i       | data-dir                      | INDEXER_DATA_DIR                      |


I left this comment on Shiqi's PR also. Could we add a viper alias so that this is mapped to INDEXER_DATA? That would allow us to skip the manual env variable checking.

@AlgoStephenAkiki does exporting INDEXER_DATA_DIR work for you? after adding the alias, viper.RegisterAlias("data-dir", "data"), only INDEXER_DATA is working for me.

That's a good point. Do we want both? Performing the alias I think removes the DATA_DIR variant.

I don't think we need both, lets update the docs to refer to INDEXER_DATA. Nice catch.

README.md

cmd/algorand-indexer/daemon.go

cmd/algorand-indexer/main.go

go.mod

cmd/algorand-indexer/daemon.go

cmd/algorand-indexer/main.go

Makefile

README.md

cmd/algorand-indexer/main.go

winder

CI is failing. Otherwise, it looks good.

codecov · 2022-06-02T23:06:07Z

Codecov Report

Merging #1012 (9c16950) into develop (7c19c7d) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #1012      +/-   ##
===========================================
- Coverage    59.58%   59.52%   -0.06%     
===========================================
  Files           45       45              
  Lines         8353     8353              
===========================================
- Hits          4977     4972       -5     
- Misses        2918     2922       +4     
- Partials       458      459       +1

Impacted Files	Coverage Δ
config/config.go	`0.00% <ø> (ø)`
fetcher/fetcher.go	`33.77% <0.00%> (-2.20%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c19c7d...9c16950. Read the comment docs.

winder

Noticed a few more small things

cmd/algorand-indexer/daemon.go

Resolves #997 This PR does the following: - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory.

* return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]>

* Local Ledger (#1011) * integrate block processor * Local Ledger Deployment (#1013) * add simple local ledger migration * add deleted opts * fast catchup (#1023) * add fast catchup * Localledger merge (#1036) * return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> * recovery scenario (#1024) * handle ledger recovery scenario * refactor create genesis block (#1026) * refactor create genesis block * Adds Local Ledger Readme (#1035) * Adds Local Ledger Readme Resolves #4109 Starts Readme docs * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Removed troubleshooting section Co-authored-by: Will Winder <[email protected]> * update ledger file path and migration (#1042) * LocalLedger Refactoring + Catchpoint Service (#1049) Part 1 cleanup genesis file access. put node catchup into a function that can be swapped out with the catchup service. pass the indexer logger into the block processor. move open ledger into a util function, and move the initial state util function into a new ledger util file. add initial catchupservice implementation. move ledger init from daemon.go to constructor. Merge multiple read genesis functions. Part 2 Merge local_ledger migration package into blockprocessor. Rename Migration to Initialize Use logger in catchup service catchup Part 3 Update submodule and use NewWrappedLogger. Make util.CreateInitState private * build: merge develop into localledger/integration (#1062) * Ledger init status (#1058) * Generate an error if the catchpoint is not valid for initialization. (#1075) * Use main logger in handler and fetcher. (#1077) * Switch from fullNode catchup to catchpoint catchup service. (#1076) * Refactor daemon, add more tests (#1039) Refactors daemon cmd into separate, testable pieces. * Merge develop into localledger/integration (#1083) * Misc Local Ledger cleanup (#1086) * Update processor/blockprocessor/initialize.go Co-authored-by: Zeph Grunschlag <[email protected]> * commit * fix function call args * RFC-0001: Rfc 0001 impl (#1069) Adds an Exporter interface and a noop exporter implementation with factory methods for construction * Fix test errors * Add/fix tests * Add postgresql_exporter tests * Update config loading * Change BlockExportData to pointers * Move and rename ExportData * Add Empty func to BlockData * Add comment Co-authored-by: shiqizng <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

Update go-algorand and go.mod

87e1ee2

AlgoStephenAkiki added the Team Lamprey label Jun 1, 2022

AlgoStephenAkiki requested review from winder, Eric-Warehime and shiqizng June 1, 2022 20:26

AlgoStephenAkiki self-assigned this Jun 1, 2022

AlgoStephenAkiki changed the title ~~Adds Data Directory Support~~ New Feature: Adds Data Directory Support Jun 1, 2022

AlgoStephenAkiki added the New Feature label Jun 1, 2022

AlgoStephenAkiki force-pushed the 997-use-indexer-data-dir branch from 3eb33a1 to 3e23145 Compare June 1, 2022 20:44

winder suggested changes Jun 1, 2022

View reviewed changes

AlgoStephenAkiki requested a review from winder June 2, 2022 14:50

winder reviewed Jun 2, 2022

View reviewed changes

cmd/algorand-indexer/main.go Outdated Show resolved Hide resolved

AlgoStephenAkiki requested a review from winder June 2, 2022 17:32

winder reviewed Jun 2, 2022

View reviewed changes

AlgoStephenAkiki requested a review from winder June 3, 2022 16:50

chaihoang removed the Team Lamprey label Jun 6, 2022

winder suggested changes Jun 6, 2022

View reviewed changes

cmd/algorand-indexer/daemon.go Outdated Show resolved Hide resolved

cmd/algorand-indexer/daemon.go Outdated Show resolved Hide resolved

cmd/algorand-indexer/daemon.go Outdated Show resolved Hide resolved

cmd/algorand-indexer/daemon.go Outdated Show resolved Hide resolved

AlgoStephenAkiki requested a review from winder June 7, 2022 15:41

AlgoStephenAkiki added 8 commits June 7, 2022 12:34

Add integration data dir flags

7fde180

Updated make file for make test

f56f877

Mock for test-generate

304b812

PR comments

d6cefe1

PR comments

2102f22

e2e fix

465f544

PR comments

9c16950

AlgoStephenAkiki force-pushed the 997-use-indexer-data-dir branch from 7c1760f to 9c16950 Compare June 7, 2022 16:34

winder approved these changes Jun 7, 2022

View reviewed changes

winder merged commit c9f5dc0 into develop Jun 7, 2022

winder deleted the 997-use-indexer-data-dir branch June 7, 2022 23:37

algobarb mentioned this pull request Jun 16, 2022

FOR REVIEW ONLY: indexer 2.12.0 into master #1033

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Feature: Adds Data Directory Support #1012

New Feature: Adds Data Directory Support #1012

AlgoStephenAkiki commented Jun 1, 2022

AlgoStephenAkiki commented Jun 1, 2022

Eric-Warehime commented Jun 1, 2022

winder left a comment

winder Jun 1, 2022

shiqizng Jun 6, 2022

AlgoStephenAkiki Jun 6, 2022

winder Jun 6, 2022

winder left a comment

codecov bot commented Jun 2, 2022 •

edited

Loading

winder left a comment

New Feature: Adds Data Directory Support #1012

New Feature: Adds Data Directory Support #1012

Conversation

AlgoStephenAkiki commented Jun 1, 2022

AlgoStephenAkiki commented Jun 1, 2022

Eric-Warehime commented Jun 1, 2022

winder left a comment

Choose a reason for hiding this comment

winder Jun 1, 2022

Choose a reason for hiding this comment

shiqizng Jun 6, 2022

Choose a reason for hiding this comment

AlgoStephenAkiki Jun 6, 2022

Choose a reason for hiding this comment

winder Jun 6, 2022

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 2, 2022 • edited Loading

Codecov Report

winder left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 2, 2022 •

edited

Loading