DO NOT MERGE: indexer 2.8.0 #800

Algo-devops-service · 2021-11-29T17:52:28Z

Important Notice

NOTE: This release of the indexer REQUIRES a full rebuild with a new database.

Indexer 2.8.0 is NOT backward compatible with 2.6.4 and must be rebuilt with a new database. This release contains significant changes to unify the accounting logic used by Indexer and Algod. A side effect of this work is that data from earlier releases do not work for the new release. If you attempt to upgrade an older database, Indexer will exit with an error.

We made this decision to simplify maintenance long-term of keeping Indexer and Algod's accounting consistent.

Highlights

Complete overhaul of indexer accounting.
Enhancements to import performance.
Bug fixes for special cases

Enhancements

Add upgrade warning when that upgrade is not supported. (Add a migration that fails telling the user that upgrade is not supported #711)
Add error msg in rewind about inner txns. (Add error msg in rewind about inner txns #780)
Add version to health endpoint. (Add version to health endpoint. #756)
Adding HTTP Error responses to documentation. (Adding HTTP Error responses #666)
Return a 404 status code instead of 500 when a block is not found. (Return a 404 when a block is not found. #719)
Cleanup in the DB schema (Cleanup #714, Cleanup in import state #717, Cleanup in the DB schema #718)
Support special accounts. (Enable special accounts. #746)
Support search by zero address. (Support search by zero address. #771)
Encode teal key-value as map in JSON (Encode teal key-value as map in json #755)
Fetch block concurrently with import (Fetch block concurrently with import #668)
Improve DB indexes for better worst case accounting time (Improve DB indexes for better worst case accounting time #760)
Inner Transaction Support (Inner Transaction Search Pt 1 #684, Inner Transaction Search #721)
Return extra program pages in app params (Return extra program pages in app params #696)
Use integer for root intra in TxnExtra (Use integer for root intra in TxnExtra #763)
Use our own JSON handle for decoding (Use our own json handle for decoding #765)
Write app call addresses in txn_participation table (Write app call addresses in txn_participation table #770)

Bug Fixes

Add a check in idb.DecodeTxnRowNext() to prevent indexer crash (Add a check in idb.DecodeTxnRowNext() to prevent indexer crash #796)
Clear out error in catchup loop (Clear out error in catchup loop #689)
Display b64 fields if their content is zero bytes (Display b64 fields if their content is zero bytes #785)
Move timeout handling into handlers. (Move timeout handling into handlers. #786)
Return TotalAppSchema and TotalExtraAppPages from AccountData (Return TotalAppSchema and TotalExtraAppPages from AccountData #784)

* Make EncodeJSON() private. * Run go mod tidy. * Add missing file (facepalm).

* Fetch block concurrently. * PR feedback.

…rted (#711) * Add a migration that fails telling the user that upgrade is not supported. * Replace all other migrations with upgradeNotSupported(). * PR suggestion.

* Cleanup in import and migration state. * Restore MigrationState.

* Bump timeout to 59 seconds * Bump version to 2.6.2 * Update indexer build to include static library flags. * Fixed reference to OS_TYPE environment variable. * Added additional GOTAGLIST and GOBUILDMODE flags. * Added static ld flags to release.py script. * Updated makefile static ld flags. * Revert formatting change. Co-authored-by: Brice Rising <[email protected]> Co-authored-by: Barbara Poon <[email protected]>

…d number (#728) * Allow 0 retries during validation in block generator. * Return 200 retries. * The accounts endpoint should return the correct round number.

* fixes * one more * PR feedback * another small change

* Add LogsOverride and integration test to parse null-character and bad UTF8 logs * Add inner txns conversions and decoding for logs * Add and modify tests for inner txn and log encoding * Remove helper function as per PR comment * Move encoding tests to newer function

* Enable special accounts. * Update comment for the metastate table. * Update go-algorand. * Remove integration tests related to special accounts.

This reverts commit 315edac.

* Adding HTTP Error responses * Rebase on master resolving conflicts + minor copy paste error + regenerating go files. * Rebase on master to resolve conflicts + running: generate.sh * Rebase on master to resolve conflicts + running: generate.sh * Adding 404 Response for LookupBlock * Removing 404 on /v2/applications Co-authored-by: Tolik Zinovyev <[email protected]>

Postgres transaction query now returns the root transaction when matching an inner transaction. Response processor logic to build the root transaction + deduplication. Transaction paging token offsets for inner transaction count.

* Improve DB indexes for better worst case accounting time. * Trigger CI.

* Use integer for root intra in TxnExtra. * Update comment.

* Search for serialization error in function's return error. * Add a test.

* Write app call addresses in txn_participation table. * Merge tests.

* Write zero address in txn_participation table. * Add comments. * Add a restriction. * Move error msg to error_messages.go. * Take out parameter validation out of transactionParamsToTransactionFilter(). * Run make fmt.

algobarb and others added 30 commits September 29, 2021 17:40

Merge branch 'release/2.7.1' into develop

4bed2cf

Better error when validator fails to parse data. (#694)

2b29630

Return extra program pages in app params (#696)

b865bdc

Clear out error in catchup loop. (#689)

5c34c31

Call Commit() in TxWithRetry(). (#691)

d6f667d

Make EncodeJSON() private (#697)

9053c11

* Make EncodeJSON() private. * Run go mod tidy. * Add missing file (facepalm).

Fetch block concurrently with import (#668)

21d3c31

* Fetch block concurrently. * PR feedback.

Cleanup (#714)

a40b361

Use COPY to insert into txn and txn_participation tables. (#667)

59523c0

Add a migration that fails telling the user that upgrade is not suppo…

a1415b8

…rted (#711) * Add a migration that fails telling the user that upgrade is not supported. * Replace all other migrations with upgradeNotSupported(). * PR suggestion.

e2equeries test workaround (#715)

b382d94

Add inner transactions to transaction table (#684)

be1cece

Add e2e extra pages test (#712)

7e2eff0

Return status 404 when a block is not found. (#719)

a46708f

Cleanup in import state (#717)

18004b0

* Cleanup in import and migration state. * Restore MigrationState.

Cleanup in the DB schema. (#718)

25058e5

Always give nil maps to evaluator when empty. (#729)

bc484b7

Implement the new indexerLedgerForEval interface in go-algorand. (#652)

293ce68

Install libffi-dev package in docker. (#742)

4be382e

block-generator: the accounts endpoint should return the correct roun…

cdde87c

…d number (#728) * Allow 0 retries during validation in block generator. * Return 200 retries. * The accounts endpoint should return the correct round number.

fixes (#727)

4307826

* fixes * one more * PR feedback * another small change

Update go-algorand. (#744)

a19d782

Added dummy circleci config until we have a proper one

9a005c4

Enable special accounts. (#746)

936e52e

* Enable special accounts. * Update comment for the metastate table. * Update go-algorand. * Remove integration tests related to special accounts.

Revert "Statically link glib libraries (#730)" (#751)

b9b77c3

This reverts commit 315edac.

Fix block generator. (#752)

782b0bc

Inner Transaction Search (#721)

38557a1

Postgres transaction query now returns the root transaction when matching an inner transaction. Response processor logic to build the root transaction + deduplication. Transaction paging token offsets for inner transaction count.

tolikzinovyev and others added 18 commits November 3, 2021 13:42

Improve DB indexes for better worst case accounting time (#760)

242e7eb

* Improve DB indexes for better worst case accounting time. * Trigger CI.

Use integer for root intra in TxnExtra (#763)

ad96531

* Use integer for root intra in TxnExtra. * Update comment.

Search for serialization error in function's return error (#764)

e3cc302

* Search for serialization error in function's return error. * Add a test.

merge error response objects (#769)

9833f05

Write app call addresses in txn_participation table (#770)

7dff12e

* Write app call addresses in txn_participation table. * Merge tests.

Update go-algorand. (#776)

d2a28b6

Add missing checksums. (#779)

e18b4c1

Add error msg in rewind about inner txns. (#780)

830e1ba

Support search by zero address. (#771)

1fdbd3f

* Write zero address in txn_participation table. * Add comments. * Add a restriction. * Move error msg to error_messages.go. * Take out parameter validation out of transactionParamsToTransactionFilter(). * Run make fmt.

Make inner transaction creatable ID lookup more resilient. (#777)

2f003fb

Add a test for the txn.asset column. (#781)

0eee086

Return TotalAppSchema and TotalExtraAppPages from AccountData. (#784)

aff0804

Display b64 fields if their content is zero bytes. (#785)

e45af39

Move timeout handling into handlers. (#786)

da24444

Add a check in idb.DecodeTxnRowNext() to preven indexer crash. (#796)

a70dece

Add libboost-math-dev to mule's Dockerfile (#798)

2b47e9d

Updating README.MD with CircleCI status badges (#793)

41079d3

Bump version to 2.8.0

06e7ca8

algobarb added the Team Hyper Flow label Nov 30, 2021

algobarb self-assigned this Nov 30, 2021

algobarb requested review from tolikzinovyev, winder, algojack, bricerisingalgorand, egieseke and onetechnical November 30, 2021 14:49

winder approved these changes Nov 30, 2021

View reviewed changes

bricerisingalgorand approved these changes Nov 30, 2021

View reviewed changes

algobarb merged commit cf2a387 into master Nov 30, 2021

onetechnical deleted the release/2.8.0 branch February 1, 2022 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DO NOT MERGE: indexer 2.8.0 #800

DO NOT MERGE: indexer 2.8.0 #800

Algo-devops-service commented Nov 29, 2021 •

edited by algobarb

Loading

DO NOT MERGE: indexer 2.8.0 #800

DO NOT MERGE: indexer 2.8.0 #800

Conversation

Algo-devops-service commented Nov 29, 2021 • edited by algobarb Loading

Important Notice

Highlights

Enhancements

Bug Fixes

Algo-devops-service commented Nov 29, 2021 •

edited by algobarb

Loading