Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement: refactor idb module #1334

Merged
merged 102 commits into from
Dec 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
3c8c150
Conduit: Postgres exporter initial implementation (#1114)
Eric-Warehime Jul 21, 2022
e2c50a0
Add all PluginTypes (#1136)
Eric-Warehime Jul 21, 2022
2c945b4
Conduit: Rfc updates (#1139)
Eric-Warehime Jul 25, 2022
c0bf99a
Conduit: Merge conduit develop (#1157)
Eric-Warehime Aug 1, 2022
f2b422f
docs: Plugin docs (#1150)
Eric-Warehime Aug 1, 2022
595d3bf
Conduit: Algod importer implementation (#1135)
algoganesh Aug 1, 2022
3d2720c
Create RFC for processor (#1140)
AlgoStephenAkiki Aug 5, 2022
34986db
Use config to control noop exporter round start (#1176)
Eric-Warehime Aug 12, 2022
959c576
Set pgsql exporter round from db on init (#1177)
Eric-Warehime Aug 12, 2022
45d5c4b
1165 create initial conduit binary (#1167)
AlgoStephenAkiki Aug 12, 2022
a0b6ecc
Add waits/retries to algod importer (#1178)
Eric-Warehime Aug 15, 2022
9cc3d36
conduit: pipeline run loop implementation (#1183)
Eric-Warehime Aug 17, 2022
adbae7e
Adds unit tests (#1202)
AlgoStephenAkiki Aug 26, 2022
bf8b441
succeed close calls when init fails (#1204)
Eric-Warehime Aug 26, 2022
13aaba6
silence usage data on error (#1205)
Eric-Warehime Aug 26, 2022
d0046cb
conduit: Conduit merge upstream (#1206)
Eric-Warehime Aug 29, 2022
beaf4c6
Adds cpu profile and pid flag (#1209)
AlgoStephenAkiki Sep 1, 2022
88fca5c
conduit: Convert daemon to conduit pipeline (#1208)
Eric-Warehime Sep 9, 2022
e8fa891
file exporter (#1213)
shiqizng Sep 9, 2022
73fdf28
conduit: Conduit/develop merge (#1222)
Eric-Warehime Sep 14, 2022
f93bd51
Conduit: e2e tests (#1221)
Eric-Warehime Sep 16, 2022
7446007
Initial Filter Processor (#1220)
AlgoStephenAkiki Sep 16, 2022
756be9b
Add wait-for-block to block-gen (#1229)
Eric-Warehime Sep 19, 2022
7cf43e8
Conduit: prometheus metrics (#1230)
Eric-Warehime Sep 20, 2022
a2e7bdc
Add filter_processor to all imports for processors (#1233)
Eric-Warehime Sep 21, 2022
ce54ff3
File Exporter: save metadata more often (#1232)
winder Sep 26, 2022
c809d35
Conduit: init subcommand (#1235)
winder Sep 27, 2022
3bc8f52
Changed tag format (#1238)
AlgoStephenAkiki Oct 3, 2022
a6b7473
Add panic recovers (#1251)
AlgoStephenAkiki Oct 3, 2022
eaab3a1
Condensed Duplicate logger output (#1256)
AlgoStephenAkiki Oct 5, 2022
6a6fd87
Fix configuration of logger (#1259)
AlgoStephenAkiki Oct 7, 2022
5f35da3
Conduit: ensure thread safety across multiple loggers. (#1271)
AlgoStephenAkiki Oct 13, 2022
6e13534
Add context to exporter interface (#1272)
AlgoStephenAkiki Oct 13, 2022
d7c2365
Conduit: improve config naming consistency (#1279)
AlgoStephenAkiki Oct 19, 2022
9284c97
enhancement: prune historical data (#1277)
shiqizng Oct 19, 2022
d0aef01
Conduit Documentation (#1275)
winder Oct 20, 2022
c5fb831
Conduit: Update submodule (#1291)
Eric-Warehime Oct 24, 2022
ace1808
Conduit: Dynamic on complete (#1285)
winder Oct 25, 2022
99a848c
Add block metadata to pipeline (#1258)
shiqizng Oct 25, 2022
997b8c7
Conduit: filereader / filewriter plugins and compatibility changes. (…
winder Oct 26, 2022
ea8c774
adding metrics endpoint (#1284)
shiqizng Oct 27, 2022
bd5a482
e2e scripts readme (#1293)
winder Oct 31, 2022
d30cfc1
plugin metrics (#1290)
shiqizng Oct 31, 2022
8f165aa
Add some basic documentation about lifecycle hooks. (#1299)
winder Oct 31, 2022
750754d
Conduit: config updates (#1295)
shiqizng Nov 1, 2022
388d00b
fix e2e conduit (#1301)
shiqizng Nov 1, 2022
84e2f12
conduit: plugin constructor (#1298)
shiqizng Nov 1, 2022
3c8938e
update import
shiqizng Nov 1, 2022
2cb79ff
Merge branch 'develop' into will/merge
winder Nov 1, 2022
330f27b
Update submodule, regenerate mapping.
winder Nov 1, 2022
0cd10ab
Update util/test/account_testutil.go
winder Nov 2, 2022
04e272a
Fix some merge issues.
winder Nov 2, 2022
2da54f0
More merge errors.
winder Nov 2, 2022
4ab2a90
More merge errors.
winder Nov 2, 2022
5043497
Added none operation (#1297)
AlgoStephenAkiki Nov 2, 2022
964200e
Merge develop into conduit. (#1303)
winder Nov 2, 2022
28ce7ca
Merge branch 'will/merge' into conduit
winder Nov 2, 2022
556e377
Merge branch 'develop' into conduit
winder Nov 2, 2022
9a2f836
conduit: next-round command line flag (#1311)
shiqizng Nov 4, 2022
74a9954
List subcommand (#1296)
AlgoStephenAkiki Nov 6, 2022
9214b07
Minor pipeline test cleanup. (#1316)
winder Nov 6, 2022
c0e1146
Reorganize plugin packages. (#1317)
winder Nov 7, 2022
b343cae
misc(release): add support for conduit (#1330)
algolucky Nov 16, 2022
49ac91d
remove unnecessary method
shiqizng Nov 17, 2022
d86c10a
refactoring postgres
shiqizng Nov 18, 2022
3048e70
more changes
shiqizng Nov 18, 2022
0fcd4cf
add a note
shiqizng Nov 18, 2022
2e6c0cc
make lint failing
shiqizng Nov 19, 2022
da3e1fe
update dummy
shiqizng Nov 21, 2022
45d9bcd
update stateproof
shiqizng Nov 21, 2022
02ba49e
tests failing
shiqizng Nov 21, 2022
46b6b75
update idb interface
shiqizng Nov 21, 2022
632e563
revert exporter interface changes
shiqizng Nov 21, 2022
67b5451
make lint
shiqizng Nov 21, 2022
7d2485c
fetch txn failing
shiqizng Nov 21, 2022
df7e65b
fix txn id
shiqizng Nov 22, 2022
26d1794
remove misc files
shiqizng Nov 22, 2022
de72bb8
updating sprf txn
shiqizng Nov 22, 2022
68de79b
fix writer_test
shiqizng Nov 23, 2022
80dc1c1
disable sprf handler tests, fix failing tests
shiqizng Nov 23, 2022
3e29296
fix postgres int tests
shiqizng Nov 23, 2022
33ec8b8
fix mock idb
shiqizng Nov 23, 2022
b4d8069
disable sprf tests
shiqizng Nov 23, 2022
5595c0a
notes
shiqizng Nov 29, 2022
28a5f23
temporary stateproof type override
shiqizng Nov 29, 2022
897d72b
make lint
shiqizng Nov 29, 2022
6d00c19
update sdk version
shiqizng Nov 29, 2022
a79df64
misc updates
shiqizng Nov 29, 2022
6d4b030
update sprf txn, fmt
shiqizng Nov 30, 2022
8ee6c70
digsest update
shiqizng Nov 30, 2022
c4d939d
update todos
shiqizng Dec 1, 2022
cc8b801
add unit tests
shiqizng Dec 1, 2022
e924c72
make lint
shiqizng Dec 1, 2022
bfe67be
update comments
shiqizng Dec 2, 2022
0728bcd
update import alias
shiqizng Dec 2, 2022
a894a0b
Merge branch 'develop' into shiqi/refactor
shiqizng Dec 5, 2022
f413e7d
merge develop
shiqizng Dec 5, 2022
761edcf
fix submodule
shiqizng Dec 5, 2022
59890ab
go mod tidy
shiqizng Dec 5, 2022
80983c4
go fmt
shiqizng Dec 6, 2022
ce4a9ca
make lint fix
shiqizng Dec 6, 2022
d2a196e
fix int test
shiqizng Dec 6, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions accounting/accounting.go
Original file line number Diff line number Diff line change
@@ -1,26 +1,24 @@
package accounting

import (
"github.com/algorand/go-algorand/data/basics"
"github.com/algorand/go-algorand/data/transactions"
"github.com/algorand/go-algorand/protocol"
sdk "github.com/algorand/go-algorand-sdk/types"
)

// GetTransactionParticipants calls function `add` for every address referenced in the
// given transaction, possibly with repetition.
func GetTransactionParticipants(stxnad *transactions.SignedTxnWithAD, includeInner bool, add func(address basics.Address)) {
func GetTransactionParticipants(stxnad *sdk.SignedTxnWithAD, includeInner bool, add func(address sdk.Address)) {
txn := &stxnad.Txn

add(txn.Sender)

switch txn.Type {
case protocol.PaymentTx:
case sdk.PaymentTx:
add(txn.Receiver)
// Close address is optional.
if !txn.CloseRemainderTo.IsZero() {
add(txn.CloseRemainderTo)
}
case protocol.AssetTransferTx:
case sdk.AssetTransferTx:
// If asset sender is non-zero, it is a clawback transaction. Otherwise,
// the transaction sender address is used.
if !txn.AssetSender.IsZero() {
Expand All @@ -31,9 +29,9 @@ func GetTransactionParticipants(stxnad *transactions.SignedTxnWithAD, includeInn
if !txn.AssetCloseTo.IsZero() {
add(txn.AssetCloseTo)
}
case protocol.AssetFreezeTx:
case sdk.AssetFreezeTx:
add(txn.FreezeAccount)
case protocol.ApplicationCallTx:
case sdk.ApplicationCallTx:
for _, address := range txn.ApplicationCallTxnFields.Accounts {
add(address)
}
Expand Down
41 changes: 20 additions & 21 deletions accounting/rewind.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@ import (
"context"
"fmt"

"github.com/algorand/go-algorand/data/basics"
"github.com/algorand/go-algorand/data/transactions"
"github.com/algorand/go-algorand/protocol"
models "github.com/algorand/indexer/api/generated/v2"

"github.com/algorand/indexer/idb"
"github.com/algorand/indexer/types"

sdk "github.com/algorand/go-algorand-sdk/types"
)

// ConsistencyError is returned when the database returns inconsistent (stale) results.
Expand Down Expand Up @@ -60,15 +59,15 @@ func (sare *SpecialAccountRewindError) Error() string {
return fmt.Sprintf("unable to rewind the %s", sare.account)
}

var specialAccounts *transactions.SpecialAddresses
var specialAccounts *types.SpecialAddresses

// AccountAtRound queries the idb.IndexerDb object for transactions and rewinds most fields of the account back to
// their values at the requested round.
// `round` must be <= `account.Round`
func AccountAtRound(ctx context.Context, account models.Account, round uint64, db idb.IndexerDb) (acct models.Account, err error) {
// Make sure special accounts cache has been initialized.
if specialAccounts == nil {
var accounts transactions.SpecialAddresses
var accounts types.SpecialAddresses
accounts, err = db.GetSpecialAccounts(ctx)
if err != nil {
return models.Account{}, fmt.Errorf("unable to get special accounts: %v", err)
Expand All @@ -77,8 +76,8 @@ func AccountAtRound(ctx context.Context, account models.Account, round uint64, d
}

acct = account
var addr basics.Address
addr, err = basics.UnmarshalChecksumAddress(account.Address)
var addr sdk.Address
addr, err = sdk.DecodeAddress(account.Address)
if err != nil {
return
}
Expand Down Expand Up @@ -126,34 +125,34 @@ func AccountAtRound(ctx context.Context, account models.Account, round uint64, d
fmt.Errorf("rewinding past inner transactions is not supported")
}
if addr == stxn.Txn.Sender {
acct.AmountWithoutPendingRewards += stxn.Txn.Fee.ToUint64()
acct.AmountWithoutPendingRewards -= stxn.SenderRewards.ToUint64()
acct.AmountWithoutPendingRewards += uint64(stxn.Txn.Fee)
acct.AmountWithoutPendingRewards -= uint64(stxn.SenderRewards)
}
switch stxn.Txn.Type {
case protocol.PaymentTx:
case sdk.PaymentTx:
if addr == stxn.Txn.Sender {
acct.AmountWithoutPendingRewards += stxn.Txn.Amount.ToUint64()
acct.AmountWithoutPendingRewards += uint64(stxn.Txn.Amount)
}
if addr == stxn.Txn.Receiver {
acct.AmountWithoutPendingRewards -= stxn.Txn.Amount.ToUint64()
acct.AmountWithoutPendingRewards -= stxn.ReceiverRewards.ToUint64()
acct.AmountWithoutPendingRewards -= uint64(stxn.Txn.Amount)
acct.AmountWithoutPendingRewards -= uint64(stxn.ReceiverRewards)
}
if addr == stxn.Txn.CloseRemainderTo {
// unwind receiving a close-to
acct.AmountWithoutPendingRewards -= stxn.ClosingAmount.ToUint64()
acct.AmountWithoutPendingRewards -= stxn.CloseRewards.ToUint64()
acct.AmountWithoutPendingRewards -= uint64(stxn.ClosingAmount)
acct.AmountWithoutPendingRewards -= uint64(stxn.CloseRewards)
} else if !stxn.Txn.CloseRemainderTo.IsZero() {
// unwind sending a close-to
acct.AmountWithoutPendingRewards += stxn.ClosingAmount.ToUint64()
acct.AmountWithoutPendingRewards += uint64(stxn.ClosingAmount)
}
case protocol.KeyRegistrationTx:
case sdk.KeyRegistrationTx:
// TODO: keyreg does not rewind. workaround: query for txns on an account with typeenum=2 to find previous values it was set to.
case protocol.AssetConfigTx:
case sdk.AssetConfigTx:
if stxn.Txn.ConfigAsset == 0 {
// create asset, unwind the application of the value
assetUpdate(&acct, txnrow.AssetID, 0, stxn.Txn.AssetParams.Total)
}
case protocol.AssetTransferTx:
case sdk.AssetTransferTx:
if addr == stxn.Txn.AssetSender || addr == stxn.Txn.Sender {
assetUpdate(&acct, uint64(stxn.Txn.XferAsset), stxn.Txn.AssetAmount+txnrow.Extra.AssetCloseAmount, 0)
}
Expand All @@ -163,7 +162,7 @@ func AccountAtRound(ctx context.Context, account models.Account, round uint64, d
if addr == stxn.Txn.AssetCloseTo {
assetUpdate(&acct, uint64(stxn.Txn.XferAsset), 0, txnrow.Extra.AssetCloseAmount)
}
case protocol.AssetFreezeTx:
case sdk.AssetFreezeTx:
default:
err = fmt.Errorf("%s[%d,%d]: rewinding past txn type %s is not currently supported", account.Address, txnrow.Round, txnrow.Intra, stxn.Txn.Type)
return
Expand Down
19 changes: 10 additions & 9 deletions accounting/rewind_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ import (
"errors"
"testing"

sdk "github.com/algorand/go-algorand-sdk/types"
"github.com/algorand/go-algorand/data/basics"
"github.com/algorand/go-algorand/data/transactions"
"github.com/algorand/go-algorand/protocol"
"github.com/algorand/indexer/types"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"

Expand All @@ -17,7 +18,7 @@ import (
)

func TestBasic(t *testing.T) {
var a basics.Address
var a sdk.Address
a[0] = 'a'

account := models.Account{
Expand All @@ -29,13 +30,13 @@ func TestBasic(t *testing.T) {

txnRow := idb.TxnRow{
Round: 7,
Txn: &transactions.SignedTxnWithAD{
SignedTxn: transactions.SignedTxn{
Txn: transactions.Transaction{
Type: protocol.PaymentTx,
PaymentTxnFields: transactions.PaymentTxnFields{
Txn: &sdk.SignedTxnWithAD{
SignedTxn: sdk.SignedTxn{
Txn: sdk.Transaction{
Type: sdk.PaymentTx,
PaymentTxnFields: sdk.PaymentTxnFields{
Receiver: a,
Amount: basics.MicroAlgos{Raw: 2},
Amount: sdk.MicroAlgos(2),
},
},
},
Expand All @@ -48,7 +49,7 @@ func TestBasic(t *testing.T) {
var outCh <-chan idb.TxnRow = ch

db := &mocks.IndexerDb{}
db.On("GetSpecialAccounts", mock.Anything).Return(transactions.SpecialAddresses{}, nil)
db.On("GetSpecialAccounts", mock.Anything).Return(types.SpecialAddresses{}, nil)
db.On("Transactions", mock.Anything, mock.Anything).Return(outCh, uint64(8))

account, err := AccountAtRound(context.Background(), account, 6, db)
Expand Down
Loading