Skip to content

Add a readme for import validator #811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 21, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions cmd/import-validator/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Import validation tool

The import validator tool imports blocks into indexer database and algod's
sqlite database in lockstep and checks that the modified accounts are the same
in the two databases.
It lets us detect the first round where an accounting discrepancy occurs
and it prints out the difference before crashing.

There is a small limitation, however.
The set of modified accounts is computed using the sqlite database.
Thus, if indexer's accounting were to modify a superset of those accounts,
this tool would not detect it.
This, however, should be unlikely.

# Using the tool

Running the tool is similar to running `algorand-indexer` with the exception
that one has to specify the path to the sqlite (go-algorand) database.

Create a new postgres (indexer) database.
```
createdb db0
```

Create a directory for the sqlite (go-algorand) database.
This will essentially serve as a non-archival algod database.
```
mkdir ledger
```

Run the tool.
```
./import-validator --algod-net localhost:8080 --algod-token ALGOD_TOKEN --algod-ledger ledger --postgres "user=ubuntu host=localhost database=db0 sslmode=disable"
```

The tool can be safely stopped and started again.
Also, the provided indexer database can be created by indexer itself, and the
provided ledger sqlite database can be created by algod.
Since the import validator essentially exercises indexer's and go-algorand's
database code, this should work.
However, the sqlite database must be at the same round as the postgres database
or one round behind; otherwise, the import validator will fail to start.

# Performance

Reading and writing to/from the sqlite database is negligible compared to
importing blocks into the postgres database.
However, the tool has to read the modified accounts after importing each block.
Thus, we can expect the import validator to be 1.5 times slower than indexer.
8 changes: 0 additions & 8 deletions cmd/import-validator/main.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,3 @@
// The import validator tool imports blocks into indexer database and algod's sqlite
// database in lockstep and checks that the modified accounts are the same in the two
// databases. It lets detect the first round where an accounting discrepancy occurs
// and it prints out what the difference is before crashing.
// There is a small limitation, however. The set of modified accounts is computed using
// the sqlite database. Thus, if indexer's accounting were to modify a superset of
// those accounts, this tool would not detect it. This, however, should be unlikely.

package main

import (
Expand Down