Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slashing: supply invariance broken #2619

Closed
cwgoes opened this issue Oct 28, 2018 · 4 comments
Closed

Slashing: supply invariance broken #2619

cwgoes opened this issue Oct 28, 2018 · 4 comments

Comments

@cwgoes
Copy link
Contributor

cwgoes commented Oct 28, 2018

Caught by #2617 (multi-seed simulation ranges over random genesis parameter combinations).

Replicable with:

go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=989182 -v -timeout 24h

on 3884b13

Fails with:

I[10-29|00:14:53.709] Absent validator 027E8AF0A5D7BC9508C550BA4097BA219703B32F at height 3, 3 missed, threshold 276 module=x/slashing
I[10-29|00:14:53.709] Absent validator 0F8DCCE06C47BC5313C334DCA3DC5092FB4AA140 at height 3, 1 missed, threshold 276 module=x/slashing 
I[10-29|00:14:53.709] Confirmed double sign from 33A2A55325BB76E45D801F94D8A853FE40FB687D at height 3, age of 0 less than max age of 214588000000000 module=x/slashing
I[10-29|00:14:53.709] Fraction slashed capped by slashing period from 500000000 to 500000000 module=x/slashing 
I[10-29|00:14:53.710] validator cosmosvaloper1xw3225e9hdmwghvqr72d32znleq0k6ra44kg0y slashed by slash factor of 0.0500000000; burned 223980000000000 tokens module=x/stake
I[10-29|00:14:53.710] validator cosmosvalcons1xw3225e9hdmwghvqr72d32znleq0k6rapx95r9 jailed module=x/stake 
I[10-29|00:14:53.710] Confirmed double sign from 0F8DCCE06C47BC5313C334DCA3DC5092FB4AA140 at height 0, age of 14568000000000 less than max age of 214588000000000 module=x/slashing
I[10-29|00:14:53.710] Fraction slashed capped by slashing period from 500000000 to 500000000 module=x/slashing 
I[10-29|00:14:53.710] validator cosmosvaloper1p7xuecrvg779xy7rxnw28hzsjta54g2qltg0m4 slashed by slash factor of 0.0500000000; burned 203604000000000 tokens module=x/stake
I[10-29|00:14:53.710] validator cosmosvalcons1p7xuecrvg779xy7rxnw28hzsjta54g2qtcmnh5 jailed module=x/stake 
Invariants broken after BeginBlock
bonded token invariance:
  pool.BondedTokens: 188465635000000000
  sum of account tokens: 188467622500000000
Begin block 0
@alexanderbez
Copy link
Contributor

welp, I cannot reproduce this with the given seed/command. I created a new branch off of the latest develop and merged in your changes and it passed every time (I tried 6 or so times).

I did however, rerun the CI pipeline and failures still do occur in the multi-seed and the fast sim. I do not see this error however. So I will be closing for now unless it occurs again.

Perhaps we have some non-determinism again.

@cwgoes
Copy link
Contributor Author

cwgoes commented Nov 1, 2018

welp, I cannot reproduce this with the given seed/command. I created a new branch off of the latest develop and merged in your changes and it passed every time (I tried 6 or so times).

Did you try the specific commit referenced? Likely the specific seeds required on develop have changed, since we've merged some PRs since which change the simulation a bit (but that didn't address whatever the underlying cause of this problem is).

Like so:

git checkout 3884b13c004d1cd1936575f03b3e31d5bd07feb9
go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=989182 -v -timeout 24h

I really don't think this is non-determinism, I can replicate it multiple times on different machines.

@cwgoes cwgoes reopened this Nov 1, 2018
@alexanderbez
Copy link
Contributor

You're correct. I should work off of that exact commit. I don't think anything committed recently went into addressing this potentially.

@rigelrozanski
Copy link
Contributor

I would just run through a whole bunch of different seeds - I bet a a failing seed exists in < 100

jaekwon pushed a commit that referenced this issue Nov 5, 2018
* Print out initial update on every block
* Randomize simulation parameters
* Randomize initial liveness weightings
* Randomize genesis parameters
* fixed power store invariant
* IterateValidatorsBonded -> IterateBondedValidatorsByPower
* WriteValidators uses IterateLastValidators rather than IterateBondedValidatorsByPower
* fixed democoin interface

Closes #2556
Closes #2396

Via #2671:
closes #2669
closes #2670
closes #2620

Offshoot issues:
#2618
#2619
#2620
#2661
@cwgoes cwgoes closed this as completed Nov 5, 2018
MarcelMWS added a commit to MarcelMWS/cosmos-sdk that referenced this issue Nov 15, 2018
* Back to 50 initially bonded

* Add query deposits cmds

* Update test

* Update PENDING.md

* Back to on-operation=false, update PENDING.md

* Remove unnecessary change, clarify amt in sim_test.go

* Cleanup, additional item in PENDING.md

* Update PENDING.md

Co-Authored-By: alessio <[email protected]>

* Update client/keys/utils.go

Co-Authored-By: alessio <[email protected]>

* update depositer addr

* Address @rigelrozanski comments

* Make linter happy

* Update PENDING.md

* Remove shorthand

* Make "multi" const

* Store last total power as sdk.Int, not sdk.Dec

* Merge PR cosmos#2553: Renamed msg.Name() and msg.Type() to msg.Type() and msg.Route()

* Fix stupid mistake

* s/number/weight/

* LastValidatorPower is also an Int

* Merge PR cosmos#2462: Add swagger-ui for gov, stake and slashing

* 'make format'

* Move PENDING to CHANGELOG

* Linkify changelog

* Fix db write perm

* Additional cleanup

* Remove logs from prior debugging

* Merge PR cosmos#2599 from cosmos/jae/dist_refactor

* Rename Pool -> DelRewards; PoolCommission -> ValCommision
* FeePool.Pool -> FeePool.ValPool
* WithdrawalHeight->DelPoolWithdrawalHeight
* OnValidatorBeginUnbonding
* Caught the bug's tail
* Update vi.FeePoolWithdrawalHeight upon bonding
* Fix staking slashUnbondingDelegation bug; fixes simulator failure cosmos#9

* Merge PR cosmos#2597: Add distribution accum invariants

* PENDING.md => CHANGELOG.md

* Manually linkify

* Manually fix some links

* Docs fixes in progress while running through the release process

* More docs fixes

* '--voter' is no longer required

* Rectify validator setup documentation

* Merge PR cosmos#2596: Cmds for validator unbondings and redelegations

* Make simulation use a transition matrix for block size

This enables simulating periods of high load, and periods of low to no load.
(low load because future ops will still terminate in that time frame)

* address bez's comments

* fix flags in docs, closes cosmos#2530

* Merge PR cosmos#2616: Block redelegations to the same validator

* Merge PR cosmos#2623: Speedup simulator by switching to goleveldb

Due to requiring app.Commit() at the moment, golevel db is significantly faster than a memdb

* fix block offsets in printing simulation block number

* Merge PR cosmos#2644: Simulation: Print last block when there is an error

There was an off by one error in the log printing function previously

* Merge PR cosmos#2642: Add todo diagrams

* Various sign command improvements

- Exit with error if the user is attempting to sign with a key
  whose address is not among those who are expected to sign
  the transaction.

- Add --print-signature-only to output only the generated
  signature.

* Check sanity of signatures and report errors when run with --print-sigs

* Improve errors reporting

* Improve online docs

* Refresh PENDING.md

* Find better name for --print-signature-only

* Fix integration tests

* Validate --name

* Fix integration tests

* s/--print-sigs/--validate-signatures/

* s/--sig-only/--signature-only/

* Docs updated

* Update PENDING.md

* Rename append, it's go builtin

* Set success = false when it fails

* Apply suggestions from bez

* Nest switches

* Fix rebase

* Document what --validate-signatures does

* perform minor doc and function cleanup

* move typedef

* Merge PR cosmos#2614: Configurable Bech32 prefix for SDK users

* Merge PR cosmos#2643: AppendTag function usage error. append elements do not work

* simulation: Make validator choice use validator set

This also had to change the default seed, since with the previous one it
actually got into a state where there were no validators left bonded, lol.

This also changes Unbond msgs from failing with almost 100% probability to now
only failing with 33% probability.
Thus more of the state machine is getting tested!

* Update changelog

* Merge PR cosmos#2657: Fix config.js

* Merge PR cosmos#2589: Update Vesting Spec

* Merge PR cosmos#2656: Revert read-only leveldb database

* Revert read-only leveldb database

Waiting on a fix for syndtr/goleveldb#240.

* Update client/keys/utils.go

* Include DNS alt name in certificate

Closes: cosmos#2664

* Gaialite signal handling is broken, repair it

* Merge PR cosmos#2665: simulation: Remove header from Invariant

This got introduced recently, but wasn't actually needed, hence the reversion

* Merge PR cosmos#2653: Add benchmark for get and set account

* Fix test

* Refactor TrapSignal

* Fix lint

* enforcing @jaekwon mergemaster

* added querier redelegation

* added validatorDelegations querier endpoint

* LCD and CLI

* cli fixes

* removed redelegation stuff

* address other comments

* rebased

* addressed comments

* Make the simulator create the new comission rate sensibly

* Update to TM v0.26.0 - Part I (cosmos#2679)

* Update to TM v0.26.0

* Bez/tm0.26 update pt 2 redux (cosmos#2684)

* Update to TM v0.26.0
* Update TODOs
* Proof and verification updates
* Fix linting
* Fix key path creation
* Temporarily fix tendermint revision to make tests pass

* Fix merge conflict bug; Update PENDING

* New genesis workflow (cosmos#2602)

New genesis workflow:
* `gaiad init` is now used to generate an empty `genesis.json`.
* Genesis accounts need to be populated manually before running
  `gaiad collect-gentxs`.
* This should support starfish too, see cosmos#2615 for more info.
* Closes: cosmos#2596 cosmos#2615
* Validate validator address and address against respective account ex ante
* Fix local testnet failures
* New genesis tests
* Run make format
* Add --pubkey flag
* gaiad collect-gentxs takes no args

* Simulation improvements (logging fix, random genesis parameters) (cosmos#2617)

* Print out initial update on every block
* Randomize simulation parameters
* Randomize initial liveness weightings
* Randomize genesis parameters
* fixed power store invariant
* IterateValidatorsBonded -> IterateBondedValidatorsByPower
* WriteValidators uses IterateLastValidators rather than IterateBondedValidatorsByPower
* fixed democoin interface

Closes cosmos#2556
Closes cosmos#2396

Via cosmos#2671:
closes cosmos#2669
closes cosmos#2670
closes cosmos#2620

Offshoot issues:
cosmos#2618
cosmos#2619
cosmos#2620
cosmos#2661

* Fix simulation bugs; Incorprates cosmos#2676 from Sunny (cosmos#2677)

* Fix simulation bugs; Incorprates cosmos#2676 from Sunny
* Address review feedback; Update PENDING

* 'make format'

* Revert "enforcing @jaekwon mergemaster"

This reverts commit 15c2093.

* Update x/stake/client/rest/query.go

Co-Authored-By: sunnya97 <[email protected]>

* addressed fede's comment

* Switch gov proposal-queues to use iterators (cosmos#2638)

* switched gov proposals queue to use iterators
* update gov spec
* update proposal.Equal
* Amino api change
* switched proposalID to uint64
* renamed Gov Procedures to Params
* s/ActiveProposalQueueProposalKey/KeyActiveProposalQueueProposal/g
* numLatestProposals -> Limit
* fixed staking invariant breakage because of gov deposits
* Send deposits to DepositedCoinsAccAddr or BurnedDepositCoinsAccAddr

* Add general merkle absence proof (also for empty substores) (cosmos#2685)

* Fix coins.IsLT() impl (cosmos#2686)

* Fix coins.IsLT() impl
* Fix coin.IsLT() impl
* Coins.IsLT -> Coins.IsAllLT etc

* Update testnet to use canonical genesis time (cosmos#2692)

* Update testnet to use canonical genesis time
* Fix linting in genesis test

* Do not allow nil values to be set in CacheKVStore (cosmos#2708)

* Do not allow nil values to be set in CacheKVStore

* Makefile OS compatibility update

* Merge PR cosmos#2714: Add commission data to MsgCreateValidator signature bytes

* PENDING => CHANGELOG

* Linkify changelog

* Cleanup bank keeper

* whitespacing

* rand utile

...

* moving stuff around a bit, trying to get rid of types

* reorganize more

* rename ambig naming of queueOperations

* minimizing indentation

* fix some duplicate to get passing

* Address style comments

* Reorganize CLI command structure. Fixes cosmos#2575

* Fix missing flags issue

* Address linting issues

* Fix gobash CLI testing

* Fix typo

* Cross-compiling get_tools Makefile added

* operations functions

* assertAllInvarients changes, Operation reorg

* mock tendermint

* util cleanup

* event stats object, more general cleanup

* compiling

* pending

* Removed comment from Makefile as per bez's request

* val comments

* Address PR comments

* Update cmd/gaia/cmd/gaiacli/main.go

Co-Authored-By: jackzampolin <[email protected]>

* PENDING

* Fix state export/import, add to CI (cosmos#2690)

* Update slashing import/export
* More slashing.WriteGenesis
* Add test import/export to CI
* Store equality comparison.
* Fix validator bond intra-tx counter
* Set timeslices for unbonding validators
* WriteGenesis => ExportGenesis
* Delete validators from unbonding queue when re-bonded
* Hook for validator deletion, fix staking genesis tests

* Merge 0.26.0 back to develop (cosmos#2718)

* PENDING => CHANGELOG
* Linkify changelog
* Merge PR cosmos#2716: Temporarily disable gaia lite insecure mode
* TODO: need to update CHANGELOG w/ import-export PR cosmos#2690

* Update CHANGELOG/PENDING for straggling PR cosmos#2690

* Add small utility to add account to genesis.json after gaiad init

* Update CHANGELOG.md

* s/WriteGenesisFile/ExportGenesisFile/

* Update PENDING.md

* Add --chain-id to testnet command

* Address remaining comments from cosmos#2690

* Update PENDING.md

* add back in PeriodicInvariant

* Linter fix

* Fix TimeoutCommit (cosmos#2743)

* Fix TimeoutCommit to 5 seconds instead of whatever it was before which was too short.

* Gaia-9000: Update to TM 0.26.1-rc2 (cosmos#2753)

* Update to tm 0.26.1-rc2 to fix prometheus issue and node disconnect issue.

* Gaia-9000: Update to TM 0.26.1-rc3 -- pex SeedMode fix

* fix typo

I think it might be a spelling mistake

* Slight distribution spec cleanup

* More cleanup

* use defer

* Use correct Bech32 prefix for show-address command (cosmos#2746)

* Use consensus address bech32 prefix
* Update show-address CLI description

* Generate random moniker when missing

* Update moniker prefix

* Require moniker instead of generating a random one

* update to tendermint v0.26.1

* Fix test coverage

* Correctly set return code

* Fix date to be cross platform

* Merge PR cosmos#2752: Don't hardcode bondable denom

* R4R: Fix unbonding command flow (cosmos#2727)

* Fix required flag

* Fix redelegation command

* Add pending entry

* update swagger.yaml

* use newQuery...Params

* Link to issue

* Fix DiffKVStore

* Address PR review

* Working on stake import/export

* Only apply validator set updates on initial genesis

* Clarify comment

* Fix failing test

* add back in CLI command after rebase

* Fix CLI tests

* update to amino 0.14.1

* pending

* R4R:  Query Gov Params (cosmos#2576)

* gov query params

* Merge PR cosmos#2744: Fix Makefile targets dependencies

* Fix Makefile targets dependencies
* Remove unnecessary build deps from install targets
* Create a rule for each tool
* Don't dep test_lint on tools

* Update docs/spec/distribution/overview.md

Co-Authored-By: alexanderbez <[email protected]>

* Update docs/spec/distribution/overview.md

Co-Authored-By: alexanderbez <[email protected]>

* Update docs/spec/distribution/overview.md

Co-Authored-By: alexanderbez <[email protected]>

* Update overview.md

* Documentation Structure Change and Cleanup (cosmos#2808)

* Update docs/sdk/clients.md
* organize ADR directory like tendermint
* docs: move spec-proposals into spec/
* remove lotion, moved to website repo
* move getting-started to cosmos-hub, and voyager to website
* docs: move lite/ into clients/lite/
* move introduction/ content to website repo
* move resources/ content to website repo
* mv sdk/clients.md to clients/clients.md
* mv validators to cosmos-hub/validators
* move deprecated sdk/ content to _attic
* sdk/modules.md is duplicate with modules/README.md
* consolidate remianing sdk/ files into a single sdk.md
* move examples/ to docs/examples/
* mv docs/cosmos-hub to docs/gaia
* Add keys/accounts section to localnet docs

* Bring back banner (cosmos#2814)

* Build docs in CircleCI  (cosmos#2810)

* error checking the API call
* added docs build trigger to circleci job

* Update contributing.md with new merge policy (cosmos#2789)

* Update contribuiting.md with new merge policy

* deleted obsolete file (cosmos#2817)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants