celestia-appd export broken on main

[rootulp]

Problem

On a v1.x node (specifically v1.8.0): I run single-node.sh for two blocks and then export:

$ celestia-appd export
{"app_hash":"","app_state":{"auth":{"accounts":[{"@type":"/cosmos.auth.v1beta1.ModuleAccount","base_account":{"account_number":"3","address":"celestia1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3y3clr6","pub_key":null,"sequence":"0"},"name":"bonded_tokens_pool","permissions":["burner","staking"]},{"@type":"/cosmos.auth.v1beta1.ModuleAccount","base_account":{"account_number":"4","address":"celestia1tygms3xhhs3yv487phx3dw4a95jn7t7ls3yw4w","pub_key":null,"sequence":"0"},"name":"not_bonded_tokens_pool","permissions":["burner","staking"]},{"@type":"/cosmos.auth.v1beta1.ModuleAccount","base_account":{"account_number":"5","address":"celestia10d07y265gmmuvt4z0w9aw880jnsr700jtgz4v7","pub_key":null,"sequence":"0"},"name":"gov","permissions":["burner"]},{"@type":"/cosmos.auth.v1beta1.ModuleAccount","base_account":{"account_number":"2","address":"celestia1jv65s3grqf6v6jl3dp4t6c9t9rk99cd8k44vnj","pub_key":null,"sequence":"0"},"name":"distribution","permissions":[]},{"@type":"/cosmos.auth.v1beta1.ModuleAccount","base_account":{"account_number":"6","address":"celestia1m3h30wlvsf8llruxtpukdvsy0km2kum8emkgad","pub_key":null,"sequence":"0"},"name":"mint","permissions":["minter"]},{"@type":"/cosmos.auth.v1beta1.ModuleAccount","base_account":{"account_number":"1","address":"celestia17xpfvakm2amg962yls6f84z3kell8c5lpnjs3s","pub_key":null,"sequence":"0"},"name":"fee_collector","permissions":[]},{"@type":"/cosmos.auth.v1beta1.BaseAccount","account_number":"0","address":"celestia17a4w3k4auujd9sjdqj8fnwpymg9ve0ttulm7xs","pub_key":{"@type":"/cosmos.crypto.secp256k1.PubKey","key":"Am03r++PXommier+JdZf8YU9xMSaFKAjC9hlYtxK3C59"},"sequence":"2"}],"params":{"max_memo_characters":"256","sig_verify_cost_ed25519":"590","sig_verify_cost_secp256k1":"1000","tx_sig_limit":"7","tx_size_cost_per_byte":"10"}},"authz":{"authorization":[]},"bank":{"balances":[{"address":"celestia1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3y3clr6","coins":[{"amount":"5000000000","denom":"utia"}]},{"address":"celestia1jv65s3grqf6v6jl3dp4t6c9t9rk99cd8k44vnj","coins":[{"amount":"28988863","denom":"utia"}]},{"address":"celestia17xpfvakm2amg962yls6f84z3kell8c5lpnjs3s","coins":[{"amount":"30000","denom":"utia"}]},{"address":"celestia17a4w3k4auujd9sjdqj8fnwpymg9ve0ttulm7xs","coins":[{"amount":"999994999970000","denom":"utia"}]}],"denom_metadata":[{"base":"utia","denom_units":[{"aliases":["microtia"],"denom":"utia","exponent":0},{"aliases":[],"denom":"TIA","exponent":6}],"description":"The native token of the Celestia network.","display":"TIA","name":"TIA","symbol":"TIA","uri":"","uri_hash":""}],"params":{"default_send_enabled":true,"send_enabled":[]},"supply":[{"amount":"1000000028988863","denom":"utia"}]},"blob":{"params":{"gas_per_blob_byte":8,"gov_max_square_size":"64"}},"capability":{"index":"2","owners":[{"index":"1","index_owners":{"owners":[{"module":"ibc","name":"ports/transfer"},{"module":"transfer","name":"ports/transfer"}]}}]},"crisis":{"constant_fee":{"amount":"1000","denom":"utia"}},"distribution":{"delegator_starting_infos":[{"delegator_address":"celestia17a4w3k4auujd9sjdqj8fnwpymg9ve0ttulm7xs","starting_info":{"height":"0","previous_period":"1","stake":"5000000000.000000000000000000"},"validator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk"}],"delegator_withdraw_infos":[],"fee_pool":{"community_pool":[{"amount":"579777.260000000000000000","denom":"utia"}]},"outstanding_rewards":[{"outstanding_rewards":[{"amount":"28409085.740000000000000000","denom":"utia"}],"validator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk"}],"params":{"base_proposer_reward":"0.000000000000000000","bonus_proposer_reward":"0.000000000000000000","community_tax":"0.020000000000000000","withdraw_addr_enabled":true},"previous_proposer":"celestiavalcons1tksk79xcg5e699hk7cyg68sv4gtghxu4u36vy6","validator_accumulated_commissions":[{"accumulated":{"commission":[{"amount":"2840908.574000000000000000","denom":"utia"}]},"validator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk"}],"validator_current_rewards":[{"rewards":{"period":"2","rewards":[{"amount":"25568177.166000000000000000","denom":"utia"}]},"validator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk"}],"validator_historical_rewards":[{"period":"1","rewards":{"cumulative_reward_ratio":[],"reference_count":2},"validator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk"}],"validator_slash_events":[]},"evidence":{"evidence":[]},"feegrant":{"allowances":[]},"genutil":{"gen_txs":[]},"gov":{"deposit_params":{"max_deposit_period":"604800s","min_deposit":[{"amount":"10000000000","denom":"utia"}]},"deposits":[],"proposals":[],"starting_proposal_id":"1","tally_params":{"quorum":"0.334000000000000000","threshold":"0.500000000000000000","veto_threshold":"0.334000000000000000"},"votes":[],"voting_params":{"voting_period":"604800s"}},"ibc":{"channel_genesis":{"ack_sequences":[],"acknowledgements":[],"channels":[],"commitments":[],"next_channel_sequence":"0","receipts":[],"recv_sequences":[],"send_sequences":[]},"client_genesis":{"clients":[],"clients_consensus":[],"clients_metadata":[],"create_localhost":false,"next_client_sequence":"0","params":{"allowed_clients":["06-solomachine","07-tendermint"]}},"connection_genesis":{"client_connection_paths":[],"connections":[],"next_connection_sequence":"0","params":{"max_expected_time_per_block":"75000000000"}}},"mint":{"bond_denom":"utia"},"params":null,"qgb":{"params":{"data_commitment_window":"400"}},"slashing":{"missed_blocks":[{"address":"celestiavalcons1tksk79xcg5e699hk7cyg68sv4gtghxu4u36vy6","missed_blocks":[]}],"params":{"downtime_jail_duration":"60s","min_signed_per_window":"0.750000000000000000","signed_blocks_window":"5000","slash_fraction_double_sign":"0.020000000000000000","slash_fraction_downtime":"0.000000000000000000"},"signing_infos":[{"address":"celestiavalcons1tksk79xcg5e699hk7cyg68sv4gtghxu4u36vy6","validator_signing_info":{"address":"celestiavalcons1tksk79xcg5e699hk7cyg68sv4gtghxu4u36vy6","index_offset":"1","jailed_until":"1970-01-01T00:00:00Z","missed_blocks_counter":"0","start_height":"0","tombstoned":false}}]},"staking":{"delegations":[{"delegator_address":"celestia17a4w3k4auujd9sjdqj8fnwpymg9ve0ttulm7xs","shares":"5000000000.000000000000000000","validator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk"}],"exported":true,"last_total_power":"5000","last_validator_powers":[{"address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk","power":"5000"}],"params":{"bond_denom":"utia","historical_entries":10000,"max_entries":7,"max_validators":100,"min_commission_rate":"0.050000000000000000","unbonding_time":"1814400s"},"redelegations":[],"unbonding_delegations":[],"validators":[{"commission":{"commission_rates":{"max_change_rate":"0.010000000000000000","max_rate":"0.200000000000000000","rate":"0.100000000000000000"},"update_time":"2024-04-26T18:24:30.919087Z"},"consensus_pubkey":{"@type":"/cosmos.crypto.ed25519.PubKey","key":"oJhc+QzY5gwWJRopHl2Jt7VL2mOUJMXC5N14XoFDv1s="},"delegator_shares":"5000000000.000000000000000000","description":{"details":"","identity":"","moniker":"private","security_contact":"","website":""},"jailed":false,"min_self_delegation":"1","operator_address":"celestiavaloper17a4w3k4auujd9sjdqj8fnwpymg9ve0tteqe8sk","status":"BOND_STATUS_BONDED","tokens":"5000000000","unbonding_height":"0","unbonding_time":"1970-01-01T00:00:00Z"}]},"transfer":{"denom_traces":[],"params":{"receive_enabled":true,"send_enabled":true},"port_id":"transfer"},"vesting":{}},"chain_id":"private","consensus_params":{"block":{"max_bytes":"1974272","max_gas":"-1","time_iota_ms":"1"},"evidence":{"max_age_duration":"1814400000000000","max_age_num_blocks":"120961","max_bytes":"1048576"},"validator":{"pub_key_types":["ed25519"]},"version":{}},"genesis_time":"2024-04-26T18:24:30.919087Z","initial_height":"3","validators":[{"address":"5DA16F14D84533A296F6F6088D1E0CAA168B9B95","name":"private","power":"5000","pub_key":{"type":"tendermint/PubKeyEd25519","value":"oJhc+QzY5gwWJRopHl2Jt7VL2mOUJMXC5N14XoFDv1s="}}]}

On main, the same actions result in a panic.

$ celestia-appd export
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x8 pc=0x102ea0f14]

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).NewContext(_, _, {{0x0, 0x0}, {0x0, 0x0}, 0x35, {0x0, 0x0, 0x0}, ...})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/[email protected]/baseapp/test_helpers.go:42 +0x44
github.com/celestiaorg/celestia-app/v2/app.(*App).ExportAppStateAndValidators(0x140011b0008, 0x0, {0x1063d0fc0, 0x0, 0x0})
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/app/export.go:22 +0x9c
github.com/celestiaorg/celestia-app/v2/cmd/celestia-appd/cmd.createAppAndExport({0x104fbf8d0, 0x1400133acb0}, {0x104fd1cc0, 0x14000306108}, {0x0, 0x0}, 0xffffffffffffffff, 0x0, {0x1063d0fc0, 0x0, ...}, ...)
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/cmd/celestia-appd/cmd/root.go:274 +0x258
github.com/cosmos/cosmos-sdk/server.ExportCmd.func1(0x14001350908, {0x1063d0fc0?, 0x0?, 0x0?})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/[email protected]/server/export.go:70 +0x3cc
github.com/spf13/cobra.(*Command).execute(0x14001350908, {0x1063d0fc0, 0x0, 0x0})
	/Users/rootulp/go/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x14001253b08)
	/Users/rootulp/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/rootulp/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/Users/rootulp/go/pkg/mod/github.com/spf13/[email protected]/command.go:1032
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x14001253b08, {0x103d9d691, 0x8}, {0x140006d29c0, 0x1c})
	/Users/rootulp/go/pkg/mod/github.com/celestiaorg/[email protected]/server/cmd/execute.go:36 +0x158
main.main()
	/Users/rootulp/git/rootulp/celestiaorg/celestia-app/cmd/celestia-appd/main.go:14 +0x3c

[rootulp]

Just confirmed, still present on main

[rootulp]

On v1.x, app.checkState is non-nil when exporting:

3:28PM INF app.checkState &{{0x140005d4800 map[0x14000614480:0x140005d4fc0 0x14000614490:0x140005d4f00 0x140006144a0:0x140005d4b40 0x140006144b0:0x140005d4d80 0x140006144c0:0x140005d4b00 0x140006144d0:0x140005d4d00 0x140006144e0:0x140005d4880 0x140006144f0:0x140005d4d40 0x14000614500:0x140005d4a80 0x14000614510:0x140005d48c0 0x14000614520:0x140005d49c0 0x14000614530:0x140005d4e80 0x14000614540:0x140005d4a40 0x14000614550:0x140005d4dc0 0x14000614560:0x140005d4ec0 0x14000614570:0x140005d4900 0x14000614580:0x140005d4840 0x140006145c0:0x140005d4a00 0x140006145a0:0x140005d4ac0] map[acc:0x14000614480 authz:0x14000614490 bank:0x140006144a0 blob:0x14000614550 capability:0x14000614540 distribution:0x140006144d0 evidence:0x14000614530 feegrant:0x14000614520 gov:0x140006144f0 ibc:0x14000614580 mem_capability:0x140006145c0 mint:0x140006144c0 params:0x14000614500 qgb:0x14000614560 slashing:0x140006144e0 staking:0x140006144b0 transfer:0x14000614570 transient_params:0x140006145a0 upgrade:0x14000614510] <nil> map[]} {{{}} {0x140005d4800 map[0x14000614480:0x140005d4fc0 0x14000614490:0x140005d4f00 0x140006144a0:0x140005d4b40 0x140006144b0:0x140005d4d80 0x140006144c0:0x140005d4b00 0x140006144d0:0x140005d4d00 0x140006144e0:0x140005d4880 0x140006144f0:0x140005d4d40 0x14000614500:0x140005d4a80 0x14000614510:0x140005d48c0 0x14000614520:0x140005d49c0 0x14000614530:0x140005d4e80 0x14000614540:0x140005d4a40 0x14000614550:0x140005d4dc0 0x14000614560:0x140005d4ec0 0x14000614570:0x140005d4900 0x14000614580:0x140005d4840 0x140006145c0:0x140005d4a00 0x140006145a0:0x140005d4ac0] map[acc:0x14000614480 authz:0x14000614490 bank:0x140006144a0 blob:0x14000614550 capability:0x14000614540 distribution:0x140006144d0 evidence:0x14000614530 feegrant:0x14000614520 gov:0x140006144f0 ibc:0x14000614580 mem_capability:0x140006145c0 mint:0x140006144c0 params:0x14000614500 qgb:0x14000614560 slashing:0x140006144e0 staking:0x140006144b0 transfer:0x14000614570 transient_params:0x140006145a0 upgrade:0x14000614510] <nil> map[]} {{0 0}  0 {0 0 <nil>} {[] {0 []}} [] [] [] [] [] [] [] [] []} []  [] {{{{0x14000132028 false  [] [] [] <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}} 1 <nil> [123] [{}] false <nil>}} [] 0x14000011130 <nil> true false [] <nil> 0x14000a26288 0 {1000 1000 1000 3 2000 30 30} {100 100 100 0 200 3 3}}}

on main, app.checkState is nil when exporting:

3:06PM INF app.checkState <nil>

Notably, both main and v1.x have app.checkState populated after init chain and on each block height so it seems like something on main has made app.checkState even more volatile in that it gets dropped when the node stops.

[rootulp]

I have a suspicion that 9db2f8b is related because the app no longer uses loadLatest anymore. See https://github.com/celestiaorg/celestia-app/blame/main/cmd/celestia-appd/cmd/root.go#L265 which previously called baseApp.Init which calls app.setCheckState.

[rootulp]

1. We need a unit test for export functionality b/c this should have been caught earlier*.
2. If we add back the LoadLatestVersion() invocation here:

	if err := app.LoadLatestVersion(); err != nil {
		panic(fmt.Sprintf("loading latest version: %s", err.Error()))
	}

then we get another nil pointer because the modules haven't been initialized in the versioned module manager 😞

[rootulp]

Found one bug in the module manager where it attempts to export modules that aren't in the currently supported app version. This results in a panic when invoking ExportGenesis on the unsupported module.