[bug]: unit flake: race in custodian from proof import

[jharveyb]

Background

Unit test failure in the custodian tests from a CI run for #768:

--- FAIL: TestTransactionConfirmedOnly (4.77s)
    test_postgres.go:11: Creating new Postgres DB for testing
    custodian_test.go:220: 
        	Error Trace:	/home/runner/work/taproot-assets/taproot-assets/tapgarden/custodian_test.go:220
        	            				/home/runner/work/taproot-assets/taproot-assets/tapgarden/custodian_test.go:740
        	            				/home/runner/work/taproot-assets/taproot-assets/tapgarden/custodian_test.go:669
        	Error:      	Received unexpected error:
        	            	event 0 has status 1 but wanted 3
        	Test:       	TestTransactionConfirmedOnly
FAIL
FAIL	github.com/lightninglabs/taproot-assets/tapgarden	31.421s
FAIL

Your environment

branch main
commit d37b48636515eaa5346bf07347fae8ec21f7a958

Steps to reproduce

Running the race detector locally for the tapgarden tests:

main unit-race pkg=tapgarden timeout=2m

We see a failure in the same test:

WARNING: DATA RACE
Write at 0x00c0004e2500 by goroutine 3770:
  github.com/lightninglabs/taproot-assets/proof.(*MultiArchiver).ImportProofs.func1()
      /home/jhb/tap/review_repo/tap/proof/archive.go:811 +0x288
  github.com/lightninglabs/taproot-assets/fn.ParSlice[go.shape.*uint8].func1()
      /home/jhb/tap/review_repo/tap/fn/concurrency.go:29 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/jhb/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x76

Previous read at 0x00c0004e2500 by goroutine 3638:
  github.com/lightninglabs/taproot-assets/tapdb.(*AssetStore).importAssetFromProof()
      /home/jhb/tap/review_repo/tap/tapdb/assets_store.go:1464 +0x7ae
  github.com/lightninglabs/taproot-assets/tapdb.(*AssetStore).ImportProofs.func1()
      /home/jhb/tap/review_repo/tap/tapdb/assets_store.go:1601 +0x10d
  github.com/lightninglabs/taproot-assets/tapdb.(*TransactionExecutor[go.shape.interface { AnchorGenesisPoint(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.AnchorGenesisPointParams) error; ApplyPendingOutput(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.ApplyPendingOutputParams) (int64, error); ConfirmChainAnchorTx(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.ConfirmChainAnchorTxParams) error; DeleteAssetWitnesses(context.Context, int64) error; DeleteExpiredUTXOLeases(context.Context, database/sql.NullTime) error; DeleteManagedUTXO(context.Context, []uint8) error; DeleteUTXOLease(context.Context, []uint8) error; FetchAssetMetaByHash(context.Context, []uint8) (github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetMetaByHashRow, error); FetchAssetMetaForAsset(context.Context, []uint8) (github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetMetaForAssetRow, error); FetchAssetProof(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetProofParams) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetProofRow, error); FetchAssetProofs(context.Context) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetProofsRow, error); FetchAssetProofsByAssetID(context.Context, []uint8) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetProofsByAssetIDRow, error); FetchAssetWitnesses(context.Context, database/sql.NullInt64) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchAssetWitnessesRow, error); FetchChainTx(context.Context, []uint8) (github.com/lightninglabs/taproot-assets/tapdb/sqlc.ChainTxn, error); FetchGenesisID(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchGenesisIDParams) (int64, error); FetchGroupedAssets(context.Context) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchGroupedAssetsRow, error); FetchManagedUTXO(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchManagedUTXOParams) (github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchManagedUTXORow, error); FetchManagedUTXOs(context.Context) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchManagedUTXOsRow, error); FetchScriptKeyIDByTweakedKey(context.Context, []uint8) (int64, error); FetchTransferInputs(context.Context, int64) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchTransferInputsRow, error); FetchTransferOutputs(context.Context, int64) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.FetchTransferOutputsRow, error); HasAssetProof(context.Context, []uint8) (bool, error); InsertAssetTransfer(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.InsertAssetTransferParams) (int64, error); InsertAssetTransferInput(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.InsertAssetTransferInputParams) error; InsertAssetTransferOutput(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.InsertAssetTransferOutputParams) error; InsertNewAsset(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.InsertNewAssetParams) (int64, error); InsertPassiveAsset(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.InsertPassiveAssetParams) error; LogProofTransferAttempt(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.LogProofTransferAttemptParams) error; QueryAssetBalancesByAsset(context.Context, []uint8) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryAssetBalancesByAssetRow, error); QueryAssetBalancesByGroup(context.Context, []uint8) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryAssetBalancesByGroupRow, error); QueryAssetTransfers(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryAssetTransfersParams) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryAssetTransfersRow, error); QueryAssets(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryAssetsParams) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryAssetsRow, error); QueryPassiveAssets(context.Context, int64) ([]github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryPassiveAssetsRow, error); QueryProofTransferAttempts(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.QueryProofTransferAttemptsParams) ([]time.Time, error); ReAnchorPassiveAssets(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.ReAnchorPassiveAssetsParams) error; SetAssetSpent(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.SetAssetSpentParams) (int64, error); UpdateUTXOLease(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpdateUTXOLeaseParams) error; UpsertAssetGroupKey(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertAssetGroupKeyParams) (int64, error); UpsertAssetGroupWitness(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertAssetGroupWitnessParams) (int64, error); UpsertAssetMeta(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertAssetMetaParams) (int64, error); UpsertAssetProof(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertAssetProofParams) error; UpsertAssetProofByID(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertAssetProofByIDParams) error; UpsertAssetWitness(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertAssetWitnessParams) error; UpsertChainTx(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertChainTxParams) (int64, error); UpsertGenesisAsset(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertGenesisAssetParams) (int64, error); UpsertGenesisPoint(context.Context, []uint8) (int64, error); UpsertInternalKey(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertInternalKeyParams) (int64, error); UpsertManagedUTXO(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertManagedUTXOParams) (int64, error); UpsertScriptKey(context.Context, github.com/lightninglabs/taproot-assets/tapdb/sqlc.UpsertScriptKeyParams) (int64, error) }]).ExecTx()
      /home/jhb/tap/review_repo/tap/tapdb/interfaces.go:241 +0x2c8
  github.com/lightninglabs/taproot-assets/tapdb.(*TransactionExecutor[github.com/lightninglabs/taproot-assets/tapdb.ActiveAssetsStore]).ExecTx()
      /home/jhb/tap/review_repo/tap/tapdb/interfaces.go:205 +0x7c
  github.com/lightninglabs/taproot-assets/tapdb.(*AssetStore).ImportProofs()
      /home/jhb/tap/review_repo/tap/tapdb/assets_store.go:1592 +0x1ed
  github.com/lightninglabs/taproot-assets/proof.(*MultiArchiver).ImportProofs()
      /home/jhb/tap/review_repo/tap/proof/archive.go:844 +0x2bb
  github.com/lightninglabs/taproot-assets/tapgarden.(*Custodian).receiveProof()
      /home/jhb/tap/review_repo/tap/tapgarden/custodian.go:529 +0xbe2
  github.com/lightninglabs/taproot-assets/tapgarden.(*Custodian).watchInboundAssets.func2()
      /home/jhb/tap/review_repo/tap/tapgarden/custodian.go:293 +0x170

Goroutine 3770 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/jhb/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x124
  github.com/lightninglabs/taproot-assets/fn.ParSlice[go.shape.*uint8]()
      /home/jhb/tap/review_repo/tap/fn/concurrency.go:28 +0x10f
  github.com/lightninglabs/taproot-assets/proof.(*MultiArchiver).ImportProofs()
      /home/jhb/tap/review_repo/tap/proof/archive.go:836 +0x1af
  github.com/lightninglabs/taproot-assets/tapgarden.(*Custodian).receiveProof()
      /home/jhb/tap/review_repo/tap/tapgarden/custodian.go:529 +0xbe2
  github.com/lightninglabs/taproot-assets/tapgarden.(*Custodian).inspectWalletTx.func1()
      /home/jhb/tap/review_repo/tap/tapgarden/custodian.go:414 +0x186

Goroutine 3638 (running) created at:
  github.com/lightninglabs/taproot-assets/tapgarden.(*Custodian).watchInboundAssets()
      /home/jhb/tap/review_repo/tap/tapgarden/custodian.go:290 +0x9c4
  github.com/lightninglabs/taproot-assets/tapgarden.(*Custodian).Start.func1.1()
      /home/jhb/tap/review_repo/tap/tapgarden/custodian.go:166 +0x33
==================
--- FAIL: TestTransactionConfirmedOnly (2.14s)
    test_sqlite.go:11: Creating new SQLite DB handle for testing: /tmp/TestTransactionConfirmedOnly3967917637/001/tmp.db
    test_sqlite.go:11: Creating new SQLite DB handle for testing: /tmp/TestTransactionConfirmedOnly3967917637/002/tmp.db
    testing.go:1465: race detected during execution of test
FAIL
FAIL    github.com/lightninglabs/taproot-assets/tapgarden       9.953s
FAIL

It looks like we can end up with multiple goroutines calling c.receiveProof() at once, and operating on the same proofs. This can lead to issues with other behavior like updating event states.

[dstadulis]

is this necessary to fix for v0.4 or merely optimal?

[guggero]

Since it is blocking the merge it would be good to fix in order to not slow down our progress.