mod: remove btcutil circular dependency with main module

[guggero]

Depends on #1824.

Removes the circular dependency between the main module and the btcutil sub module by introducing a bunch of new sub modules.

[guggero]

Rebased after dependent PR was merged.
@Roasbeef PTAL.

[chappjc]

Thanks a ton for both #1823 and #1824.
🙏
Really helpful to a lot of consumers, I think.

In this PR, I'm curious about the new submodules: btcutil/address, btcutil/bech32, btcutil/psbd, and btcutil/base58. I figured those would be cool to remain just as packages in the one btcutil module since making them into modules doesn't seem to break free of any obviously dependency issues like with wire, txscript, etc. I would just hate to see an unnecessary maintenance burden for you guys with too many modules. :P

Thanks again guys! I don't mean to overstep with my questions, just curious and wanting to help.

[guggero]

Thanks for your input, @chappjc!
I was able to remove two of the new modules by just putting btcutil/base58 and btcutil/bech32 below the new btcutil/address module.
We still require a separate module for btcutil/address because that resolves the circular dependency between txscript and btcutil.
Before it was btcutil -> txscript -> btcutil, now it's btcutil -> txscript -> btcutil/address.

[chappjc]

We still require a separate module for btcutil/address because that resolves the circular dependency between txscript and btcutil.

Ahh, yes, Address et al. is in btcutil!

Down in dcrd, @davecgh resolved this by introducing a stdaddr package under the txscript module (decred/dcrd#2610), then breaking txscript's dependency on dcrutil (decred/dcrd#2626, decred/dcrd#2628). That was a job and a half.

Anyway, a btcutil/address module makes sense to quickly break txscript's require on btcutil.

[guggero]

Rebased on top of #1831.

[coveralls]

Pull Request Test Coverage Report for Build 1994552374

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 19 of 149 (12.75%) changed or added relevant lines in 15 files are covered.
• 468 unchanged lines in 4 files lost coverage.
• Overall coverage decreased (-9.1%) to 45.106%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
config.go	0	2	0.0%
mining/mining.go	0	4	0.0%
rpcclient/mining.go	0	4	0.0%
rpcclient/notify.go	0	5	0.0%
rpcclient/extensions.go	0	6	0.0%
rpcclient/rawtransactions.go	0	8	0.0%
blockchain/indexers/addrindex.go	0	10	0.0%
rpcwebsocket.go	0	15	0.0%
rpcserver.go	0	17	0.0%
rpcclient/wallet.go	0	59	0.0%

Files with Coverage Reduction	New Missed Lines	%
rpcclient/extensions.go	1	0.0%
limits/limits_unix.go	2	0.0%
rpcserver.go	8	0.26%
rpcclient/wallet.go	457	0.0%

Totals
Change from base Build 1994295751:	-9.1%
Covered Lines:	16519
Relevant Lines:	36623

---

💛 - Coveralls

[guggero]

Did a final rebase.

After merging, please push tags for the following commits:

• wire/v1.0.0 at 0f0c17b
• chaincfg/v1.0.0 at 9b22ef3
• btcutil/address/v1.0.0 at f0bb656
• txscript/v1.0.0 at afc69c2

[chappjc]

From my point of view (a consumer), this is looking great. Looks like one of the last packages I import that will still come from the main btcd mega-module is btcjson (which imports some of the new modules you're making in this PR).

Other than btcjson, the only other thing that stands out is github.com/btcsuite/btcd/peer and maybe blockchain coming indirectly though both neutrino (neutrino.ServerPeer embeds a *peer.Peer as an exported field, so it's part of the neutrino API) and btcwallet/chain. There's probably others, but this is looking great so far!

[chappjc]

Given recent issues, txscript should really start at txscript/v2 on account of the breaking changes since v0.22.0-beta. Two issues have popped up since where a txscript build error was hit because of the API changes: #1845 and #1843 (comment)

Combined with the "ambiguous import" issues with chainhash recently (#1839), I think all new modules should start at v2 to be safe (if packages of the same name had previously existed, so not btcutil/address), and existing ones that have those types in their API's (e.g. btcutil has in it's API types from chaincfg, chainhash, and wire) should be updated and have their module import path bumped as well. The order with which all these modules are created is obviously going to require care, but I think it needs to be done if these modules are going to be created.

Lastly, I think the best move for the next btcd module tag should be a btcd/v2 module at v2.0.0, while a corresponding product tag like release-v0.23 can be created as long as it is not a semver tag to confuse the Go tooling that treats semver tags with special meaning w.r.t. modules. Potentially the root btcd/v2 module could be created before being updated to use any of the new modules, but I'm not sure.

[Roasbeef]

Care to rebase this? 😅

[guggero]

I rebased everything and created new v2 module for a couple of intertwined packages.
I think with this we can get away with NOT also creating a v2 for the main btcd module, since once we push a new tag, everything will be pulled in as v2 and the "ambiguous import" problem should go away. What do you think, @chappjc?

[Roasbeef]

One question that comes to mind here: will we maintain a top level replace for development versions, and only go through the dance of tagging and updating all the modules for releases?

[guggero]

Hmm, we've employed both strategies in different projects (replace directive for easier PRs vs. needing to do a PR for the submodule first, then tag, then second PR to use the new feature).
Not really sure which makes more sense for this project.

Since these packages are mainly used as library modules and there isn't a lot of new functionality added to the main daemon, I slightly lean toward the "no replace directive" approach.

[Roasbeef]

Here're some diagrams to visualize the before/after, and why we had a circular dep issue in the first place:

before

graph TD
    btcd[btcd<br/>main module]
    btcutil[btcutil<br/>• Address types<br/>• Amount<br/>• Block utilities]
    txscript[txscript<br/>• Script execution<br/>• Signature verification<br/>• Address validation]
    wire[wire<br/>• Protocol messages<br/>• Serialization]
    chaincfg[chaincfg<br/>• Network parameters<br/>• Genesis blocks]
    chainhash[chainhash<br/>• Hash primitives<br/>• Block/Tx hashes]
    
    %% Direct dependencies
    btcd --> btcutil
    btcd --> txscript
    btcd --> wire
    btcd --> chaincfg
    
    btcutil --> chaincfg
    btcutil --> chainhash
    btcutil --> wire
    btcutil -.->|CIRCULAR!| txscript
    
    txscript --> wire
    txscript --> chaincfg
    txscript --> chainhash
    txscript -.->|CIRCULAR!| btcutil
    
    wire --> chainhash
    chaincfg --> chainhash
    
    %% Highlight the circular dependency with high contrast colors
    style btcutil fill:#ff6b6b,stroke:#c92a2a,stroke-width:3px,color:#fff
    style txscript fill:#ff6b6b,stroke:#c92a2a,stroke-width:3px,color:#fff
    style btcd fill:#495057,stroke:#343a40,color:#fff
    style wire fill:#868e96,stroke:#495057,color:#fff
    style chaincfg fill:#868e96,stroke:#495057,color:#fff
    style chainhash fill:#adb5bd,stroke:#495057,color:#000
    
    %% Add annotation
    btcutil -.- annotation[" ⚠️ Circular Dependency<br/>btcutil needs txscript for validation<br/>txscript needs btcutil for Address types"]
    style annotation fill:#ffe066,stroke:#f08c00,stroke-dasharray: 5 5,color:#000

Loading

after

graph TD
    btcd_v2[btcd<br/>main module]
    btcutil_v2[btcutil/v2<br/>• Amount<br/>• Block utilities<br/>• Focused scope]
    address_v2[address/v2<br/>• Address types<br/>• Base58/Bech32<br/>• Encoding/Decoding<br/>✨ NEW MODULE]
    txscript_v2[txscript/v2<br/>• Script execution<br/>• Signature verification<br/>• Uses address/v2]
    wire_v2[wire/v2<br/>• Protocol messages<br/>• Serialization]
    chaincfg_v2[chaincfg/v2<br/>• Network parameters<br/>• Genesis blocks]
    chainhash_v2[chainhash/v2<br/>• Hash primitives<br/>• Block/Tx hashes]
    
    %% Clean dependency flow - bottom to top
    chainhash_v2 --> wire_v2
    chainhash_v2 --> chaincfg_v2
    
    wire_v2 --> address_v2
    chaincfg_v2 --> address_v2
    chainhash_v2 --> address_v2
    
    address_v2 --> txscript_v2
    wire_v2 --> txscript_v2
    chaincfg_v2 --> txscript_v2
    chainhash_v2 --> txscript_v2
    
    txscript_v2 --> btcutil_v2
    address_v2 --> btcutil_v2
    wire_v2 --> btcutil_v2
    chaincfg_v2 --> btcutil_v2
    chainhash_v2 --> btcutil_v2
    
    btcutil_v2 --> btcd_v2
    txscript_v2 --> btcd_v2
    wire_v2 --> btcd_v2
    chaincfg_v2 --> btcd_v2
    
    %% Highlight with high contrast colors
    style address_v2 fill:#51cf66,stroke:#2f9e44,stroke-width:3px,color:#000
    style btcutil_v2 fill:#4dabf7,stroke:#1971c2,stroke-width:2px,color:#000
    style txscript_v2 fill:#4dabf7,stroke:#1971c2,stroke-width:2px,color:#000
    style btcd_v2 fill:#495057,stroke:#343a40,color:#fff
    style wire_v2 fill:#868e96,stroke:#495057,color:#fff
    style chaincfg_v2 fill:#868e96,stroke:#495057,color:#fff
    style chainhash_v2 fill:#adb5bd,stroke:#495057,color:#000
    
    %% Add success annotation
    address_v2 -.- success[" ✅ Problem Solved!<br/>Address types extracted to new module<br/>No more circular dependencies<br/>Clean dependency hierarchy"]
    style success fill:#8ce99a,stroke:#51cf66,stroke-dasharray: 5 5,color:#000

Loading

[Roasbeef]

LGTM 📝

Needs a rebase.

[guggero]

Rebased.

[litecomb]

Thank you for the hard work, this make sense, if it makes us closer to eliminating circular go module dependencies.

[MichaelMure]

Could this be merged? It looks like a great improvement to use btcd modularly.