Merge bitcoin#17261: Make ScriptPubKeyMan an actual interface and the wallet to have multiple

3f37365 Refactor: Replace SigningProvider pointers with unique_ptrs (Andrew Chow)
3afe53c Cleanup: Drop unused GUI learnRelatedScripts method (Andrew Chow)
e2f02aa Refactor: Copy CWallet signals and print function to LegacyScriptPubKeyMan (Andrew Chow)
c729afd Box the wallet: Add multiple keyman maps and loops (Andrew Chow)
4977c30 refactor: define a UINT256_ONE global constant (Andrew Chow)
415afcc HD Split: Avoid redundant upgrades (Andrew Chow)
01b4511 Make UpgradeKeyMetadata work only on LegacyScriptPubKeyMan (Andrew Chow)
4a7e43e Store p2sh scripts in AddAndGetDestinationForScript (Andrew Chow)
501acb5 Always try to sign for all pubkeys in multisig (Andrew Chow)
81610ed List output types in an array in order to be iterated over (Andrew Chow)
eb81fc3 Refactor: Allow LegacyScriptPubKeyMan to be null (Andrew Chow)
fadc08a Locking: Lock cs_KeyStore instead of cs_wallet in legacy keyman (Andrew Chow)
f5be479 wallet: Improve CWallet:MarkDestinationsDirty (João Barbosa)

Pull request description:

  Continuation of wallet boxes project.

  Actually makes ScriptPubKeyMan an interface which LegacyScriptPubkeyMan. Moves around functions and things from CWallet into LegacyScriptPubKeyMan so that they are actually separate things without circular dependencies.

  ***

  Introducing the `ScriptPubKeyMan` (short for ScriptPubKeyManager) for managing scriptPubKeys and their associated scripts and keys. This functionality is moved over from `CWallet`. Instead, `CWallet` will have a pointer to a `ScriptPubKeyMan` for every possible address type, internal and external. It will fetch the correct `ScriptPubKeyMan` as necessary. When fetching new addresses, it chooses the `ScriptPubKeyMan` based on address type and whether it is change. For signing, it takes the script and asks each `ScriptPubKeyMan` for whether that `ScriptPubKeyMan` considers that script `IsMine`, whether it has that script, or whether it is able to produce a signature for it. If so, the `ScriptPubKeyMan` will provide a `SigningProvider` to the caller which will use that in order to sign.

  There is currently one `ScriptPubKeyMan` - the `LegacyScriptPubKeyMan`. Each `CWallet` will have only one `LegacyScriptPubKeyMan` with the pointers for all of the address types and change pointing to this `LegacyScriptPubKeyMan`. It is created when the wallet is loaded and all keys and metadata are loaded into it instead of `CWallet`. The `LegacyScriptPubKeyMan` is primarily made up of all of the key and script management that used to be in `CWallet`. For convenience, `CWallet` has a `GetLegacyScriptPubKeyMan` which will return the `LegacyScriptPubKeyMan` or a `nullptr` if it does not have one (not yet implemented, but callers will check for the `nullptr`). For purposes of signing, `LegacyScriptPubKeyMan`'s `GetSigningProvider` will return itself rather than a separate `SigningProvider`. This will be different for future `ScriptPubKeyMan`s.

  The `LegacyScriptPubKeyMan` will also handle the importing and exporting of keys and scripts instead of `CWallet`. As such, a number of RPCs have been limited to work only if a `LegacyScriptPubKeyMan` can be retrieved from the wallet. These RPCs are `sethdseed`, `addmultisigaddress`, `importaddress`, `importprivkey`, `importpubkey`, `importmulti`, `dumpprivkey`, and `dumpwallet`. Other RPCs which relied on the wallet for scripts and keys have been modified in order to take the `SigningProvider` retrieved from the `ScriptPubKeyMan` for a given script.

  Overall, these changes should not effect how everything actually works and the user should experience no difference between having this change and not having it. As such, no functional tests were changed, and the only unit tests changed were those that were directly accessing `CWallet` functions that have been removed.

  This PR is the last step in the [Wallet Structure Changes](https://github.com/bitcoin-core/bitcoin-devwiki/wiki/Wallet-Class-Structure-Changes).

ACKs for top commit:
  instagibbs:
    re-utACK bitcoin@3f37365
  Sjors:
    re-utACK 3f37365 (it still compiles on macOS after bitcoin#17261 (comment))
  meshcollider:
    Tested re-ACK 3f37365

Tree-SHA512: f8e2b8d9efa750b617691e8702d217ec4c33569ec2554a060141d9eb9b9a3a5323e4216938e2485c44625d7a6e0925d40dea1362b3af9857cf08860c2f344716

Author: achow101

src/Makefile.am

@@ -246,6 +246,7 @@ BITCOIN_CORE_H = \
   node/transaction.h \
   node/utxo_snapshot.h \
   noui.h \
+  outputtype.h \
   policy/feerate.h \
   policy/fees.h \
   policy/policy.h \
@@ -673,6 +674,7 @@ libbitcoin_common_a_SOURCES = \
   netaddress.cpp \
   netbase.cpp \
   net_permissions.cpp \
+  outputtype.cpp \
   policy/feerate.cpp \
   policy/policy.cpp \
   protocol.cpp \

src/bench/coin_selection.cpp

@@ -31,7 +31,8 @@ static void CoinSelection(benchmark::Bench& bench)
 {
     NodeContext node;
     auto chain = interfaces::MakeChain(node);
-    const CWallet wallet(chain.get(), "", CreateDummyWalletDatabase());
+    CWallet wallet(chain.get(), "", CreateDummyWalletDatabase());
+    wallet.SetupLegacyScriptPubKeyMan();
     std::vector<std::unique_ptr<CWalletTx>> wtxs;
     LOCK(wallet.cs_wallet);
 
@@ -63,7 +64,7 @@ static void CoinSelection(benchmark::Bench& bench)
 typedef std::set<CInputCoin> CoinSet;
 static NodeContext testNode;
 static auto testChain = interfaces::MakeChain(testNode);
-static const CWallet testWallet(testChain.get(), "", CreateDummyWalletDatabase());
+static CWallet testWallet(testChain.get(), "", CreateDummyWalletDatabase());
 std::vector<std::unique_ptr<CWalletTx>> wtxn;
 
 // Copied from src/wallet/test/coinselector_tests.cpp
@@ -92,6 +93,7 @@ static CAmount make_hard_case(int utxos, std::vector<OutputGroup>& utxo_pool)
 static void BnBExhaustion(benchmark::Bench& bench)
 {
     // Setup
+    testWallet.SetupLegacyScriptPubKeyMan();
     std::vector<OutputGroup> utxo_pool;
     CoinSet selection;
     CAmount value_ret = 0;

src/bench/wallet_balance.cpp

@@ -21,6 +21,7 @@ static void WalletBalance(benchmark::Bench& bench, const bool set_dirty, const b
     std::unique_ptr<interfaces::Chain> chain = interfaces::MakeChain(node);
     CWallet wallet{chain.get(), "", CreateMockWalletDatabase()};
     {
+        wallet.SetupLegacyScriptPubKeyMan();
         bool first_run;
         if (wallet.LoadWallet(first_run) != DBErrors::LOAD_OK) assert(false);
     }

src/coinjoin/util.cpp

@@ -94,6 +94,7 @@ CTransactionBuilderOutput::CTransactionBuilderOutput(CTransactionBuilder* pTxBui
 {
     assert(pTxBuilder);
     CTxDestination txdest;
+    LOCK(pwalletIn->cs_wallet);
     dest.GetReservedDestination(txdest, false);
     script = ::GetScriptForDestination(txdest);
 }

src/interfaces/wallet.cpp

@@ -192,15 +192,15 @@ class WalletImpl : public Wallet
     }
     bool getPubKey(const CScript& script, const CKeyID& address, CPubKey& pub_key) override
     {
-        const SigningProvider* provider = m_wallet->GetSigningProvider(script);
+        std::unique_ptr<SigningProvider> provider = m_wallet->GetSigningProvider(script);
         if (provider) {
             return provider->GetPubKey(address, pub_key);
         }
         return false;
     }
     bool getPrivKey(const CScript& script, const CKeyID& address, CKey& key) override
     {
-        const SigningProvider* provider = m_wallet->GetSigningProvider(script);
+        std::unique_ptr<SigningProvider> provider = m_wallet->GetSigningProvider(script);
         if (provider) {
             return provider->GetKey(address, key);
         }
@@ -254,7 +254,6 @@ class WalletImpl : public Wallet
         }
         return result;
     }
-    //void learnRelatedScripts(const CPubKey& key, OutputType type) override { if (spk_man) m_wallet->GetLegacyScriptPubKeyMan()->LearnRelatedScripts(key, type); }
     bool addDestData(const CTxDestination& dest, const std::string& key, const std::string& value) override
     {
         LOCK(m_wallet->cs_wallet);

src/outputtype.cpp

@@ -0,0 +1,21 @@
+// Copyright (c) 2009-2010 Satoshi Nakamoto
+// Copyright (c) 2009-2019 The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#include <outputtype.h>
+
+#include <script/signingprovider.h>
+#include <script/standard.h>
+
+#include <assert.h>
+
+CTxDestination AddAndGetDestinationForScript(FillableSigningProvider& keystore, const CScript& script)
+{
+    // Add script to keystore
+    keystore.AddCScript(script);
+    ScriptHash sh(script);
+    // Note that scripts over 520 bytes are not yet supported.
+    keystore.AddCScript(GetScriptForDestination(sh));
+    return sh;
+}

src/outputtype.h

@@ -0,0 +1,19 @@
+// Copyright (c) 2009-2010 Satoshi Nakamoto
+// Copyright (c) 2009-2019 The Bitcoin Core developers
+// Distributed under the MIT software license, see the accompanying
+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
+
+#ifndef BITCOIN_OUTPUTTYPE_H
+#define BITCOIN_OUTPUTTYPE_H
+
+#include <script/signingprovider.h>
+#include <script/standard.h>
+
+/**
+ * Get a destination of the requested type (if possible) to the specified script.
+ * This function will automatically add the script (and any other
+ * necessary scripts) to the keystore.
+ */
+CTxDestination AddAndGetDestinationForScript(FillableSigningProvider& keystore, const CScript& script);
+
+#endif // BITCOIN_OUTPUTTYPE_H

src/qt/test/addressbooktests.cpp

@@ -60,6 +60,7 @@ void TestAddAddressesToSendBook(interfaces::Node& node)
     TestChain100Setup test;
     node.setContext(&test.m_node);
     std::shared_ptr<CWallet> wallet = std::make_shared<CWallet>(node.context()->chain.get(), "", CreateMockWalletDatabase());
+    wallet->SetupLegacyScriptPubKeyMan();
     bool firstRun;
     wallet->LoadWallet(firstRun);
 

src/qt/test/wallettests.cpp

@@ -114,9 +114,8 @@ void TestGUI(interfaces::Node& node)
     bool firstRun;
     wallet->LoadWallet(firstRun);
     {
-        auto spk_man = wallet->GetLegacyScriptPubKeyMan();
-        LOCK(wallet->cs_wallet);
-        AssertLockHeld(spk_man->cs_wallet);
+        auto spk_man = wallet->GetOrCreateLegacyScriptPubKeyMan();
+        LOCK2(wallet->cs_wallet, spk_man->cs_KeyStore);
         wallet->SetAddressBook(PKHash(test.coinbaseKey.GetPubKey()), "", "receive");
         spk_man->AddKeyPubKey(test.coinbaseKey, test.coinbaseKey.GetPubKey());
         wallet->SetLastBlockProcessed(105, ::ChainActive().Tip()->GetBlockHash());

src/rpc/misc.cpp

@@ -279,8 +279,9 @@ static UniValue createmultisig(const JSONRPCRequest& request)
     }
 
     // Construct using pay-to-script-hash:
+    FillableSigningProvider keystore;
     CScript inner;
-    const CTxDestination dest = AddAndGetMultisigDestination(required, pubkeys, inner);
+    const CTxDestination dest = AddAndGetMultisigDestination(required, pubkeys, keystore, inner);
 
     UniValue result(UniValue::VOBJ);
     result.pushKV("address", EncodeDestination(dest));