From f3eeaeef3409019fda488aa3a3ccd7f31ab3696f Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Sat, 23 Jan 2021 12:58:13 -0700 Subject: [PATCH 01/16] First draft of HIPE for plugin removal helpers Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 134 +++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 text/0162-plugin-removal-helpers/README.md diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md new file mode 100644 index 00000000..3a95d86b --- /dev/null +++ b/text/0162-plugin-removal-helpers/README.md @@ -0,0 +1,134 @@ +# Indy HIPE 0162: Features to Help Remove Ledger Plugins +- Author: [Alexandr Kolesov](alexander.kolesov@evernym.com), [Richard Esplin](mailto:richard.esplin@evernym.com), and [Renata Toktar](renata.toktar@evernym.com) +- Start Date: 2020-12-01 + +## Status +- Status: [ADOPTED](/README.md#hipe-lifecycle) +- Status Date: 2020-01-22 +- Status Note: The changes in this HIPE were discussed in the Indy Maintainers calls on [2020-12-22](https://wiki.hyperledger.org/display/indy/2020-12-22+Indy+Contributors+Call) and [2020-01-19](https://wiki.hyperledger.org/display/indy/2021-01-19+Indy+Contributors+Call). The other maintainers approved the plan, but requested a HIPE to be submitted. + +## Summary + +In this HIPE, we propose two new features for Indy Node that will assist with removing ledger plugins that are no longer used: frozen ledgers and default fee handlers. + +## Motivation + +Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](). + +Removing the plugin has three important consequences: +* The data stored on the custom ledger created by the plugin will be deleted. +* The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. +* Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. + +The first consequence is intended. This HIPE proposes feature to address the remaining two consequences to ensure that the network remains functional after the plugin is removed. + +We include both features in the same HIPE because they share a common motivation and are likely to be used together. + +## Frozen Ledgers + +### Tutorial + +If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: + +* It preserves the ability to do rolling updates. If an update removes a plugin which includes custom ledgers, consensus would be interrupted and new transactions would not be able to be ordered until all nodes completed the update. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. +* It allows the removal of old plugins. Frozen ledgers can be removed and there will be no left over data from the removed plugin. The removal of old plugins will also help reduce the cost of keeping the network stable and secure. + +Freezing ledgers requires the following workflow: +* Upgrade all nodes to a version of Indy that supports frozen ledgers. +* Send a LEDGERS_FREEZE transaction for the ledgers that you want to drop. +* Upgrade all nodes to remove the plugin. +* Execute a migration script to drop the ledgers. + +Permissions around the LEDGERS_FREEZE transaction are managed in the auth_rules with a default permission requiring the approval of three trustees ([the same default auth_rule for upgrading the ledger](https://github.com/hyperledger/indy-node/blob/master/docs/source/auth_rules.md)). + +### Reference + +If a ledger is frozen it can be used neither for reading nor for writing. It will not be caught up by new nodes and can be safely removed. Default ledgers such as the domain, config, pool, and audit ledgers cannot be frozen. + +Implementation notes: +* Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. +* We were nervous that replacing a ledger with a frozen ledger hash would break foreign keys in auth_rules, but our testing showed that this is not an issue. + +Implementation steps: +1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state +2. Implement GET_FROZEN_LEDGERS transaction handler that will return information about frozen ledgers from state +3. Add dynamic validation to all handlers: check if the ledger is frozen +4. Update audit logic by taking into account the root hashes of frozen ledgers +5. Fix catch up to not catch up frozen ledgers **TODO: mirror requests or not?** + +#### State schema: +There will be a key in CONFIG_LEDGER that will store a list of frozen ledgers in the following format: + +``` +2:FROZEN_LEDGERS = [ + : { + ledger: , +        state: , +        seq_no: + }, + ... +] +``` + +#### New transactions: + +Freeze ledgers that have the provided IDs: +``` +LEDGERS_FREEZE { + LEDGERS_IDS: [int] +} +``` + +Return frozen ledgers and their root hashes. +``` +GET_FROZEN_LEDGERS +``` + +### Drawbacks + +The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to replayed with the correct ledger hash. + +### Rationale and alternatives + +We considered implementing the frozen ledger hash in a plugin, but we consider it a generally useful feature and believe it should belong in Indy. + +### Prior art + +No relevant prior art. + +### Unresolved questions + +All questions raised during the development of this proposal have been answered in this draft of the proposal. + + +## Default Fee Handlers + +### Tutorial + +[Indy Authorization Rules](https://github.com/hyperledger/indy-node/blob/master/docs/source/requests.md#auth_rule) provide a flexible way of authorizing write transactions to the domain ledger. This authorization can [depend on the payment of fees tracked by ledger plugins](https://github.com/sovrin-foundation/libsovtoken/blob/master/doc/fees.md). + +Because Indy currently has no native concept of fees, if a plugin that implemented fees is removed, historical transactions can no longer be validated and will prevent new nodes from catching up on the ledger. In many cases it is sufficient to know that the historical transactions were valid at the time of writing. In these cases a default fee handler can approve the historical transactions. + +### Reference + +The default fee handler is implemented in a new class ... +**TODO** + +### Drawbacks + +* Historically there were concerns about having payment functionality included in Hyperledger projects, but as distributed payments have become more mainstream, it now seems acceptable to include into Indy generic primitives related to payments. +* The default fee handler will not perform the same validation on a historical transaction as was done in the removed plugin at the time that the transaction was originally completed. Therefore, it is most suitable for ledgers that do not make strong guarantees of historical validity, such as ledgers used for development and testing. +* If auth_rules with fees are defined, but no plugin is installed to process them, the rules will fall back to the default fee handler and authorization will be granted. This permissive behavior could be dangerous in some circumstances, and network administrators should ensure that auth_rules do not rely on fees when no plugin is installed to handle them. This is consistent with other Indy configuration options which are permissive by default to assist new users, but expected to be locked down for production use. + +### Prior art + +No relevant prior art. + +### Unresolved questions + +All questions raised during the development of this proposal have been answered in this draft of the proposal. + +## Both + +### Rationale and alternatives +There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger. But the features proposed in this HIPE would still be useful for the development and maintenance of other ledger plugins. From e4f1c6322b341678f78014779dc9dd2051b25b53 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Sat, 23 Jan 2021 13:45:40 -0700 Subject: [PATCH 02/16] Minor improvements: anchor tags and links Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md index 3a95d86b..3401afde 100644 --- a/text/0162-plugin-removal-helpers/README.md +++ b/text/0162-plugin-removal-helpers/README.md @@ -8,12 +8,14 @@ - Status Note: The changes in this HIPE were discussed in the Indy Maintainers calls on [2020-12-22](https://wiki.hyperledger.org/display/indy/2020-12-22+Indy+Contributors+Call) and [2020-01-19](https://wiki.hyperledger.org/display/indy/2021-01-19+Indy+Contributors+Call). The other maintainers approved the plan, but requested a HIPE to be submitted. ## Summary +[summary]: #summary In this HIPE, we propose two new features for Indy Node that will assist with removing ledger plugins that are no longer used: frozen ledgers and default fee handlers. ## Motivation +[motivation]: #motivation -Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](). +Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). Removing the plugin has three important consequences: * The data stored on the custom ledger created by the plugin will be deleted. @@ -25,8 +27,10 @@ The first consequence is intended. This HIPE proposes feature to address the rem We include both features in the same HIPE because they share a common motivation and are likely to be used together. ## Frozen Ledgers +[frozen-ledgers]: #frozen-ledgers ### Tutorial +[frozen-ledgers-tutorial]: #frozen-ledgers-tutorial If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: @@ -42,6 +46,7 @@ Freezing ledgers requires the following workflow: Permissions around the LEDGERS_FREEZE transaction are managed in the auth_rules with a default permission requiring the approval of three trustees ([the same default auth_rule for upgrading the ledger](https://github.com/hyperledger/indy-node/blob/master/docs/source/auth_rules.md)). ### Reference +[frozen-ledgers-reference]: #frozen-ledgers-reference If a ledger is frozen it can be used neither for reading nor for writing. It will not be caught up by new nodes and can be safely removed. Default ledgers such as the domain, config, pool, and audit ledgers cannot be frozen. @@ -85,50 +90,63 @@ GET_FROZEN_LEDGERS ``` ### Drawbacks +[frozen-ledgers-drawbacks]: #frozen-ledgers-drawbacks The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to replayed with the correct ledger hash. ### Rationale and alternatives +[frozen-ledgers-rationale-and-alternatives]: #frozen-ledgers-rationale-and-alternatives We considered implementing the frozen ledger hash in a plugin, but we consider it a generally useful feature and believe it should belong in Indy. ### Prior art +[frozen-ledgers-prior-art]: #frozen-ledgers-prior-art No relevant prior art. ### Unresolved questions +[frozen-ledgers-unresolved-questions]: #frozen-ledgers-unresolved-questions All questions raised during the development of this proposal have been answered in this draft of the proposal. ## Default Fee Handlers +[default-fee-handler]: #default-fee-handler ### Tutorial +[default-fee-handler-tutorial]: #default-fee-handler-tutorial [Indy Authorization Rules](https://github.com/hyperledger/indy-node/blob/master/docs/source/requests.md#auth_rule) provide a flexible way of authorizing write transactions to the domain ledger. This authorization can [depend on the payment of fees tracked by ledger plugins](https://github.com/sovrin-foundation/libsovtoken/blob/master/doc/fees.md). Because Indy currently has no native concept of fees, if a plugin that implemented fees is removed, historical transactions can no longer be validated and will prevent new nodes from catching up on the ledger. In many cases it is sufficient to know that the historical transactions were valid at the time of writing. In these cases a default fee handler can approve the historical transactions. ### Reference +[default-fee-handler-reference]: #default-fee-handler-reference The default fee handler is implemented in a new class ... **TODO** ### Drawbacks +[default-fee-handler-drawbacks]: #default-fee-handler-drawbacks * Historically there were concerns about having payment functionality included in Hyperledger projects, but as distributed payments have become more mainstream, it now seems acceptable to include into Indy generic primitives related to payments. * The default fee handler will not perform the same validation on a historical transaction as was done in the removed plugin at the time that the transaction was originally completed. Therefore, it is most suitable for ledgers that do not make strong guarantees of historical validity, such as ledgers used for development and testing. * If auth_rules with fees are defined, but no plugin is installed to process them, the rules will fall back to the default fee handler and authorization will be granted. This permissive behavior could be dangerous in some circumstances, and network administrators should ensure that auth_rules do not rely on fees when no plugin is installed to handle them. This is consistent with other Indy configuration options which are permissive by default to assist new users, but expected to be locked down for production use. ### Prior art +[default-fee-handler-prior-art]: #default-fee-handler-prior-art No relevant prior art. ### Unresolved questions +[default-fee-handler-unresolved-questions]: #default-fee-handler-unresolved-questions All questions raised during the development of this proposal have been answered in this draft of the proposal. ## Both +[both]: #both ### Rationale and alternatives +[both-rationale-and-alternatives]: #both-rationale-and-alternatives + There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger. But the features proposed in this HIPE would still be useful for the development and maintenance of other ledger plugins. From a6db684b8a116a791fbd1442ae6e29a8234bd5bd Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Sat, 23 Jan 2021 22:03:06 -0700 Subject: [PATCH 03/16] Additional alternative approach and rationale. Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md index 3401afde..9732cccc 100644 --- a/text/0162-plugin-removal-helpers/README.md +++ b/text/0162-plugin-removal-helpers/README.md @@ -149,4 +149,6 @@ All questions raised during the development of this proposal have been answered ### Rationale and alternatives [both-rationale-and-alternatives]: #both-rationale-and-alternatives -There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger. But the features proposed in this HIPE would still be useful for the development and maintenance of other ledger plugins. +Both of these features could be avoided by moving the token transactions into Indy as deprecated historical transactions. This would allow the history to be validated, but we don't want Indy to become a graveyard for every transaction type a plugin author defines. Implementing the features proposed in this HIPE is a more generic way to address the problem. + +There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger, but no one has volunteered to do that work. Even if Indy does one day have a generic token, the features proposed in this HIPE would still be useful for the development and maintenance of other ledger plugins. From 570fa50ef68d048aa0543a225508490603d1f88f Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Sat, 23 Jan 2021 22:07:03 -0700 Subject: [PATCH 04/16] Clarification about unfreezing ledgers. Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md index 9732cccc..64804e39 100644 --- a/text/0162-plugin-removal-helpers/README.md +++ b/text/0162-plugin-removal-helpers/README.md @@ -53,6 +53,7 @@ If a ledger is frozen it can be used neither for reading nor for writing. It wil Implementation notes: * Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. * We were nervous that replacing a ledger with a frozen ledger hash would break foreign keys in auth_rules, but our testing showed that this is not an issue. +* Frozen ledgers cannot be unfrozen, but dropped ledgers can be recreated with a different ledger ID. Implementation steps: 1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state From b79e5426cbb695459f6242aebfcd39c26d50c2cd Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Mon, 25 Jan 2021 19:14:50 -0700 Subject: [PATCH 05/16] Incorporating feedback from Alex K. Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 35 +++++++++++++--------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md index 64804e39..962fa7a6 100644 --- a/text/0162-plugin-removal-helpers/README.md +++ b/text/0162-plugin-removal-helpers/README.md @@ -34,7 +34,7 @@ We include both features in the same HIPE because they share a common motivation If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: -* It preserves the ability to do rolling updates. If an update removes a plugin which includes custom ledgers, consensus would be interrupted and new transactions would not be able to be ordered until all nodes completed the update. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. +* It preserves the ability to do rolling updates. Without this capability, attempting to remove a plugin during a rolling update risks breaking consensus if a transaction is submitted to the ledger during the roll out period. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. * It allows the removal of old plugins. Frozen ledgers can be removed and there will be no left over data from the removed plugin. The removal of old plugins will also help reduce the cost of keeping the network stable and secure. Freezing ledgers requires the following workflow: @@ -50,17 +50,17 @@ Permissions around the LEDGERS_FREEZE transaction are managed in the auth_rules If a ledger is frozen it can be used neither for reading nor for writing. It will not be caught up by new nodes and can be safely removed. Default ledgers such as the domain, config, pool, and audit ledgers cannot be frozen. -Implementation notes: +#### Implementation notes: * Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. -* We were nervous that replacing a ledger with a frozen ledger hash would break foreign keys in auth_rules, but our testing showed that this is not an issue. +* We were nervous that removing transactions would break foreign keys in auth_rules, but our testing showed that this is not an issue. * Frozen ledgers cannot be unfrozen, but dropped ledgers can be recreated with a different ledger ID. -Implementation steps: -1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state -2. Implement GET_FROZEN_LEDGERS transaction handler that will return information about frozen ledgers from state -3. Add dynamic validation to all handlers: check if the ledger is frozen -4. Update audit logic by taking into account the root hashes of frozen ledgers -5. Fix catch up to not catch up frozen ledgers **TODO: mirror requests or not?** +#### Implementation steps: +1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state. +2. Implement GET_FROZEN_LEDGERS transaction handler that will return information about frozen ledgers from state. +3. Add dynamic validation to all handlers: check if the ledger is frozen. +4. Update audit logic by taking into account the root hashes of frozen ledgers. +5. Fix catch up to not catch up frozen ledgers. #### State schema: There will be a key in CONFIG_LEDGER that will store a list of frozen ledgers in the following format: @@ -93,7 +93,7 @@ GET_FROZEN_LEDGERS ### Drawbacks [frozen-ledgers-drawbacks]: #frozen-ledgers-drawbacks -The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to replayed with the correct ledger hash. +The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to be replayed with the correct ledger hash. ### Rationale and alternatives [frozen-ledgers-rationale-and-alternatives]: #frozen-ledgers-rationale-and-alternatives @@ -124,15 +124,22 @@ Because Indy currently has no native concept of fees, if a plugin that implement ### Reference [default-fee-handler-reference]: #default-fee-handler-reference -The default fee handler is implemented in a new class ... -**TODO** +The default fee handler will be a copy of [the sovtoken fee handler (FeesAuthorizer)](https://github.com/sovrin-foundation/token-plugin/blob/master/sovtokenfees/sovtokenfees/fees_authorizer.py#L26) with the validation disabled. + +Any transaction handler consists of two parts: validation logic (static and dynamic) and business logic. + +For fee transactions, transaction validation will be performed on new transactions, but no validation will be performed during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. By copying the handler and disabling validation, it will allow catchup but will not allow new transactions with fees. + +Business logic for fee related transactions will continue to be determined by the auth_rules and logic implemented in a plugin which defines a payment implementation. ### Drawbacks [default-fee-handler-drawbacks]: #default-fee-handler-drawbacks * Historically there were concerns about having payment functionality included in Hyperledger projects, but as distributed payments have become more mainstream, it now seems acceptable to include into Indy generic primitives related to payments. -* The default fee handler will not perform the same validation on a historical transaction as was done in the removed plugin at the time that the transaction was originally completed. Therefore, it is most suitable for ledgers that do not make strong guarantees of historical validity, such as ledgers used for development and testing. -* If auth_rules with fees are defined, but no plugin is installed to process them, the rules will fall back to the default fee handler and authorization will be granted. This permissive behavior could be dangerous in some circumstances, and network administrators should ensure that auth_rules do not rely on fees when no plugin is installed to handle them. This is consistent with other Indy configuration options which are permissive by default to assist new users, but expected to be locked down for production use. +* The default fee handler replays transactions without performing validation. This is fine for catching up validator nodes, but an audit of the ledger history would need to take into account the historical validation expected by any removed plugins. +* If auth_rules with fees are defined, but no plugin is installed to process them, the validation will fall back to the default fee handler which will ignore them and consequently grant authorization. This permissive behavior could be dangerous in some circumstances, and network administrators should ensure that auth_rules do not rely on fees when no plugin is installed to handle them. This is consistent with other Indy configuration options which are permissive by default to assist new users, but expected to be locked down for production use. +* The existence of a default fee handler in Indy Node could in theory be exploited by an attacker, but we consider this a small risk for a distributed ledger as any attack would have to simultaneously succeed on a majority of validation nodes in order to change what gets written. + ### Prior art [default-fee-handler-prior-art]: #default-fee-handler-prior-art From 2a86beb498015d5ae1501756b69c0123dec3e151 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Tue, 26 Jan 2021 16:37:55 -0700 Subject: [PATCH 06/16] Incorporating more feedback from Alex K. Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md index 962fa7a6..c3115801 100644 --- a/text/0162-plugin-removal-helpers/README.md +++ b/text/0162-plugin-removal-helpers/README.md @@ -126,11 +126,9 @@ Because Indy currently has no native concept of fees, if a plugin that implement The default fee handler will be a copy of [the sovtoken fee handler (FeesAuthorizer)](https://github.com/sovrin-foundation/token-plugin/blob/master/sovtokenfees/sovtokenfees/fees_authorizer.py#L26) with the validation disabled. -Any transaction handler consists of two parts: validation logic (static and dynamic) and business logic. +Any Indy transaction handler consists of two parts: validation logic (static and dynamic) and business logic. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. The default fee handler will contain a copy of the sovtoken fee handler but with validation disabled, so it will allow catchup but will not allow new transactions with fees. -For fee transactions, transaction validation will be performed on new transactions, but no validation will be performed during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. By copying the handler and disabling validation, it will allow catchup but will not allow new transactions with fees. - -Business logic for fee related transactions will continue to be determined by the auth_rules and logic implemented in a plugin which defines a payment implementation. +Indy will look to the default fee handler to determine the business logic for fee related transactions, but the default fee handler will not allow any new fee transactions and so doesn't need any specific business logic. An implementation of a payment plugin will need to define this logic to enable fee transactions. ### Drawbacks [default-fee-handler-drawbacks]: #default-fee-handler-drawbacks From b2714baeedd20d1917cee98bc09825e8621b9bf9 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Wed, 27 Jan 2021 20:05:47 -0700 Subject: [PATCH 07/16] Further improvements suggested by Alex K. Signed-off-by: Richard Esplin --- text/0162-plugin-removal-helpers/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md index c3115801..d980b9ff 100644 --- a/text/0162-plugin-removal-helpers/README.md +++ b/text/0162-plugin-removal-helpers/README.md @@ -124,18 +124,18 @@ Because Indy currently has no native concept of fees, if a plugin that implement ### Reference [default-fee-handler-reference]: #default-fee-handler-reference -The default fee handler will be a copy of [the sovtoken fee handler (FeesAuthorizer)](https://github.com/sovrin-foundation/token-plugin/blob/master/sovtokenfees/sovtokenfees/fees_authorizer.py#L26) with the validation disabled. +Implementation will require adding to Indy [the SET_FEE transaction copied from the sovtoken plugin](https://github.com/sovrin-foundation/token-plugin/blob/a635780361a7478ea4b2f47fcffee78b150fdea3/sovtokenfees/sovtokenfees/req_handlers/write_handlers/set_fees_handler.py#L20) and [a default fee handler copied from the sovtoken fee handler (FeesAuthorizer)](https://github.com/sovrin-foundation/token-plugin/blob/master/sovtokenfees/sovtokenfees/fees_authorizer.py#L26). The fee handler will have the validation logic replaced with `throw NotAllowedException()`. -Any Indy transaction handler consists of two parts: validation logic (static and dynamic) and business logic. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. The default fee handler will contain a copy of the sovtoken fee handler but with validation disabled, so it will allow catchup but will not allow new transactions with fees. +Any Indy transaction handler consists of two parts: validation logic and execution logic. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. The default fee handler will contain a copy of the sovtoken fee handler but with validation disabled, so it will allow catchup but will not allow new transactions with fees. -Indy will look to the default fee handler to determine the business logic for fee related transactions, but the default fee handler will not allow any new fee transactions and so doesn't need any specific business logic. An implementation of a payment plugin will need to define this logic to enable fee transactions. +Indy will also look to the default fee handler to determine the execution logic for fee related transactions, but the default fee handler will not allow any new fee transactions and so doesn't need any specific execution logic. Any future implementation of a payment plugin will need to override the default fee handler with a functional implementation of validation and execution logic. ### Drawbacks [default-fee-handler-drawbacks]: #default-fee-handler-drawbacks * Historically there were concerns about having payment functionality included in Hyperledger projects, but as distributed payments have become more mainstream, it now seems acceptable to include into Indy generic primitives related to payments. * The default fee handler replays transactions without performing validation. This is fine for catching up validator nodes, but an audit of the ledger history would need to take into account the historical validation expected by any removed plugins. -* If auth_rules with fees are defined, but no plugin is installed to process them, the validation will fall back to the default fee handler which will ignore them and consequently grant authorization. This permissive behavior could be dangerous in some circumstances, and network administrators should ensure that auth_rules do not rely on fees when no plugin is installed to handle them. This is consistent with other Indy configuration options which are permissive by default to assist new users, but expected to be locked down for production use. +* The ledger could be erroneously configured with auth_rules that define fees without having a plugin installed to process them. The exact behavior in this scenario will depend on the specific rules that mention fees, and could lead to confusing and hard to diagnose behavior. However this is a niche case because SET_FEE transactions will be rejected if there is no plugin to define them. * The existence of a default fee handler in Indy Node could in theory be exploited by an attacker, but we consider this a small risk for a distributed ledger as any attack would have to simultaneously succeed on a majority of validation nodes in order to change what gets written. From 64cae31f74004a82b7861b05752c9390ad7ef533 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Fri, 29 Jan 2021 17:29:47 -0700 Subject: [PATCH 08/16] Splitting this HIPE into two. Signed-off-by: Richard Esplin --- text/0162-freeze-ledgers/README.md | 118 +++++++++++++++ text/0162-plugin-removal-helpers/README.md | 160 --------------------- text/0163-default-fee-handler/README.md | 75 ++++++++++ 3 files changed, 193 insertions(+), 160 deletions(-) create mode 100644 text/0162-freeze-ledgers/README.md delete mode 100644 text/0162-plugin-removal-helpers/README.md create mode 100644 text/0163-default-fee-handler/README.md diff --git a/text/0162-freeze-ledgers/README.md b/text/0162-freeze-ledgers/README.md new file mode 100644 index 00000000..46141539 --- /dev/null +++ b/text/0162-freeze-ledgers/README.md @@ -0,0 +1,118 @@ +# Indy HIPE 0162: Freeze Ledgers +- Author: [Alexandr Kolesov](alexander.kolesov@evernym.com), [Richard Esplin](mailto:richard.esplin@evernym.com), and [Renata Toktar](renata.toktar@evernym.com) +- Start Date: 2020-12-22 + + +## Status +- Status: [PROPOSED](/README.md#hipe-lifecycle) +- Status Date: 2020-01-22 + + +## Summary +[summary]: #summary + +Ledgers that are created by Indy plugins cannot be removed without breaking consensus. This HIPE describes functionality to freeze a ledger so that it can be safely removed. + + +## Motivation +[motivation]: #motivation + +Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). + +Removing the plugin has three important consequences: +* The data stored on the custom ledger created by the plugin will be deleted. +* The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. +* Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. + +The first consequence is intended. This HIPE proposes a feature to address the second consequences to ensure that the network remains functional after the plugin is removed. The last consequence is [treated in a separate HIPE](https://github.com/hyperledger/indy-hipe/tree/master/text/0163-default-fee-handler). + + +## Tutorial +[tutorial]: #tutorial + +If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: + +* It preserves the ability to do rolling updates. Without this capability, attempting to remove a plugin during a rolling update risks breaking consensus if a transaction is submitted to the ledger during the roll out period. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. +* It allows the removal of old plugins. Frozen ledgers can be removed and there will be no left over data from the removed plugin. The removal of old plugins will also help reduce the cost of keeping the network stable and secure. + +Freezing ledgers requires the following workflow: +* Upgrade all nodes to a version of Indy that supports frozen ledgers. +* Send a LEDGERS_FREEZE transaction for the ledgers that you want to drop. +* Upgrade all nodes to remove the plugin. +* Execute a migration script to drop the ledgers. + +Permissions around the LEDGERS_FREEZE transaction are managed in the auth_rules with a default permission requiring the approval of three trustees ([the same default auth_rule for upgrading the ledger](https://github.com/hyperledger/indy-node/blob/master/docs/source/auth_rules.md)). + + +## Reference +[reference]: #reference + +If a ledger is frozen it can be used neither for reading nor for writing. It will not be caught up by new nodes and can be safely removed. Default ledgers such as the domain, config, pool, and audit ledgers cannot be frozen. + +### Implementation notes: +* Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. +* We were nervous that removing transactions would break foreign keys in auth_rules, but our testing showed that this is not an issue. +* Frozen ledgers cannot be unfrozen, but dropped ledgers can be recreated with a different ledger ID. + +### Implementation steps: +1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state. +2. Implement GET_FROZEN_LEDGERS transaction handler that will return information about frozen ledgers from state. +3. Add dynamic validation to all handlers: check if the ledger is frozen. +4. Update audit logic by taking into account the root hashes of frozen ledgers. +5. Fix catch up to not catch up frozen ledgers. + +### State schema: +There will be a key in CONFIG_LEDGER that will store a list of frozen ledgers in the following format: + +``` +2:FROZEN_LEDGERS = [ + : { + ledger: , +        state: , +        seq_no: + }, + ... +] +``` + +### New transactions: + +Freeze ledgers that have the provided IDs: +``` +LEDGERS_FREEZE { + LEDGERS_IDS: [int] +} +``` + +Return frozen ledgers and their root hashes. +``` +GET_FROZEN_LEDGERS +``` + + +## Drawbacks +[drawbacks]: #drawbacks + +The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to be replayed with the correct ledger hash. + + +## Rationale and alternatives +[alternatives]: #alternatives + +The current need for freezing a ledger can be avoided by moving the token transactions into Indy as deprecated historical transactions. This would allow the history to be validated, but we don't want Indy to become a graveyard for every transaction type a plugin author defines. Implementing frozen ledgers is a more generic way to address the problem. + +There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger, but no one has volunteered to do that work. Even if Indy does one day have a generic token, the ability to freeze ledgers would still be useful for the development and maintenance of other ledger plugins. + +We considered implementing the frozen ledger hash in a plugin, but we consider it a generally useful feature and believe it should belong in Indy. + + +## Prior art +[prior-art]: #prior-art + +No relevant prior art. + + +## Unresolved questions +[unresolved-questions]: #unresolved-questions + +All questions raised during the development of this proposal have been answered in this draft of the proposal. diff --git a/text/0162-plugin-removal-helpers/README.md b/text/0162-plugin-removal-helpers/README.md deleted file mode 100644 index d980b9ff..00000000 --- a/text/0162-plugin-removal-helpers/README.md +++ /dev/null @@ -1,160 +0,0 @@ -# Indy HIPE 0162: Features to Help Remove Ledger Plugins -- Author: [Alexandr Kolesov](alexander.kolesov@evernym.com), [Richard Esplin](mailto:richard.esplin@evernym.com), and [Renata Toktar](renata.toktar@evernym.com) -- Start Date: 2020-12-01 - -## Status -- Status: [ADOPTED](/README.md#hipe-lifecycle) -- Status Date: 2020-01-22 -- Status Note: The changes in this HIPE were discussed in the Indy Maintainers calls on [2020-12-22](https://wiki.hyperledger.org/display/indy/2020-12-22+Indy+Contributors+Call) and [2020-01-19](https://wiki.hyperledger.org/display/indy/2021-01-19+Indy+Contributors+Call). The other maintainers approved the plan, but requested a HIPE to be submitted. - -## Summary -[summary]: #summary - -In this HIPE, we propose two new features for Indy Node that will assist with removing ledger plugins that are no longer used: frozen ledgers and default fee handlers. - -## Motivation -[motivation]: #motivation - -Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). - -Removing the plugin has three important consequences: -* The data stored on the custom ledger created by the plugin will be deleted. -* The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. -* Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. - -The first consequence is intended. This HIPE proposes feature to address the remaining two consequences to ensure that the network remains functional after the plugin is removed. - -We include both features in the same HIPE because they share a common motivation and are likely to be used together. - -## Frozen Ledgers -[frozen-ledgers]: #frozen-ledgers - -### Tutorial -[frozen-ledgers-tutorial]: #frozen-ledgers-tutorial - -If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: - -* It preserves the ability to do rolling updates. Without this capability, attempting to remove a plugin during a rolling update risks breaking consensus if a transaction is submitted to the ledger during the roll out period. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. -* It allows the removal of old plugins. Frozen ledgers can be removed and there will be no left over data from the removed plugin. The removal of old plugins will also help reduce the cost of keeping the network stable and secure. - -Freezing ledgers requires the following workflow: -* Upgrade all nodes to a version of Indy that supports frozen ledgers. -* Send a LEDGERS_FREEZE transaction for the ledgers that you want to drop. -* Upgrade all nodes to remove the plugin. -* Execute a migration script to drop the ledgers. - -Permissions around the LEDGERS_FREEZE transaction are managed in the auth_rules with a default permission requiring the approval of three trustees ([the same default auth_rule for upgrading the ledger](https://github.com/hyperledger/indy-node/blob/master/docs/source/auth_rules.md)). - -### Reference -[frozen-ledgers-reference]: #frozen-ledgers-reference - -If a ledger is frozen it can be used neither for reading nor for writing. It will not be caught up by new nodes and can be safely removed. Default ledgers such as the domain, config, pool, and audit ledgers cannot be frozen. - -#### Implementation notes: -* Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. -* We were nervous that removing transactions would break foreign keys in auth_rules, but our testing showed that this is not an issue. -* Frozen ledgers cannot be unfrozen, but dropped ledgers can be recreated with a different ledger ID. - -#### Implementation steps: -1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state. -2. Implement GET_FROZEN_LEDGERS transaction handler that will return information about frozen ledgers from state. -3. Add dynamic validation to all handlers: check if the ledger is frozen. -4. Update audit logic by taking into account the root hashes of frozen ledgers. -5. Fix catch up to not catch up frozen ledgers. - -#### State schema: -There will be a key in CONFIG_LEDGER that will store a list of frozen ledgers in the following format: - -``` -2:FROZEN_LEDGERS = [ - : { - ledger: , -        state: , -        seq_no: - }, - ... -] -``` - -#### New transactions: - -Freeze ledgers that have the provided IDs: -``` -LEDGERS_FREEZE { - LEDGERS_IDS: [int] -} -``` - -Return frozen ledgers and their root hashes. -``` -GET_FROZEN_LEDGERS -``` - -### Drawbacks -[frozen-ledgers-drawbacks]: #frozen-ledgers-drawbacks - -The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to be replayed with the correct ledger hash. - -### Rationale and alternatives -[frozen-ledgers-rationale-and-alternatives]: #frozen-ledgers-rationale-and-alternatives - -We considered implementing the frozen ledger hash in a plugin, but we consider it a generally useful feature and believe it should belong in Indy. - -### Prior art -[frozen-ledgers-prior-art]: #frozen-ledgers-prior-art - -No relevant prior art. - -### Unresolved questions -[frozen-ledgers-unresolved-questions]: #frozen-ledgers-unresolved-questions - -All questions raised during the development of this proposal have been answered in this draft of the proposal. - - -## Default Fee Handlers -[default-fee-handler]: #default-fee-handler - -### Tutorial -[default-fee-handler-tutorial]: #default-fee-handler-tutorial - -[Indy Authorization Rules](https://github.com/hyperledger/indy-node/blob/master/docs/source/requests.md#auth_rule) provide a flexible way of authorizing write transactions to the domain ledger. This authorization can [depend on the payment of fees tracked by ledger plugins](https://github.com/sovrin-foundation/libsovtoken/blob/master/doc/fees.md). - -Because Indy currently has no native concept of fees, if a plugin that implemented fees is removed, historical transactions can no longer be validated and will prevent new nodes from catching up on the ledger. In many cases it is sufficient to know that the historical transactions were valid at the time of writing. In these cases a default fee handler can approve the historical transactions. - -### Reference -[default-fee-handler-reference]: #default-fee-handler-reference - -Implementation will require adding to Indy [the SET_FEE transaction copied from the sovtoken plugin](https://github.com/sovrin-foundation/token-plugin/blob/a635780361a7478ea4b2f47fcffee78b150fdea3/sovtokenfees/sovtokenfees/req_handlers/write_handlers/set_fees_handler.py#L20) and [a default fee handler copied from the sovtoken fee handler (FeesAuthorizer)](https://github.com/sovrin-foundation/token-plugin/blob/master/sovtokenfees/sovtokenfees/fees_authorizer.py#L26). The fee handler will have the validation logic replaced with `throw NotAllowedException()`. - -Any Indy transaction handler consists of two parts: validation logic and execution logic. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. The default fee handler will contain a copy of the sovtoken fee handler but with validation disabled, so it will allow catchup but will not allow new transactions with fees. - -Indy will also look to the default fee handler to determine the execution logic for fee related transactions, but the default fee handler will not allow any new fee transactions and so doesn't need any specific execution logic. Any future implementation of a payment plugin will need to override the default fee handler with a functional implementation of validation and execution logic. - -### Drawbacks -[default-fee-handler-drawbacks]: #default-fee-handler-drawbacks - -* Historically there were concerns about having payment functionality included in Hyperledger projects, but as distributed payments have become more mainstream, it now seems acceptable to include into Indy generic primitives related to payments. -* The default fee handler replays transactions without performing validation. This is fine for catching up validator nodes, but an audit of the ledger history would need to take into account the historical validation expected by any removed plugins. -* The ledger could be erroneously configured with auth_rules that define fees without having a plugin installed to process them. The exact behavior in this scenario will depend on the specific rules that mention fees, and could lead to confusing and hard to diagnose behavior. However this is a niche case because SET_FEE transactions will be rejected if there is no plugin to define them. -* The existence of a default fee handler in Indy Node could in theory be exploited by an attacker, but we consider this a small risk for a distributed ledger as any attack would have to simultaneously succeed on a majority of validation nodes in order to change what gets written. - - -### Prior art -[default-fee-handler-prior-art]: #default-fee-handler-prior-art - -No relevant prior art. - -### Unresolved questions -[default-fee-handler-unresolved-questions]: #default-fee-handler-unresolved-questions - -All questions raised during the development of this proposal have been answered in this draft of the proposal. - -## Both -[both]: #both - -### Rationale and alternatives -[both-rationale-and-alternatives]: #both-rationale-and-alternatives - -Both of these features could be avoided by moving the token transactions into Indy as deprecated historical transactions. This would allow the history to be validated, but we don't want Indy to become a graveyard for every transaction type a plugin author defines. Implementing the features proposed in this HIPE is a more generic way to address the problem. - -There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger, but no one has volunteered to do that work. Even if Indy does one day have a generic token, the features proposed in this HIPE would still be useful for the development and maintenance of other ledger plugins. diff --git a/text/0163-default-fee-handler/README.md b/text/0163-default-fee-handler/README.md new file mode 100644 index 00000000..cbae3375 --- /dev/null +++ b/text/0163-default-fee-handler/README.md @@ -0,0 +1,75 @@ +# Indy HIPE 0163: Default Fee Handler +- Author: [Alexandr Kolesov](alexander.kolesov@evernym.com), [Richard Esplin](mailto:richard.esplin@evernym.com), and [Renata Toktar](renata.toktar@evernym.com) +- Start Date: 2020-12-01 + + +## Status +- Status: [ADOPTED](/README.md#hipe-lifecycle) +- Status Date: 2020-01-22 +- Status Note: The changes in this HIPE were discussed in the Indy Maintainers calls on [2020-12-22](https://wiki.hyperledger.org/display/indy/2020-12-22+Indy+Contributors+Call) and [2020-01-19](https://wiki.hyperledger.org/display/indy/2021-01-19+Indy+Contributors+Call). The other maintainers approved the plan, but requested a HIPE to be submitted. + + +## Summary +[summary]: #summary + +Adding a default fee handler to Indy will make it easier to maintain ledgers with payment plugins. + + +## Motivation +[motivation]: #motivation + +Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). + +Removing the plugin has three important consequences: +* The data stored on the custom ledger created by the plugin will be deleted. +* The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. +* Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. + +The first consequence is intended. The second consequence is [treated in a separate HIPE](https://github.com/hyperledger/indy-hipe/tree/master/text/0162-freeze-ledgers). This HIPE proposes a feature to address the last consequence. + + +## Tutorial +[tutorial]: #tutorial + +[Indy Authorization Rules](https://github.com/hyperledger/indy-node/blob/master/docs/source/requests.md#auth_rule) provide a flexible way of authorizing write transactions to the domain ledger. This authorization can [depend on the payment of fees tracked by ledger plugins](https://github.com/sovrin-foundation/libsovtoken/blob/master/doc/fees.md). + +Because Indy currently has no native concept of fees, if a plugin that implemented fees is removed, historical transactions can no longer be validated and will prevent new nodes from catching up on the ledger. In many cases it is sufficient to know that the historical transactions were valid at the time of writing. In these cases a default fee handler can approve the historical transactions. + + +## Reference +[reference]: #reference + +Implementation will require adding to Indy [the SET_FEE transaction copied from the sovtoken plugin](https://github.com/sovrin-foundation/token-plugin/blob/a635780361a7478ea4b2f47fcffee78b150fdea3/sovtokenfees/sovtokenfees/req_handlers/write_handlers/set_fees_handler.py#L20) and [a default fee handler copied from the sovtoken fee handler (FeesAuthorizer)](https://github.com/sovrin-foundation/token-plugin/blob/master/sovtokenfees/sovtokenfees/fees_authorizer.py#L26). The fee handler will have the validation logic replaced with `throw NotAllowedException()`. + +Any Indy transaction handler consists of two parts: validation logic and execution logic. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger, so we know that validation was already performed and we don't need to do it again. The default fee handler will contain a copy of the sovtoken fee handler but with validation disabled, so it will allow catchup but will not allow new transactions with fees. + +Indy will also look to the default fee handler to determine the execution logic for fee related transactions, but the default fee handler will not allow any new fee transactions and so doesn't need any specific execution logic. Any future implementation of a payment plugin will need to override the default fee handler with a functional implementation of validation and execution logic. + + +## Drawbacks +[drawbacks]: #drawbacks + +* Historically there were concerns about having payment functionality included in Hyperledger projects, but as distributed payments have become more mainstream, it now seems acceptable to include into Indy generic primitives related to payments. +* The default fee handler replays transactions without performing validation. This is fine for catching up validator nodes, but an audit of the ledger history would need to take into account the historical validation expected by any removed plugins. +* The ledger could be erroneously configured with auth_rules that define fees without having a plugin installed to process them. The exact behavior in this scenario will depend on the specific rules that mention fees, and could lead to confusing and hard to diagnose behavior. However this is a niche case because SET_FEE transactions will be rejected if there is no plugin to define them. +* The existence of a default fee handler in Indy Node could in theory be exploited by an attacker, but we consider this a small risk for a distributed ledger as any attack would have to simultaneously succeed on a majority of validation nodes in order to change what gets written. + + +## Rationale and alternatives +[alternatives]: #alternatives + +The current need for a default fee handler could be avoided by moving the token transactions into Indy as deprecated historical transactions. This would allow the history to be validated, but we don't want Indy to become a graveyard for every transaction type a plugin author defines. Implementing the features proposed in this HIPE is a more generic way to address the problem. + +There is a [proposal to incorporate the token plugin as an optional part of Indy](https://github.com/hyperledger/indy-hipe/tree/master/text/0161-generic-token). This would remove the need for the removal of the token plugin from the Sovrin ledger, but no one has volunteered to do that work. + + +## Prior art +[prior-art]: #handler-prior-art + +No relevant prior art. + + +## Unresolved questions +[unresolved-questions]: #unresolved-questions + +All questions raised during the development of this proposal have been answered in this draft of the proposal. From 4cd83dba16a0e15937bffdb44020108b638175d6 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Fri, 29 Jan 2021 20:03:05 -0700 Subject: [PATCH 09/16] Improving explanation about drawback Trying to be clear about why only unused ledgers should be dropped, based on feedback from Kevin W. Signed-off-by: Richard Esplin --- text/0162-freeze-ledgers/README.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/text/0162-freeze-ledgers/README.md b/text/0162-freeze-ledgers/README.md index 46141539..e7e02189 100644 --- a/text/0162-freeze-ledgers/README.md +++ b/text/0162-freeze-ledgers/README.md @@ -30,7 +30,7 @@ The first consequence is intended. This HIPE proposes a feature to address the s ## Tutorial [tutorial]: #tutorial -If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: +If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers in the state trie, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: * It preserves the ability to do rolling updates. Without this capability, attempting to remove a plugin during a rolling update risks breaking consensus if a transaction is submitted to the ledger during the roll out period. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. * It allows the removal of old plugins. Frozen ledgers can be removed and there will be no left over data from the removed plugin. The removal of old plugins will also help reduce the cost of keeping the network stable and secure. @@ -93,7 +93,18 @@ GET_FROZEN_LEDGERS ## Drawbacks [drawbacks]: #drawbacks -The LEDGERS_FREEZE transaction allows removal of the plugin that created the ledger if the ledger was never used and the hash on the audit ledger never changed. Otherwise, the plugin will need to remain installed to allow historical transactions to be replayed with the correct ledger hash. +A frozen ledger should only be removed from the system if it has never been used, because in that scenario the hash on the audit ledger never changed. Understanding this limitation requires consideration of three separate properties of the distributed ledger: +1. The ability to achieve consensus when writing a new transaction to the ledger. +2. The ability to catch up nodes through consensus. +3. The ability to audit the history of the ledger to prove that there was no tampering. + +When a ledger is frozen, the root hash of the deprecated ledger is available to be used by the audit ledger when computing consensus. So the first property is preserved. + +During catchup, the transaction validation logic is not executed. This is because we are catching up transactions that have the ledger BLS signature, so we know it was already validated and we don't need to do it again. So the second property is preserved. + +If a ledger is added (so its root hash is expected by the audit ledger), but it was never used (so the root hash never changed), then LEDGERS_FREEZE will preserve the third property even when the ledger is removed. + +But the third property will not be preserved if there is a history of transactions on a ledger and then the ledger is removed. This is because the LEDGERS_FREEZE transaction only stores the most recent root hash, and not the entire history of root hashes, so that history can not be recreated after the plugin ledger is removed. Instead, the frozen ledger should be retained in its read-only state. ## Rationale and alternatives From a0ce52b6ca21802f6d5c55772fbdf230f8319426 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Fri, 29 Jan 2021 20:06:47 -0700 Subject: [PATCH 10/16] Making the HIPE name more natural. Signed-off-by: Richard Esplin --- text/{0162-freeze-ledgers => 0162-frozen-ledgers}/README.md | 2 +- text/0163-default-fee-handler/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename text/{0162-freeze-ledgers => 0162-frozen-ledgers}/README.md (99%) diff --git a/text/0162-freeze-ledgers/README.md b/text/0162-frozen-ledgers/README.md similarity index 99% rename from text/0162-freeze-ledgers/README.md rename to text/0162-frozen-ledgers/README.md index e7e02189..75290a0b 100644 --- a/text/0162-freeze-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -1,4 +1,4 @@ -# Indy HIPE 0162: Freeze Ledgers +# Indy HIPE 0162: Frozen Ledgers - Author: [Alexandr Kolesov](alexander.kolesov@evernym.com), [Richard Esplin](mailto:richard.esplin@evernym.com), and [Renata Toktar](renata.toktar@evernym.com) - Start Date: 2020-12-22 diff --git a/text/0163-default-fee-handler/README.md b/text/0163-default-fee-handler/README.md index cbae3375..fe9a1ed7 100644 --- a/text/0163-default-fee-handler/README.md +++ b/text/0163-default-fee-handler/README.md @@ -25,7 +25,7 @@ Removing the plugin has three important consequences: * The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. * Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. -The first consequence is intended. The second consequence is [treated in a separate HIPE](https://github.com/hyperledger/indy-hipe/tree/master/text/0162-freeze-ledgers). This HIPE proposes a feature to address the last consequence. +The first consequence is intended. The second consequence is [treated in a separate HIPE](https://github.com/hyperledger/indy-hipe/tree/master/text/0162-frozen-ledgers). This HIPE proposes a feature to address the last consequence. ## Tutorial From 275a322c3625281af4d8cf3dd4f276fd297b019f Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Mon, 1 Feb 2021 10:56:50 -0700 Subject: [PATCH 11/16] Listing error messages, and other minor fix Signed-off-by: Richard Esplin --- text/0162-frozen-ledgers/README.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/text/0162-frozen-ledgers/README.md b/text/0162-frozen-ledgers/README.md index 75290a0b..30019ff5 100644 --- a/text/0162-frozen-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -65,7 +65,7 @@ If a ledger is frozen it can be used neither for reading nor for writing. It wil There will be a key in CONFIG_LEDGER that will store a list of frozen ledgers in the following format: ``` -2:FROZEN_LEDGERS = [ +4:FROZEN_LEDGERS = [ : { ledger: ,         state: , @@ -89,6 +89,16 @@ Return frozen ledgers and their root hashes. GET_FROZEN_LEDGERS ``` +### Error messages: + +Error message for all client actions with a frozen ledger. + +``` +InvalidClientRequest(request.identifier, request.reqId, + "'{}' transaction is forbidden because of " + "'{}' ledger is frozen".format(self.txn_type, self.ledger_id)) +``` + ## Drawbacks [drawbacks]: #drawbacks From cade81a80885c3907c66dac8e994cb32055634c5 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Tue, 2 Feb 2021 10:09:06 -0700 Subject: [PATCH 12/16] Incorporating feedback from Lynn Signed-off-by: Richard Esplin --- text/0162-frozen-ledgers/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0162-frozen-ledgers/README.md b/text/0162-frozen-ledgers/README.md index 30019ff5..ec88a155 100644 --- a/text/0162-frozen-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -33,13 +33,13 @@ The first consequence is intended. This HIPE proposes a feature to address the s If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers in the state trie, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: * It preserves the ability to do rolling updates. Without this capability, attempting to remove a plugin during a rolling update risks breaking consensus if a transaction is submitted to the ledger during the roll out period. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. -* It allows the removal of old plugins. Frozen ledgers can be removed and there will be no left over data from the removed plugin. The removal of old plugins will also help reduce the cost of keeping the network stable and secure. +* It allows the removal of old plugins. Frozen ledgers can optionally be removed and there will be no left over data from the removed plugin. The removal of old plugins will help reduce the cost of keeping the network stable and secure. Freezing ledgers requires the following workflow: * Upgrade all nodes to a version of Indy that supports frozen ledgers. -* Send a LEDGERS_FREEZE transaction for the ledgers that you want to drop. +* Send a LEDGERS_FREEZE transaction for the ledgers that you want to deprecate. * Upgrade all nodes to remove the plugin. -* Execute a migration script to drop the ledgers. +* Optionally, execute a migration script to drop the ledgers that are no longer used. Permissions around the LEDGERS_FREEZE transaction are managed in the auth_rules with a default permission requiring the approval of three trustees ([the same default auth_rule for upgrading the ledger](https://github.com/hyperledger/indy-node/blob/master/docs/source/auth_rules.md)). @@ -52,7 +52,7 @@ If a ledger is frozen it can be used neither for reading nor for writing. It wil ### Implementation notes: * Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. * We were nervous that removing transactions would break foreign keys in auth_rules, but our testing showed that this is not an issue. -* Frozen ledgers cannot be unfrozen, but dropped ledgers can be recreated with a different ledger ID. +* Frozen ledgers cannot be unfrozen, but dropped ledgers can be replaced by creating a new ledger with a different ledger ID. ### Implementation steps: 1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state. From 245da85c4404b87fd3a682ae59b3c2c864a5dac4 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Wed, 24 Feb 2021 23:00:24 -0700 Subject: [PATCH 13/16] Incorporating feedback from WadeBarnes and swcurran. Signed-off-by: Richard Esplin --- text/0162-frozen-ledgers/README.md | 6 +++--- text/0163-default-fee-handler/README.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0162-frozen-ledgers/README.md b/text/0162-frozen-ledgers/README.md index ec88a155..8223c75b 100644 --- a/text/0162-frozen-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -17,9 +17,9 @@ Ledgers that are created by Indy plugins cannot be removed without breaking cons ## Motivation [motivation]: #motivation -Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). +Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [a plan was made to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). -Removing the plugin has three important consequences: +Removing the Sovrin token plugin has three important consequences: * The data stored on the custom ledger created by the plugin will be deleted. * The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. * Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. @@ -30,7 +30,7 @@ The first consequence is intended. This HIPE proposes a feature to address the s ## Tutorial [tutorial]: #tutorial -If a ledger has been created, but will never be used, it can be frozen. The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers in the state trie, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: +If a ledger has been created, but will no longer be used, it can be frozen (refer to the [Drawbacks](#drawbacks) section for important details). The LEDGERS_FREEZE transaction will record the root hashes of one or more ledgers in the state trie, and perpetually use those hashes when computing audit ledger validity. This has two important advantages: * It preserves the ability to do rolling updates. Without this capability, attempting to remove a plugin during a rolling update risks breaking consensus if a transaction is submitted to the ledger during the roll out period. This is because updated nodes would not be able to process new transactions that were ordered by the non-updated nodes. But if we freeze all ledgers associated with the plugin, we can do rolling updates because new transactions related to the plugin will be impossible. * It allows the removal of old plugins. Frozen ledgers can optionally be removed and there will be no left over data from the removed plugin. The removal of old plugins will help reduce the cost of keeping the network stable and secure. diff --git a/text/0163-default-fee-handler/README.md b/text/0163-default-fee-handler/README.md index fe9a1ed7..db88cf9f 100644 --- a/text/0163-default-fee-handler/README.md +++ b/text/0163-default-fee-handler/README.md @@ -18,9 +18,9 @@ Adding a default fee handler to Indy will make it easier to maintain ledgers wit ## Motivation [motivation]: #motivation -Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [they decided to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). +Indy Plenum can be extended through the use of [ledger plugins](https://github.com/hyperledger/indy-plenum/blob/master/docs/source/plugins.md). Ledger plugins can introduce new transaction types that are either stored on existing ledgers, or on new ledgers created by the plugins. The [Sovrin Network](http://sovrin.org) used this capability to develop a [token plugin for Indy](https://github.com/sovrin-foundation/token-plugin) that stored fee and value transfer transactions on a token ledger. After experimenting with the plugin, [a plan was made to remove it](https://github.com/sovrin-foundation/sovrin-sip/tree/master/text/5005-token-removal/README.md). -Removing the plugin has three important consequences: +Removing the Sovrin token plugin has three important consequences: * The data stored on the custom ledger created by the plugin will be deleted. * The audit ledger will no longer be able to obtain the root hash of any custom ledgers from the plugin. * Any authorization rules (auth_rules) on the config ledger that define fees as being necessary to authorize writes to the domain ledger can no longer be handled by the plugin. From 3291be423209ba8425df31d8f3e97cf1a67ddfa3 Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Mon, 8 Mar 2021 20:18:02 -0700 Subject: [PATCH 14/16] Ilcluding additional alternative pointed out by @anton.denishchenko Signed-off-by: Richard Esplin --- text/0162-frozen-ledgers/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/text/0162-frozen-ledgers/README.md b/text/0162-frozen-ledgers/README.md index 8223c75b..337973d3 100644 --- a/text/0162-frozen-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -126,6 +126,11 @@ There is a [proposal to incorporate the token plugin as an optional part of Indy We considered implementing the frozen ledger hash in a plugin, but we consider it a generally useful feature and believe it should belong in Indy. +An auth_rule could be used to prohibit writing to a ledger, but this approach seems more fragile than the frozen ledger transaction because: +* It will not be obvious to the user that transactions are not being recorded because the ledger is deprecated. +* If the auth_rule was changed, writes could be re-enabled for the ledger. If the ledger has not been used for an extended period of time, a hidden error could be uncovered. +* The ledger cannot be removed from the system and so would always need to be maintained. + ## Prior art [prior-art]: #prior-art From 79b394a3799671f1a5085afa816cbd75dd0a404e Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Mon, 8 Mar 2021 20:29:55 -0700 Subject: [PATCH 15/16] Added clarification on how to perform an audit of history. Signed-off-by: Richard Esplin --- text/0162-frozen-ledgers/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0162-frozen-ledgers/README.md b/text/0162-frozen-ledgers/README.md index 337973d3..c2bc7f6c 100644 --- a/text/0162-frozen-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -116,6 +116,8 @@ If a ledger is added (so its root hash is expected by the audit ledger), but it But the third property will not be preserved if there is a history of transactions on a ledger and then the ledger is removed. This is because the LEDGERS_FREEZE transaction only stores the most recent root hash, and not the entire history of root hashes, so that history can not be recreated after the plugin ledger is removed. Instead, the frozen ledger should be retained in its read-only state. +Note that Indy Node does not include functionality to perform an audit of the ledger history based on the original validation rules. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger (it has a state signature), so we know that validation was already performed and we don't need to do it again. An audit of the history would have to be done outside of Indy Node by replaying the transactions using the original validation logic, using the audit ledger to compare the state among the various ledgers. Freezing a ledger does not change this process, but dropping a frozen ledger that contains historical transactions would eliminate data that would be necessary to perform a complete audit of the ledger history. + ## Rationale and alternatives [alternatives]: #alternatives From c41304734fb859dfe0b54dbeb922e5623205f1ad Mon Sep 17 00:00:00 2001 From: Richard Esplin Date: Mon, 8 Mar 2021 20:52:06 -0700 Subject: [PATCH 16/16] Added note about frozen ledgers and catchup. Signed-off-by: Richard Esplin --- text/0162-frozen-ledgers/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/text/0162-frozen-ledgers/README.md b/text/0162-frozen-ledgers/README.md index c2bc7f6c..5b6e1c97 100644 --- a/text/0162-frozen-ledgers/README.md +++ b/text/0162-frozen-ledgers/README.md @@ -53,6 +53,7 @@ If a ledger is frozen it can be used neither for reading nor for writing. It wil * Implementation of this feature is possible because the catchup of the config ledger happens before catchup of other ledgers. * We were nervous that removing transactions would break foreign keys in auth_rules, but our testing showed that this is not an issue. * Frozen ledgers cannot be unfrozen, but dropped ledgers can be replaced by creating a new ledger with a different ledger ID. +* Data from frozen ledgers will not be propagated to new nodes in the pool during catchup. ### Implementation steps: 1. Implement LEDGERS_FREEZE transaction handler that will write information about frozen ledger to state. @@ -114,7 +115,7 @@ During catchup, the transaction validation logic is not executed. This is becaus If a ledger is added (so its root hash is expected by the audit ledger), but it was never used (so the root hash never changed), then LEDGERS_FREEZE will preserve the third property even when the ledger is removed. -But the third property will not be preserved if there is a history of transactions on a ledger and then the ledger is removed. This is because the LEDGERS_FREEZE transaction only stores the most recent root hash, and not the entire history of root hashes, so that history can not be recreated after the plugin ledger is removed. Instead, the frozen ledger should be retained in its read-only state. +But the third property will not be preserved if there is a history of transactions on a ledger and then the ledger is removed. This is because the LEDGERS_FREEZE transaction only stores the most recent root hash, and not the entire history of root hashes, so that history can not be recreated after the plugin ledger is removed. Instead, the frozen ledger should be retained in its read-only state and new nodes in the pool should manually backup the data from frozen ledgers since they are not included in catchup. Note that Indy Node does not include functionality to perform an audit of the ledger history based on the original validation rules. Indy transaction handlers perform validation on new transactions, but not during the catchup process. This is because the transaction is already on the ledger (it has a state signature), so we know that validation was already performed and we don't need to do it again. An audit of the history would have to be done outside of Indy Node by replaying the transactions using the original validation logic, using the audit ledger to compare the state among the various ledgers. Freezing a ledger does not change this process, but dropping a frozen ledger that contains historical transactions would eliminate data that would be necessary to perform a complete audit of the ledger history.