Update remove_ledger.py to protect against removing a ledger that has been used

[WadeBarnes]

Update remove_ledger.py to protect against removing a ledger that has been used.

If a ledger has been used (it contains transactions) it's history is critical to the historical integrity if the network, even if the ledger has been frozen. Without the historical data for a given ledger the ability to audit the history of the ledger to prove that there was no tampering is not possible. Therefore it is important to protect such ledgers from being deleted.

Refer to the Drawbacks section of the 0162-frozen-ledgers HIPE in hyperledger/indy-hipe#162.

[Toktar]

Hello.
@WadeBarnes Thanks for your ideas. But why do you say that removing the plugin ledger (non-empty) will compromise the integrity of the system?
From the point of view of the system, if the LEDGERS_FREEZE transaction was applied, then it happened by consensus. And for the complete loss of the ledger, all stewards on all nodes must start this script manually.
From the point of view of business logic, audit problems will only be for the removed ledger. But this is obvious, since when we delete a database, we lose the ability to read from this database.

[WadeBarnes]

@Toktar, Exactly, that is the concern, the impact of not being able to read the data. The conversation and answers are best discussed on the HIPE. I think we need to better understand the impact of 1) Freezing a Ledger, 2) Removing the associated plug-in, and 3) Deleting the Ledger. It's not 100% clear what the overall impact is, especially to the audit history of the ledger (and/or associated ledgers), and how this will present itself under various conditions.

One such scenario; "Without the plugin (once the ledger has been frozen and the plugin removed) how can the ledger history be validated when there have been transactions processed by it?"

Lets continue the conversation over on the PR for the HIPE; hyperledger/indy-hipe#162

[esplinr]

@m00sey pointed out that there are various developer use cases where people will likely want to delete ledgers with data in them.

Specifically, the goal of the task is to delete the token history for the Sovrin test ledgers. This is safe because they are not intended to have a permanent history (see my reply in the associated sovrin-sip).

But I agree that there should be a warning. I suggest:

• By default, the script should fail if the ledger is not empty,
• To proceed with the deletion, the user should need to add a --delete-data flag.

This this address your concern @WadeBarnes ?

Is this reasonably easy @Toktar ?

[Toktar]

@esplinr This can be not easy due to the opening of the database in multiple threads. But we believe that this can be solved and will take several days.

[WadeBarnes]

@m00sey pointed out that there are various developer use cases where people will likely want to delete ledgers with data in them.

I'm not disputing there are legitimate use cases. However it is imperative the ramifications of such actions are clear and fully understood before actions are taken.

Specifically, the goal of the task is to delete the token history for the Sovrin test ledgers. This is safe because they are not intended to have a permanent history (see my reply in the associated sovrin-sip).

I'll have a look.

But I agree that there should be a warning. I suggest:

• By default, the script should fail if the ledger is not empty,
• To proceed with the deletion, the user should need to add a --delete-data flag.

Yes, that would address the concerns with this script.

[esplinr]

@Toktar Can you elaborate on why checking if the ledger is empty will take several days of development? What do you mean "opening a database in several threads"?

It's not worth multiple days of effort. Our work is better focused on resolving the blockers to release. If it can't be done quickly, then a warning should be added to the top of the script and we should move on.