Tessera timeout can cause inconsistent state when fetching private state

[antonydenyer]

When processing private transactions if a private payload is missing from tessera goquorum will ignore the problem and continue on the assumption that you are not a party to that private transaction. This means that private senders need to ensure that the private payload is in the receivers tessera instance before submitting the ethereum transaction. If, however, goquroum can not communicate with tessera then this is a different problem, you can not know if you should or shouldn't have access to a transaction.

Expected behaviour

When communication with tessera fails, either a 5xx error or a transport layer problem the GoQ node should panic and exit or attempt to retry.

Actual behaviour

It is treated as if the transaction is not found.

This error https://github.com/ConsenSys/quorum/blob/master/private/engine/tessera/tessera.go#L321 is passed up the call stack but is not handled correctly.

[yorkerhodes3]

When can we get the exit/retry logic developed and tested?