Computation failed due to loss of intermediate data

[renjiezh]

Describe the bug
Computation failed due to empty input to a certain stage.
The failure reason from kingdom is "Computation Participant failed. INVALID_ARGUMENT: There is neither actual data nor effective noise in the request." It is caused by an empty input to PHASE_THREE.
After check the cloud storage, it appears worker1 PHASE_TWO and following stages' folders are empty.
From the log, neither worker1 or aggregator's mill has any error message or retry.

The aggregator's PHASE_TWO completes successfully but the output is empty given that PHASE_TWO_INPUT from worker1 is not empty.

Steps to reproduce
Run stress test and there is a chance to reproduce.

Component(s) affected
Duchy

Version
v0.5.6

Environment
QA env

Additional context
MeasurementId = 8441441368107331282

[renjiezh]

Related issue:
All duchies are sharing one bucket in GCS. They also use the same folder name (ComputationGlobalId) for a given Computation. Due to interruption and retry mechanism, it is possible to overwrite each other's target file content.