BLOCKED: Large collection (druid:qr157jm7296) is failing to complete collection release

[andrewjbtw]

NOTE: this ticket is blocked while we wait for a resolution of the load-balancer/networking issue being worked on by Ops and UIT.

Describe the bug
A collection object with 14,000+ items is failing to complete the releaseWF: https://argo.stanford.edu/view/druid:qr157jm7296

The release-members step runs for a few minutes and then hits an error. Retrying the step just results in an error a few minutes later. The error message is always:

unable to reach dor-services-app: Connection reset by peer

User Impact
It's not going to be possible to release large collections.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://argo.stanford.edu/view/druid:qr157jm7296
2. Click on releaseWF to view the workflow
3. Choose "rerun" from the dropdown menu.
4. Wait a few minutes
5. See the error again

Expected behavior
The release-members step for a collection release will trigger the release of all items in the collection. That step may take a while and once it's complete, the collection moves on to the next step in WF.

In the past, we have successfully released much larger collections, with 140,000 being the largest that I remember.

Additional context
This might be a similar class of errors to what we've seen in accessionWF.

[justinlittman]

See https://github.com/sul-dlss/operations-tasks/issues/3519

[andrewjbtw]

Based on further observation following repeated retries, release-members runs for about two minutes before failing.

[ndushay]

Julian is supposed to be pinging UIT for help with this.

Under load, the load-balancer drops connections without queuing the requests. (It should send 503 responses at a minimum)

[ndushay]

Does this open a new http connection for each item, or does it send all the requests over one connection? It should work either way.