reduce number of pool connections

[terrazoon]

Description

We saw some BrokenPipeErrors that caused messages not to be sent. Some of the stack traces pointed in the direction of multiprocessing.Manager, which handles downloading S3 objects to build up our jobs cache.

One recommendation when seeing BrokenPipeError is to reduce the number of connections in the connection pool, and that is the easiest fix to try.

Others are:

1. Increase memory and CPU
2. Use more retry logic (yuck)
3. Consider replacing multiprocessing.Manager with multiprocessing.Queue or Pipe

NOTE: The multiprocessing.Manager is used in app/aws/s3.py and the whole cache-building workflow also relies on concurrent.futures.ThreadPoolExecutor. Both rely on the AWS_CLIENT_CONFIG, which defines the maximum number of pool connections.

Security Considerations

N/A

[ccostino]

Thank you, @terrazoon!