Uncaught exceptions causing site instability

[cms21]

There is an issue of uncaught exceptions causing site processes to die and the site to go down. Some appear to be specific to PBS (e.g. issue #323 and #324 ) but some seem to be arising in parts of the code not associated with PBS calls.

[cms21]

An example error:
2023-02-28 18:09:52.380 | 46166 | ERROR | balsam:120] Uncaught Exception <class 'requests.exceptions.HTTPError'>: 521 Server Error: for url: https://balsam-dev.alcf.anl.gov/transfers/?site_id=431&state=active&state=inactive&limit=100000&offset=0 Traceback (most recent call last): File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/util/process.py", line 17, in run self._run() File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/site/service/service_base.py", line 23, in _run self.run_cycle() File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/site/service/transfer.py", line 171, in run_cycle logger.debug(f"In-flight TransferItems: {[(item.id, item.state, item.task_id) for item in in_flight]}") File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/_api/query.py", line 114, in __iter__ self._fetch_cache() File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/_api/query.py", line 124, in _fetch_cache instances, count = self._manager._get_list( File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/_api/manager.py", line 202, in _get_list count, results = self._fetch_pages(filter_chunk, ordering, limit, offset) File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/_api/manager.py", line 169, in _fetch_pages response_data = self._client.get(self._api_path, **query_params) File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/client/rest_base_client.py", line 65, in get return self.request(url, "GET", params=kwargs) File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/client/requests_client.py", line 97, in request response = self._do_request(absolute_url, http_method, params, json, data) File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/client/requests_client.py", line 129, in _do_request self._raise_with_explanation(response) File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/balsam/client/requests_client.py", line 149, in _raise_with_explanation response.raise_for_status() File "/eagle/datascience/parton/conda/2023-02/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 521 Server Error: for url: https://balsam-dev.alcf.anl.gov/transfers/?site_id=431&state=active&state=inactive&limit=100000&offset=0

[cms21]

Addressed with #334