Investigate sqlalchemy.exc:NoResultFound, which could be related to the botocore.errorfactory:NoSuchKey errors

[ccostino]

We're still seeing some sqlalchemy.exc:NoResultFound come through and these could be indicative of jobs not having been properly processed/written because they couldn't be written and/or retrieved from S3. We need to understand why these are happening to track down where any potential breakdowns are taking place.

Here is a stack trace when attempting to retrieve job info:

Traceback (most recent call last):
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/eventlet/greenthread.py", line 265, in main
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/gunicorn/workers/geventlet.py", line 157, in handle
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/gunicorn/workers/base_async.py", line 55, in handle
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/newrelic/api/wsgi_application.py", line 665, in _nr_wsgi_application_wrapper_
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/flask/app.py", line 1498, in __call__
File "/home/vcap/app/notifications_utils/request_helper.py", line 81, in __call__
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/newrelic/api/wsgi_application.py", line 560, in _nr_wsgi_application_wrapper_
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/newrelic/hooks/framework_flask.py", line 79, in _nr_wrapper_handler_
File "/home/vcap/app/app/job/rest.py", line 46, in get_job_by_service_and_job_id
File "/home/vcap/app/app/dao/jobs_dao.py", line 43, in dao_get_job_by_service_id_and_job_id
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/sqlalchemy/engine/result.py", line 1810, in one
File "/home/vcap/deps/0/python/lib/python3.12/site-packages/sqlalchemy/engine/result.py", line 752, in _only_one_row

Implementation Sketch and Acceptance Critiera

• Trace the code to understand how this condition can occur in the first place - what would lead a job to not be found in the first place?
• Check to see if this is related at all to Research why botocore.errorfactory:NoSuchKey errors continue to be thrown and what can be done to prevent them #1392 - is a symptom or cause of that, or related some other way?
• Check to see if we need to update our handling of when this situation occurs, if we cannot prevent it from happening
• Write new or update existing tests as necessary

Security Considerations

• We need to make sure our application is handling exceptions and edge cases properly so it remains stable and functional, and so users know what's happening and how to correct it.