Detect dead jobs and set to 'failed' status #806
Conversation
|
#585 does mention other stages, so we should implement this for the other three job_types as well. |
| $locked = intval( $wpdb->get_row( $query )->lck ); | ||
| if ( ! $locked ) { | ||
| WsLog::l( 'Failed to acquire "wp2static_jobs_deploying" lock.' ); | ||
| return; |
There was a problem hiding this comment.
Will this still let registered post deployment hooks fire? ie, if S3 deploy fails and we return here, might it still call CF invalidation unnecessarily?
There was a problem hiding this comment.
No, we're not catching the exceptions so it will not run post-deploy hooks in that case.
| $deployer | ||
| ); | ||
| } finally { | ||
| $wpdb->query( "DO RELEASE_LOCK('wp2static_jobs_deploying')" ); |
There was a problem hiding this comment.
If this ever fails, do we have a way to manually clean up, like with caches?
There was a problem hiding this comment.
In testing, releasing here was not even necessary because WP apparently closes the connection after the request, which implicitly releases the locks. If we ever run into a problem with a customized WP or something, we can add a config option to turn this off.
|
@john-shaffer added cpl minor comments/questions. Also, if we implement this for the crawling stage, will we still be able to catch and nicely log specific crawl errors, like network timeouts? You suggested we do for other job types - wanna do in this PR or create new tickets for later? |
I've only implemented this for deploy jobs since that's the only place I've seen a problem, but we could pretty easily use this pattern for every job type if needed.