Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: ability to cancel a running build from dashboard #8850

Merged
merged 10 commits into from
Mar 1, 2022

Conversation

humitos
Copy link
Member

@humitos humitos commented Jan 26, 2022

We tried to implement this in another opportunities (see
#7031) but the build process
was really complex and we had to manage the exception in multiple places.

After implementing Celery Handlers, we can just raise the exception when
attending to the proper signal coming from app.control.revoke and handle it
properly from on_failure task's method.

All the initial local tests were great!

Closes #7660
Closes #8772

@humitos
Copy link
Member Author

humitos commented Jan 26, 2022

We could also revoke a task that hasn't started yet as noted by @astrojuanlu in #7660 (comment)

Button to cancel a queued build that hasn't started yet (i.e. remove it from the queue)

@humitos
Copy link
Member Author

humitos commented Jan 27, 2022

@agjohnson since you are reviewing the PR for the new Celery handlers, it would be good to take a quick look at this one as well to get some feedback about the implementation.

# itself to be executed in the `on_failure` handler.
terminate = True

app.control.revoke(build.task_id, signal=signal.SIGINT, terminate=terminate)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be a helper cancel_build that we could use from other places as well.

Base automatically changed from humitos/celery-handlers to master February 7, 2022 23:39
Copy link
Contributor

@agjohnson agjohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes here looks good to me, I'll see about adding this to the new UI as well

@humitos
Copy link
Member Author

humitos commented Feb 21, 2022

Yay! I'm happy that you are +1 on this. The only thing that I want to test a little more is what happens with supervisord and the celery process itself. I tested it locally and it worked fine, but I'd expect supervisor to restart celery process after a build was canceled because terminate=True, in theory, kills the celery process.

We tried to implement this in another opportunities (see
#7031) but the build process
was really complex and we had to manage the exception in multiple places.

After implementing Celery Handlers, we can just raise the exception when
attending to the proper signal coming from `app.control.revoke` and handle it
properly from `on_failure` task's method.

All the initial local tests were great!
@humitos humitos marked this pull request as ready for review February 22, 2022 14:51
@humitos humitos requested a review from a team as a code owner February 22, 2022 14:51
@humitos
Copy link
Member Author

humitos commented Feb 22, 2022

I QAed ths locally and the celery process was never killed. The PID is always the same even after canceling multiple builds (while building and before starting them). I think the implementation is good enough and is worth testing it in production. Since the process is not killed at all, I'm not expecting supervisord to interfere here --it should not notice anything different.

I'm targeting this feature for next deploy, on 2022-03-01.

`task.id` is a mock.Mock object and it's not possible to save it into the
database. I've tried using `.return_value` and others but I failed when trying
to mock it and return a valid str/int.

So, for now, we can keep this hack here and only save the value into the
database if it's a str/int.
Comment on lines +234 to +241
# FIXME: I'm using `isinstance` here because I wasn't able to mock this
# properly when running tests and it fails when trying to save a
# `mock.Mock` object in the database.
#
# Store the task_id in the build object to be able to cancel it later.
if isinstance(task.id, (str, int)):
build.task_id = task.id
build.save()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stsewd do you have an idea how we can mock this? At this point update_docs_task and task are mock.Mock objects. It would be good if we can make task.id to return an integer or a string. I've tried using mock.return_value and others and I wasn't able to make it work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which test is running this code? Something like update_docs_task_mock.apply_async.return_value = MagicMock(id=1) should do it.

maybe update_docs_task_mock.apply_async().id = 1 could work as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which test is running this code?

Most of the tests from readthedocs/projects/tests/test_build_tasks.py. For example, test_successful_build

The solution that you suggested is what I've tried and I wasn't able to mock them properly 😞

I'd appreciate it if you find out the way to mock it. Otherwise, I think I will commit the code with this if for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All tests from that file pass for me. But some tests from rtd_tests/tests/test_builds.py fail, but with a mock like this update_docs_task.signature().apply_async().id = 1 they pass

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! I'm merging it for now because I want to deploy this change today. I'll come back to this in the following days. If you have the diff required at hand, can you open a PR with the changes?

@humitos humitos merged commit eba4025 into master Mar 1, 2022
@humitos humitos deleted the humitos/cancel-build branch March 1, 2022 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RTD PR builds get stuck Feature Request: Allow maintainers to cancel PR builds
3 participants