Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: make sure to update commit info #158

Merged

Conversation

giovanni-guidini
Copy link
Contributor

The pre-process commit task doesn't check if the commit data has
been already pulled from the provider. This can cause an error where
the parent commit is not found to carryforward the reports.

These changes make sure that we try to update the commit info if necessary.
It also moves the function - that was repeated in a few places - to
RepositoryService.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. In 2022 this entity acquired Codecov and as result Sentry is going to need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

The pre-process commit task doesn't check if the commit data has
been already pulled from the provider. This can cause an error where
the parent commit is not found to carryforward the reports.

These changes make sure that we try to update the commit info if necessary.
It also moves the function - that was repeated in a few places - to
RepositoryService.
@giovanni-guidini giovanni-guidini force-pushed the gio/possibly-update-commit-before-carryforward-reports branch from ad2d337 to 01473f1 Compare October 24, 2023 20:15
@codecov-staging
Copy link

codecov-staging bot commented Oct 24, 2023

Codecov Report

Merging #158 (01473f1) into main (b9e2bf6) will decrease coverage by 0.01%.
The diff coverage is 90.90%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #158      +/-   ##
==========================================
- Coverage   98.39%   98.38%   -0.01%     
==========================================
  Files         348      348              
  Lines       27464    27451      -13     
==========================================
- Hits        27022    27009      -13     
  Misses        442      442              
Flag Coverage Δ
integration 98.38% <90.90%> (-0.01%) ⬇️
latest-uploader-overall 98.38% <90.90%> (-0.01%) ⬇️
unit 98.38% <90.90%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.89% <88.88%> (-0.01%) ⬇️
OutsideTasks 98.18% <84.61%> (-0.01%) ⬇️
Files Coverage Δ
tasks/commit_update.py 100.00% <100.00%> (ø)
tasks/preprocess_upload.py 79.71% <100.00%> (+0.29%) ⬆️
tasks/tests/unit/test_commit_update.py 100.00% <ø> (ø)
tasks/tests/unit/test_preprocess_upload.py 100.00% <ø> (ø)
tasks/tests/unit/test_upload_task.py 99.44% <100.00%> (-0.01%) ⬇️
tasks/upload.py 99.07% <100.00%> (+0.81%) ⬆️
services/repository.py 95.95% <84.61%> (-0.80%) ⬇️

@codecov-qa
Copy link

codecov-qa bot commented Oct 24, 2023

Codecov Report

Merging #158 (01473f1) into main (b9e2bf6) will decrease coverage by 0.01%.
The diff coverage is 90.90%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #158      +/-   ##
==========================================
- Coverage   98.39%   98.38%   -0.01%     
==========================================
  Files         348      348              
  Lines       27464    27451      -13     
==========================================
- Hits        27022    27009      -13     
  Misses        442      442              
Flag Coverage Δ
integration 98.38% <90.90%> (-0.01%) ⬇️
latest-uploader-overall 98.38% <90.90%> (-0.01%) ⬇️
unit 98.38% <90.90%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.89% <88.88%> (-0.01%) ⬇️
OutsideTasks 98.18% <84.61%> (-0.01%) ⬇️
Files Coverage Δ
tasks/commit_update.py 100.00% <100.00%> (ø)
tasks/preprocess_upload.py 79.71% <100.00%> (+0.29%) ⬆️
tasks/tests/unit/test_commit_update.py 100.00% <ø> (ø)
tasks/tests/unit/test_preprocess_upload.py 100.00% <ø> (ø)
tasks/tests/unit/test_upload_task.py 99.44% <100.00%> (-0.01%) ⬇️
tasks/upload.py 99.07% <100.00%> (+0.81%) ⬆️
services/repository.py 95.95% <84.61%> (-0.80%) ⬇️

@codecov-public-qa
Copy link

codecov-public-qa bot commented Oct 24, 2023

Codecov Report

Merging #158 (01473f1) into main (b9e2bf6) will decrease coverage by 0.01%.
The diff coverage is 90.90%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #158      +/-   ##
==========================================
- Coverage   98.39%   98.38%   -0.01%     
==========================================
  Files         348      348              
  Lines       27464    27451      -13     
==========================================
- Hits        27022    27009      -13     
  Misses        442      442              
Flag Coverage Δ
integration 98.38% <90.90%> (-0.01%) ⬇️
latest-uploader-overall 98.38% <90.90%> (-0.01%) ⬇️
unit 98.38% <90.90%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.89% <88.88%> (-0.01%) ⬇️
OutsideTasks 98.18% <84.61%> (-0.01%) ⬇️
Files Coverage Δ
tasks/commit_update.py 100.00% <100.00%> (ø)
tasks/preprocess_upload.py 79.71% <100.00%> (+0.29%) ⬆️
tasks/tests/unit/test_commit_update.py 100.00% <ø> (ø)
tasks/tests/unit/test_preprocess_upload.py 100.00% <ø> (ø)
tasks/tests/unit/test_upload_task.py 99.44% <100.00%> (-0.01%) ⬇️
tasks/upload.py 99.07% <100.00%> (+0.81%) ⬆️
services/repository.py 95.95% <84.61%> (-0.80%) ⬇️

@codecov
Copy link

codecov bot commented Oct 24, 2023

Codecov Report

Merging #158 (01473f1) into main (b9e2bf6) will decrease coverage by 0.01%.
The diff coverage is 90.90%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #158      +/-   ##
==========================================
- Coverage   98.35%   98.35%   -0.01%     
==========================================
  Files         374      374              
  Lines       27960    27947      -13     
==========================================
- Hits        27499    27486      -13     
  Misses        461      461              
Flag Coverage Δ
integration 98.38% <90.90%> (-0.01%) ⬇️
latest-uploader-overall 98.38% <90.90%> (-0.01%) ⬇️
unit 98.38% <90.90%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
NonTestCode 96.78% <88.88%> (-0.01%) ⬇️
OutsideTasks 98.18% <84.61%> (-0.01%) ⬇️
Files Coverage Δ
tasks/commit_update.py 100.00% <100.00%> (ø)
tasks/preprocess_upload.py 80.00% <100.00%> (+0.28%) ⬆️
tasks/tests/unit/test_commit_update.py 100.00% <ø> (ø)
tasks/tests/unit/test_preprocess_upload.py 100.00% <ø> (ø)
tasks/tests/unit/test_upload_task.py 99.44% <100.00%> (-0.01%) ⬇️
tasks/upload.py 99.08% <100.00%> (+0.81%) ⬆️
services/repository.py 95.95% <84.61%> (-0.80%) ⬇️
Related Entrypoints
run/app.tasks.upload.Upload
run/app.tasks.commit_update.CommitUpdate

Copy link
Contributor

@scott-codecov scott-codecov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change looks good. I think we have a more general issue w/ our architecture around this though - just sharing some thoughts here.

The preprocess commit task is enqueued during the create-report stage of an upload. The task gets put on a queue and a subsequent do-upload is run (which then enqueues an upload task). Say the preprocess task gets popped by a worker that is slow to process the task ahead of it (tasks are prefetched in bulk). Then the upload task might execute before preprocess and we have the same issue. Granted this is probably not going to happen very often but I think we are not strictly enforcing the order of execution when we don't use Celery's chain functionality.

Am I overthinking this?

@giovanni-guidini
Copy link
Contributor Author

Change looks good. I think we have a more general issue w/ our architecture around this though - just sharing some thoughts here.

The preprocess commit task is enqueued during the create-report stage of an upload. The task gets put on a queue and a subsequent do-upload is run (which then enqueues an upload task). Say the preprocess task gets popped by a worker that is slow to process the task ahead of it (tasks are prefetched in bulk). Then the upload task might execute before preprocess and we have the same issue. Granted this is probably not going to happen very often but I think we are not strictly enforcing the order of execution when we don't use Celery's chain functionality.

Am I overthinking this?

You are not overthinking. But I'd say strictly speaking we cannot assume a task will run before another. That is part of the celery architecture. But the upload task does have these same checks in place, so we shouldn't have problems.

@giovanni-guidini giovanni-guidini merged commit 1d2e5a9 into main Oct 24, 2023
13 of 30 checks passed
@giovanni-guidini giovanni-guidini deleted the gio/possibly-update-commit-before-carryforward-reports branch October 24, 2023 20:51
@scott-codecov
Copy link
Contributor

we cannot assume a task will run before another. That is part of the celery architecture

Not even when using chain? I figured the next task was not even enqueued until the previous task is completed.

@giovanni-guidini
Copy link
Contributor Author

Oh definitely you are right about chain.
But these tasks come from different CLI commands. I don't think we can chain them together.

Copy link

sentry-io bot commented Nov 2, 2023

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ TorngitServer5xxCodeError: Github is having 5xx issues app.tasks.upload.Upload View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants