Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enrich_cntrb method not checking for additional tokens #1423

Closed
sgoggins opened this issue Aug 13, 2021 · 3 comments · Fixed by #1427 or #1471
Closed

enrich_cntrb method not checking for additional tokens #1423

sgoggins opened this issue Aug 13, 2021 · 3 comments · Fixed by #1427 or #1471
Assignees
Labels
critical-fix Should be addressed before any other issue/PRs workers Related to data workers

Comments

@sgoggins
Copy link
Member

Please help us help you by filling out the following sections as thoroughly as you can.

Description:
If GH/GL worker collection fails on retrieval of contributor_id from the platform, it issues this error and stops the worker.

2021-08-12 16:25:43,170,170ms [PID: 279416] workers.github_worker.57516 [INFO] Contributor data: {'message': 'API rate limit exceeded for user ID 75419635.', 'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'}

Embedded in this error:

2021-08-12 16:25:42,887,887ms [PID: 279416] workers.github_worker.57516 [INFO] Hitting endpoint: https://api.github.com/users/toufikImk ...

2021-08-12 16:25:43,170,170ms [PID: 279416] workers.github_worker.57516 [INFO] Contributor data: {'message': 'API rate limit exceeded for user ID 75419635.', 'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'}
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] Worker ran into an error for task: {'job_type': 'MAINTAIN', 'models': ['issues'], 'display_name': 'issues model for url: https://github.com/angular/angular', 'given': {'github_url': 'https://github.com/angular/angular'}, 'focused_task': 1}
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] Worker was processing this task for -2498.578952550888 seconds.
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] Printing traceback...
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] 'login'
2021-08-12 16:25:43,173,173ms [PID: 279416] workers.github_worker.57516 [ERROR] Traceback (most recent call last):
  File "/home/sean/github/operatefirst/workers/worker_base.py", line 180, in collect
    model_method(message, repo_id)
  File "/home/sean/github/operatefirst/workers/github_worker/github_worker.py", line 192, in issues_model
    pk_source_issues = self._get_pk_source_issues()
  File "/home/sean/github/operatefirst/workers/github_worker/github_worker.py", line 174, in _get_pk_source_issues
    pk_source_issues_increment_insert(source_issues,action_map)
  File "/home/sean/github/operatefirst/workers/github_worker/github_worker.py", line 90, in pk_source_issues_increment_insert
    inc_source_issues['insert'] = self.enrich_cntrb_id(
  File "/home/sean/github/operatefirst/workers/worker_git_integration.py", line 384, in enrich_cntrb_id
    "cntrb_login": contributor['login'],
KeyError: 'login'
2021-08-12 16:25:42,887,887ms [PID: 279416] workers.github_worker.57516 [INFO] Hitting endpoint: https://api.github.com/users/toufikImk ...

2021-08-12 16:25:43,170,170ms [PID: 279416] workers.github_worker.57516 [INFO] Contributor data: {'message': 'API rate limit exceeded for user ID 75419635.', 'documentation_url': 'https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting'}
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] Worker ran into an error for task: {'job_type': 'MAINTAIN', 'models': ['issues'], 'display_name': 'issues model for url: https://github.com/angular/angular', 'given': {'github_url': 'https://github.com/angular/angular'}, 'focused_task': 1}
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] Worker was processing this task for -2498.578952550888 seconds.
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] Printing traceback...
2021-08-12 16:25:43,171,171ms [PID: 279416] workers.github_worker.57516 [ERROR] 'login'
2021-08-12 16:25:43,173,173ms [PID: 279416] workers.github_worker.57516 [ERROR] Traceback (most recent call last):
  File "/home/sean/github/operatefirst/workers/worker_base.py", line 180, in collect
    model_method(message, repo_id)
  File "/home/sean/github/operatefirst/workers/github_worker/github_worker.py", line 192, in issues_model
    pk_source_issues = self._get_pk_source_issues()
  File "/home/sean/github/operatefirst/workers/github_worker/github_worker.py", line 174, in _get_pk_source_issues
    pk_source_issues_increment_insert(source_issues,action_map)
  File "/home/sean/github/operatefirst/workers/github_worker/github_worker.py", line 90, in pk_source_issues_increment_insert
    inc_source_issues['insert'] = self.enrich_cntrb_id(
  File "/home/sean/github/operatefirst/workers/worker_git_integration.py", line 384, in enrich_cntrb_id
    "cntrb_login": contributor['login'],
KeyError: 'login'

How to reproduce:

  1. You have to run the GL/GH workers for some time to his this error. The PR worker has logic to wait for the next available token. The issue workers do not seem to. Haven't verified GL Merge request worker:
  2. Stops worker Collection
  3. Will upload full logs in slack.

Expected behavior:
Wait for the next available token.

Screenshots
If applicable, add screenshots to help explain your problem. If your bug is related to the UI, you must include screenshots.

Log files
Attach the relevant log files here. Server and installation logs can be found in the logs/ directory in the root augur/ directory, and the logs for each worker are stored in their respective directories. If the logs are pretty long (> 50ish lines or just use your best judgement) please use a Gist or a pastebin. These logs file are required if you would like help solving your issue.

Software versions:

  • Augur: v20.0.1
  • OS: UBUNTU 20.04; but probably universal
@sgoggins sgoggins added workers Related to data workers critical-fix Should be addressed before any other issue/PRs labels Aug 13, 2021
@sgoggins
Copy link
Member Author

@IsaacMilarky : Can you make this a top priority today?

@sgoggins
Copy link
Member Author

did not mean to close issues. Buttons are hard.

@IsaacMilarky
Copy link
Contributor

Yes, I will work on a fix first thing. It seems to not be too hard of a fix but I won't jinx it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
critical-fix Should be addressed before any other issue/PRs workers Related to data workers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants