403 API rate limit exceeded for jissues-bot on saving an issue

[zero-24]

Steps to reproduce the issue

Try to save a issue on issues.joomla.org

Expected result

issue gets saved

Actual result

array(7) { ["exception"]=> string(44) "403 API rate limit exceeded for jissues-bot." ["user"]=> string(6) "joomla" ["project"]=> string(10) "joomla-cms" ["issueNo"]=> string(4) "5984" ["state"]=> string(4) "open" ["old_state"]=> string(4) "open" ["data"]=> array(11) { ["status"]=> string(1) "3" ["priority"]=> string(1) "3" ["rel_type"]=> string(1) "0" ["rel_number"]=> string(0) "" ["build"]=> string(7) "staging" ["easy"]=> string(1) "1" ["milestone_id"]=> string(1) "0" ["title"]=> string(69) "Set a 500 header if we hit the 'Error display the error page' message" ["description_raw"]=> string(570) "In error conditions that cannot render the error template, we fall back to just echoing a message. However, it is possible that this message will send a HTTP 200 OK header. In reality, we should be sending an error header back. This PR will try to send a HTTP 500 error header if the headers have not already been sent. ### Test Instructions The easiest way to get here is to have your database server offline. Without the patch, you should get a HTTP 200 header when you see this message. Apply the patch and try the request again; a 500 header should be sent." ["id"]=> string(4) "6770" ["issue_number"]=> string(4) "5984" } } Unrecoverable GitHub error - sry ;(

403 API rate limit exceeded for jissues-bot

[mbabker]

We're going to need folks to stop working for the hour 😝

Maybe the CMS repo is getting big enough that we need another bot?

[zero-24]

hmm and/or reduce the github api calls?

The issue comes from here:
https://github.com/joomla/jissues/blob/master/src/App/Tracker/Controller/Issue/Save.php#L100-118

[zero-24]

also no sync of comments here: http://issues.joomla.org/tracker/joomla-cms/5984 from here joomla/joomla-cms#5984

[zero-24]

maybe also related to: #611 and similiar?

[zero-24]

and Brian can perform it as he use his own account and not the bot:
https://github.com/joomla/jissues/blob/master/src/App/Tracker/Controller/Issue/Save.php#L273-280

[mbabker]

I've resent the missing events.

It's hard to cut back on the number of calls to the GitHub API. The block you highlighted is what sends updates from the app to GitHub immediately upon save. The first request tries as the authenticated user and if they don't have permission to perform the action, the bot picks up the slack.

We're only making requests as needed. An issue we're hitting is there are cron jobs now running every 5 minutes except for :25 and :55 after the hour to support all of the repositories in the live app. If I'm not mistaken, all use the @jissues-bot account. So, we either need more bots or we have to give more users permissions on the main repos to perform the same actions, and the latter is not preferred.

[mbabker]

For reference, the current cron schedule:

0,30 * * * * php cli/tracker.php get project --all --project=2 --log=cron.log >/dev/null 2>&1
5,35 * * * * php cli/tracker.php get project --all --project=5 --log=cron.log >/dev/null 2>&1
10,40 * * * * php cli/tracker.php get project --all --project=1 --log=cron.log >/dev/null 2>&1
15,45 * * * * php cli/tracker.php get project --all --project=6 --log=cron.log >/dev/null 2>&1
20,50 * * * * php cli/tracker.php get project --all --project=4 --log=cron.log >/dev/null 2>&1

ID mapping:

• 1 => CMS
• 2 => Issue Tracker
• 4 => Tests
• 5 => Patch Tester
• 6 => Weblinks

[zero-24]

So, we either need more bots or we have to give more users permissions on the main repos to perform the same actions, and the latter is not preferred.

hmm so i think we have tree options:

• one bot per repo (like: jcmsbot; jissuesbot; Jtestsbot, ...)
• bots per action: so we have one or two for the crons and one or two for the app (like edit as JBS member)
• more (from the aktive members) get limited access to the repo to perform only the needed actions without a bot.

One of the problems is that if more users gets enabled as JBS and can change status and comments via jissues the limit will be exceeded faster than now. ATM we have only a smal number of users that uses jissues for actions that needs the bot. If i see it correct.

[elkuku]

Folks we have this functionality actually implemented.

We have one (or more) GitHub (bot) account settings in /etc/config.json as well as an edit-bot setting for every project that is stored in the database.
The idea is to use the former for CLI scripts and the latter for project specific "edit actions".

So the thing to do here would be to create a project specific bot user account (a cms-bot in this case) and set the credentials in the project settings.

That should be it for the moment, unless those JBS folks are getting high on pizza and pass the 5k/hour limit in one of those global bug squashing sessions 😉

@mbabker regarding the cron script: maybe you could write the output to a log file instead of /dev/null including stderr, so maybe PHP errors get caught here that do not enter in our regular cron log, so maybe we get some hints about what is going wrong currently..

[mbabker]

I can do that. Just need to remember to cycle them off the server; I cleared 8 months (and something like 3 GB) of un-read log files that were in the mail queue out when I added that dev/null piece.

[elkuku]

No logrotate or similar available?...

The most important thing right now is to not use the bot account from the config.json in project edit bot settings.

[mbabker]

No logrotate command unfortunately.

[mbabker]

I've got the cron jobs updated to capture output and logging on the server. If we want to add some sort of UI for them, I'm writing them as .txt files to a cronlogs folder which would be at the repo root here.

[mbabker]

Welcome @joomla-cms-bot to GitHub, this bot shall take on responsibility for the CMS repo.

[elkuku]

I've got the cron jobs updated to capture output and logging on the server. If we want to add some sort of UI for them, I'm writing them as .txt files to a cronlogs folder which would be at the repo root here.

That sounds terrific :)

Welcome on board @joomla-cms-bot 😋