Use push notifications for repository updates

[s-ludwig]

The registry is currently living on the edge with it's GitHub API use. 60 non-cached requests per hour are allowed and jus registering two projects at once that contain some tags/branches will hit that limit.

When using OAuth2 to access the API, the limit gets increased to 2000 and it also becomes possible to use the PubSubHubbub API to avoid polling, making the registry more responsive and saving bandwith.

Bitbucket probably has something similar.

[eskimor]

I personally would not mind to release my software manually, via some "dub release" command, thus dub could be totally independent from github?

[s-ludwig]

Well I would ;) For sure I will not remove that feature, it's one of the most important ones for me. Enabling releases that are not publically hosted may be a good enhancement, though. But that will require disk space to host the uploaded .zip files somewhere.

[eskimor]

Hosting packages seems to be a good idea:

• Released versions are completely independent from the github repo, could be deleted, renamed, ...
• Support for not publicly hosted packages/ or packages hosted on a non supported platform (no github/bitbucket)
• Packages can contain different files than the repository. (E.g. only a sub-set, package.json gets enriched with the version extracted from the tag name directly, or applicable version() statements, ...)

Of course the resources would need to be provided somehow, maybe I could figure something out, I'll try as soon as I have some prototype of a building/hosting server implementation.

[p0nce]

Do the registry absolutely need to poll github in the first place?
What I miss is a manual notification button in the dub registry website to avoid to wait 30 minutes when fixing an error. My current work-around is to delete package/re-add it which doesn't sound like a good idea in retrospect :)

[MartinNowak]

I'll have a look at this when I find some time.

[MartinNowak]

OK, I played around a little with the github API.
Webhooks are a nice thing, but we'd need the repo owner's OAuth access for the write:repo_hook scope to install them.
I still think, it's a good idea to implement this, so if I find even more time I'll do it.

[s-ludwig]

Hm okay, that's a bit of a downer. Maybe there is a way to abuse something else? Maybe the "watch" functionality can be used to aggregate all changes, so that a single request gets all updates at once?

[MartinNowak]

We could have a pseudo user, that watches all those repos. But from what I read you can't get notifications about commits. Watching only notifies about pull requests and comments.

[s-ludwig]

I was hoping the the data of the "News feed" could be accessed, as that also contains "git push" notifications. But I'm not sure if those are for all watched repositories or just for the ones where the user has push access.