Automate and improve issues transfer

[xavt]

Following your suggestion to open an issue to make a sugestion, here's one idea:

"Automate and improve issues transfer"

Here's a goal:

• be able to move all the issues of one repo to another (over 10K issues of the same org)
• It should not send notifications
• It should mimic the behavior of the github "transfer issue" button : create redirects, keep comments, all PR/interactions
• keep milestones, labels, assignees ...

We got a few parts here and there:

• https://docs.github.com/en/graphql/reference/mutations#transferissue
• https://docs.github.com/en/rest/reference/issues#create-an-issue
  • we can pass labels ans miletsones
• https://docs.github.com/en/github/managing-your-work-on-github/transferring-an-issue-to-another-repository
  • through the UI, magic "transfer issue" button
  • This will create a redirection from the old rep URL to new rep URL and this is a must.
• https://github.com/ahadik/git_mover
• https://gist.github.com/jonmagic/5282384165e0f86ef105
  • does not notify

But no definite solution! What would be your take on it?
Maybe this could be a nice experiment to stream about.

[gr2m]

be able to move all the issues of one repo to another (over 10K issues of the same org)

That shouldn't be a problem, it will just take a while

It should mimic the behavior of the github "transfer issue" button : create redirects, keep comments, all PR/interactions

That should be possible with https://docs.github.com/en/graphql/reference/mutations#transferissue that you already linked

keep milestones, labels, assignees

that will be more tricky. You'd have to copy over milestones and labels. You will only be able to assign people who are either collaborators on the issue or who commented on the issue

It should not send notifications

I don't think that will be possible. I'm not sure if the transfer itself will send any notifications, but I think adding labels, milestones, and assignees will.

But no definite solution! What would be your take on it?
Maybe this could be a nice experiment to stream about.

Is this for a single repository or multiple repositories from which you want to move issues from? Are the repositories public?

[Zimmi48]

Transfer does generate notifications, labelling and milestoning don't. Assigning doesn't except maybe for the person who gets assigned (not sure about that one).

Massive issue transfers from anywhere to anywhere without triggering notifications is also possible with the issue import API but it doesn't keep the authors of the issue and comments (see https://gist.github.com/jonmagic/5282384165e0f86ef105 / https://www.theozimmermann.net/2017/10/bugzilla-to-github/ for details).

[xavt]

Thanks for your answers :)

It is from a single repository to a single repository, both private.

So basically it will be a "2 steps", first transfering the issues, then adding the missing info (labels etc...)
I guess it will be better to get the info before the transfer as it won't be available in the new issue and the old one will be redirected...

[gr2m]

Overall this seems to be a common enough problem, I'd be happy to do an episode on it. We would need a test setup with public repositories, one that we can easily reproduce for testing. I think we'll have to create a script which creates two repositories with some labels and milestones, and assignees. Otherwise it will be hard to test the script. We can create that test scenario script part of the show, too.

Can you create a test GitHub organization with two repositories that we can use as templates? That would be a good starting point.

Once we have that we should agree on what the script will do exactly. We could start a repository with just a README that documents the non-existing CLI, and then make it real during a show?

the issue import API

That API isn't documented, I don't know the state of it, it might likely be removed if it hasn't already? Could you inquire GitHub support about its state? https://support.github.com/contact

You might also ask support if they can do the transfer for you without the noise from the transfer notifications?

[xavt]

Awesome for this idea to start an episode!

• we have this org to do some testings :
  • https://github.com/Another-Practice-Organization
• two repositories
  • https://github.com/Another-Practice-Organization/repository1
  • https://github.com/Another-Practice-Organization/repository2

Github support confirmed me this endpoint is still working ans won't create notifications: https://gist.github.com/jonmagic/5282384165e0f86ef105
But also confirmed this endpoint https://docs.github.com/en/graphql/reference/mutations#transferissue is the way to "transfer" issues

[gr2m]

Sounds great, thanks for confirming! I'd prefer to use the official transfer issue APIs, it will make it future proof and more usable to others, does that work for you?

I'd suggest we rename the repositories

1. template-issues-source
2. template-issues-target

Then create some issues in the source, with labels, milestones, assignees

then create some of the labels and milestones in the target repositories to make sure the script handles conflicts correctly.

Next we have to create a script that creates a new repository based on an existing one, including issues, labels, milestones, and collaborators, for our use case. We could make this part of the show, or the first part of it, might be interesting in and of itself?

How is your availability? I'm trying to do two shows per week, I've nothing scheduled after this week yet:
https://www.twitch.tv/gregorcodes/schedule

[xavt]

Nice!

The transfer issue API would be the way;
I think for a few scenarios, having the redirect links is necessary (from the old issue in old repo to the new issue in new repo); it would have been a requirement in my case.

The repositories have been renamed (and I discover in the process the "template repository" option)

Ok to generate some content in both, I'll do it later today;
I'd also like to add "projects" as well while we're at it... what do you think?

Next we have to create a script that creates a new repository based on an existing one

Or "targets another repository" instead of "creates a new repository" ;)
we'll transfer issue from template-issues-source to existing template-issues-target
(the script could have a flag to create a new repository first though)

like @Zimmi48 mentionned we'll have to check for notifications (what they are when we transfer the issue/comment/assign) AND the rate limitations for mass transfer (I'm thinking of 10k issues transfer); is there such a thing as an organization/repository wide notification setting that we could toggle with the script?

You decide what you'd like to put on your show :) !
As for the show I'll try to watch it live (any time after 1PM PDT) or post-live for sure!

[Zimmi48]

the rate limitations for mass transfer

That's a good point. One of the advantages of the issue import API was precisely to get higher rate limits. Notification-triggering actions generally have lower rate limits. As I note in the blog post mentioned above, "the GitHub API was limiting issue creation to 300, after what it refused any new posting for the next 30 minutes" before I converted it to use the issue import API.

[xavt]

Nice post and details on this blog post
I didn't know about the updated_at field! And good job keeping the numberings as well. In the use case I've in mind it won't be possible and that's why a transfer with redirects will do the trick.

This API does not allow you to choose the author of issues / comments

and that's why we should avoid this API then. The transfer feature on the other end (triggered through the UI at least) does move a lot of things like one would like!

[gr2m]

sorry I'm a little late, but I'm planning on doing a show tomorrow at 1pm, you are all welcome to join! I'll post details soon. But I'll focus on creating a script that will copy an existing repository including issues, labels, assignees and milestones, as a means of preparation for another show about the actual issue transfer script

[gr2m]

Massive issue transfers from anywhere to anywhere without triggering notifications is also possible with the issue import API but it doesn't keep the authors of the issue and comments (see https://gist.github.com/jonmagic/5282384165e0f86ef105 / https://www.theozimmermann.net/2017/10/bugzilla-to-github/ for details).

I only now realize this is not the original request by @xavt. The issue import API is not helpful when the source of the issues is another repository, because the import API does not close existing issues in that source repository.

I'm happy to make a separate show about the issue import API once it's documented by GitHub and considered stable. Until then it's better to reach out to support directly, because they have much more information on the APIs than I do.

@Zimmi48 Can you reach out to support and ask them about the state of the issue import API, if there is any plan to make it official?

[gr2m]

I created an issue for tomorrows show at 1pm Pacific Time: #16

@xavt would you like to join?

[johnjsatter]

@gr2m i would live to join if possible

[gr2m]

@johnjsatter can you give me a short intro and a way how I can send you an invite link later? What's your interest in the episode, what do you hope to get out of it?

[johnjsatter]

@gr2m hi! Well I stumbled into the community and making every effort to be righty within the community. I may have upset some people along the way. I want learn as much as possible but here are my priorities. Secure email w/ ProtonVPn (Ashestohelens@....) I need an API key, I also think I had one with google but am a little reluctant to be outside of GitHub or DevHub. I am sure I can get a lot out of it

[gr2m]

Secure email w/ ProtonVPn (Ashestohelens@....)

I'm not sure what that means? Is the email domain @protonvpn.com? Can you join 15 min before the show (12:45pm pacific time) to chat? I just want to make sure we are on the same page with today's episode

[johnjsatter]

I hope I can get back there

[gr2m]

@johnjsatter I got an error message

[johnjsatter]

Yea. It’s not set up yet. I’m sorry. This is the only place I can communicate

[gr2m]

On Thursday, I'll do a show on actually creating the script:
#20

You can ask questions there if you like, before and after the show