Discovery on issues around boot/dev env repo, git changes, and merge conflicts

[abayer]

We've seen a number of issues stemming from jx boot, jx upgrade boot, or jx step verify environments, relating to changes being made to the local boot clone/dev env repo that shouldn't be happening, or merge conflicts syncing up with the upstream jenkins-x-boot-config repo. This issue is to track investigative work on how jx boot is intended to behave w.r.t. the boot repo, what it's doing now, and what needs to be reworked to get it to correctly, consistently, and reliably do what it needs to do.

Related issues include:

• jx upgrade boot fails when nexus is disabled #6106
• jx boot should not mutate files on disk if the user doesn't change anything explicitly #6056
• jx boot verify-jenkins-x-environment fatal: refusing to merge unrelated histories #6037
• Merge conflict in environment repo during jx boot run #5827
• Unable to re-run 'jx boot' from the initial boot config directory #5772
• upgrade boot should have a mechanism to continue when a cherry pick fails during the update #5582

And closed issues like the ones listed below will be considered too - while they're "fixed" currently, their original problems were probably symptoms of the wider issues with the boot repo:

• jenkins-x-boot-config BDD test fails due to merge conflict #6024
• jx boot throws up on <.Requirements.repository>  #6011
• jx upgrade boot doesn't fetch the tag to upgrade to, so hits cherry pick error #5974
• Jx boot failing to modify its own configuration during execution #5929
• jx upgrade boot wipes user requirements  #5896

...

[bitfactory-henno-schooljan]

When setting up a new Jenkins X install, after a fresh jx boot I'd expect jx upgrade boot to do nothing or update if needed, instead I get a merge error on jx-requirements.yml:

19:54 $ jx upgrade boot
? A local Jenkins X versions repository already exists, pulling the latest: Yes
Upgrade available
using default boot config https://github.com/jenkins-x/jenkins-x-boot-config.git
? A local Jenkins X versions repository already exists, pulling the latest: Yes
? A local Jenkins X versions repository already exists, pulling the latest: Yes
No boot config upgrade available
Upgrading version stream ref to e0007c3e977ff40fa6bd438ff3facefe3aa9ce9b
? A local Jenkins X versions repository already exists, pulling the latest: Yes
Updating pipeline agent images to gcr.io/jenkinsxio/builder-go:2.0.993-323
Skipping builder image update as no changes were detected
Updating template builder images to gcr.io/jenkinsxio/builder-go:2.0.993-323
Skipping template builder image update as no changes were detected
error: failed to create a merge commit for jx-requirements.yml: error creating a commit with the merged jx-requirements.yml file from dir /var/folders/36/xdf6fq696d5f337p2nhnvcvm0000gp/T/697884180/jx-requirements.yml: git output: On branch 4f3f5fc3-b629-4254-bffe-6f6ec037ebc4
Changes not staged for commit:
	modified:   jenkins-x-bdd-local.yml
	modified:   jenkins-x.yml

no changes added to commit: failed to run 'git commit -m Merge jx-requirements.yml -- /var/folders/36/xdf6fq696d5f337p2nhnvcvm0000gp/T/697884180/jx-requirements.yml' command in directory '/var/folders/36/xdf6fq696d5f337p2nhnvcvm0000gp/T/697884180', output: 'On branch 4f3f5fc3-b629-4254-bffe-6f6ec037ebc4
Changes not staged for commit:
	modified:   jenkins-x-bdd-local.yml
	modified:   jenkins-x.yml

no changes added to commit'

I obviously have changes in that file, it would be nice if I somehow could resolve this conflict.

[haysclark]

Please let me know if I need to add anything else to this ticket.

☺  jx upgrade boot                                                                                                                                                                                                master 585812c
? A local Jenkins X versions repository already exists, pulling the latest: Yes
Upgrade available
using default boot config https://github.com/jenkins-x/jenkins-x-boot-config.git
? A local Jenkins X versions repository already exists, pulling the latest: Yes
? A local Jenkins X versions repository already exists, pulling the latest: Yes
boot config upgrade available
Upgrading from v1.0.42 to v1.0.48
cherry picking commits in the range 712b65e05ce98272e4bcab4ea10808f43de6732c..bfc4da8aeb74d1b4f56d97574e51f823096c4800
6b99c30b1d6ded3336646be32f6d3059cd74e5ef - fix: added velero configuration for iks
84ba05b583dd6eea3a31013dd6a9861626727177 - fix: Add verify-fmt and fmt, fail PR if not formatted
error: failed to update boot configuration: failed to cherry pick upgrade commits: cherry-picking 6065e8417e538cc4e53d719feb3b0337edb76ea1: git output: The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

Otherwise, please use 'git reset'
On branch 5f5c35c5-20a3-4140-b436-6191262ddf72
You are currently cherry-picking commit 6065e84.

nothing to commit, working tree clean: failed to run 'git cherry-pick 6065e8417e538cc4e53d719feb3b0337edb76ea1 --strategy=recursive -X theirs' command in directory '/var/folders/dp/8pv_k2nn0b3_ylsfbpdly9100000gn/T/996188035', output: 'The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

Otherwise, please use 'git reset'
On branch 5f5c35c5-20a3-4140-b436-6191262ddf72
You are currently cherry-picking commit 6065e84.

nothing to commit, working tree clean'

[abayer]

Thanks - that’s exactly the datapoint I was looking for!

[abayer]

Ok, first thing that needs to be done, I think, is removing

jx/pkg/cmd/step/verify/step_verify_environments.go

Line 197 in fa2e0aa

func (o *StepVerifyEnvironmentsOptions) modifyPipelineGitEnvVars(dir string) error {

- well, more accurately,

jx/pkg/cmd/step/verify/step_verify_environments.go

Lines 233 to 236 in fa2e0aa

	err = projectConf.SaveConfig(fileName)
	if err != nil {
	return errors.Wrapf(err, "failed to write to %s", fileName)
	}

- we don't need to actually write the GIT_* env vars into the jenkins-x.yml, since they're getting set within the environment of the running steps now. That cuts down one of the cases of files getting modified in multiple histories.

[abayer]

How to deal with jx-requirements.yaml seems pretty clear to me - see comments on #6056, but the gist is the idea of moving it in jenkins-x-boot-config to a new file, like jx-requirements.base.yaml, so that changes upstream never cause git conflicts downstream. We'd then be able to reconcile upstream changes to local changes via mergo. I need to see what other files besides jenkins-x.yml and jx-requirements.yaml are causing conflicts...

[abayer]

Trying to lay out scenarios in which conflicts can and can't appear:

First, when running jx boot itself (before it enters the pseudo-pipeline step execution):

• You won't hit any errors here if you pass in --git-ref, regardless of whether you have an existing clone or not.
• You won't hit any errors here if you're booting from an existing dev env repo, because any existing dev env repo created via jx boot won't have a common commit history with jenkins-x-boot-config, thanks to jx step verify environments squashing all the commits down into one. We don't even try to update the local clone vs upstream in that situation.
• If you clone the upstream jenkins-x-boot-config repo, make changes, and then run jx boot (or run jx boot to the point where it clones the jenkins-x-boot-config repo, quit jx boot, and then make changes), it is possible that you can hit errors, if the version stream changes the tag in jenkins-x-boot-config it points to between the clone/first run of jx boot, and the run of jx boot after local changes are made. The most likely places to see conflicts are jx-requirements.yaml and jenkins-x.yml - far more likely to be in jx-requirements.yaml now that we've ensured that jenkins-x.yml will be consistently formatted in jenkins-x-boot-config. Local changes to jx-requirements.yaml can result in conflicts due to new fields being added upstream, default values for fields changing, or the user changing order in their local jx-requirements.yaml.
• The same scenarios can result in errors with -r (requirements file) without a local clone.

[abayer]

Error cases during the pseudo-pipeline step execution are all in jx step verify environments:

• Due to how gits.ForkAndPullRepo works with private repositories (not 100% sure if this applies to public repositories at well, but I think it probably does - the logic is the way it is due to private repo credentials, but I believe we use the same logic regardless of private or public), running jx boot for the second (or third, etc) time from the initial jenkins-x-boot-config repository can hit conflicts that aren't really conflicts. See Unable to re-run 'jx boot' from the initial boot config directory #5772 for more details. This doesn't happen when running jx boot the first time, or if you're running jx boot a second/third/etc time from a clone of the dev env repo created by the first jx boot run, just if you try to run from the initial boot config clone more than once. I'm actually not 100% sure this is due to the private repo handling, though - the created dev env repo has a different history than the initial local clone, so I think it's very possible that they just can't be matched up. Needs more investigation.
• Previously, you could hit errors in the cases laid out for jx boot's pre-pseudo-pipeline execution in jx step verify environments instead, but we moved the relevant logic earlier in the process.
• It is still theoretically possible to hit a conflict here on first run, when the dev env repo is created from a base of the upstream squashed into a single commit and any local changes are pushed as a pull request to the dev env repo, but since moving the stash/unstash logic earlier, I haven't seen this happen.

[abayer]

jx upgrade boot now seems to have the most/worst merge conflicts, which makes sense - in theory, at least, jx boot and jx step verify environments are dealing with the same upstream jenkins-x-boot-config baseline throughout their run, with only the local changes to add to that. But jx upgrade boot inherently is changing the upstream being used, so conflicts are much more likely. More detail to come...

[abayer]

Alright, jx upgrade boot can go wrong in enough ways that it's not easy to just lay them all out like I could for jx boot and jx step verify environments, so I'm going to be more general here.

Even in the best case scenario, if we can get rid of all the things that can result in false merge conflicts in jx upgrade boot, there are still going to be legitimate cases of merge conflicts. #5582 points out that you can't resolve the conflicts and then re-run jx upgrade boot using the same local clone/branch. We need to allow something along those lines, perhaps by adding a flag to pass an existing branch and resume from there. Gotta think more on this.

[hferentschik]

I have a very bad feeling about this. IMO a lot of the problems stem from #5722 and the broken gits.ForkAndPullRepo. Before we go down the road of doing manual merges of specific files etc, I'd start with solving this well known issue. This is a problem for public as well as private repos. Once this is out of the way, I'd take another pass at looking at the remaining issues.

[abayer]

I'm not sure #5772 is responsible for many of the reported issues - it's only going to show up when you're booting using an existing dev env repo and a local clone of upstream boot config. Upgrade has its own mechanism for combining the different histories in upstream and the dev env, when you don't have an existing dev env repo, it doesn't hit this, and when you run boot from a clone of the dev env, it shouldn't hit this either.

[abayer]

I'm marking this as done - the epic has three tangible issues to work on in it.