Jx boot failing to modify its own configuration during execution

[tdcox]

Summary

jx boot fails during the installation of jx components due to git conflicts within jenkins-x.yml:

STEP: verify-jenkins-x-environment command: /bin/sh -c jx step verify env in dir: .

Validating git repository for dev environment at URL https://github.com/bootstrap-jersey/environment-d72-dev.git
Duplicated Git repository https://github.com/jenkins-x/jenkins-x-boot-config to https://github.com/Bootstrap-Jersey/environment-d72-dev
Setting upstream to https://github.com/Bootstrap-Jersey/environment-d72-dev

WARNING: Waiting for the fork of bootsbot/environment-d72-dev to appear...
Forked Git repository to https://github.com/bootsbot/environment-d72-dev
Attempting to cherry pick commits that were on master but not yet pushed
  Cherry-picked 9bbea55c7 release 1.0.32

error: error updating dev environment for environment-d72-dev: forking and pulling https://github.com/Bootstrap-Jersey/environment-d72-dev.git: unable to pop the stash: git output: Auto-merging jenkins-x.yml
CONFLICT (content): Merge conflict in jenkins-x.yml: failed to run 'git stash pop' command in directory '/Users/terry/Documents/code/jxtesting/d72/jenkins-x-boot-config', output: 'Auto-merging jenkins-x.yml
CONFLICT (content): Merge conflict in jenkins-x.yml'
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step verify env' command in directory '.', output: ''

➜ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 2 and 279 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	modified:   jx-requirements.yml
	modified:   systems/acme/Chart.yaml

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add <file>..." to mark resolution)

	both modified:   jenkins-x.yml

If you manually merge the changes and restart, you just get the same problem in systems/acme/Chart.yaml.

apiVersion: v1
<<<<<<< HEAD
appVersion: 1.0.32
=======
>>>>>>> b9e8cff... initial config based of jenkins-x/jenkins-x-boot-config with ref origin/master
description: ACME Chart
maintainers:
- name: Team
name: acme
<<<<<<< HEAD
version: 1.0.32
=======
version: "1"
>>>>>>> b9e8cff... initial config based of jenkins-x/jenkins-x-boot-config with ref origin/master

Jx version

The output of jx version is:

2.0.905

[abayer]

So the second one seems to, in the immediate, be a clear ordering issue in chart generation, but yeah, the real problem seems to be that the dev env repo is being modified (presumably by a gitops PR?) while the local repo isn't picking up that update before it tries to change things.

[abayer]

Waaaaaiiiit - that Cherry-picked 9bbea55c7 release 1.0.32 is catching my eye. I know you were running this today, so I know that you were running it after the update of jenkins-x-boot-config in jenkins-x-versions at around 2:45am UTC, and that update was taking jenkins-x-boot-config from 1.0.32 to 1.0.35. Which explains the cherrypick merge conflict - your local dev env repo checked out the v1.0.32 tag as master, meaning that there's an extra commit on it (since jx puts an empty commit down for releases and tags that, with the empty commit never making it to master). I'm not entirely sure how that empty commit is causing the conflict, since there are no changes in it, but it definitely is.

So here's what I think is happening:

• Your ~/.jx/jenkins-x-versions is sliiiiiightly "out of date".
• So the jenkins-x-boot-config that gets cloned locally is pointing to v1.0.32 as the basis for its master branch. Changes get made to it, as we'd expect boot to do.
• The forking of jenkins-x-boot-config is just getting regular ol' master from jenkins-x-boot-config as its master, unsurprisingly.
• Ok, so now jx step verify env tries to cherrypick the changes on your local master on top of master from the forked env repo...but while they share a common ancestor (the last real commit before the v1.0.32 empty commit), they've both made changes since that common ancestor, so they can't be cleanly reconciled.

In this case, there were legit changes to the files you're seeing conflicts in - jenkins-x/jenkins-x-boot-config#81 and jenkins-x/jenkins-x-boot-config#88 - but the cherrypicking logic is just failing horribly on them...not sure if that's because of the empty commit or our logic for cherrypicking is flawed, though.

Need to think more on this...

[tdcox]

I suspect that changing the version stream value in jx-requirements.yml to master caused some of the code to refresh to 1.0.35 but doesn’t refresh ~/.jx/jenkins-x-versions itself.

[abayer]

Ah ha ! I have reproduced it with https://github.com/abayer/jx/blob/is-version-stream-clone-verifying-remote/jx/bdd/boot-vault/ci.sh. Now that I've got a reproduction in the wild, I should be able to nail down exactly what's not doing what it's supposed to tomorrow.

[abayer]

dangit, I didn't actually reproduce it. That attempt used an existing dev env repo by accident. Sigh, more digging...

[tdcox]

I can confirm that ~/.jx/jenkins-x-versions was borked. I tried a jx install on a fresh cluster from this machine and it also failed with git errors:

? A local Jenkins X versions repository already exists, pull the latest? Yes
FATAL: ingress init failed: failed to load version of chart stable/nginx-ingress: : pulling the latest: non-fast-forward update

I deleted the folder and was able to run the install.

[abayer]

Ok, that may be part of it, but from my experiments, I don't know if that's all of it. I verified that if you've already fetched the latest in ~/.jx/jenkins-x-versions but your master is pointing at an older commit than the latest in origin, it won't pull the branch, but that didn't result in reproducing the error. Gonna try rigging the repo to have an extra commit on local master now to see what happens.

[abayer]

I did finally, decisively reproduce - if you run jx boot from within an existing clone of jenkins-x-boot-config (say, because you started a boot, stopped it to edit the requirements, and then started it again) based on an older tag of jenkins-x-boot-config than is referred to by jenkins-x-versions, you get a merge conflict creating the dev env repo. I thought maybe I could fix it by not including the empty commit in the local clone, but that did not fix it. So...more to do fix it, but I'm confident that I can fix it now. =)

[abayer]

Ok, so the problem is that we make uncommited changes in the local clone that then get stashed and unstashed onto the created dev env repo's master branch, and there's always a chance that those changes conflict with any changes that have happened in the master branch since the commit we're using as the baseline in the clone. I'm honestly not yet sure what to do about that.

We could handle the most simple case easily enough - if we kick off jx boot and the local boot config is out of date w.r.t. what jenkins-x-versions is pointing to, just update the local boot config. But if there are changes in the local boot config, it could still run into the conflict then.

I'm wondering if the "right" solution is to do that update, and if there are conflicts then (i.e., before we actually execute any steps), error out with a message saying "we had to update your local boot config, but there are conflicts. Please update your local boot config by running git stash; git pull (boot URL) (boot ref); git stash pop and resolve the conflicts". Though that would still have issues due to the empty commits, so maybe we still need to checkout the commit before the empty commit, and have (boot ref) there be the commit before the new tag's empty commit...

[abayer]

cc @daveconde @macox @garethjevans @pmuir @jstrachan - thoughts on the above?

[abayer]

My reasoning here is that if we're going to have a conflict updating the local boot config to the version specified in jenkins-x-versions, it's better that we do it sooner than later, with a more clear error message and directions for rectifying it.

[tdcox]

Or, if we wanted to give the customer a better user experience, we could just beat them over the head repeatedly with a rusty bar?

[abayer]

Hey, I'd say that failing before actually doing anything with an error message that actually tells you how to fix it is an improvement. =)

[abayer]

So I'm about to put a WIP PR up with my semi-solution for this. It will try to update the local boot clone, let the user know it's doing so, and fail with what I hope is a useful error if it's unable to do so due to conflicts. The output will look like:

Attempting to resolve version for boot config https://github.com/jenkins-x/jenkins-x-boot-config from https://github.com/jenkins-x/jenkins-x-versions.git
Local boot clone is out of date. It is based on v1.0.32, but the version stream is using v1.0.35. The clone will now be updated to v1.0.35.
Stashing any changes made in local boot clone.
error: Could not update local boot clone due to conflicts between local changes and v1.0.35.
To fix this, resolve the conflicts listed below manually, run 'git reset HEAD', and run 'jx boot' again.
    git output: Auto-merging jenkins-x.yml
    CONFLICT (content): Merge conflict in jenkins-x.yml