Unable to re-run 'jx boot' from the initial boot config directory

[hferentschik]

Summary

When using jx boot to bootstrap a cluster, one should be allowed to re-run jx boot from within the initial boot config directory, however, this currently results in errors of the form:

Attempting to cherry pick commits that were on master but not yet pushed
WARNING: Unable to cherry-pick 1c384abadb71921d01971922930decfecc03e6f3: git output: error: could not apply 1c384ab... initial config based of hf-bee-bot/environment-hardy-bdd-dev-test with ref v1.0.25
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit': failed to run 'git cherry-pick 1c384abadb71921d01971922930decfecc03e6f3' command in directory '/Users/hardy/tmp/dummy-boot/jenkins-x-boot-config', output: 'error: could not apply 1c384ab... initial config based of hf-bee-bot/environment-hardy-bdd-dev-test with ref v1.0.25
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit''

Steps to reproduce the behavior

Install Jenkins-X on a new cluster via jx boot, then re-run jx boot from within the initial boot config directory.

Expected behavior

The second jx boot succeeds as well, resulting in no changes to the cluster.

Actual behaviour

See above.

The problem is how we try to merge / cherry-pick local changes during verify environment. Effectively we get the latest master and then cherry-pick any local changes which are not in the remotes on top of it. This happens in helpers.ForkAndPullRepo or the more particular in gitter.GetCommitsNotOnAnyRemote. Under the hood this calls git log master --not --remotes. This will fail for us due to the way we are pushing to the remote. To avoid having to store credentials for the remote environment repository, we internally (in code) construct an authenticated push URL to which we push. This pushes the code to the remote correctly, but it does not update the remote ref in .git/refs/remotes/origin. This means git log master --not --remotes will report the local commits which are actually already on the remote as well missing and we will attempt to cherry pick them leading to a merge conflict.

The problematic code is:

	userDetails := provider.UserAuth()
	duplicatePushURL, err := gitter.CreateAuthenticatedURL(duplicateInfo.CloneURL, &userDetails)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to create push URL for %s", duplicateInfo.CloneURL)
	}
	err = gitter.Push(dir, duplicatePushURL, true, fmt.Sprintf("%s:%s", "HEAD", toBranch))
	if err != nil {
		return nil, errors.Wrapf(err, "failed to push HEAD to %s", toBranch)
	}

[hferentschik]

This relates to #5463. A solution to the problem would be proper handling of credentials locally, eg via a git credential helper.

As a temporary solution, we could manually update the ref. This might unlock this particular use case, but won't scale.

[tdcox]

This is also breaking in the same way on legacy jx install clusters in the environments folder under .jx, for example when attempting to use jx add app.

[tdcox]

As of 2.0.866, this is now failing during the first execution of jx boot as well:

STEP: verify-jenkins-x-environment command: /bin/sh -c jx step verify env in dir: .

Validating git repository for dev environment at URL https://github.com/bootsbot/environment-d70-dev.git
Duplicated Git repository https://github.com/jenkins-x/jenkins-x-boot-config to https://github.com/bootsbot/environment-d70-dev
Setting upstream to https://github.com/bootsbot/environment-d70-dev

Attempting to cherry pick commits that were on master but not yet pushed
  Cherry-picked 24397f1bf release 1.0.27

error: error updating dev environment for environment-d70-dev: forking and pulling https://github.com/bootsbot/environment-d70-dev.git: unable to pop the stash: git output: Auto-merging jenkins-x.yml
CONFLICT (content): Merge conflict in jenkins-x.yml: failed to run 'git stash pop' command in directory '/Users/terry/Documents/code/jxtesting/d70/jenkins-x-boot-config', output: 'Auto-merging jenkins-x.yml
CONFLICT (content): Merge conflict in jenkins-x.yml'
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step verify env' command in directory '.', output: ''

[tdcox]

Hmmm, looks like this may have been due to a jx release occurring during a cluster deployment.

[abayer]

I think this is partially dealt with by #5949 merging.

[deanesmith]

@hferentschik will pick up in the coming week. Only happens when running from clone of boot config when you have a dev env repo. It is assumed git is configured which contributes to matter but this issue is not relevant to that. Issues with regards to upgrade boot are prioritized in this context as well.

[ccojocar]

I tried today to run boot multiple times in the same folder which was created by the initial run. Second run was successful but without making any change in the config. In the third run, I enabled the TLS which was successfully configured for vault but I encountered the first error mentioned above related to local commits cherry picking during the environments verification:

STEP: verify-jenkins-x-environment command: /bin/sh -c jx step verify env in dir: /Users/cosmin/projects/tmp/cluster-vault/jenkins-x-boot-config

Storing the requirements in team settings in the dev environment
Validating git repository for dev environment at URL https://github.com/ccojocar-org/environment-cosmin-splitsecrets-dev.git
Attempting to cherry pick commits that were on master but not yet pushed
  Cherry-picked 5a971d89d fix: enable TLS for all ingress resources
  Cherry-picked 6d59c3070 chore: remove the exec permissions
error: handle dev environment repository: error updating dev environment for environment-cosmin-splitsecrets-dev: forking and pulling https://github.com/ccojocar-org/environment-cosmin-splitsecrets-dev.git: Unable to cherry-pick 0f34eba0d4076450002cccf89a6e89c6feaf184b: git output: error: commit 0f34eba0d4076450002cccf89a6e89c6feaf184b is a merge but no -m option was given.
fatal: cherry-pick failed: failed to run 'git cherry-pick 0f34eba0d4076450002cccf89a6e89c6feaf184b' command in directory '/Users/cosmin/projects/tmp/cluster-vault/jenkins-x-boot-config', output: 'error: commit 0f34eba0d4076450002cccf89a6e89c6feaf184b is a merge but no -m option was given.
fatal: cherry-pick failed'
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step verify env' command in directory '.', output: ''

[ccojocar]

I am wondering whether git merge --ff-only is not a better option instead of cherry picking.