JENKINS-48431 Support both lightweight checkout AND build parameters #772

tgatinea · 2019-10-14T10:37:26Z

JENKINS-48431 Support both lightweight checkout AND build parameters

To support both lightweight checkout AND build parameters, we need to:

recover the environment parameters of the build
evaluate the SCM structure according to the build environment parameters

This leads to implement a new method into scm-api-plugin:
public static SCMFileSystem of(@nonnull Run build, @nonnull SCM scm, @checkfornull SCMRevision rev)

And it leads to implement:
public SCMFileSystem build(@nonnull Run build, @nonnull SCM scm, @checkfornull SCMRevision rev)

This method recovers from the current build the environment parameters and evaluate the SCM structure with the environment parameters.

It also does a git fetch with the refspec if the refspec is set in SCM.

It also take into account the case of a gerrit trigger with:

SCM refspec set to $GERRIT_REFSPEC
SCM branch set to FETCH_HEAD

This is done between lines 360 to 369

Note that this modification is in relation with two other pull requests that concerns:

workflow-cps-plugin: JENKINS-48431 Support both lightweight checkout AND build parameters workflow-cps-plugin#329
scm-api-plugin: [JENKINS-48431] Support both lightweight checkout AND build parameters scm-api-plugin#78

Note that this pull request will not work if the pull request for scm-api-plugin is not valdated

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. This is simply a reminder of what we are going to look for before merging your code. If a checkbox or line does not apply to this pull request, delete it. We prefer all checkboxes to be checked before a pull request is merged

I have read the CONTRIBUTING doc
I have referenced the Jira issue related to my changes in one or more commit messages
I have added tests that verify my changes
Unit tests pass locally with my changes
I have added documentation as necessary
No Javadoc warnings were introduced with my changes
No spotbugs warnings were introduced with my changes
I have interactively tested my changes
=> I have tested my changes though workflow-cps-plugin under hpi
Any dependent changes have been merged and published in upstream modules (like git-client-plugin)
=> This pull request depends closely to the scm-api-plugin

Types of changes

What types of changes does your code introduce? Put an x in the boxes that apply. Delete the items in the list that do not apply

Dependency or infrastructure update
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Further comments

If this is a relatively large or complex change, start the discussion by explaining why you chose the solution you did and what alternatives you considered.

tgatinea · 2019-10-22T16:27:03Z

@MarkEWaite I'm familiar with delivering plugins that gets dependencies.
Do I need to wait the end of the pull request jenkinsci/scm-api-plugin#78 before having some feedback for the git-plugin code?
Thanks by advance

MarkEWaite · 2019-10-22T17:19:36Z

You don't need to wait until the end of pull request jenkinsci/scm-api-plugin#78 before receiving feedback on the git plugin code. A reviewer could provide feedback based on the combination of pull requests.

I have limited time for review of pull requests and many other pull requests that I consider higher priority than your request to support lightweight checkout and build parameters. I will not review your pull request for a long time.

Top themes that I've prioritized include:

Other maintainers may have different priorities and different objectives. You're certainly welcome to check with @rsandell and @fcojfernandez on their priorities.

hkrpavan · 2019-11-24T08:24:39Z

@tgatinea were you able to generate plugin with this change? Do you have a private .hpi file to share with? I too waiting for this bug fix.

I tried building these three plugins locally and generated plugins. They seems to be not working. Looks like git client is crashing inspite of downgrading it to desired version.

java.lang.NullPointerException
at jenkins.plugins.git.GitSCMFileSystem.lambda$new$0(GitSCMFileSystem.java:117)
at jenkins.plugins.git.GitSCMFileSystem$2.invoke(GitSCMFileSystem.java:191)
at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:78)
at jenkins.plugins.git.GitSCMFileSystem.invoke(GitSCMFileSystem.java:187)
at jenkins.plugins.git.GitSCMFileSystem.(GitSCMFileSystem.java:117)
at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:369)
at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:197)
at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:173)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:113)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:299)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

tgatinea · 2019-11-27T17:04:04Z

@hkrpavan Were you able to recompile the 3 plugins :

scm-api-plugin should be version 2.6.4-SNAPSHOT and need to be installed on jenkins first
I patched git-plugin from version stable-3.x
=> But now last version is 4.0.0 and I cannot recompile anymore
=> I probably have to rework
This plugin is to be installed after previous plugin
workflow-cps-plugin was coming from 2.74 and it is now at 2.77

When I'll have time I will move to the last versions and push my code

hkrpavan · 2019-11-27T22:53:15Z

@tgatinea , Thanks for the reply. I could able to build these plugins (git-plugin stable-3.x) with your patch and make it working. This works good with declarative pipelines. Now i bumped into https://issues.jenkins-ci.org/browse/JENKINS-56943, though a different problem altogether. This too is part of light weight checkout while pulling resource file after Jenkinsfile pull.

Something like below in jenkinsfile,

agent {
kubernetes {
yamlFile 'build/pipelines/resources/KubernetesPodPipeline.yaml'
}
}

Started by user
Rebuilds build #242
Obtained build/Jenkinsfile from git ssh://XYZ/XYZ
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] readTrusted
[Pipeline] End of Pipeline
hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune -- origin +refs/heads/FETCH_HEAD:refs/remotes/origin/FETCH_HEAD" returned status code 128:
stdout:
stderr: fatal: Couldn't find remote ref refs/heads/FETCH_HEAD
fatal: the remote end hung up unexpectedly

tgatinea · 2019-11-28T10:53:26Z

@hkrpavan I have found why git-plugin does not compile anymore.
In my patch, I have :
String refspec = environment.expand(config.getRefspec()).trim();
String head = headName;
if (refspec == null) { ...

The spotbugs "mvn spotbugs:gui" complain that refspec cannot be null because String.trim() cannot be with a String null.
But in fact it can be the case as config.getRefspec() can return null value.

Now, if I code something like:
String refspec = environment.expand(config.getRefspec());
if (refspec != null) {
refspec = refspec.trim();
}
String head = headName;
if (refspec == null) { ...

Then it compiles because "refspec = environment.expand(config.getRefspec())" can be null.

hkrpavan · 2019-11-30T01:41:07Z

@tgatinea Thank you, i could able to build git-plugin with your latest changes from top if master branch. Also, i could able to resolve my other issue by updating workflow-multibranch-plugin to point to your SCMFileSystem method. Now i could able to resolve Kubernetes resource file pull issue.

https://issues.jenkins-ci.org/browse/JENKINS-56943

vpod · 2019-12-06T13:34:25Z

@hkrpavan : so, let's vote and contribute both changes :-)

jetersen · 2020-05-25T21:38:05Z

I have expanded the changes to include GitSCMTelescope and GitSCMFileSystem and removed the duplication :)

jetersen · 2020-05-25T23:25:16Z

@vpod @hkrpavan
I created jenkinsci/workflow-multibranch-plugin#98 to solve the readTrusted step 😅

jetersen · 2020-05-26T11:02:18Z

FYI I just deployed all the bits to our production server 💪 and our job passed successfully. #898 (comment)

jetersen · 2020-05-28T05:41:58Z

if anyone else wants the bits. I have deployed them to repo.jenkins-ci.org

https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/scm-api/2.6.4-SNAPSHOT/scm-api-2.6.4-20200525.215035-2.hpi
https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/workflow/workflow-cps/2.81-SNAPSHOT/workflow-cps-2.81-20200526.095117-1.hpi
https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/workflow/workflow-multibranch/2.22-SNAPSHOT/workflow-multibranch-2.22-20200526.094121-1.hpi
https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/git/4.3.0-SNAPSHOT/git-4.3.0-20200525.225217-1.hpi

fcojfernandez

Hi @tgatinea ! thanks for the PR. First, you're overriding some default behaviours, so it'd be nice to have some unitary testing.

MarkEWaite · 2020-06-06T11:58:39Z

Superseded by #898

JENKINS-48431 Support both lightweight checkout AND build parameters

5bd9173

This was referenced Oct 14, 2019

JENKINS-48431 Support both lightweight checkout AND build parameters jenkinsci/workflow-cps-plugin#329

Closed

[JENKINS-48431] Support both lightweight checkout AND build parameters jenkinsci/scm-api-plugin#78

Closed

Thierry added 2 commits October 14, 2019 19:22

Bugfix for tests

bf791e2

Trim whitespaces

96f458d

MarkEWaite added the enhancement Improvement or new feature label Oct 19, 2019

Bugfix spotbugs

cb53320

MarkEWaite added 2 commits March 10, 2020 11:34

Merge branch 'master' into JENKINS-48431

efe53fd

Merge branch 'master' into JENKINS-48431

4e4d5fd

This was referenced May 25, 2020

[JENKINS-48431] - Support both lightweight checkout AND build parameter #898

Closed

[JENKINS-53705] - add support for build parameters jenkinsci/workflow-multibranch-plugin#98

Draft

This was referenced May 26, 2020

[JENKINS-48431] - Support both lightweight checkout AND build parameters jenkinsci/scm-api-plugin#83

Closed

[JENKINS-56063] (Expansion of variables in refspec in case of honor refspec on initial clone) #830

Closed

fcojfernandez requested changes Jun 1, 2020

View reviewed changes

MarkEWaite closed this Jun 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JENKINS-48431 Support both lightweight checkout AND build parameters #772

JENKINS-48431 Support both lightweight checkout AND build parameters #772

tgatinea commented Oct 14, 2019 •

edited

Loading

tgatinea commented Oct 22, 2019

MarkEWaite commented Oct 22, 2019 •

edited

Loading

hkrpavan commented Nov 24, 2019 •

edited

Loading

tgatinea commented Nov 27, 2019

hkrpavan commented Nov 27, 2019 •

edited

Loading

tgatinea commented Nov 28, 2019

hkrpavan commented Nov 30, 2019 •

edited

Loading

vpod commented Dec 6, 2019

jetersen commented May 25, 2020

jetersen commented May 25, 2020 •

edited

Loading

jetersen commented May 26, 2020

jetersen commented May 28, 2020

fcojfernandez left a comment

MarkEWaite commented Jun 6, 2020

JENKINS-48431 Support both lightweight checkout AND build parameters #772

JENKINS-48431 Support both lightweight checkout AND build parameters #772

Conversation

tgatinea commented Oct 14, 2019 • edited Loading

Checklist

Types of changes

Further comments

tgatinea commented Oct 22, 2019

MarkEWaite commented Oct 22, 2019 • edited Loading

hkrpavan commented Nov 24, 2019 • edited Loading

tgatinea commented Nov 27, 2019

hkrpavan commented Nov 27, 2019 • edited Loading

tgatinea commented Nov 28, 2019

hkrpavan commented Nov 30, 2019 • edited Loading

vpod commented Dec 6, 2019

jetersen commented May 25, 2020

jetersen commented May 25, 2020 • edited Loading

jetersen commented May 26, 2020

jetersen commented May 28, 2020

fcojfernandez left a comment

Choose a reason for hiding this comment

MarkEWaite commented Jun 6, 2020

tgatinea commented Oct 14, 2019 •

edited

Loading

MarkEWaite commented Oct 22, 2019 •

edited

Loading

hkrpavan commented Nov 24, 2019 •

edited

Loading

hkrpavan commented Nov 27, 2019 •

edited

Loading

hkrpavan commented Nov 30, 2019 •

edited

Loading

jetersen commented May 25, 2020 •

edited

Loading