continuous delivery CD pipeline for releases

[monperrus]

Requirements

• a new script for normal releases (shell or python) (spec below)
• add CI job to handle changes in the release spec set up declarative releases for Spoon SpoonLabs/spoon-deploy#6

Normal release is everything which is not a beta

[monperrus]

FYI, CD of gumtree-spoon is also still Travis-based;

[slarse]

I've got this on next week's TODO list.

[slarse]

By next week I of course meant next month when Travis CI has already stopped working.

Looking at this right now. It's of course possible to just port the CD script to some other platform, but while we're at it, we may want to consider simplifying the CD process. The script is fairly complex and fragile. One example of extreme fragility is that it relies on the version number in pom.xml being on line 21. If I were to run the same formatting on the spoon-core pom file as I do on the parent pom file in #3994, then the version would be bumped down to line 22, breaking the CD script. That seems unreasonably fragile to me.

So there are two goals here: make the CD less fragile, and make it work on some other platform. We can deploy either from GitHub Actions or Jenkins. As this is a critical process, I think we should run it on Jenkins for complete control.

@monperrus I suggest the following changes.

Separate the beta releases and normal releases into different scripts/jobs

These are two fundamentally different concerns. We want to do a beta release every X days, while we want to do a normal release on specific commits (bump commits).

Permanently fix the Javadoc plugin problems (actually, it may already be fixed)

Currently, the CD script is injecting stuff into the pom to fix some crash on Travis. I don't see why we can't fix this permanently by just fixing the pom file. According to the linked issue, a workable solution is to just set the maven.compiler.source property to 8. This may clash with Spoon overriding the source version in the parent pom with the custom java.src.version property, but that's fixable.

However, as the source version is already set to 1.8 in the parent pom, this crash may not even be a problem anymore, and the fix in the CD script is just a leftover from before the source version was explicitly set. The docs for the Javadoc plugin say that it uses the source parameter of maven-compiler-plugin to resolve API links, and since it's set to 1.8, it should be all good.

Normal release changes

Currently, the normal release is where we have the severe fragility I mentioned above. We can get rid of that by running the normal release script on every commit, but exiting immediately if the commit in question is not a bump commit. Then, instead of searching for changes on a specific line, we just compare the results of mvn help:evaluate -Dexpression=project.version -q -DforceStdout on the current and previous commit. If they differ, we have a bump!

Instead of semi-manually fiddling around in the pom with xmlstarlet, I also think it would be much better to use the Maven release plugin.

Current priorities

As we don't plan to make a normal release in the near future, setting up a standalone beta release job is the top priority. I think we should deploy from Jenkins for complete control, but using GitHub Actions like Sorald does is actually simpler.

@monperrus Any thoughts before I get to work on this?

[monperrus]

We should update the Continuous Delivery script to create a new release on Github.

[slarse]

I've done that in Sorald's release script, although that's in Python. It's essentially just a single API call, so it's easy enough to do with curl.

[monperrus]

FYI, CD of spoon-maven-plugin is off because we're out of Travis gas again. Starts to be pressing to move CD to Actions or Jenkins.

[slarse]

Okidoki, I think I can spare some time during the weekend to pick up this thread again.

[slarse]

FYI, CD of spoon-maven-plugin is off because we're out of Travis gas again. Starts to be pressing to move CD to Actions or Jenkins.

I'm a bit confused, I can't find that spoon-maven-plugin has any continuous delivery? Did you mean just the CI running on Travis?

[monperrus]

you're right, there is no CD for spoon-maven-plugin.

[slarse]

But there is CI, and I suppose that's what we want to migrate there?

And gumtree-spoon has already been migrated over to GitHub Actions, so I don't really think there's all that much urgency. AFAIK, we have nothing crucial running on Travis anymore.

[monperrus]

yes, it'd be great to migrate spoon-maven-plugin's CI.

[algomaster99]

Spoon's auto-deployment of beta release is broken since October 2021. I think that puts a priority on this issue. I will try having a look next week. I have a lot on my plate for the time being.

[monperrus]

did a manual delivery of 10.0.1-beta-2 from 3cdf267

[algomaster99]

@khaes-kth we can now use the beta version to run our experiments. :)

[monperrus]

@algomaster99 the current CD code which is broken is at https://github.com/SpoonLabs/spoon-deploy

It's broken because the SponnLabs org has no more free Travis credits.

[algomaster99]

On first look, it seems similar to how gumtree-spoon's CI was set up with travis. Here are the PRs which might be helpful - SpoonLabs/gumtree-spoon-ast-diff#203, SpoonLabs/gumtree-spoon-ast-diff#205, SpoonLabs/gumtree-spoon-ast-diff#207, and SpoonLabs/gumtree-spoon-ast-diff#209.

[algomaster99]

@monperrus I am taking this one up. This is very similar to what we have done for gumtree-spoon.

[slarse]

Fixed assignee.

[algomaster99]

The beta deployment script for Spoon has been fixed - SpoonLabs/spoon-deploy#3. It should push beta-5 on Saturday at 0100.

One further action which we might need to take on Spoon is to change the version of nexus-staging-maven-plugin. In spoon-deploy, I had to change it from 1.6.10 to 1.6.8 to make a deployment. However, @monperrus and I feel the maintainers might have fixed the issue in 1.6.11. Hence, we are not downgrading it in spoon-pom as of now.

[algomaster99]

spoon-deploy is disabled if there is no activity on the repository for 60 days. Should we do something about it? For example, push an empty commit once in while.

[monperrus]

Do you mean a commit in spoon-core (always very active) or a commit in spoon-deploy (indeed calm)? What does "disabled" mean in this context?

[MartinWitt]

Why don't we use an already built solution, https://github.com/marketplace/actions/keepalive-workflow? The 60-day problem should be solved by this.

Do you mean a commit in spoon-core (always very active) or a commit in spoon-deploy (indeed calm)? What does "disabled" mean in this context?

In any repo after 60 days without a commit, GitHub disables all workflows and emails you about it. To keep them active you either need to click the link in the mail or create some activity in the repo like a commit.

To prevent unnecessary workflow runs, scheduled workflows may be disabled automatically. When a public repository is forked, scheduled workflows are disabled by default. In a public repository, scheduled workflows are automatically disabled when no repository activity has occurred in 60 days.
https://docs.github.com/en/actions/managing-workflow-runs/disabling-and-enabling-a-workflow

[algomaster99]

Why don't we use an already built solution, github.com/marketplace/actions/keepalive-workflow? The 60-day problem should be solved by this.

This seems like a good idea. I did not look up existing solutions because I thought it was quite straightforward and hence, we should avoid any more dependencies.

[slarse]

I did not look up existing solutions because I thought it was quite straightforward and hence, we should avoid any more dependencies.

I agree with this. This is so incredibly simple, it should more or less be something like this in a workflow:

git config --local user.email github-actions[bot]@users.noreply.github.com
git config --local user.name github-actions[bot]
git commit --allow-empty -m "Empty commit to keep workflows alive"
git push origin master

While adopting ready-to-use solutions is great and all, if it's a small enough problem (cough, left-pad, cough) you might as well solve it yourself :)

[algomaster99]

Thanks @slarse! I was also thinking of something like this and schedule in every 59 days.

…

On Thu, 3 Nov, 2022, 6:24 pm Simon Larsén, ***@***.***> wrote: I did not look up existing solutions because I thought it was quite straightforward and hence, we should avoid any more dependencies. I agree with this. This is so incredibly simple, it should more or less be something like this in a workflow: git config --local user.email ***@***.*** git config --local user.name github-actions[bot] git commit --allow-empty -m "Empty commit to keep workflows alive" git push origin master While adopting ready-to-use solutions is great and all, if it's a small enough problem (cough, left-pad, cough) you might as well solve it yourself :) — Reply to this email directly, view it on GitHub <#3890 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIMPTOLPBVN64BHWQENZVOLWGPYNNANCNFSM43SW4NZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[I-Al-Istannen]

Somebody requested that we provide a jar file in the github releases so you don't have to download it from maven central. This might be useful as spoon can be run stand-alone, directly from your terminal. On the other hand, I am not quite sure why you would use the CLI in the first place.

We would probably want to provide a fat jar in that case, as it doesn't really help you much otherwise.

[algomaster99]

@I-Al-Istannen GitHub releases only have the non-beta releases. Since they are deployed manually on maven central, I think the maintainer needs to keep in mind that they also push the jar to the release.

However, if you think creating GitHub releases for beta releases is a good idea, we can discuss that.

[I-Al-Istannen]

Then I might have misunderstood the purpose of this issue, I thought it was about releases in general. But yea, if they are done manually I am not sure where we'd discuss whether we want this or document it.

[slarse]

I see no reason not to automate the entirety of the release cycle, including full releases.

[monperrus]

we should be compatible with reproducible builds, see #5139 (comment)

[MartinWitt]

Is there any reason to not push the commit for each release/beta release to the master? I can try to integrate a state of the art java solution(https://jreleaser.org/) later this week.

[monperrus]

I can try to integrate a state of the art java solution(jreleaser.org) later this week.

nice, thanks a lot!

[MartinWitt]

https://github.com/chains-project/maven-lockfile/actions/workflows/jreleaser.yml I build here a solution which is foolproof. If you want to create a release, you simply select the semantic next version and start the action. I will try to create this script for spoon also.
Related chains-project/maven-lockfile#63

[monperrus]

Great, I understand it is click-based. Can we make it Infrastructure-as-code where merging metadata = deploy?

[MartinWitt]

You mean if we change the version in a commit (HEAD) in comparison to the commit before(HEAD~1) the release script should start? Sure, I can try to build this.

[monperrus]

let's keep it simple, and set up jreleaser as you propose. if the whole workflow works as expected, we'll automate with IaaC later.

[MartinWitt]

Blocked by #5164. JReleaser and maven central want this tag normally.

[monperrus]

#5164 is merged, we can make progress here @MartinWitt

[algomaster99]

@MartinWitt can be assigned now.

[monperrus]

the new release pipeline works