[WIP] New build tool #299

mderriey · 2022-09-23T14:42:42Z

Fixes #298

This is my first attempt at trying to come up with something that fits the bill.
Mostly to get something in front of you and get feedback.

For now, I've created different workflows to make it easier for me to test things; in the future, we could look at consolidating them and including some logic within the workflow itself.
They're mostly the same, the things that change are:

The trigger: a pull request is opened, a pull request is merged (push), a tag is created (either for the main Carter package, or the Newtonsoft one)
Which package we push, and where it's pushed: this is done through the arguments we pass to the push.sh script

Here's the list of actions I took on the repo, and what would happen:

Action taken	Result	Link to workflow
Opened a PR	Carter prerelease package pushed to feedz	https://github.com/mderriey/Carter/actions/runs/3113410773/jobs/5048006852#step:6:1
Merged a PR	Carter prerelease package pushed to NuGet	https://github.com/mderriey/Carter/actions/runs/3113457288/jobs/5048111598#step:6:1
Created the `6.1.3` tag	Carter release package pushed to NuGet	https://github.com/mderriey/Carter/actions/runs/3113464170/jobs/5048127102#step:6:1
Carted the `newtonsoft-6.0.0` tag	Newtonsoft release package pushed to NuGet	https://github.com/mderriey/Carter/actions/runs/3113483201/jobs/5048168976#step:6:1

…Action workflow files See https://www.meziantou.net/enabling-intellisense-for-github-actions-workflows-in-vs-code.htm

See https://github.com/mderriey/Carter/actions/runs/3113376250/jobs/5047929435#step:3:1

Dummy change to open PR

…efaultinterface branch

mderriey · 2022-09-23T14:44:03Z

.github/workflows/publish-prerelease.yml

+on:
+  push:
+    branches:
+    - defaultinterface


This will change to main but I wanted to build on top of your defaultinterface branch where you had already moved Newtonsoft into its own project.

mderriey · 2022-09-23T14:44:34Z

.github/workflows/publish-release.yml

+on:
+  push:
+    tags-ignore:
+    - 'newtonsoft-*'


This makes sure that it doesn't run for when we want to release a new version of the Newtonsoft package

Is that looking for a tag containing the term "newtonsoft"

Apologies, that should have read "for when we want to push a release of the Carter package".

And you're correct, this is ignoring tags that start with newtonsoft-

Thing is tags will probably just be v7.0.0 for example

I used different tags as I thought you wanted to be able to have different versions of the Carter and the Carter.ResponseNegotiators.Newtonsoft packages while the source stays in the same repo.

"Regular" tags, like v7.0.0, would release Carter, while newtonsoft-v7.0.0 would release the Carter.ResponseNegotiators.Newtonsoft package.

Did I make a wrong assumption thinking you might want to keep their versioning separate?
My thinking was that if we keep them in sync, new versions of Carter.ResponseNegotiators.Newtonsoft could be released without any changes in them.

mderriey · 2022-09-23T14:44:58Z

.github/workflows/publish-newtonsoft-release.yml

+on:
+  push:
+    tags:
+    - 'newtonsoft-*'


This makes sure that it runs only for when we want to release the Newtonsoft package.

mderriey · 2022-09-23T14:45:51Z

push.sh

+TARGET_PACKAGE=$1
+if [ "$TARGET_PACKAGE" = "carter" ]; then
+  TARGET_PACKAGE_PATH="./src/Carter/**/*.nupkg"
+elif [ "$TARGET_PACKAGE" = "newtonsoft" ]; then
+  TARGET_PACKAGE_PATH="./src/Carter.ResponseNegotiators.Newtonsoft/**/*.nupkg"
+fi


Not a Bash expert by any means.

This is the way I found to target either the main Carter or the Newtonsoft package,

mderriey · 2022-09-25T08:53:59Z

@jchannon would you mind taking a look and letting me know if that sort of fits what you had in mind?

Very keen for feedback and happy to make changes to adjust things.

Cheers.

jchannon · 2022-09-26T10:12:26Z

Looks good, the only thing I wouldnt want is a push to nuget on a PR merge. For pre-releases I'd probably tag the repo as 7.0.0-beta1 for example

jchannon · 2022-09-26T10:13:29Z

.github/workflows/publish-release.yml

+on:
+  push:
+    tags-ignore:
+    - 'newtonsoft-*'


Is that looking for a tag containing the term "newtonsoft"

jchannon · 2022-09-26T10:14:08Z

.github/workflows/publish-release.yml

+    - name: Build Reason
+      run: echo ${{github.ref}} and ${{github.event_name}}
+    - name: Build with dotnet
+      run: find . -path ./template -prune -false -o -name '*.csproj' -execdir dotnet build --configuration Release --nologo {} \;


I might have to change this or move the newtonsoft stuff out of the main repo otherwise this will find the csproj.

I suppose it depends on how split out you want them to be. Right now it'll find all csprojs and build them; if that's an issue, we could look at using path filters in the GitHub Actions workflow to only build projects we're interested in.

I find it overkill, that project doesn't add much overhead now, but it's up to you.

mderriey · 2022-09-26T11:46:48Z

Looks good, the only thing I wouldnt want is a push to nuget on a PR merge. For pre-releases I'd probably tag the repo as 7.0.0-beta1 for example

OK, got it, I'll push an update for this.

jchannon · 2022-09-26T12:04:19Z

No you were right, I just couldn't envisage how we'd do two separate tag approaches :) I am wondering if I move it out into it's own repo whether that might be clearer however I'd still somehow want to bump those external packages to use the latest carter when we release a new carter version

…

On Mon, 26 Sept 2022 at 12:59, Mickaël Derriey ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In .github/workflows/publish-release.yml <#299 (comment)> : > @@ -0,0 +1,29 @@ +name: Main tag created workflow + +on: + push: + tags-ignore: + - 'newtonsoft-*' I used different tags as I thought you wanted to be able to have different versions of the Carter and the Carter.ResponseNegotiators.Newtonsoft packages while the source stays in the same repo. "Regular" tags, like v7.0.0, would release Carter, while newtonsoft-v7.0.0, would release the Carter.ResponseNegotiators.Newtonsoft package. Did I make a wrong assumption thinking you might want to keep their versioning separate? My thinking was that if we keep them in sync, new versions of Carter.ResponseNegotiators.Newtonsoft could be released without any changes in them. — Reply to this email directly, view it on GitHub <#299 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAZVJTADOJPVEIW4KOUKCDWAGFZNANCNFSM6AAAAAAQUBPU6Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

mderriey · 2022-09-26T12:25:44Z

No you were right, I just couldn't envisage how we'd do two separate tag approaches :)

Cool 👍

I am wondering if I move it out into its own repo whether that might be clearer however I'd still somehow want to bump those external packages to use the latest carter when we release a new carter version.

I think there are several options here:

In the same workflow as the one where we release the Carter package, we can find/replace references to the Carter with the latest version; this works for things that are in the same repository.
We could also imagine a separate workflow that queries NuGet for the latest Carter version, update package references, then run some kind of tests; this could work for things in the same repo and in different repositories.

Whether to move it in its own repository is up to you, I'd say.
If you don't foresee many other projects with a different release cycle, keeping it in here doesn't add too much "engineering" overhead in my opinion.
If/when we hit issues, we can always move it.
Maybe some folks at NDC will have opinions based on how painful it was for them 😉

For the template project, one thing I did in the past that worked fairly well was the following:

The CarterTemplate project would have a project reference to the Carter project — the benefit of this is that you get immediate feedback if you make an API change that affects the template project.
At release time, swap the project reference for a package reference (let's say 7.0.0 if that's what we're releasing).
Then pack and push the template project.

This way, you always keep an updated template project with the latest Carter version.

jchannon · 2022-09-26T13:16:51Z

I mean we already have a couple of external repos that depend on carter as a package reference but have not been kept up to date really because of the complexities here and me being bothered to work out to do it best :) So if we did move the newtonsoft package out then we'd have 3 or 4 repos that need the package refence updated. Whilst you say we could query nuget, how would you do that, a timer or I wonder if there is a cross repo trigger eg/Carter gets tagged and can then somehow notify other repos.

Regarding the template I also came up with that idea but never got around to doing it, if we did swap the references I'd also want to do that automatically as I already manually change it once Carter is released and I don't like that :)

mderriey · 2022-09-26T13:25:59Z

Whilst you say we could query nuget, how would you do that, a timer or I wonder if there is a cross repo trigger eg/Carter gets tagged and can then somehow notify other repos.

I'm not aware of cross-repo way of triggering workflows.

My initial idea was manually triggering the workflow run (I didn't know there were other repos with package references to Carter), but scheduled runs could also work.

Regarding the template I also came up with that idea but never got around to doing it, if we did swap the references I'd also want to do that automatically as I already manually change it once Carter is released and I don't like that :)

I wasn't clear enough, apologies. Yes, the swapping was done automatically when I implemented that at work a while ago.

jchannon · 2022-09-26T13:39:34Z

A manually triggered workflow that did a find and replace, commit and asked for a tag and then pushed and released to nuget could be very handy indeed!

mderriey · 2022-09-27T14:25:32Z

OK, we can have a look at that in the future.

How would you like to go forward with this PR? Do you want to wait for .NET 7 to be out and your changes to be merged to main, or should we try and get something merged soon-ish and adapt when the split-out Newtonsoft project gets merged to main?

Remove the publishing of a prerelease package when a PR is merged to `main` as per #299 (comment).

mderriey · 2022-09-30T08:54:25Z

@jchannon friendly bump about #299 (comment)

jchannon · 2022-09-30T08:58:40Z

👍 will take a look next week. At NDC at the mo 😁

…

On Fri, 30 Sep 2022 at 10:54, Mickaël Derriey ***@***.***> wrote: @jchannon <https://github.com/jchannon> friendly bump about #299 (comment) <#299 (comment)> — Reply to this email directly, view it on GitHub <#299 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAZVJWPMCONT7MOVFR3XITWA2TEXANCNFSM6AAAAAAQUBPU6Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

jchannon · 2022-10-01T19:04:15Z

As your PR is going to be merged into my branch which has all the .NET 7 changes on it I wonder if we can get it all working in your branch? I just approved a thing in GH Actions so any changes made in the workflow files should build here 🤞

This reverts commit 6eb0461.

mderriey · 2022-10-03T12:31:15Z

I've made a few changes to test out the real thing, but I believe it will need to be tested on your repo as I'm fairly certain I won't have access to your repo's secrets, like the feedz.io API key.

mderriey · 2022-10-05T09:06:47Z

It now says that the workflow is awaiting approval, so I suppose you need to do something on your end.

To test the whole thing, you'll need to add:

The feedz API key as FEEDZ_KEY, which you should already have.
The NuGet API key as NUGET_KEY.

The scenarios to test those would be:

Open a PR to trigger a prerelease package pushed to feedz.io (this PR should do the trick)
Create a tag, like X.X.X to trigger a release package pushed to NuGet.
Create a tag, like newtonsoft-X.X.X to trigger a Newtonsoft release package pushed to NuGet.

It's a bit awkward testing this as it will push packages to NuGet; can you unlist them afterwards? Or maybe you could create tags with versions lower than the current one so that consumers don't get asked to update?

You also mentioned earlier that you would create tags with a leading v, like v7.0.0; if you still want to go this way, I'll need to make a slight change to let MinVer know that the tags contain a prefix.

jchannon · 2022-10-05T09:09:03Z

Ok, approved the workflow.

Tags don't seem to contain the prefix on the repo, I just checked.

mderriey · 2022-10-05T09:12:31Z

Ok, approved the workflow.

Sweet, thanks.

The run failed, supposedly because the workflow didn't have access to the FEEDZ_KEY secret, see https://github.com/CarterCommunity/Carter/actions/runs/3174168115/jobs/5201477628#step:6:8.

Maybe you need to enable forks having access to the secrets? It's a bit sketchy, unless you can do this on a case-by-case basis. Otherwise, we could look at making me a temporary contributor to this repo and I open a PR from a separate branch in this repo?

Tags don't seem to contain the prefix on the repo, I just checked.

OK, so you want to keep it this way, with no prefix?

jchannon · 2022-10-05T09:16:08Z

Just given you write access! 😄
See if that helps, if not I can bump the access.

Yeah no prefix required

mderriey · 2022-10-05T09:19:49Z

Just given you write access! 😄

Sweet, thanks, opened #300 and closing this PR.

mderriey added 9 commits September 23, 2022 15:11

Add VS Code settings and extensions to get auto-completion on GitHub …

4816a43

…Action workflow files See https://www.meziantou.net/enabling-intellisense-for-github-actions-workflows-in-vs-code.htm

First attempt with plenty of different workflows

1a76aae

Dummy change to open PR

6eb0461

Fix dotnet-version

1a6d32e

See https://github.com/mderriey/Carter/actions/runs/3113376250/jobs/5047929435#step:3:1

Skip tests as they're failing

0fb27ab

Skip tests in all workflows

858fade

Merge pull request #1 from mderriey/defaultinterface-pr

1e90d77

Dummy change to open PR

Update target branch for prerelease since I'm still working on that d…

142b9ac

…efaultinterface branch

Differentiate 'tag created' workflows

f46036a

mderriey commented Sep 23, 2022

View reviewed changes

jchannon reviewed Sep 26, 2022

View reviewed changes

Consolidate workflows

c1349ca

Remove the publishing of a prerelease package when a PR is merged to `main` as per #299 (comment).

mderriey added 3 commits October 3, 2022 14:16

Revert "Dummy change to open PR"

b48d4cb

This reverts commit 6eb0461.

Remove unneeded reference to the Carter package

ce1ebd4

Wire up API keys

299a472

For now trigger on PRs to the defaultinterface branch

121c0e1

mderriey mentioned this pull request Oct 5, 2022

New build tool #300

Merged

mderriey closed this Oct 5, 2022

This was referenced Oct 16, 2022

Publish the CarterTemplate package automatically #302

Closed

Publish the CarterTemplate package automatically (with MSBuild) #303

Merged

mderriey mentioned this pull request Nov 4, 2022

Add workflow to automate updating references to Carter NuGet package #304

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] New build tool #299

[WIP] New build tool #299

mderriey commented Sep 23, 2022 •

edited

Loading

mderriey Sep 23, 2022

mderriey Sep 23, 2022

jchannon Sep 26, 2022

mderriey Sep 26, 2022 •

edited

Loading

jchannon Sep 26, 2022

mderriey Sep 26, 2022 •

edited

Loading

mderriey Sep 23, 2022

mderriey Sep 23, 2022

mderriey commented Sep 25, 2022

jchannon commented Sep 26, 2022

jchannon Sep 26, 2022

jchannon Sep 26, 2022

mderriey Sep 26, 2022

mderriey commented Sep 26, 2022

jchannon commented Sep 26, 2022 via email

mderriey commented Sep 26, 2022

jchannon commented Sep 26, 2022

mderriey commented Sep 26, 2022

jchannon commented Sep 26, 2022

mderriey commented Sep 27, 2022

mderriey commented Sep 30, 2022

jchannon commented Sep 30, 2022 via email

jchannon commented Oct 1, 2022

mderriey commented Oct 3, 2022

mderriey commented Oct 5, 2022

jchannon commented Oct 5, 2022

mderriey commented Oct 5, 2022

jchannon commented Oct 5, 2022

mderriey commented Oct 5, 2022

[WIP] New build tool #299

[WIP] New build tool #299

Conversation

mderriey commented Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mderriey Sep 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mderriey Sep 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mderriey commented Sep 25, 2022

jchannon commented Sep 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mderriey commented Sep 26, 2022

jchannon commented Sep 26, 2022 via email

mderriey commented Sep 26, 2022

jchannon commented Sep 26, 2022

mderriey commented Sep 26, 2022

jchannon commented Sep 26, 2022

mderriey commented Sep 27, 2022

mderriey commented Sep 30, 2022

jchannon commented Sep 30, 2022 via email

jchannon commented Oct 1, 2022

mderriey commented Oct 3, 2022

mderriey commented Oct 5, 2022

jchannon commented Oct 5, 2022

mderriey commented Oct 5, 2022

jchannon commented Oct 5, 2022

mderriey commented Oct 5, 2022

mderriey commented Sep 23, 2022 •

edited

Loading

mderriey Sep 26, 2022 •

edited

Loading

mderriey Sep 26, 2022 •

edited

Loading