Provide ability to trigger CI once flux applies changes

[abbi-gaurav]

Describe the feature
Flux is a great tool for GitOps. However, what I am missing is the ability to roll-over from say stage to production in a true Gitops fashion.

I would like to keep the git as my source of truth whether it is for a stage or a production deployment.

Here is my generic idea:

• A flux controller is deployed in the stage namespace.

• A flux controller is deployed in the prod namespace.

• Have a stage namespace and a corresponding stage Github branch.

• Developer commits changes to the stage branch.

• stage flux will pick up the changes and deploy to stage namespace.

• Run the smoke tests. If the test passes, merge the changes to prod Github branch.

  • currently, I would have to do some random wait which may be volatile.
  • what is want is to get notified and be able to trigger my smoke tests one flux have deployed any changes.

• prod flux will pick up the changes and deploy to prod namespace.

[chancez]

Perhaps if Flux had a CRD, or other resource which could be watched it would be relatively easy to write something which would allow you to be notified however you like, eg: webhooks, or a queue, etc.

I've wanted some way to monitor Flux' progress in rolling out changes myself, not for automation, but more for monitoring. I know ArgoCD operates on CRs and it maintains some information in the status, so something similar could be nice.

[gecube]

Hello!

I found flux too useful very much. And I think that the proposed feature is out of scope of Flux tooling.
Right now I implement the analogical scheme with GitLab-CI:

1. the prod environment is located in prod branch and stage environment in stage branch
2. When I change the stage branch, GitLab-CI process triggers the sync with Flux in stage cluster
3. I make checks that everything is fine.
4. After that I promote manually changes from stage to prod branches.
5. The same as step 2 proceeds with prod cluster

Maybe I am wrong here and I shall use incoming webhook from GitLab - not sure

The next my step is to change the manual step in Gitlab-CI pipeline to automatic one. I already collect metrics from prometheus so the only thing I need is to add the correct checks to pipeline.

Also I wonder if Flux can correctly wait until deployment is finish - because I am sure that this is again out of scope the tooling (let's imagine that I deploy with the help of the Flux automated things like prometheus resources managed by prometheus operator and so on) and can we use upstream service like https://github.com/justinbarrick/fluxcloud or use Flagger

So it is very interesting topic anyway and I will look after the progress.

[nabadger]

I was going to suggest fluxcloud - but you have already.

@chancez we use fluxcloud in combination with https://github.com/mintel/fluxcloud-filebeat

We then ship events from filebeat to elasticsearch, and then pick them up in grafana (elasticsearch is a datasource). We display flux events as annotations - it gives us a way to monitor releases, but doesn't give any information on the rollout-status of the release (although combined with other kubernetes metrics, it's possible to get closer to that goal).

[2opremio]

@abbi-gaurav It's not clear to me what you would expect Flux to do. Could you please clarify in detail how you would expect Flux to behave?

You outlined a set of steps but didn't indicate what changes from the current behavior. It seems you want Flux to automatically wait for the tests to be done and automatically promote the changes to the production cluster, but I am not sure that's what you mean.

[abbi-gaurav]

Hi @2opremio, I would expect Flux to provide some kind of mechanism to notify via some webhooks the result of applying the changes. Then such webhooks can then be configured to trigger pipelines that can e.g run smoke tests.
Does it help?

[abbi-gaurav]

I can imagine this issue #2696 could help in solving the problem.

[2opremio]

Does it help?

Yes, that's more specific, thanks

I can imagine this issue #2696 could help in solving the problem.

Yes, I think so. Check if the ArgoCD hooks would some the problem for you, which is what the gitops engine will provide.

[ryedin]

@2opremio glad you brought up the ArgoCD hooks. Current dilemma I'm having (related at least tangentially to this one) is e.g. having flux wait for the istio HelmRelease to complete, before rolling out the app's HelmRelease (indeterminate right now, so sometimes don't get envoy sidecar injection). This specific case is really only an issue when bootstrapping a new cluster, but the general form of that problem must be fairly common. It looks like the Argo hooks (sync waves) concept would apply well, but AFAICT flux is a good deal easier to get setup than Argo (and other than this problem we are set up pretty well with flux now).

waiting with bated breath for the gitOps engine release

[2opremio]

You can already test it, using the image at #2886

[abbi-gaurav]

@2opremio , Is the issue still vague? If not please consider removing the lablel

[kingdonb]

I think this functionality is all present in Flux v2 now.

You can order dependencies with Kustomization dependencies, you can provide event triggers to third-party services with the Notification provider. We could use some documentation around the specific use case that you've described, since it is likely a very common one, but in terms of missing primitives that were not available in Flux v1, the rewrite has taken all of this into consideration and I think you'll like what you see, if you haven't upgraded yet! It is a breaking change as you may have heard and there is some work involved to upgrade. The migration guide is a great place to start.

Flux v1 is in maintenance mode now, and is not adding any new features unless they are critical.

As Flux contrib efforts have been focused on Flux v2, the Flux project has moved to a new repo, fluxcd/flux2

In the interest of reducing the number of open issues not directly related to supporting Flux v1 in maintenance mode, and respecting you may have moved on already, I will go ahead and close out this issue for now.

If you have a use case for Flux that isn't covered well in the new Flux v2 (which is a total rewrite), we want to hear about it.

If you've been following our development efforts then of course we hope you are able to upgrade, here's more info on how to find support with that: https://fluxcd.io/support/