Add support to track Git repo branches by polling / using webhooks to act on branch updates

[metral]

In order to use a branch as a desired state for a Git repo, we'll want to add the ability to poll branches to receive updates through webhooks.

This approach is used to not have stale branches after an update, and for the operator to continuously drive towards the commit state of the tracked branch.

[nesl247]

Do you have a timeframe in which this may be done? Out of curiosity, without this, is there a purpose to the operator at all? Asking as we're starting to look at this and want to understand more.

[metral]

Do you have a timeframe in which this may be done?

We're aiming to get to this in Q3 :-)

Out of curiosity, without this, is there a purpose to the operator at all?

We believe so -- it's still possible to place the operator in your delivery pipeline with a Stack CustomResource that deploys the current git commit SHA being evaluated. The branch polling is more oriented for Stacks that aren't as actively updated / deployed in a pipeline.

Asking as we're starting to look at this and want to understand more.

Let us know if you have any further questions!

[metral]

We did not get to this in Q3 and are reevaluating when to pick it back up.

We'd love to get input from the community on expected use cases and assumptions to help mold the design, so feel free to let us know any thoughts you may have.

[UnstoppableMango]

I use pulumi to deploy to my homelab, so my use case may not fit the majority of customers. But essentially, if I merge a change to the master branch of my app repo, I would like the operator to deploy that change. To me, it sounds like this feature would handle that use case.

I fully admit I may not be using the operator correctly right now, but as I understand it in the current state I would:
Merge my change to the master branch of my app repo
Grab the commit hash from the master commit containing my change
Update my pulumi stack containing the stack resource to deploy my app
Deploy the "top-level" pulumi stack

From my admittedly naive perspective, the update to the "top-level" stack seems redundant. I certainly understand the use case for using the hash, but this feature would be great for less critical environments where automation is the priority.

[metral]

Thank you for the input @UnstoppableMango.

Your understanding of the flow is correct.

As you point out, deploying it automatically by watching the branch is a better experience.

Would you expect the branch to always roll out the latest change? Or are there instances where you'd be more selective about which changes go out?

[UnstoppableMango]

I can't think of a case where I would. My personal workflow is very iterative, if I break something I'll merge the fix into the branch I want deployed.

[tedchang77]

I think there are use cases where you want to deploy based on a git tag instead of the latest changes in a branch. For example, we are deploying to different stacks/envs based on a specific tag. Associating a stack with a git tag regular expression would allow us to use the operator with our current deployment workflow.

[tedchang77]

I think getting rid of the "top-level" stack would be ideal as it removes the need to mange another moving piece. However, if we want to support more complex rules such as the deploying based on git tags, where would we specifiy the rules? Could this be part of the metadata of the stack that we want to deploy?

[lth82]

Our use-case is that we need a pull based deployment model for stacks inside multiple k8 clusters instead of having a pipeline that uses a fan-out push model. Tools like Argo or Flux does this but then we are back to YAML again...

[eljoth]

I also think branch polling like flux already does is an imprtant feature to keep the operator comprehensive in the row of gitops tools. Especially for dev-environments it is a convenient feature so devs do not have any boilerplate to deploy the code. Simply push it and see the magic done by the pulumi operator.

[stevenklar]

Poor workaround for now could be to use git-sync and pulumi watch.
Not sure how reliable pulumi watch is right now. But this would at least allow a pull model deployment.

Would be nice to have this git source polling pretty soon.

[lukehoban]

As noted in #158, we could likely build upon some of the core components of Flux or Argo as part of adding this support, instead of implementing it from scratch for this operator.