Skip to content

chore: incorporate gitops-engine lib preserving history#22341

Closed
leoluz wants to merge 341 commits intoargoproj:masterfrom
leoluz:import-gitops-engine
Closed

chore: incorporate gitops-engine lib preserving history#22341
leoluz wants to merge 341 commits intoargoproj:masterfrom
leoluz:import-gitops-engine

Conversation

@leoluz
Copy link
Collaborator

@leoluz leoluz commented Mar 13, 2025

fix #10774

This PR incorporates the gitops-engine library in the Argo CD main repository preserving the its history. It leverages the Go Workspace feature to make the gitops-engine library work seamlessly without major package refactoring changes.

First the gitops-engine repo needs to be prepared for the migration. The preparation consists of:

  1. moving all necessary files in that repo under a single folder (gitops-engine)
  2. Removing all unnecessary files
  3. Pushing the changes to a branch.

The following commands were executed for the steps 1 and 2 above:

mkdir gitops-engine
find . -type d \( -path ./gitops-engine -o -path ./.git -o -path ./.github -o -path ./.gitignore \) -prune -o -type f \( -path ./.gitignore -o -path ./.golangci.yaml -o -path ./codecov.yml -o -path ./sonar-project.properties -o -path . \) -prune -o -name '*' -maxdepth 1 -exec echo {} \;
git rm -r .github
git rm .gitignore
git rm .golangci.yaml
git rm codecov.yml
git rm sonar-project.properties

Then we push the changes in a separate branch: git push origin migrate

Steps executed in this repo after executing the procedure above:

git remote add ge-leo git@github.com:leoluz/gitops-engine.git
git fetch ge-leo
git merge ge-leo/migrate --allow-unrelated-histories
# edit the go.mod and remove all references from gitops-engine from it
go work init ./gitops-engine
go work use .
go work vendor
go mod tidy

At this stage we can build Argo CD locally using the local gitops-engine as part of the GO workspace:

make cli-local

Alexander Matyushentsev and others added 30 commits July 13, 2020 10:21
…j#78)

Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 0.0.5 to 0.0.7.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Changelog](https://github.com/spf13/cobra/blob/master/CHANGELOG.md)
- [Commits](spf13/cobra@0.0.5...0.0.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from v1 to v2.1.0.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@v1...1616116)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
)

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from v1 to v1.0.10.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Commits](codecov/codecov-action@v1...f357072)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/cache](https://github.com/actions/cache) from v1 to v2.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](actions/cache@v1...b820478)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
apimachinery provides library code to parse
multi document YAML, there is no need to resort
to regexes.
…goproj#77)

Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.4.0 to 1.6.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](stretchr/testify@v1.4.0...v1.6.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: added prehook for creating ns

* feat: added prehook for creating ns
Initial Draft

* feat: added prehook for creating ns

* feat: added prehook for creating ns

* feat: added prehook for creating ns checking the health of namespace created.

* feat: added prehook for creating ns. checking if ns existed already.

* feat: getSyncTasks returns the same list each time. added checking if resources contains ns already.

* feat: move const variable to right location; added additional checking if namespace is included in resources.

* feat: fixing compile issue.

* feat: moved code closer together.

* feat: adding test cases.

* feat: auto create only for sc.namespace

* feat: fix failed test

* feat: update livObj

* feat: added error handling

* feat: added error handling

* feat: move into its own fuction

* feat: fixing compile error
ctxCompleted is read and written from different
goroutines, which is a data race.

This change also makes the sleep interruptible.
* refactor: remove global variable

handlerKey does not need to be global,
field is just fine.
Atomic access is no longer needed because field
is protected by mutex.

* fix: use the correct mutex

* refactor: pre-allocate slice
* feat: added prehook for creating ns

* feat: added prehook for creating ns
Initial Draft

* feat: added prehook for creating ns

* feat: added prehook for creating ns

* feat: added prehook for creating ns checking the health of namespace created.

* feat: added prehook for creating ns. checking if ns existed already.

* feat: getSyncTasks returns the same list each time. added checking if resources contains ns already.

* feat: move const variable to right location; added additional checking if namespace is included in resources.

* feat: fixing compile issue.

* feat: moved code closer together.

* feat: adding test cases.

* feat: auto create only for sc.namespace

* feat: fix failed test

* feat: update livObj

* feat: added error handling

* feat: added error handling

* feat: move into its own fuction

* feat: fixing compile error

* fix: auto namespace creation

* fix: simplify sort method

* fix: remove sorting of namespace
…oproj#106)

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from v1.0.10 to v1.0.12.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Commits](codecov/codecov-action@v1.0.10...07127fd)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-go](https://github.com/actions/setup-go) from v2.1.0 to v2.1.1.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@v2.1.0...d0c5def)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1. Instead of global semaphore use a
per-cache semaphore. Removes thread-safety
issues, allows fine control over limiting
if multiple caches are used in a program

2. Use the semaphore to guard whole
sections that use expensive list
operations, not just the list API call.
This ensures that memory usage is capped,
not the list operations.

3. Allow to control list pager.
Reduce default prefetch limit to 1 page
from 10.

Co-authored-by: Alexander Matyushentsev <Alexander_Matyushentsev@intuit.com>
…rgoproj#109)

* fix: panic for data type conversion to *unstructured.Unstructured.

* fix: panic for data type conversion to *unstructured.Unstructured.
* feat: detect PVC StatefulSet ownership
* feat: use Kubernetes v1.18.6 libraries

* support switching between server and client side dry run mode

Co-authored-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
* chore: bump kubernetes deps

* refactor: remove dependency on pkg/errors

* refactor: remove indirect dependency version

* refactor: remove dependency on github.com/google/shlex
as it was only used in tests
* remove Flux mention on main README

* FAQ: explain backstory of Argo+Flux collaboration

* remove sentence about 'natural evolution'
* Fix data race on err variable

==================
WARNING: DATA RACE
Write at 0x00c000621c30 by goroutine 29:
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1()
      /gitops-engine/pkg/cache/cluster.go:548 +0x3a8
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).processApi()
      /gitops-engine/pkg/cache/cluster.go:479 +0x2cc
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).sync.func1()
      /gitops-engine/pkg/cache/cluster.go:536 +0x430
  github.com/argoproj/gitops-engine/pkg/utils/kube.RunAllAsync.func1()
      /gitops-engine/pkg/utils/kube/ctl.go:510 +0x45
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /Users/mikhail/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57 +0x85

Previous write at 0x00c000621c30 by goroutine 28:
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).sync.func1.1()
      /gitops-engine/pkg/cache/cluster.go:548 +0x3a8
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).processApi()
      /gitops-engine/pkg/cache/cluster.go:479 +0x2cc
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).sync.func1()
      /gitops-engine/pkg/cache/cluster.go:536 +0x430
  github.com/argoproj/gitops-engine/pkg/utils/kube.RunAllAsync.func1()
      /gitops-engine/pkg/utils/kube/ctl.go:510 +0x45
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /Users/mikhail/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57 +0x85

Goroutine 29 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /Users/mikhail/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:54 +0x73
  github.com/argoproj/gitops-engine/pkg/utils/kube.RunAllAsync()
      /gitops-engine/pkg/utils/kube/ctl.go:509 +0x1a1
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).sync()
      /gitops-engine/pkg/cache/cluster.go:526 +0x98b
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).EnsureSynced()
      /gitops-engine/pkg/cache/cluster.go:592 +0xe9
  github.com/argoproj/gitops-engine/pkg/cache.TestEnsureSynced()
      /gitops-engine/pkg/cache/cluster_test.go:140 +0xe2
  testing.tRunner()
      /usr/local/Cellar/go/1.14.3/libexec/src/testing/testing.go:991 +0x1eb

Goroutine 28 (finished) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /Users/mikhail/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:54 +0x73
  github.com/argoproj/gitops-engine/pkg/utils/kube.RunAllAsync()
      /gitops-engine/pkg/utils/kube/ctl.go:509 +0x1a1
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).sync()
      /gitops-engine/pkg/cache/cluster.go:526 +0x98b
  github.com/argoproj/gitops-engine/pkg/cache.(*clusterCache).EnsureSynced()
      /gitops-engine/pkg/cache/cluster.go:592 +0xe9
  github.com/argoproj/gitops-engine/pkg/cache.TestEnsureSynced()
      /gitops-engine/pkg/cache/cluster_test.go:140 +0xe2
  testing.tRunner()
      /usr/local/Cellar/go/1.14.3/libexec/src/testing/testing.go:991 +0x1eb
==================

* More type safety

Make runState a new type rather
than a type alias

* Fix data race on runState variable

==================
WARNING: DATA RACE

Write at 0x00c0000d0b40 by goroutine 83:
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).runTasks.func5.1()
      /gitops-engine/pkg/sync/sync_context.go:786 +0x68c

Previous write at 0x00c0000d0b40 by goroutine 84:
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).runTasks.func5.1()
      /gitops-engine/pkg/sync/sync_context.go:786 +0x68c

Goroutine 83 (running) created at:
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).runTasks.func5()
      /gitops-engine/pkg/sync/sync_context.go:778 +0x165
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).runTasks()
      /gitops-engine/pkg/sync/sync_context.go:807 +0xb32
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).Sync()
      /gitops-engine/pkg/sync/sync_context.go:265 +0x1b3d
  github.com/argoproj/gitops-engine/pkg/sync.TestSyncFailureHookWithFailedSync()
      /gitops-engine/pkg/sync/sync_context_test.go:532 +0x4e5
  testing.tRunner()
      /usr/local/Cellar/go/1.14.3/libexec/src/testing/testing.go:991 +0x1eb

Goroutine 84 (running) created at:
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).runTasks.func5()
      /gitops-engine/pkg/sync/sync_context.go:778 +0x165
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).runTasks()
      /gitops-engine/pkg/sync/sync_context.go:807 +0xb32
  github.com/argoproj/gitops-engine/pkg/sync.(*syncContext).Sync()
      /gitops-engine/pkg/sync/sync_context.go:265 +0x1b3d
  github.com/argoproj/gitops-engine/pkg/sync.TestSyncFailureHookWithFailedSync()
      /gitops-engine/pkg/sync/sync_context_test.go:532 +0x4e5
  testing.tRunner()
      /usr/local/Cellar/go/1.14.3/libexec/src/testing/testing.go:991 +0x1eb
==================

* Simplify
Bumps [github.com/evanphx/json-patch](https://github.com/evanphx/json-patch) from 4.2.0+incompatible to 4.9.0+incompatible.
- [Release notes](https://github.com/evanphx/json-patch/releases)
- [Commits](evanphx/json-patch@v4.2.0...v4.9.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…odical K8S API state resynchronization (argoproj#133)

* fix: leverage RetryWatcher to watch cluster events and introduce periodical K8S API state resynchronization

* Apply reviewer notes

* enable race detection in tests
dependabot bot and others added 22 commits February 7, 2025 11:06
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.26.0 to 0.33.0.
- [Commits](golang/net@v0.26.0...v0.33.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rgoproj#662)

* fix: New kube applier for server side diff dry run with refactoring

Part of a fix for: argoproj#21488

Separate logic to handle server side diff dry run applies.

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Break backwards compatibility for a better code

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Don't put applier constructor in the interface

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Address comments

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Address more comments

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* chore: enable testifylint linter (argoproj#657)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* chore: enable gofumpt, gosimple and whitespace linters (argoproj#666)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* chore: enable use-any from revive (argoproj#667)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* chore: bump golangci-lint to v1.63.4 and list argo-cd linters (argoproj#670)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* chore: enable unused-parameter and var-declaration from revive (argoproj#668)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* chore: remove actions/cache duplicated behavior with actions/setup-go (argoproj#658)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Add Leo's code

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Make linter happy

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

---------

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Co-authored-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
…om revive (argoproj#674)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
…ive (argoproj#679)

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
…ed field set (argoproj#646)



Signed-off-by: Dejan Zele Pejchev <pejcev.dejan@gmail.com>
Signed-off-by: Alexandre Gaudreault <alexandre_gaudreault@intuit.com>
Co-authored-by: Alexandre Gaudreault <alexandre_gaudreault@intuit.com>
…rgoproj#688)

* fix: Server-side diff shows incorrect diffs for list related changes

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>

* Update docs for removeWebHookMutation

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>

* Update docs

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>

* Update docs

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>

---------

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>
Signed-off-by: sivchari <shibuuuu5@gmail.com>
Signed-off-by: Michael Crenshaw <350466+crenshaw-dev@users.noreply.github.com>
* chore(deps): Update some package versions

Helps with argoproj#22104

Update some versions trying to avoid legacy dependencies. Bump go to 1.23.5.

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

* Fix versions

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>

---------

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>
…rgoproj#696)

Signed-off-by: Andrii Korotkov <andrii.korotkov@verkada.com>
* fix: properly check dryRun flag for server side apply

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>

* Remove debug logging

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>

---------

Signed-off-by: Peter Jiang <peterjiang823@gmail.com>
Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>
Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>
@leoluz leoluz requested a review from a team as a code owner March 13, 2025 19:08
@bunnyshell
Copy link

bunnyshell bot commented Mar 13, 2025

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

  • 🚀 /bns:deploy to deploy the environment

Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>
@leoluz leoluz changed the title import gitops engine chore: import gitops engine Mar 14, 2025
@leoluz
Copy link
Collaborator Author

leoluz commented Mar 14, 2025

There are a few things that needs to be fixed:

Basically with Go workspaces we can't run go mod vendor anymore. We must run go work vendor instead. I am not too sure if we still need that vendor folder in the project.

The following Github check are failing with the error below:

  • Integration tests / Build & cache Go code (pull_request)
  • Integration tests / Check changes to generated code (pull_request)
  • Integration tests / Ensure Go modules synchronicity (pull_request)
Run go mod download
2  go mod download
3  shell: /usr/bin/bash -e {0}
4  env:
5    GOLANG_VERSION: 1.24.1
6go: k8s.io/externaljwt@v0.0.0: invalid version: unknown revision v0.0.0  

The docker image build job is also failing. Fix seems to be simple as the gitops-engine folder needs to be added to the Dockerfile:

Image/ build-only / publish (pull_request)

@leoluz leoluz changed the title chore: import gitops engine chore: incorporate gitops-engine lib preserving history Mar 14, 2025
Signed-off-by: Leonardo Luz Almeida <leonardo_almeida@intuit.com>
@leoluz leoluz closed this Sep 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incorporate gitops-engine back in Argo CD repo