Ability to preserve apps individuality in a monorepo after kustomize-build

[asafhm]

Checklist

• I've searched the issue queue to verify this is not a duplicate feature request.
• I've pasted the output of kargo version, if applicable.
• I've pasted logs, if applicable.

Proposed Feature

Given the following monorepo structure example:

base/
├─ external-dns/
│  ├─ kustomization.yaml
├─ keda/
│  ├─ kustomization.yaml
├─ prometheus-operator/
│  ├─ kustomization.yaml
envs/
├─ prod/
│  ├─ kustomization.yaml
├─ staging/
│  ├─ kustomization.yaml

After running kustomize-build, all manifests are outputed to a single file, or a file per manifest.
When using the Rendered Manifests pattern, it would be great to have a way to control the output of this action (or maybe with a novel action) so that the final directory structure of an env/prod branch would look like this:

external-dns/
   ...
keda/
   ...
prometheus-operator/
   ...

Motivation

We gather more and more apps with time, and managing each in a dedicated repository sounds like a management nightmare.
This is why a monorepo sounds like a good pick for this case.
It would also make sense to use an ApplicationSet that generate the apps relevant for a specific env by using a the git generator for all directories in the env/prodbranch. The problem is that kustomize-build doesn't let us control which directory we want to output some resources to.

Perhaps I miss something big time, but I can't seem to find an example out there of how to harness Kargo with a monorepo approach that handles this simple requirement.

If there's some workaround, or a better structuring suggestion, please share.

Suggested Implementation

[krancour]

Is there a reason you can't just set the path on the kustomize-build step to be a path within the repo instead of the root of the repo?

If it's just a matter of the possibility that the path doesn't exist, I think we could easily fix that by doing the programmatic equivalent of mkdir -p before rendering the manifests to that location.

[asafhm]

Is there a reason you can't just set the path on the kustomize-build step to be a path within the repo instead of the root of the repo?

When I apply changes to the env/ENV_NAME branch during the Kargo Stage run, I expect to have all apps listed there as directories, and not just a single app appearing there, so that's why I assumed I'd have to run the kustomize-build step at the root.
I'm leaning towards this approach because I worry of overloading Kargo with multiple pipelines/projects for each 3rd party app I use.
Perhaps I misunderstood your meaning though. Can you maybe share an example of the hierarchy you envision?

If it's just a matter of the possibility that the path doesn't exist, I think we could easily fix that by doing the programmatic equivalent of mkdir -p before rendering the manifests to that location.

Will this work when running the kustomize-build at the root?

At this point, after struggling so much with this, I'm starting to wonder if my approach to monorepos is wrong at the core...

[krancour]

I understand the end result you want, but my previous response was based on what I now think was a flawed understanding of your repo's layout. Let me look at this again with fresh 👀 in the morning.

[asafhm]

@krancour Much appreciated.
With that said, I'm open to restructuring my layout to any that fulfills this requirement.

[hiddeco]

If I understand correctly, you have a repository with one or more Kustomize overlays in a directory. You then use kustomize-build to render the final resources for those overlays, but find the current writing options too limited because they either bundle everything in a single file or a single directory instead of e.g. writing them to a logical place based on the Kustomize (sub-)overlay they originate from.

This is especially impractical because you want to render everything at once, but still have individual Applications manage a subset of the resources in Argo CD. For which having a separate file or directory is a requirement.

The problem for us to solve here is that we do not have anything to trace the origin of a Kubernetes resource from the Kustomize build output at the time it is done building the resources, as Kustomize truncates this information before returning the final resources.

One option I could see working is that we allow some annotation to be set on resources to define logical units, which we could then use to write things to sub-directories and/or files.

[asafhm]

@hiddeco My thoughts exactly. I hoped kustomize had some kind of psuedo-label/annotation that could help me output my resources to specific directories, but none such feature exists.
Your suggestion is indeed feasible if there were a new step whose purpose is to find these manifests and output them accordingly, but my honest fear is that I haven't seen anyone else complains about this hardship.
Usually when I find myself the only one struggling with something, I end up realizing I was just doing things wrong. That's why before I dare ask for this feature, I really want to know if my repo layout even makes sense and isn't inferior somehow.

[asafhm]

Consider the following alternative file hierarchy, where each addon gets the full base+variants+env-specific details in its own directory:

keda/
├─ base/
│  ├─ kustomization.yaml
├─ envs/
│  ├─ dev/
│  │  ├─ kustomization.yaml
│  ├─ staging/
│  │  ├─ kustomization.yaml
│  ├─ prod/
│  │  ├─ kustomization.yaml
external-dns/
├─ base/
│  ├─ kustomization.yaml
├─ envs/
│  ├─ dev/
│  │  ├─ kustomization.yaml
│  ├─ staging/
│  │  ├─ kustomization.yaml
│  ├─ prod/
│  │  ├─ kustomization.yaml
...

The following assumptions will have to be made:

1. Each addon will require its own Kargo Warehouse subscribed to changes in this git repo for the addon's specific subdirectory (e.g. for keda's Warehouse - set includePaths: [keda] in the git subscription)
2. Each addon-env combination will require a Kargo Stage, that will produce its output manifests in its own branch. E.g.: Manifests of keda in staging env will be generated in a branch like env/staging/addon/keda
3. An ArgoCD Application set to track each such branch mentioned in (2) will have to be created as well.
   I.e. an ApplicationSet with a git generator configured to generate an application for each directory in an env/staging branch for example, could not be harnessed.

I can see this working, but it requires a lot of extra Stages, Warehouses, branches and Applications, than I hoped to create for a monorepo, when most addons, if not all, don't require any special treatment that justifies dedicated Warehouse & Stages resources.
There must be a better way to use Kargo with a monorepo, am I right?