Check for circular dependencies in buildConfigs

[0xmichalis]

No description provided.

[0xmichalis]

@bparees @mfojtik fyi

[0xmichalis]

Eventually I think I will have to move all this code into the buildConfig registry as there I will have total control over the updates and deletions of buildConfigs.

[smarterclayton]

circular dependencies aren't necessarily bad. you might be doing a test / validation flow and reusing input from the old image. I think the way we would implement openshift build on openshift is via a circular build.

[bparees]

what we're trying to avoid here is buildA outputting imageA which triggers buildB which outputs imageB which triggers buildA.

that's definitely bad, the build loop would never end.

[smarterclayton]

I'm saying there are use cases for that. The second build might block waiting for an upstream dependency and then manually trigger the base image. We also will eventually have squash images.

The tool should flag them but they're not (yet) verboten.

On Mar 26, 2015, at 1:05 PM, Ben Parees [email protected] wrote:

what we're trying to avoid here is buildA outputting imageA which triggers buildB which outputs imageB which triggers buildA.

that's definitely bad, the build loop would never end.

—
Reply to this email directly or view it on GitHub.

[bparees]

if you were doing the openshift on openshift scenario you describe, i'd expect you to be differentiating by tags. A config like:
ImageA:testme triggers BuildA(runs tests) outputs ImageA:tested would be fine, that's not a circular chain.

The idea that you might have an intentional chain that's "broken" by having a manual step somewhere in there is interesting, i guess i can see the argument for that though i still think differentiating on tags seems like the more likely way to handle flowing images through test/validation steps.

[smarterclayton]

Probably. Andy has been working on the proposals here so once we know more there we can come back to circularity.

On Mar 26, 2015, at 1:14 PM, Ben Parees [email protected] wrote:

if you were doing the openshift on openshift scenario you describe, i'd expect you to be differentiating by tags. A config like:
ImageA:testme triggers BuildA(runs tests) outputs ImageA:tested would be fine, that's not a circular chain.

The idea that you might have an intentional chain that's "broken" by having a manual step somewhere in there is interesting, i guess i can see the argument for that though i still think differentiating on tags seems like the more likely way to handle flowing images through test/validation steps.

—
Reply to this email directly or view it on GitHub.

[0xmichalis]

I have added simple warnings instead of erroring for the time being, but I share the same thoughts as @bparees, we shouldn't allow circular deps by default. If there are any use-cases, we should probably add an option of enabling/disabling them (still the default should be not supporting them).

[smarterclayton]

There's no way to prevent circular dependencies in OpenShift without a lot of overhead. So it's likely there will always be circular dependencies and we should be able to handle them gracefully and warn users.

On Mar 27, 2015, at 7:44 AM, Michail Kargakis [email protected] wrote:

I have added simple warnings instead of erroring for the time being, but I share the same thoughts as @bparees, we shouldn't allow circular deps by default. If there are any use-cases, we should probably add an option of enabling/disabling them (still the default should be not supporting them).

—
Reply to this email directly or view it on GitHub.

[0xmichalis]

I have moved the code under the buildConfig directory where it makes more sense for it to be.

[0xmichalis]

Added tests and they do pass:)
@bparees @smarterclayton PTAL

[bparees]

needs to include the config namespace.

[bparees]

rather than maintaining an in memory datastructure (which is problematic since you're not rebuilding it at startup), i'd rather see this just use the same logic as the CLI build-chain tool. Can't that share some logic here? Just produce the build tree for the ICT From imagerepo, checking for cycles as you go.

you'll want that check in the CLI tool logic anyway since w/o it, if you invoke the CLI tool on a tree w/ cycles it'll never terminate, so it would be good to abort with an error message when it hits cycle.

[smarterclayton]

Um, we should not be adding this. This is way too expensive to calculate.

[smarterclayton]

And by that I mean we have to run an O(N) load on the cluster for every create, which is extremely expensive. Ben and I talked through a number of options, but none of them are really efficient or easy to implement, and they don't solve the problem for end users.

At a minimum, osc status will show some of this for end users and that will reduce the possibility this happens.

[0xmichalis]

Running O(N) loads in maps of maps is expensive, agreed, but I suspect this can be refactored to use a cheaper structure where it won't really matter.

[0xmichalis]

Just produce the build tree for the ICT From imagerepo, checking for cycles as you go

@bparees so what you are suggesting is rebuilding the tree every time? Because in the build-chain tool we just build the tree and are done. Here we need to maintain one (add nodes, update nodes, delete nodes) which is different and more challenging.

[bparees]

@Kargakis you can't really maintain a universal structure anyway since visibility rules could change, for example (A->B->C->A, previously I didn't have permission to view C so it wasn't a cycle, suddenly I get permissions added, which creates a cycle). So you'd have to rebuild your structure whenever visibility rules changed, at a minimum.

In general there seemed to be a lot of challenges here, and @smarterclayton convinced me the right thing to w/ respect to cycles is:

1. enable users to see if they have cycles (via status information)
2. control cycles by our general resource consumption limits (ie if you define a cycle, you will get lots of builds, but their impact on the system will be limited by your quota)

[0xmichalis]

So you'd have to rebuild your structure whenever visibility rules changed, at a minimum.

FWIW that's not true. The whole structure nevers needs to be rebuilt because every time we add or remove a relationship, we traverse all related nodes.

[bparees]

@Kargakis the visibility rules changing is not an add/removal of a relationship.

[smarterclayton]

We should not do this as part of API calls (it violates our O(1) API design goals for complexity of API ops). Let's defer this until post 3.0.

----- Original Message -----

Just produce the build tree for the ICT From imagerepo, checking for cycles
as you go

@bparees so what you are suggesting is rebuilding the tree every time?
Because in the build-chain tool we just build the tree and are done. Here we
need to maintain one (add nodes, update nodes, delete nodes) which is
different and more challenging.

---

Reply to this email directly or view it on GitHub:
#1468 (comment)

[0xmichalis]

We should not do this as part of API calls (it violates our O(1) API design goals for complexity of API ops).

Agreed. Closing this...