Output build dependencies of a specific image repository

[0xmichalis]

Supported output formats: json, dot, and ast.

$ openshift ex build-chain ruby-20-centos7
{
    "fullname": "default/ruby-20-centos7",
    "tags": [
        ""
    ],
    "edges": [
        {
            "fullname": "default/ruby-sample-build",
            "to": "default/origin-ruby-sample"
        }
    ],
    "children": [
        {
            "fullname": "default/origin-ruby-sample",
            "tags": [
                ""
            ]
        }
    ]
}

$ openshift ex build-chain ruby-20-centos7 -o dot
digraph ruby_20_centos7 {
    ruby_20_centos7->origin_ruby_sample[ label=<ruby_sample_build<BR /><FONT POINT-SIZE="10">default</FONT>> ];
    origin_ruby_sample [ comment="", label=<origin_ruby_sample<BR /><FONT POINT-SIZE="10">default</FONT>> ];
    ruby_20_centos7 [ comment="", label=<ruby_20_centos7<BR /><FONT POINT-SIZE="10">default</FONT>> ];

}

$ openshift ex build-chain ruby-20-centos7 -o ast
ruby-20-centos7(origin-ruby-sample)

DOT output visualized
http://i.imgur.com/LVse6hX.png

[0xmichalis]

Also how should I name this utility?

[bparees]

it should be a command under the experimental command set... "build-tree" or "build-chain" would be my first name ideas.

btw i don't know that we need an issue and a trello card for this, we can discuss it in the card.

as for the specific format, let's get Jay to weigh in. i'll send a note.

[0xmichalis]

btw i don't know that we need an issue and a trello card for this, we can discuss it in the card.

WIth some luck I'll turn this into a pull request.

[smarterclayton]

We should have three formats:

internally, in memory, a structure that represents the data
externally, a versioned JSON format that can be dumped
externally, a graphviz transformation of the in memory format.

the first two should be very closely coupled - they don't have to be formal API objects, but it would be better to structure them that way so that in the future we can version them. I would think of the graphviz as a special "-o" argument that visualizes them (-o viz)

----- Original Message -----

cc: @csrwng @smarterclayton @bparees

Opening this issue to sync with
this
Trello card.

@smarterclayton I was thinking of going straight to the dot stuff. There is a
library that can make things very easy:
https://godoc.org/code.google.com/p/gographviz
What do you think?

---

Reply to this email directly or view it on GitHub:
#1229

[0xmichalis]

@bparees @csrwng ptal

I have created the basic structure of the code (data structures, flow, etc). I haven't thoroughly tested it though except starting two builds from the same configuration file just to see if it can find the pushed images (and it does).

[vagrant@openshiftdev sample-app]$ openshift cli get images
NAME                DOCKER REF
[vagrant@openshiftdev sample-app]$ openshift cli get buildConfigs
NAME                TYPE                SOURCE
ruby-sample-build   STI                 git://github.com/openshift/ruby-hello-world.git
[vagrant@openshiftdev sample-app]$ openshift cli start-build ruby-sample-build
ruby-sample-build-422f94ba-c407-11e4-9426-080027c5bfa9
[vagrant@openshiftdev sample-app]$ openshift cli start-build ruby-sample-build
ruby-sample-build-5a35194d-c408-11e4-9426-080027c5bfa9
[vagrant@openshiftdev sample-app]$ openshift cli get images
NAME                                                               DOCKER REF
26393db67314c93f2a42ef2f2cc6266b872a55443fc57f6a126d48d31a6b74b3   172.30.17.83:5000/default/origin-ruby-sample:26393db67314c93f2a42ef2f2cc6266b872a55443fc57f6a126d48d31a6b74b3
ff5ea1a3e3c7dd7ea8869cdee53be964a2460357b313bc724aecf3aa6440e432   172.30.17.83:5000/default/origin-ruby-sample:ff5ea1a3e3c7dd7ea8869cdee53be964a2460357b313bc724aecf3aa6440e432
[vagrant@openshiftdev sample-app]$ openshift ex build-chain -i origin-ruby-sample
Image repo origin-ruby-sample has 2 image(s)

For testing all of the code, I have to create multiple Dockerfiles that depend on each other, right? :soundsboring: :)

[bparees]

@Kargakis Well this is an interesting question... do we want to illustrate

1. any build that would be triggered by an upstream image change, even if that build doesn't explicitly consume the upstream image
   or
2. only the cases where the build actually consumes the upstream image?

I think the (1) is sufficient, and given that, you don't need do anything w/ actual dockerfiles..you can just create a set of BuildConfigs that have some ImageChangeTrigger relationship. It doesn't really matter if the actual BuildConfig consumes the new/changed image or not, just that the upstream image changing causes that BuildConfig to run (which in turn produces some new image that may or may not actually be based on the upstream image)

[0xmichalis]

1. any build that would be triggered by an upstream image change, even if that build doesn't explicitly consume the upstream image

I don't understand this. How can a build be triggered by an upstream image if that image is not a part of the build chain (ie. one of the images that can be found in FROM of one of the Dockerfiles higher in the tree) ?

[bparees]

@Kargakis nothing prevents me from setting up an ImageChangeTrigger on a build in which the ICT watches image "foo" for changes but the BuildConfig itself does not consume "foo" as part of the build process.

[bparees]

@Kargakis similarly you could setup a Github webhook that triggered your build on pushes to repo Foo even though your build was building repo Bar

[0xmichalis]

@bparees Ok, so to make this clear I am going to check for triggers in configuration files as well as images that explicitly depend on other images (which I already do), right?

[bparees]

@Kargakis no, just triggers. @smarterclayton @csrwng please confirm.

[csrwng]

@bparees @Kargakis - that's my understanding. Only triggers.

[0xmichalis]

Until deciding how are we going to look for dependencies (see Jay's comment on Trello), I've started writting tests for parts of the code that are independent [of the way we inspect for dependencies] . Most important being the building of the tree structure (buildTree).

[0xmichalis]

When running go test in the buildchain directory I get the following error message:

$ cd pkg/cmd/experimental/buildchain/
$ go test
# github.com/GoogleCloudPlatform/kubernetes/pkg/api/validation
../../../../../../GoogleCloudPlatform/kubernetes/pkg/api/validation/schema.go:129: invalid operation: fieldDetails.DataTypeFields.Items[0] (type *swagger.Item does not support indexing)
FAIL    github.com/kargakis/origin/pkg/cmd/experimental/buildchain [build failed]

Is anybody familair with it? Tests should pass. Here is a slightly edited version of the algorithm (and tests) using strings instead of api.Images: http://play.golang.org/p/qEo_nGt5Ko

[smarterclayton]

I cleared this up with jay - the approach Ben relayed is the one we are doing for now.

On Mar 9, 2015, at 8:12 AM, Michail Kargakis [email protected] wrote:

Until deciding how are we going to look for dependencies (see Jay's comment on Trello), I've started writting tests for parts of the code that are independent [of the way we inspect for dependencies] . Most important being the building of the tree structure (buildTree).

—
Reply to this email directly or view it on GitHub.

[0xmichalis]

Since we probably haven't run any buildConfigs, these image repositories do not exist, so anything that goes here will be baptized as invalid. So I think we should check for image repositories in buildConfigs right?

(cc: @bparees @csrwng )

[csrwng]

The image repositories will exist. Only the images/tags won't.

[0xmichalis]

Are we addressing circular dependencies in buildConfigs atm?

[csrwng]

Shouldn't you just take the first argument as the image repo name? (line 45 implies that you do)

[smarterclayton]

There should be a mode for this command that is --all that iterates over all image repositories in the system and creates a graph with multiple roots.

----- Original Message -----

+Output build dependencies of a specific image repository.
+
+Examples:
+

• $ openshift ex build-chain [image-repository]
  +`

+func NewCmdBuildChain(f *clientcmd.Factory, parentName, name string)
*cobra.Command {

• cmd := &cobra.Command{

•   Use:   fmt.Sprintf("%s [image-repository]", name),
  

•   Short: "Output build dependencies of a specific image repository",
  

•   Long:  longDescription,
  

•   Run: func(cmd *cobra.Command, args []string) {
  

•       imageRepoName := cmdutil.GetFlagString(cmd, "imagerepo")
  

•       if imageRepoName == "" {
  

•           usageError(cmd, "Must pass an image repository name with the
  

  --imagerepo (-i) flag")

Shouldn't you just take the first argument as the image repo name? (line 45
implies that you do)

---

Reply to this email directly or view it on GitHub:
https://github.com/openshift/origin/pull/1229/files#r26130209

[0xmichalis]

Good catch, I'll just use the first argument.

[bparees]

@Kargakis no, we are not.

[0xmichalis]

@Kargakis no, we are not.

@bparees fyi I just created a Trello card for this

[0xmichalis]

JSON is in, next thing tbd DOT

[0xmichalis]

@bparees @csrwng @smarterclayton ptal

[bparees]

I believe it's valid for Output.To to specify a namespace (Output.To.Namespace) so this should take that into account. (if not specified it defaults to this namespace)

[0xmichalis]

@bparees the same goes for the DockerImageReference, right?

[bparees]

no, DockerImageReference is a docker pull spec for a registry, it has no association to an openshift namespace.

[bparees]

I think we need to specify a tag when running this, because a particular ICT isn't triggered just by an ImageRepo changing, the specific Tag that the ICT watches needs to change.

[0xmichalis]

I think we need to specify a tag when running this, because a particular ICT isn't triggered just by an ImageRepo changing, the specific Tag that the ICT watches needs to change.

The idea is that the user specifies an image repository and gets back a dependency tree. Now, the tags you are talking about can be found in the tree, ie. each node has alongside its name and namespace, the tags by which it depends on another node.

[bparees]

again needs to include namespace in de-duplication checking.

[0xmichalis]

I fixed the labeling plus since we want to distinguish between namespaces I have added them in the name of a node (now called fullName). I don't output namespaces in the graph though because we cannot distinguish the name from the namespace. [1]
http://i.imgur.com/IAxE89y.png

[1] The DOT parser eventually doesn't accept any special character except underscores. So imagine a node id like : the_fox_jumps_the_fence. Where does the namespace ends and the name starts?

ps. the label of an edge is really the name of a buildConfig
ps2. An idea to output namespaces in the graph is by using colors

[smarterclayton]

Namespace will be important to most users

On Mar 17, 2015, at 8:26 AM, Michail Kargakis [email protected] wrote:

An idea is to drop namespaces from DOT output altogether and use colors instead to distinguish between namespaces

—
Reply to this email directly or view it on GitHub.

[0xmichalis]

@smarterclayton sure, take a look below where I output namespaces in smaller font
http://i.imgur.com/LVse6hX.png

[smarterclayton]

Looks good

----- Original Message -----

@smarterclayton sure, take a look below where I output namespaces in smaller
font
http://i.imgur.com/LVse6hX.png

---

Reply to this email directly or view it on GitHub:
#1229 (comment)

[0xmichalis]

Is defaultNamespace the right namespace to assign here or should I check first if cfg.Namespace is defined, if so, assign that, if not, assign defaultNamespace ? @bparees

[0xmichalis]

Nvm, got my answer on IRC

[0xmichalis]

Doc: openshift/openshift-docs#225

[0xmichalis]

Is ref.Name the name of the image or the image repository? @bparees

[0xmichalis]

Nvm v.2, it's the image repo name

[0xmichalis]

I think this is ready for one more (hopefully the last) review:)

cc: @bparees @csrwng @smarterclayton

[bparees]

it isn't really invalid, it's just an image repo that doesn't trigger any builds, right?

[0xmichalis]

Well it may be that or it may be invalid

[bparees]

lgtm and it's experimental so we can iterate a little if we need to. great work @Kargakis

let me know if you want it merged, wasn't sure if you had any final tweaks or that's just doc updates.

[0xmichalis]

let me know if you want it merged, wasn't sure if you had any final tweaks or that's just doc updates.

@bparees I want to add tags on node names for the DOT output and was thinking about adding an extra flag where we would output all the trees generated by a specific repository which we currently don't support and it's probably something that the RCM guys will probably want.

eg.

$ openshift ex ruby-20-centos7 --complete

generates all trees from all tags in this repo.

Thinking about it, we support outputing all trees but for all image repos and not for one specific.
What do you think? Also any suggestions about the name of the new flag?
cc: @smarterclayton @csrwng

[bparees]

@karkgakis sounds reasonable. Maybe "--all-tags"

[0xmichalis]

@bparees have a look in the latest commits and I'll squash if everything is fine. I won't add tags next to names for the time being because it gets complicated with children with multiple tags.

[0xmichalis]

I have injected tags into root names (only) because I feel it's better to show root tags than not
http://i.imgur.com/2GxyL8d.png

[bparees]

@Kargakis still lgtm.

[0xmichalis]

@bparees cool, you can merge it now

[bparees]

[merge]

[openshift-bot]

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_openshift3/1259/) (Image: devenv-fedora_1107)

[openshift-bot]

Evaluated for origin up to a2212c3

[coveralls]

Changes Unknown when pulling a2212c3 on kargakis:output-dep into * on openshift:master*.

[coveralls]

Changes Unknown when pulling a2212c3 on kargakis:output-dep into * on openshift:master*.