Skip to content

Justfile: make running the graph-builder and e2e tests simpler #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 2 commits into from
Mar 25, 2020
Merged

Justfile: make running the graph-builder and e2e tests simpler #258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d8a0fb0
Justfile: use toml config for graph-builder
steveej Mar 10, 2020
cf34b32
Justfile: refactor the get-and-display-graph recipe
steveej Mar 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 42 additions & 7 deletions Justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,18 +45,53 @@ run-ci-tests:

path_prefix := "api/upgrades_info/"

# Reads a graph on stdin, creates an SVG out of it and opens it with SVG-associated default viewer. Meant to be combined with one of the `get-graph-*` recipes.
display-graph:
#!/usr/bin/env bash
required_tools=("xdg-open" "dot" "jq" "adfasdf")
for tool in "${required_tools[@]}"; do
type ${tool} >/dev/null 2>&1 || {
printf "ERROR: program '%s' not found, please install it.\n" "${tool}"
exit 1
}
done

get-and-display-graph:
just get-graph-gb | jq -cM | {{invocation_directory()}}/hack/graph.sh | dot -Tsvg > graph.svg; xdg-open graph.svg
jq -cM . | {{invocation_directory()}}/hack/graph.sh | dot -Tsvg > graph.svg; xdg-open graph.svg
Copy link
Member

@LalatenduMohanty LalatenduMohanty Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to add the graph.svg to .gitignore.

Copy link
Member

@LalatenduMohanty LalatenduMohanty Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason this is not working in my tmux session. But thats not a blocker for the PR.

Copy link
Contributor Author

@steveej steveej Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please paste the command and its output here. It's supposed to work for you as well.

Copy link
Member

@LalatenduMohanty LalatenduMohanty Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I am using $ just display-graph in a wrong way. Can you add an example in the dev docs?

Copy link
Member

@LalatenduMohanty LalatenduMohanty Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running $ just get-and-display-graph works fine. But as expected $ just display-graph would hang as it need json input. If this is right, we should make $ just display-graph a private recipe.

Copy link
Member

@LalatenduMohanty LalatenduMohanty Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@steveej mentioned that the right usage is just get-graph-pe-production "stable-4.2" "amd64" | just display-graph. As discussed we should add docs for display-graph in the Justfile for just --list to show. I will also suggest that we add just --list to the dev docs.

Copy link
Contributor Author

@steveej steveej Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not yet ready to document all of this. I would like to, but I'm not convinced this is a stable interface. Right now we're observing a divergence, or at best a duplication, between the Justfile and the scripts in dist/ and hack/. I would like to unify this as much as possible soon along with updating the documentation.

Copy link
Member

@LalatenduMohanty LalatenduMohanty Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the divergence exists because we have not accepted the Justfile yet as the default way for local development. By making it easier to use the Justfile will be a positive step in that direction. I would suggest lets add the doc required for this PR in $ just --list and then take up the other things required to unify things.

Copy link
Contributor Author

@steveej steveej Mar 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. So you're suggesting to simply add docs to the recipes in this PR?


run-graph-builder registry="https://quay.io" repository="openshift-release-dev/ocp-release" credentials_file="${HOME}/.docker/config.json":
#!/usr/bin/env bash
export RUST_BACKTRACE=1
export RUST_LOG="graph_builder=trace,graph-builder=trace,cincinnati=trace"
# export RUST_LOG="${RUST_LOG},dkregistry=trace"
# strace -f -D -o gb.strace.lol
cargo run --package graph-builder -- --service.pause_secs 30 --address 0.0.0.0 --registry {{registry}} --repository {{repository}} -vvv --service.path_prefix {{path_prefix}} --credentials-file {{credentials_file}} # --disable-quay-api-metadata

cargo run --package graph-builder -- -c <(cat <<'EOF'
verbosity = "vvv"

[service]
pause_secs = 9999999
address = "127.0.0.1"
port = 8080
path_prefix = "{{path_prefix}}"

[status]
address = "127.0.0.1"
port = 9080

[[plugin_settings]]
name="release-scrape-dockerv2"
registry = "{{registry}}"
repository = "{{repository}}"
fetch_concurrency=128
credentials_file = "{{credentials_file}}"

[[plugin_settings]]
name="quay-metadata"
repository="{{repository}}"

[[plugin_settings]]
name="node-remove"

[[plugin_settings]]
name="edge-add-remove"
EOF
)

run-graph-builder-satellite:
just run-graph-builder 'sat-r220-02.lab.eng.rdu2.redhat.com' 'default_organization-custom-ocp'
Expand Down
8 changes: 8 additions & 0 deletions docs/developer/developing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,21 @@ just run-graph-builder
just run-policy-engine
```

## Requesting the graph from the local instance

`$ just get-graph-pe` can be run to get the graph from a local policy-engine instance.
For example, to get the graph for the "stable-4.2" channel and the "amd64" architecture, run:

```shell
just get-graph-pe "stable-4.2" "amd64"
```

If you would like to visualize the result you can pipe the result through the _display-graph_ recipe as follows:

```shell
just get-graph-pe "stable-4.2" "amd64" | just display-graph
```

To get the graph from a local graph-builder instance, run:

```shell
Expand Down