Skip to content

Conversation

@petr-muller
Copy link
Member

It's better to not prune stale jobs than remove the existing ones. We could also consider removing --from-file entirely, WDYT?

Also clean up some imports.

/cc @bbguimaraes @stevekuznetsov

@openshift-ci-robot openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jul 3, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 3, 2019
@bbguimaraes
Copy link
Contributor

--from-file has advantages that would be nice to maintain: speed and scope.

What about replacing it with a flag that operates on a sub-tree? Those benefits would remain and there should be no problem in pruning just that sub-tree.

@petr-muller
Copy link
Member Author

petr-muller commented Jul 3, 2019

--from-file has advantages that would be nice to maintain: speed and scope.

What about replacing it with a flag that operates on a sub-tree? Those benefits would remain and there should be no problem in pruning just that sub-tree.

PRs to openshift/release are subject to full --from-dir=ci-operator/config --to-dir=ci-operator/jobs check anyway, so any "locally" operating flag will need to keep that consistency. Which is hard, because in local operation you do not see the other jobs which may need modifying or pruning. Personally I don't see much use for the locally operating flag. In the end --from-dir=ci-operator/config --to-dir=ci-operator/jobs is the safe and convenient option.

@bbguimaraes
Copy link
Contributor

I hacked together a POC that operates only in ci-operator/{config,jobs}/openshift/ci-tools. Compare:

Details
$ git write-tree
de367e4c98d55056113408f9f25505d7a42662b1
$ time ci-operator-prowgen --from-dir ci-operator/config/ --to-dir ci-operator/jobs/

real    0m0.043s
user    0m0.039s
sys     0m0.019s
$ git add -u && git write-tree
0038afab92983fb6fb860b276465deaa4ca5c6dc

vs.

Details
$ git write-tree
de367e4c98d55056113408f9f25505d7a42662b1
$ time ci-operator-prowgen --from-dir ci-operator/config/ --to-dir ci-operator/jobs/
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-ansible-service-broker-master-operator-molecule-e2e
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-ansible-service-broker-master-operator-molecule-olm-e2e
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cloud-credential-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-api-actuator-pkg-master-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-api-provider-azure-master-bazel-verify
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-config-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-dns-operator-release-4.1-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-dns-operator-release-4.1-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-dns-operator-release-4.2-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-dns-operator-release-4.2-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-dns-operator-release-4.3-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-dns-operator-release-4.3-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-network-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-samples-operator-master-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-samples-operator-master-e2e-aws-image-ecosystem
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-samples-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-storage-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-update-keys-release-4.1-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-update-keys-release-4.2-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-update-keys-release-4.3-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-cluster-version-operator-master-e2e-aws-upgrade
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-descheduler-operator-release-4.1-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-descheduler-operator-release-4.2-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-descheduler-operator-release-4.3-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-image-registry-release-4.1-e2e-aws-image-registry
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-image-registry-release-4.2-e2e-aws-image-registry
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-image-registry-release-4.3-e2e-aws-image-registry
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-installer-release-4.2-e2e-restore-cluster-state
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-installer-release-4.3-e2e-restore-cluster-state
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-kubernetes-autoscaler-release-4.1-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-kubernetes-autoscaler-release-4.2-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-kubernetes-autoscaler-release-4.3-e2e-aws-operator
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-library-go-master-verify-alpha-build-machinery
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-library-go-release-4.1-verify-alpha-build-machinery
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-library-go-release-4.2-verify-alpha-build-machinery
WARN[0000] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-library-go-release-4.3-verify-alpha-build-machinery
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.1-e2e-aws-operator
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.1-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.1-e2e-rhel-scaleup
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.2-e2e-aws-operator
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.2-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.2-e2e-rhel-scaleup
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.3-e2e-aws-operator
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.3-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-api-operator-release-4.3-e2e-rhel-scaleup
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-config-operator-master-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-config-operator-master-e2e-rhel-scaleup
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-config-operator-master-e2e-etcd-quorum-loss
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-machine-config-operator-master-e2e-restore-cluster-state
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-openshift-ansible-master-e2e-aws-scaleup-rhel7
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-openshift-ansible-release-3.11-e2e-gcp-logging
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-openshift-ansible-release-4.1-e2e-aws-scaleup-rhel7
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-openshift-ansible-release-4.2-e2e-aws-scaleup-rhel7
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-openshift-ansible-release-4.3-e2e-aws-scaleup-rhel7
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-origin-aggregated-logging-master-full-integ-aws
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-prometheus-alertmanager-master-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-prometheus-operator-release-4.1-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-prometheus-operator-release-4.2-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-prometheus-operator-release-4.3-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-service-ca-operator-release-4.1-e2e-aws-operator
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-service-ca-operator-release-4.1-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-service-ca-operator-release-4.2-e2e-aws-operator
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-service-ca-operator-release-4.2-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-service-ca-operator-release-4.3-e2e-aws-operator
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-openshift-service-ca-operator-release-4.3-e2e-aws-upgrade
WARN[0001] Generated job name is longer than 63 characters. This may cause issues when Prow attempts to label resources with job name. Consider a shorter name.  name=pull-ci-operator-framework-operator-metering-master-metering-e2e-aws
WARN[0001] Failed to determine info for prow job config  error="file name was not prefixed with \"ci-operator-jobs-\": \"infra-periodics\"" source-file=ci-operator/jobs/infra-periodics.yaml

real    0m2.854s
user    0m3.925s
sys     0m0.160s
$ git add -u && git write-tree
0038afab92983fb6fb860b276465deaa4ca5c6dc

Being able to limit the tool to the relevant subset of files (with the guarantee that the pre-submit will catch any problems) is worth keeping, IMO.

@petr-muller
Copy link
Member Author

Well I did not say it's not possible, simply I don't believe it's worth the effort.

/shrug

@petr-muller
Copy link
Member Author

/close

@openshift-ci-robot
Copy link
Contributor

@petr-muller: Closed this PR.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

DennisPeriquet pushed a commit to DennisPeriquet/ci-tools that referenced this pull request Jan 6, 2022
# This is the 1st commit message:

force all disruption aggregation to skipped status while we re-gather unbugged data

# This is the commit message openshift#2:

Fix lint issue.

# This is the commit message openshift#3:

sanitize: keep jobs from the same PR on the same cluster

# This is the commit message openshift#4:

Support runIfChanged and skipIfOnlyChanged for postsubmit

Signed-off-by: clyang82 <chuyang@redhat.com>

# This is the commit message openshift#5:

Update UT

Signed-off-by: clyang82 <chuyang@redhat.com>

# This is the commit message openshift#6:

fix integration test

Signed-off-by: clyang82 <chuyang@redhat.com>

# This is the commit message openshift#7:

[DPTP-2635]cluster-display: add endpont for clusters

# This is the commit message openshift#8:

prowgen: stop tolerating changes to `optional` fields

...except for images jobs for which do not have syntax in ci-operator config

Once we pull all manual changes to ci-operator configs, we can stop tolerating these changes. One step closer to fully generated Prowjob configuration.

# This is the commit message openshift#9:

Honor semver when comparing clone target/source

# This is the commit message openshift#10:

Write a README for autoconfigbrancher

# This is the commit message openshift#11:

autoconfigbrancher: simplify a conditional

# This is the commit message openshift#12:

autoconfigbrancher: use a permanent link in README

# This is the commit message openshift#13:

Rename mis-spelled variable, add short for cobra help, add comments

# This is the commit message openshift#14:

add more comments re: GCS

# This is the commit message openshift#15:

Note two tables created in memory using a SELECT

# This is the commit message openshift#16:

fix typo

# This is the commit message openshift#17:

cast to a type so we can see more info on dryrun mode

# This is the commit message openshift#18:

Share ReleaseController's config

# This is the commit message openshift#19:

Add aggregating job results to testgrid (openshift#2576)

* Add aggregating job results to testgrid

* Add aggregating job results to testgrid

Aggregating jobs are added to blocking dashboard of the corresponding release.
Release is determined by the aggregated job prow config.

* Add test cases

* Add nil pointer check
# This is the commit message openshift#20:

Updating auto-include logic for OpenStack jobs

Adding the OpenStack jobs which recently moved from release to
shiftstack directory.
This will add them to testgrid automatically.

# This is the commit message openshift#21:

[DPTP-2660]payload-testing-prow-plugin: format errors

# This is the commit message openshift#22:

handle missing history for aggregated disruption

# This is the commit message openshift#23:

Add create-tables subcommand to create Jobs tab and debugging tips to README

# This is the commit message openshift#24:

create JobRuns table

# This is the commit message openshift#25:

Rename vars to be more generic for table creation

# This is the commit message openshift#26:

add in gofmt fixes for lint

# This is the commit message openshift#27:

comment about Jobs being initially primed

# This is the commit message openshift#28:

comment compile time checks in case go beginners have never seen it

# This is the commit message openshift#29:

prpqr: add a PR deploy makefile target

# This is the commit message openshift#30:

prpqr ui: use centos stream instead of 8 from docker

# This is the commit message openshift#31:

Refuse /payload command on PRs to repositories that do not contribute to OCP

# This is the commit message openshift#32:

allow correcting history without changing test name when we get criteria incorrect

# This is the commit message openshift#33:

switch the check for aggregation passes to use the SQL views
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants