Set of scripts and configuration options to manage tedious tasks across linux-system-roles repos.
- shellcheck
- lsr_role2collection.py
- release_collection.py
- check_rpmspec_collection.sh
- manage-role-repos.sh
- role-make-version-changelog.sh
- role-manage-prs.sh
- role-manage-dependabot-alerts.sh
- list-pr-statuses-ghapi.py
- bz-manage.sh
- manage_jenkins.py
- configure_squid
- lsr_fingerprint.py
- get-github-stats.sh
- manage_issues.py
When making edits, use shellcheck *.sh to check your scripts for common
shell script problems. On Fedora, dnf -y install ShellCheck. There is
currently no way to disable a check by using a mnemonic string or keyword
value - you must use a code like SC2034. See
koalaman/shellcheck#1948 - In the meantime, if you
need to disable a check, add a link to the ShellCheck wiki like this:
# https://github.com/koalaman/shellcheck/wiki/SC2064
# shellcheck disable=SC2064
trap "rm -f ${STDOUT} ${STDERR}; cd ${HERE}" ABRT EXIT HUP INT QUIT
So that someone can easily figure out what this code means. NOTE that the link must come first, followed by the comment with the shellcheck disable directive.
This is a tool to convert a linux-system-role style role to the collections format.
lsr_role2collection.py [-h] [--namespace NAMESPACE] [--collection COLLECTION]
[--dest-path DEST_PATH] [--tests-dest-path TESTS_DEST_PATH]
[--src-path SRC_PATH] [--src-owner SRC_OWNER] [--role ROLE]
[--new-role NEW_ROLE] [--replace-dot REPLACE_DOT]
[--subrole-prefix SUBROLE_PREFIX] [--readme README]
[--extra-mapping EXTRA_MAPPING] [--meta-runtime META_RUNTIME]
[--extra-script EXTRA_SCRIPT]
-h, --help show this help message and exit
--namespace NAMESPACE
Collection namespace; default to fedora
--collection COLLECTION
Collection name; default to system_roles
--dest-path DEST_PATH
Path to parent of collection where role should be migrated;
default to ${HOME}/.ansible/collections
--tests-dest-path TESTS_DEST_PATH
Path to parent of tests directory in which rolename directory is
created and test scripts are copied to the directory; default to
DEST_PATH/NAMESPACE/COLLECTION
--src-path SRC_PATH Path to the parent directory of the source role;
default to ${HOME}/linux-system-roles
--src-owner SRC_OWNER
Owner of the role in github. If the parent directory name in SRC_PATH is
not the github owner, may need to set to it, e.g., "linux-system-roles";
default to the parent directory of SRC_PATH
--role ROLE Role to convert to collection
--new-role NEW_ROLE The new role name to convert to
--replace-dot REPLACE_DOT
If sub-role name contains dots, replace them with the specified
value; default to '_'
--subrole-prefix SUBROLE_PREFIX
If sub-role name does not start with the specified value, change
the name to start with the value; default to an empty string
--readme README Path to the readme file used in top README.md
--extra-mapping EXTRA_MAPPING
This is a comma delimited list of extra mappings to apply when converting
the files - this replaces the given role name with collection format with
the optional given namespace and collection as sell as the given FQCN with
other FQCN.
The format is
"src_name:[[dest_namespace.]dest_collection.]dest_name,\
...,
src_FQCN:[[dest_namespace.]dest_collection.]dest_name,\
..."
The default for `dest_namespace` is the `--namespace` value,
and the default for `dest_collection` is the `--collection` value.
`src_name` is the name of a role, preferably in `namespace.rolename` format.
If just using `rolename` for `src_name`, and `rolename` is used in places
in the README that you do not want to change, you may have to change the
README in another way, not using this script, by using sed with a custom
regex.
In addition, 'fedora.linux_system_roles:NAMESPACE.COLLECTION' is in the
mapping, 'fedora.linux_system_roles' is converted to 'NAMESPACE.COLLECTION'.
--meta-runtime /path/to/runtime.yml
This is the path to the collection meta/runtime.yml - the default is
$HOME/linux-system-roles/auto-maintenance/lsr_role2collection/runtime.yml.
--extra-script /path/to/executable
This is a script to use to do custom conversion of the role. For example,
to convert uses of filter plugins to FQCN. If you do not specify anything,
then it looks for an executable file named `lsr_role2coll_extra_script` in
the role root directory, and runs it. The arguments such as dest dir,
namespace, etc. are passed in via environment variables. See the code for
the list of environment variables available.
Each option has corresponding environment variable to set.
--namespace NAMESPACE COLLECTION_NAMESPACE
--collection COLLECTION COLLECTION_NAME
--src-path SRC_PATH COLLECTION_SRC_PATH
--dest-path DEST_PATH COLLECTION_DEST_PATH
--role ROLE COLLECTION_ROLE
--new-role NEW_ROLE COLLECTION_NEW_ROLE
--replace-dot REPLACE_DOT COLLECTION_REPLACE_DOT
--subrole-prefix SUBROLE_PREFIX COLLECTION_SUBROLE_PREFIX
The default logging level is ERROR.
To increase the level to INFO, set LSR_INFO to true.
To increase the level to DEBUG, set LSR_DEBUG to true.
In this table, DEST_PATH/ansible_collections/NAMESPACE/COLLECTION is represented as COLLECTION_PATH. Assume the role name to be converted is myrole.
| Items | Original roles path | New collections path |
|---|---|---|
| README.md | SRC_PATH/myrole/README.md | COLLECTION_PATH/roles/myrole/README.md [0] |
| role | SRC_PATH/myrole/{defaults,files,handlers,meta,tasks,templates,vars} | COLLECTION_PATH/roles/myrole/* |
| subrole | SRC_PATH/myrole/roles/mysubrole | COLLECTION_PATH/roles/mysubrole [1] |
| modules | SRC_PATH/myrole/library/*.py | COLLECTION_PATH/plugins/modules/*.py |
| module_utils | SRC_PATH/myrole/module_utils/*.py | COLLECTION_PATH/plugins/module_utils/myrole/*.py [2] |
| tests | SRC_PATH/myrole/tests/*.yml | COLLECTION_PATH/tests/myrole/*.yml |
| docs | SRC_PATH/myrole/{docs,design_docs,examples,DCO}/*.{md,yml} | COLLECTION_PATH/docs/myrole/*.{md,yml} |
| license files | SRC_PATH/myrole/filename | COLLECTION_PATH/filename-myrole |
A top level README.md is created in COLLECTION_PATH and it lists the link to COLLECTION_PATH/myrole/README.md.
If a main role has sub-roles in the roles directory, the sub-roles are copied to the same level as the main role in COLLECTION_PATH/roles. To distinguish such sub-roles from the main roles, the sub-roles are listed in the Private Roles section in the top level README.md.
In the current implementation, if a module_utils program is a direct child of SRC_PATH/module_utils, a directory named "myrole" is created in COLLECTIONS_PATH and the module_utils program is copied to COLLECTIONS_PATH/plugins/module_utils/myrole. If a module_utils program is already in a sub-directory of SRC_PATH/module_utils, the program is copied to COLLECTIONS_PATH/plugins/module_utils/sub-directory.
Convert a role myrole located in the default src-path to the default dest-path with default namespace fedora and default collection name system_roles.
Source role path is /home/user/linux-system-roles/myrole. Destination collections path is /home/user/.ansible/collections.
python lsr_role2collection.py --role myrole
Convert a role myrole located in the default src-path to the default dest-path with namespace community and collection name test.
Source role path is /home/user/linux-system-roles/myrole. Destination collections path is /home/user/.ansible/collections.
python lsr_role2collection.py --role myrole \
--namespace community \
--collection test
Convert a role myrole located in a custom src-path to a custom dest-path and a custom tests-dest-path with a custom role name.
Source role path is /path/to/role_group/myrole. Destination collections path is /path/to/collections.
python lsr_role2collection.py --role myrole \
--src-path /path/to/role_group \
--dest-path /path/to/collections \
--tests-dest-path /path/to/test_dir \
--new-role mynewrole
Convert a role myrole in a github owner "linux-system-roles" located in a custom src-path to a custom dest-path and a custom tests-dest-path
Source role path is /path/to/role_group/myrole. Destination collections path is /path/to/collections.
python lsr_role2collection.py --role myrole \
--src-owner linux-system-roles \
--src-path /path/to/role_group \
--dest-path /path/to/collections \
--tests-dest-path /path/to/test_dir
This script is used to:
- Check all of the roles for new versions, and update
collection_release.ymlwith the updatedreffields. - Convert the roles to collection using lsr_role2collection.py
- Update the version in
galaxy.yml - Build the collection file using
ansible-galaxy collection build - Check the collection using
galaxy-importer - Publish the collection using
ansible-galaxy collection publish - Additionally, instead of the upstream sources, the rpm file could be used as
an input. If the option is selected, it skips the first 3 items and builds the
collection file using
ansible-galaxy collection buildwithgalaxy.ymlorMANIFEST.jsonin the rpm file.
The list of roles is specified by default in the file collection_release.yml.
You can use the --include and --exclude options (described below) to control
which roles you want to update and release in the collection. The format of this
file is a dict of role names. Each role name is a dict which must specify the
ref which is the git tag, branch, or commit hash specifying the commit in the
role repo to use to build the collection from. You can optionally specify the
org (default: linux-system-roles) and the repo (default: role name). The
use of the tag in semantic version format is strongly encouraged, as the script
can automatically update the collection version in galaxy.yml if all of the
tags are semantically versioned (see --no-auto-version, --new-version).
The other collection metadata comes from galaxy.yml - namespace, collection
name, version, etc. The script can update the version field automatically if
all updated roles are using semantic versioning (see --no-auto-version,
--new-version).
The script reads the list of roles from collection_release.yml and modifies
the list depending on --include and --exclude. For each role, it will clone
the repo from github to a local working directory (if there is no local clone).
By default, the script will attempt to determine if there are any updates to the
role by comparing the ref in the file with the latest commit. If both refs
are tags in the format of a semantic version, the script will attempt to
automatically determine what the new semantic version of the collection should
be. See the Version section below for details. If using the --no-update
flag, then the script will assume the user has already updated
collection_release.yml to the correct ref and will checkout that ref. If you
have a local working copy of the roles, you can specify it with the --src-path
argument, or the script will use a tmp directory.
The script calls lsr_role2collection.py to convert
each role to collection format using the galaxy namespace name and collection
name specified in galaxy.yml. The script will use
~/.ansible/collections/ansible_collections/$NAMESPACE/$NAME to convert the
files into and assemble the other metadata (such as the ignore files). If you
want to use a different directory, use --dest-path. After all of the roles
have been converted, the script uses ansible-galaxy collection build to build
the collection package file suitable for publishing. The file will be placed in
the ~/.ansible/collections directory. If the --dest-path exists, and you
want to replace it, use the --force argument.
The script will then run galaxy-importer against the collection package file
to check if it will import into galaxy cleanly. If you get errors, you will
probably need to go back and fix the role or add some suppressions to the
.sanity-ansible-ignore-x.y.txt in the role, where x.y is the version of
ansible that galaxy-importer is using.
By default, the script will not publish the collection to Galaxy. Specify the
--publish argument to publish the collection to Galaxy. The script will then
do something like this:
ansible-galaxy collection publish -vv ~/.ansible/collections/NAMESPACE-COLLECTION_NAME-VERSION.tar.gz
and will wait until the publish is completed. If you do not want to wait,
specify the --no-wait argument which will do something like this:
ansible-galaxy collection publish --no-wait ~/.ansible/collections/NAMESPACE-COLLECTION_NAME-VERSION.tar.gz
If the script is unable to calculate the new version for galaxy.yml you will
need to figure out what the new semantic version will be, and use the
--new-version argument.
git clone https://github.com/linux-system-roles/auto-maintenance [-b STABLE_TAG]
STABLE_TAG can be omitted to use master.
Or, figure out some way to download the correct versions of
lsr_role2collection.py, galaxy.yml,
collection_release.yml, and release_collection.py from this repo.
You should ensure that the roles have been appropriately tagged with semantic
versions. You are strongly encouraged to use
role-make-version-changelog.sh to tag, release, and
publish the individual roles. If you do this, then you can usually let the
script automatically update the versions in collection_release.yml and
galaxy.yml. Otherwise, you will need to ensure that the information in
galaxy.yml (particularly the version: field), and the information in
collection_release.yml (particularly the ref: fields), are correct and
up-to-date for the collection you want to build and publish, then use the
--no-update argument.
You will need the galaxy-importer package - pip install galaxy-importer --user.
You will need docker in order to use galaxy-importer.
ROLENAME:
ref: TAG_OR_HASH_OR_BRANCH
org: github-organization # default is linux-system-roles
repo: github-repo # default is ROLENAME
sourcenum: NWhere ROLENAME is the name of the role as it will appear in the collection
(under NAMESPACE/COLLECTION_NAME/roles/). ref is the git tag in semantic
version format (preferred), commit hash, or branch to use (in the format used as
the argument to git clone -b or git checkout). org is the github
organization (default linux-system-roles). The repo is the name of the repo
under the github organization, and the default is the ROLENAME, so only use
this if you need to specify a different name for the role in the collection. The
sourcenum is the source number in the RPM spec file. For example,
https://src.fedoraproject.org/rpms/linux-system-roles/blob/rawhide/f/linux-system-roles.spec#_93
%global rolename1 postfix
%deftag 1 1.3.4
The sourcenum for the postfix role is 1.
The sshd role currently uses both org and repo.
To add a new role - add it with a ref: null, and sourcenum as the next
available sourcenum value.
ad_integration:
ref: null
sourcenum: 23NOTE: Please keep the file in order of sourcenum.
Then the next time you run release_collection.py, it will know that this is a
new role and will update the versions accordingly.
Example:
tlog:
ref: 1.2.12
sourcenum: 8
kernel_settings:
ref: 1.1.13
sourcenum: 9
logging:
ref: 1.11.6
sourcenum: 10This will use e.g.
git clone https://github.com/linux-system-roles/tlog -b 1.2.12, etc.
Use role-make-version-changelog.sh to create new
tags/versions in each role. If you use strict semantic versioning everywhere,
in your github tags and in the collection_release.yml file, you can use the
automatic versioning feature of the script to automatically update the version
in galaxy.yml.
Basic usage:
cd auto-maintenance
python release_collection.py --publish
This will use the galaxy.yml and collection_release.yml files in the current
directory, will create a temporary working directory to clone the roles into,
will create a collection under
$HOME/.ansible/collections/ansible_collections/NAMESPACE/COLLECTION_NAME, and
will create the collection package file
$HOME/.ansible/collections/NAMESPACE-COLLECTION_NAME-VERSION.tar.gz, where
NAMESPACE, COLLECTION_NAME, and VERSION are specified in galaxy.yml, and
will publish the collection, waiting until it is completed.
--galaxy-yml- env:GALAXY_YML- defaultgalaxy.ymlin current directory- full path and filename of
galaxy.ymlto use to build the collection
- full path and filename of
--collection-release-yml- env:COLLECTION_RELEASE_YML- defaultcollection_release.ymlin current directory - full path and filename ofcollection_release.ymlto use to build the collection--src-path- env:COLLECTION_SRC_PATH- no default - Path to the directory containing the local clone of the role repos - if nothing is specified, the script will create a temporary directory--dest-path- env:COLLECTION_DEST_PATH- default$HOME/.ansible/collections- collection directory structure will be created inDIR/ansible_collection/NAMESPACE/COLLECTION_NAME- collection package file will be created inDIR--force- boolean - defaultFalse- if specified, the collection directory will be removed first if it exists, and the package file, if any, will be overwritten--include- list - default empty - by default, the script will operate on all roles in thecollection_release.yml. Use--includeto specify only those roles you want to update in the new collection. If you specify both--excludeand--includethen--excludetakes precedence.--exclude- list - default empty - by default, the script will operate on all roles in thecollection_release.yml. Use--excludeto remove the specified roles from that list. If you specify both--excludeand--includethen--excludetakes precedence.--new-version- string - The new semantic version to use for the collection. The script will update this--use-commit-hash- boolean - By default, only tags in semantic version format will be used for thereffield incollection_release.yml. If instead you want to use the latest commit hash for a role that has not been tagged, use this flag. However, this means you will not be able to use automatic versioning, and you will need to use--new-version, or manually edit thecollection_release.ymlandgalaxy.ymland use--no-update.--use-commit-hash-role- list - By default, use whatever is the value of--use-commit-hash. There are some cases where you want to use the tag for all roles except one or a few. In that case, you can use--use-commit-hash-role role1 --use-commit-hash-role role2to specify the roles for which you want to use the commit hash, and use the tag for all other roles.--no-auto-version- boolean - By default, the script will attempt to update the collection version ingalaxy.yml. Use this flag if you do not want to do that.--no-update- boolean - By default, the script will attempt to update thereffields for each role incollection_release.ymland the version ingalaxy.yml. Use this flag if you do not want to do that.--publish- boolean - By default, the script will just create the collection tarball in~/.ansible/collections. You must specify--publishif you want to publish the collection.--no-wait- boolean - By default, when publishing, the script will wait until the publishing is completed. Use--no-waitif you do not want to wait, and instead will check the import status in Galaxy.--skip-git- boolean - If set totrue, use local source. By default,false.--skip-check- boolean - If set totrue, check using galaxy-importer is skipped. By default,false.--skip-changelog- boolean - By default, the script will attempt to generate a collection changelog from the individual role changelogs. Use--skip-changelogif you do not want to do this.--debug- boolean - By default, the script will only output informational messages. Use--debugto see the details.--rpm- string - Specifies the rpm file for the input collection. When --rpm is set, other input options such as--galaxy-yml,--include,--exclude,--new-version,--use-commit-hash, and--no-auto-versionare ignored. Note: if the rpm file does not containgalaxy.ymlorMANIFEST.jsonin/path/to/ansible_collections/namespace/collection,release_collection.pyfails. No default value.--extra-mapping- string - same as the--extra-mappingargument to lsr_role2collection--changelog-rst- boolean - by default, CHANGELOG.rst will not be created - set this to create CHANGELOG.rst from docs/CHANGELOG.md. You must not use--skip-changelogif you want to use--changelog-rst.
For the version: field in galaxy.yml, we have to use X.Y.Z where X, Y,
and Z are non-negative integers. We will start at 0.0.1 until the
collection stabilizes, then we should start at 1.0.0. During this
stabilization period, we should just increase the Z number when we do a new
release.
We assume that each role will be doing regular tagged releases where the tag is the semantic version. If not, then it will be difficult to determine what sort of change the role has made, and how it should affect the collection version. The role-make-version-changelog.sh script is useful to identify what sort of changes were made in each role, update the semantic version tag, do github releases, and publish individual roles to Galaxy. You are strongly encouraged to use that script.
The collection version will be derived from all of the role versions, and will be semantic versioned.
Notation: Xr is the X number from the version of a given role r. Xc is
the X number for the collection c. Similarly for Yr, Yc, Zr, and
Zc.
Examine the versions in the updated roles and compare them to the roles in the
collection_release.yml.
If any of the Xr has changed, set the new Xc to Xc + 1 - bump major
release number to indicate there is a role which has introduced an api breaking
change, and set Yc and Zc to 0.
If none of the Xr has changed, and if any of the Yr has changed, set the new
Yc to Yc + 1 - bump minor release number to indicate there is a role which
has introduced a non-breaking api change, and set Zc to 0.
If none of the Xr or Yr has changed, and if any of the Zr has changed, set
Zc Zc + 1 - some role has changed.
If the role does not use a semantic version for the tag, or it is otherwise
unclear how to determine what sort of changes have occurred in the role, it is
up to the collection maintainer to investigate the role to determine how to
change Xc, Yc, or Zc.
This script is useful for the linux-system-roles team to make changes to multiple roles.
This script uses gh for cli/api interactions with github, and jq for
parsing and formatting the output of gh. You can install these on Fedora
like dnf -y install jq gh. The gh command also requires the use of your
github api token. See man gh for instructions.
Basic usage, with no arguments:
LSR_BASE_DIR=~/linux-system-roles ./manage-role-repos.sh
This will clone all of the repos under https://github.com/linux-system-roles
(except those listed in $EXCLIST - see the script for the default list e.g. it
excludes repos such as test-harness, auto-maintenance, etc.). These will be
cloned to $HOME/linux-system-roles using the gh repo clone command (if the
directory does not already exist). Each repo will be forked under your github
user using the gh repo fork command (if you do not already have a fork). Each
local clone will have a git remote named upstream for the source repo, and a
git remote named origin for your fork. It will also do a git fetch to pull
the latest code from the remotes (but it will not merge that code with your
local branches - you must do that if desired).
Options are passed as environment variables.
LSR_BASE_DIR - local directory holding clones of repos - no default - it is
recommended you use a temporary directory. If you do not specify this, the
script assumes you want to perform some action e.g. gh pr list in each role
that does not require a local clone, and will not clone the repo locally.
LSR_GH_ORG - the github org to use as the origin of the clones/forks e.g. the
ORG used in https://github.com/ORG - default is linux-system-roles
DEBUG - turns on set -x
MAKE_FORK - default true - this is only used if you also specified
LSR_BASE_DIR - this will ensure that you have a fork of the role repo, for
example, so that you can create working branches in your private repo in github
and use that to submit PRs. If you set this to false, then some actions may
not work.
RENAME_FORK - default false - if you use true, then the script will ensure
that your private fork will be renamed linux-system-roles-$ROLENAME.
REPOS - default none - a space delimited list of roles. Use this if you want
to operate only on the given roles.
# only manage the network and storage roles
REPOS="network storage" ./manage-role-repos.sh
EXCLIST - default none - a space delimited list of roles. Use this if you
want to operate on all roles except the given roles.
# manage all roles except the network and storage roles
EXCLIST="network storage" ./manage-role-repos.sh
You can pass commands to perform in each repo in these ways:
For each repo, checkout the main branch and make sure it is up-to-date:
LSR_BASE_DIR=~/working-lsr ./manage-role-repos.sh "git checkout main || git checkout master; git pull"
The arguments will be passed to eval to be evaluated in the context of each
repo. This is useful if you need to just refresh your local copy of the repo,
or perform a very simple task in each repo. Remember to escape
shell variables that will be set in manage-role-repos.sh.
To list all non-draft PRs in all repos:
./manage-role-repos.sh gh pr list -R \$LSR_GH_ORG/\$repo --search draft:false
You can do the same as above like this:
LSR_BASE_DIR=~/working-lsr ./manage-role-repos.sh <<EOF
git checkout main || git checkout master
git pull
echo \$repo
EOF
That is, you can pass in the commands to use in a bash here document. This
is useful when you have more complicated tasks to perform that takes multiple
commands and/or involves shell logic/looping. Whatever you specify in the here
document will be passed to eval in each repo directory. Remember to escape
shell variables that will be set in manage-role-repos.sh.
For really complex repo administration, you may want to write a shell script to be executed in each repo:
./manage-role-repos.sh /path/to/myscript.sh
For example, to update changelog in every role:
BRANCH=changelog-20230417 LSR_BASE_DIR=~/working-lsr \
./manage-role-repos.sh /path/to/role-make-version-changelog.sh
This script is to be executed in the dist-git directory. It locally builds rpm
packages with various combination of ansible and collection_artifact
options, and checks whether the built rpm count is correct or not. Then, it
verifies that README.html files are only in /usr/share/doc/ area.
Usage: ./check_rpmspec_collection.sh [ -h | --help ] | [ fedpkg | rhpkg [ branch_name ] ]
This script is used to create a new version, tag, and release for a role. It
will guide you through the process. It will show you the changes in the role
since the last tag, and ask you what will be the new semantic version to use for
the tag. If the commits are in Conventional Commits format, it will
automatically determine the new semantic version of the role and prompt you to
use it, or provide your own version. It will then put the changes in a file to
use for the update to the CHANGELOG.md file for the new version, and put you in
your editor to edit the file. If you are using this in conjunction with
manage-role-repos.sh, it will push the changes to your repo and create a pull
request for CHANGELOG.md. Once the CHANGELOG.md PR is merged, there is github
action automation to tag the repo with the version, create a github release, and
import the new version into Ansible Galaxy. You must provide a branch for the
PR, or if you are not using the script with manage-role-repos.sh, you can
create a branch in your local clone directory.
BRANCH=my_branch_name LSR_BASE_DIR=~/working-lsr ./manage-role-repos.sh `pwd`/role-make-version-changelog.sh
If the role has commits that do not follow the Conventional Commits, format, but
you still want to create a new release, use ALLOW_BAD_COMMITS=true, and then
you may need to provide your own new version if the script was unable to
determine the new version from the bad commits.
If the role has no changes since the last tag, the script will skip processing
the role. If you do not want to do this, use AUTOSKIP=false, and you will be
prompted for a new version/tag.
NOTE: You must install and configure gh in order to create the pull request.
If you want to have more control over the commit, commit msg, and PR, then you
can clone the repo manually, create the branch, and run
role-make-version-changelog.sh in the local repo clone directory. This will
not push or create a PR.
This script is highly interactive. Since we are using semantic versioning, we
use Conventional Commits format to determine the semantic changes from the
commits in the log. The script will use this to determine the new version
number, and prompt you to use this version, or you can provide your own version.
The script will show you the commits since the last tag, and will optionally
show you the detailed changes. It will then prompt for the new version which
will also be used as the git tag. If you enter 'y' here, it will use the new
version automatically determined from the Conventional Commits. You can also
provide your own version here, or hit Enter/Return to skip and go to the next
role. If you do want to provide your own version, enter a tag/version in the
form of X.Y.Z, based on the semantic changes to the role. You will then be
prompted to edit CHANGELOG.md for the release. The body will be filled in by
the commit messages from the commits since the last tag - you will need to edit
these. When you are done, it will make a commit in your local repo. If you are
using it with manage-role-repos.sh, and gh is installed and configured, it
will push the changes to your repo and create the PR.
Use this script, and ensure the CHANGELOG.md PR is merged, and the repo is tagged and released, before you run release_collection.py.
Used in conjunction with manage-role-repos.sh. This will list the non-draft
PRs for a role and prompt for merging one of them. The list will include the
checks and tests status and review status for the PR. Actions:
NUM- Enter the PR number to merge it. Usesgh pr merge NUM -r -dl- Refresh the listw- View the list of PRs in a browserv NUM- View the PR in a browsera NUM- Merge PR with admin override. This usesgh pr merge NUM -r -d --admint NUM- View detailed status of tests/checks not reporting SUCCESSci NUM- Add[citest]comment to PR to trigger CI testings NUM /path/to/script- Run a script against a PR. The role name and PR number are passed to the script. This can be used to perform whatever action you want against the PR e.g. check logs for specific errors.c NUM [comment]- Close PR with optional commentd NUM [diff args]- View diff with optional argse NUM [edit args]- Edit the PR with optional argsp NUM- Approve the PR - you cannot approve your own PRspm NUM- Approve and merge the PR - you cannot approve your own PRs- Press Enter by itself to skip to the next role
By default, the list will only include PRs authored by you. Use AUTHOR=@all
to view PRs from all authors. Use AUTHOR=username to view those PRs authored
by the given username.
Examples:
./manage-role-repos.sh ./role-manage-prs.sh - View all non-draft PRs authored
by myself
AUTHOR=@all ./manage-role-repos.sh ./role-manage-prs.sh - View all non-draft
PRs
AUTHOR=richm ./manage-role-repos.sh ./role-manage-prs.sh - View all non-draft
PRs authored by richm
AUTHOR- default is@me- look for PRs authored by this username. Use@allto list all PRs.DELETE_BRANCH- default istrue- usefalseif you do not want to delete the PR branch after merging (thegh pr mergeflag-dis used to delete the branch)MERGE_TYPE- default isr(rebase and merge) - Usemfor merge commit,sfor squash and merge, oraskwhich will prompt you for the merge type.
Used in conjunction with manage-role-repos.sh. This will list the open
dependabot alerts for a role and prompt for an action to perform. Actions:
l- Refresh the listv NUM- View the alert in a browserd NUM reason comment- Dismiss the alert with the given reason and commenta reason comment- Dismiss all alerts with the given reason and comment- Press Enter by itself to skip to the next role
Examples:
./manage-role-repos.sh ./role-manage-dependabot-alerts.sh
Allows you to query the set of all PRs open across all repos. By default, it will print out all open PRs, along with their statuses and checks, along with some other metadata. There are a number of command line options to look for specific repos, platform status, ansible version status, staging vs. production, and many more. See the help for the command for more information.
This tool is primarily for downstream maintainers for various administrative
Bugzilla tasks such as creating new system roles BZs, checking if a BZ in a
specified release has a clone in other releases, setting the ITM and DTM fields
for BZs in a given ITR, managing the devel_whiteboard field, and printing git
commit and changelog entries.
See also RHEL Development Guide for more information about the Bugzilla workflow.
You must have the bugzilla cli tool (provided by the python-bugzilla-cli
package on Fedora).
You must have an authentication API key configured
API Key - see
Authentication
and man bugzilla - the section "AUTHENTICATION CACHE AND API KEYS" for more
information.
You must have the jq cli tool (provided by the jq package on Fedora).
Use this to set the ITM or DTM for all BZ in a given ITR and given STATUS to a given value, if the ITM/DTM is less than the given value. For example, you have several BZ in the POST status for ITR 8.7.0, and you need to ensure that all of them have an ITM of at least 23 - if the ITM is not set, set it to 23 - if the ITM is set to a value less than 23, set it to 23 - if the ITM is set to a value of 23 or higher, do not touch it.
ITR=8.7.0 ITM=23 STATUS=POST ./bz-manage.sh setitm
Use this to remove qa_ack? from the Devel Whiteboard field if the BZ has
been given qa_ack+, and remove pre-verify? from the Devel Whiteboard
field if the BZ has been given Verified:Tested.
ITR=9.1.0 ./bz-manage.sh reset_dev_wb
Create a new system roles BZ. NOTE: Does not use any environment variables - all parameters are passed in on the command line. All parameters are required. Usage:
./bz-manage.sh new X.Y "SUMMARY" "COMMENT" ROLENAME
X.Y- the major.minor version (e.g. 8.7) - the ITR will be set toX.Y.0SUMMARY- the BZ summary/titleCOMMENT- the initial BZ commentROLENAME- name of role - the whiteboard field will be updated withrole:ROLENAME- use""if you want to leave the field blank e.g. the BZ corresponds to multiple roles
For example, to create a new 9.1 BZ:
./bz-manage.sh new 9.1 "kernel_settings mis-configures memory settings" \
"I can reproduce on rhel-9 but not rhel-8" kernel_settings
NOTE: You will have to edit the BZ after creation. And the RHEL Bugzilla workflow tools automatically edit the BZ several times. It may take a few minutes after you create a new BZ that you can actually edit it without having your edits overwritten or discarded.
Check that the BZs of a given ITR and STATUS all have an appropriate clone. It will check that there is a clone, that the clone has the same SUMMARY, and that the clone has the same STATUS.
ITR=8.7.0 STATUS=POST ./bz-manage clone_check
e.g.
ERROR: bz 2066876 status [POST] does not match clone 2072745 status [ON_QA]
The clone check is not perfect, so be sure to check manually.
Use USE_RST=1 to generate the new CHANGELOG text in rST. By default, it
will use Markdown (.md).
Use RELEASE_PLUS=false to list all BZs, even if they do not have release+.
By default, will only list BZs that have release+.
Use this to generate the following files:
- new-cl.md or .rst - The new text to add to the CHANGELOG.md or .rst
- cl-spec - The new text to add to the spec %changelog section
- git-commit-msg - The git commit message
For example - I have several BZ in POST that I am doing a new build for ITR 8.8.0.
The new version will be 1.22.0. NOTE that the version in CHANGELOG.md
is different from the version in the spec file - so you will need to add
the -N for the RELEASE for the spec file version in cl-spec.
I want to update the CHANGELOG.md, the spec %changelog, and the git commit
message with the information from all of these BZ, formatted in the correct
manner for all of these. You will need to edit all 3 files:
- Edit CHANGELOG.md and add the contents of new-cl.md in the right place
- Edit the spec file to add cl-spec in the right place, and ensure the name and email are correct.
- Edit the git-commit-msg with the correct git subject line. You can then use this file with git commit -F
Example:
ITR=8.8.0 ./bz-manage.sh rpm_release 1.22.0
# edit CHANGELOG.md - put the contents of new-cl.md at the top
# edit the spec file - put the contents of cl-spec at the top
# of %changelog - edit the name, email, and add the -N to the version
# edit git-commit-msg - add a descriptive subject line
git add CHANGELOG.md, the spec file, sources, .gitignore ...
git commit -F git-commit-msg
Update Doc Types and Texts in Bugzilla based on the conventional commits format.
For example, to run against BZs with ITR 9.3.0, enter ITR=9.3.0 sh ./bz-manage.sh update_doc_text.
When run, the script searches all PRs from the ITM and other filters provided and does the following:
- If a BZ has a
required_doc_textflag set to+or-, the BZ is skipped.+indicates that the doc text has already been verified by a tech writer.-indicates that no release note is required. - If a BZ has
required_doc_text?set and GitHub PR attached:- If the existing Doc Text is empty, automatically add Doc Text from PR description.
- Else, print the current Doc Text and Doc Type, print the GH PR info, and ask users if they want to update the Doc Text.
- If a BZ has
required_doc_textunset and GH PR attached:- Automatically add Doc Text from PR description
- If a BZ has
required_doc_textnot set to+or-and GitHub PR not attached:- Print the current doc text and type, and ask user if they want to post a comment asking to attach a GH PR or enter the Doc Text.
Generate a list of BZs to stdout. e.g. to list all BZs in POST state that have been acked for 9.3.0:
# ITR=9.3.0 STATUS=POST RELEASE_PLUS=true ./bz-manage.sh list_bzs
https://bugzilla.redhat.com/show_bug.cgi?id=2179460 role:selinux use restorecon -T 0 on supported platforms
https://bugzilla.redhat.com/show_bug.cgi?id=2180902 role:certificate add mode parameter to change permissions for cert files
https://bugzilla.redhat.com/show_bug.cgi?id=2211187 role:kdump support auto_reset_crashkernel, dracut_args, deprecate /etc/sysconfig/kdump
Almost all parameters are passed as environment variables. However, the new
command takes all parameters on the command line - see above.
default "8.8.0" - the Internal Target Release of the BZ to manage
Internal Target Milestone - used with setitm
Dev Target Milestone - used with setdtm
BZ status e.g. NEW, ASSIGNED, POST, etc. Most commands only operate on BZs that
have the specified STATUS - the primary exception is reset_dev_wb.
By default, the limit on the number of BZ returned by a query is 100. Use LIMIT to change that value.
By default, will list all BZs. If you only want to look for BZs that have
release+, use RELEASE_PLUS=true. Note that rpm_release by default is the
opposite, because you typically want BZs that have release+ for that
particular query.
There are other undocumented environment variables used, check the code for more details.
Manage the test tasks in a jenkins server - list and view in various ways, and cancel tasks.
package python3-jenkins-1.7.0-6.fc36.noarch
You must have Kerberos authentication set up.
You have to create a file ~/.config/jenkins.yml like this:
somename:
username: MY_USERNAME
url: "https://my.jenkins.hostname.tld"
job_name: MY_JENKINS_JOBNAME
token: "MY_JENKINS_TOKEN"
current: somenameBy default, it will print the queued tasks, the running tasks, and the completed tasks.
> ./manage_jenkins.py
Queued tasks:
QueueID Queued Since Role PR Platform Queue Reason
3915644 2022-09-14T19:27:23 timesync 119 RHEL-9.2/ansible-2.13 waiting on executor
3915643 2022-09-14T19:27:23 timesync 119 RHEL-9.2/ansible-2.9 waiting on executor
3915642 2022-09-14T19:27:23 timesync 119 RHEL-8.8/ansible-2.13 waiting on executor
3915641 2022-09-14T19:27:23 timesync 119 RHEL-8.8/ansible-2.9 waiting on executor
...
Running tasks:
TaskID Started At Role PR Platform Queue Time
15994 2022-09-14T19:23:42 cockpit 73 RHEL-9.2/ansible-2.9 11
15993 2022-09-14T19:23:41 cockpit 73 RHEL-8.8/ansible-2.13 10
15992 2022-09-14T19:23:41 cockpit 73 RHEL-8.8/ansible-2.9 10
...
Completed tasks:
TaskID Started At Role PR Platform Duration Queue Time Status
15966 2022-09-14T16:00:06 podman 14 RHEL-9.2/ansible-2.13 695 491 FAILED
15965 2022-09-14T15:58:55 podman 14 RHEL-9.2/ansible-2.9 563 420 FAILED
...
It takes a long time to run. You can shorten the duration by using the environment variable
MAX_TASK_AGE (number of seconds of maximum task age) e.g.
> MAX_TASK_AGE=7200 ./manage_jenkins.py
Queued tasks:
...
e.g. to skip tasks older than 2 hours.
You can also specify the argument print_queued_tasks, print_running_tasks, or print_completed_tasks
if you just want to look at those tasks.
> ./manage_jenkins.py print_queued_tasks
Queued tasks:
QueueID Queued Since Role PR Platform Queue Reason
3915644 2022-09-14T19:27:23 timesync 119 RHEL-9.2/ansible-2.13 waiting on executor
...
Use print_task_info TASK_NUM to show a YAML representation of a task:
> ./manage_jenkins.py print_task_info 15967
_class: hudson.model.FreeStyleBuild
actions:
- _class: hudson.model.ParametersAction
parameters:
- _class: hudson.model.StringParameterValue
name: priority
value: '3'
...
which isn't very useful unless you are debugging the script and/or Jenkins itself.
Use this to see information about the task, including the individual tests statuses.
> ./manage_jenkins.py print_task_tests_info 12345
Role:nbde_client PR:80 Platform:RHEL-8.8.0-20220921.0 Arch:x86_64
Node:production-3 IP:10.0.0.3 Workspace:/var/lib/jenkins/workspace/ci-test-jobname@6
Stage State Result GuestID Test Workdir
COMPLETE OK PASSED c3767ea3-934c-408d-b42f-d7639e3e30ca tests_bind_high_availability.yml work-tests_bind_high_availability.ymlrHBR08
COMPLETE OK PASSED 06e43cbc-0090-41ad-8477-16fe0ff43447 tests_default.yml work-tests_default.yml16nMco
COMPLETE OK PASSED d05aaf71-5075-47a1-b50a-9904cc33b699 tests_default_vars.yml work-tests_default_vars.ymlQdWTmI
GUEST_PROVISIONING OK UNDEFINED unknown tests_include_vars_from_parent.yml unknown
GUEST_PROVISIONING OK UNDEFINED unknown tests_key_rotation.yml unknown
GUEST_PROVISIONING OK UNDEFINED unknown tests_passphrase_temporary.yml unknown
CREATED OK UNDEFINED unknown tests_passphrase_temporary_keyfile.yml unknown
Node is the internal node name used by Jenkins. Workspace is the path to
the directory on the node which is used to hold the test artifacts before they
are published. For example, if you want to see the ansible results for the
completed tests_bind_high_availability.yml test:
ssh -i /path/to/key [email protected]
cd /var/lib/jenkins/workspace/ci-test-jobname@6
# from here you can see citool-debug.txt, etc.
cat work-tests_bind_high_availability.ymlrHBR08/ansible-output.txt
cat guest-setup-c3767ea3-934c-408d-b42f-d7639e3e30ca/pre-installation-artifact-workaround.txt
etc.
If you just want to see the task output without having to use the web browser:
./manage_jenkins.py print_task_console 12345 30
... bunch of text here ...
This will show the last 30 lines of the console.
This requires a valid token. Cancel the running task given by the id.
./manage_jenkins.py cancel_tasks 18859 18858 ....
This requires a valid token. There is currently a bug in Jenkins that will
create duplicate tasks. Using this command will remove duplicate tasks from the
queue and running tasks.
./manage_jenkins.py cancel_duplicate_tasks
This requires a valid token. Cancel the running task given by the PR in the
form role:number.
./manage_jenkins.py cancel_pr_tasks storage:555
Print the failed tests from the tasks that match the given role, prnum, and platform.
Platform is optional. This is useful for roles that have a lot of long running
tests, and you don't want to wait several hours to find out if any of the tests
have failed. See also print_task_tests_info.
./manage_jenkins.py print_test_failures network 535
Print the nodes - name, labels, offline status, idle status
node name=the_node_name labels=a,b,c,d offline=True idle=False
...
Used to see if fact gathering in roles is working correctly. Roles should
gather only the subset of facts required for the role, and should only gather
facts if those facts are not already present in ansible_facts. The script
parses the ansible logs from test runs looking for a task
TASK .*$role : Ensure ansible_facts used by role
or in the case of the network role
TASK .*network : Ensure ansible_facts used by role are present
If facts were gathered, the result of the task will be ok, otherwise, it will
be skipping if the facts were already present. By default, Ansible will gather
facts as the first thing it does when you run a playbook. You will see this in
the Ansible log as TASK [Gathering Facts]. If the test uses
gather_facts: false, or if the user specifies the env. var.
ANSIBLE_GATHERING=explicit (or the equivalent Ansible config), then you will
not see this in the logs.
For the normal case, the script looks to see if the Ensure ansible_facts task
has a result of skipping, except in a few special cases.
If running without fact gathering, the script looks to see if the
Ensure ansible_facts task has a result of ok, except in a few special cases,
and if the test has already gathered facts in a previous task (which are special
cases noted in the script).
If you specify a link, this link is assumed to be a Beaker job result XML file. Every log specified in this file will be scanned. If you specify a role name and a PR number, every log specified in the PR CI results will be scanned.
Used to scan logs for references to packages used with the package task, and
generate the lists of packages required for ostree images for a given
distribution and version, and also for runtime and testing. Typically used like
this:
ROLE_PARENT_DIR=~/linux-system-roles log_dir=/var/tmp/lsr ./check-logs-for-packages.sh rolename PR rolename PR ....This will download all of the integration test logs from each given rolename/PR
combination to log_dir and create the lists of packages. You must have run
the PR with the tests/callback_plugins/dump_packages.py plugin in place to
dump the packages used. The directory ROLE_PARENT_DIR should have the role
subdirectories in either the form rolename/ or linux-system-roles.rolename/.
It will update the .ostree/ directory in each role with the package files.
The configure_squid directory stores the playbook that you can use to
configure a Squid caching proxy server for caching RPM packages. The playbook
copies the squid.conf file to the managed node. The squid.conf file
configures Squid to use SSL Bump to cache RPM packages over HTTPS and does some
further configurations required for RPM packages caching. You can compare
squid.conf with squid.conf.default to see what squid.conf adds.
After you configure a squid proxy using this playbook, you must point dnf or yum to use this proxy. To do that, append the following strings to /etc/yum.conf on EL 6 or to /etc/dnf/dnf.conf on EL > 6 and Fedora:
proxy=http://<squid_server_ip>:3128
sslverify=False
Introduced to modify fingerprint in spec file to be system_role:$rolename.
lsr_fingerprint.py scans files in the templates dir under ./role["lsrrolename"],
if the file contains a string role["reponame"]:role["rolename"],
replaces it with system_role:role["lsrrolename"]. Note: role is an entry
of roles that is defined as an array of dictionaries in lsr_fingerprint.py.
For example, in metrics, performancecopilot:ansible-pcp is replaced with
system_role:metrics; in sshd, willshersystems:ansible-sshd is with
system_role:sshd.
Get statistics about system roles PRs and issues from github.
Uses the gh command, and you must have a valid github token.
Usage:
PRS_CSVFILE=prs.csv ISSUES_CSVFILE=issues.csv DATE_RANGE=YYYY-MM-DD..YYYY-MM-DD \
./manage-role-repos.sh ./get-github-stats.shExample date range - 2024-07-01..2024-09-30 for 2024 Q3.
Writes the files prs.csv and issues.csv which are CSV files and can be
imported into Google Sheets or other spreadsheet applications.