Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] make shell scripts stricter #6266

Merged
merged 35 commits into from
Apr 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
63be785
[ci] make shell scripts stricter
jameslamb Jan 11, 2024
50a31f7
more fixes
jameslamb Jan 11, 2024
6e01e25
set defaults
jameslamb Jan 11, 2024
dab8148
more defaults
jameslamb Jan 11, 2024
728d9c4
change default
jameslamb Jan 11, 2024
3f6456c
fix shebangs, more defaults
jameslamb Jan 11, 2024
fe6953c
omit -o pipefail from build-cran-package and build-python
jameslamb Jan 12, 2024
b1cc27a
revert shebang changes
jameslamb Jan 12, 2024
50b7a5d
more unbound variables
jameslamb Jan 13, 2024
8efd6de
more changes
jameslamb Jan 13, 2024
54f4578
merge master
jameslamb Feb 10, 2024
3a2db64
merge master
jameslamb Feb 19, 2024
bbc2319
unbound variables
jameslamb Feb 19, 2024
9187206
more fixes
jameslamb Feb 19, 2024
13b3233
more fixes
jameslamb Feb 19, 2024
52be3c2
more changes
jameslamb Feb 19, 2024
342967f
Merge branch 'master' into ci/stricter-scripts
jameslamb Feb 20, 2024
5051ef7
remove more libompy.dylib workarounds
jameslamb Feb 20, 2024
16f6441
Merge branch 'master' into ci/stricter-scripts
jameslamb Feb 21, 2024
f23a406
Merge branch 'master' into ci/stricter-scripts
jameslamb Feb 24, 2024
ddf2482
Merge branch 'master' into ci/stricter-scripts
jameslamb Mar 3, 2024
75ed073
debugging
jameslamb Mar 3, 2024
65c077d
Merge branch 'ci/stricter-scripts' of github.com:microsoft/LightGBM i…
jameslamb Mar 3, 2024
041b403
fix exit codes
jameslamb Mar 4, 2024
1f79149
Merge branch 'master' into ci/stricter-scripts
jameslamb Mar 20, 2024
8af2599
Merge branch 'master' into ci/stricter-scripts
jameslamb Apr 17, 2024
96930b0
merge master
jameslamb Apr 17, 2024
b13cf9a
merge master
jameslamb Apr 20, 2024
8a2e093
tweak scripts
jameslamb Apr 20, 2024
317d673
merge master
jameslamb Apr 20, 2024
8296006
Merge branch 'master' into ci/stricter-scripts
jameslamb Apr 22, 2024
86ee054
Merge branch 'master' into ci/stricter-scripts
jameslamb Apr 22, 2024
e4fa87d
clarify grep exit
jameslamb Apr 22, 2024
7ede17c
Merge branch 'ci/stricter-scripts' of github.com:microsoft/LightGBM i…
jameslamb Apr 22, 2024
9d24415
Update .ci/lint-cpp.sh
jameslamb Apr 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .ci/append_comment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#
# BODY: Text that will be appended to the original comment body.

set -e
set -e -E -u -o pipefail

if [ -z "$GITHUB_ACTIONS" ]; then
echo "Must be run inside GitHub Actions CI"
Expand Down
6 changes: 6 additions & 0 deletions .ci/check_python_dists.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
#!/bin/sh
borchero marked this conversation as resolved.
Show resolved Hide resolved

set -e -E -u

DIST_DIR=${1}

# defaults
METHOD=${METHOD:-""}
TASK=${TASK:-""}

echo "checking Python package distributions in '${DIST_DIR}'"

pip install \
Expand Down
12 changes: 11 additions & 1 deletion .ci/lint-cpp.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
#!/bin/sh
#!/bin/bash

set -e -E -u -o pipefail

echo "running cpplint"
cpplint \
Expand Down Expand Up @@ -32,9 +34,17 @@ get_omp_pragmas_without_num_threads() {
'pragma omp parallel' \
| grep -v ' num_threads'
}

# 'grep' returns a non-0 exit code if 0 lines were found.
# Turning off '-e -o pipefail' options here so that bash doesn't
# consider this a failure and stop execution of the script.
#
# ref: https://www.gnu.org/software/grep/manual/html_node/Exit-Status.html
set +e +o pipefail
PROBLEMATIC_LINES=$(
jameslamb marked this conversation as resolved.
Show resolved Hide resolved
get_omp_pragmas_without_num_threads
)
set -e -o pipefail
if test "${PROBLEMATIC_LINES}" != ""; then
get_omp_pragmas_without_num_threads
echo "Found '#pragma omp parallel' not using explicit num_threads() configuration. Fix those."
Expand Down
4 changes: 3 additions & 1 deletion .ci/lint-python.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
#!/bin/sh
#!/bin/bash

set -e -E -u -o pipefail

echo "running pre-commit checks"
pre-commit run --all-files || exit 1
Expand Down
2 changes: 1 addition & 1 deletion .ci/rerun_workflow.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
#
# PR_BRANCH: Name of pull request's branch.

set -e
set -e -E -u -o pipefail

if [ -z "$GITHUB_ACTIONS" ]; then
echo "Must be run inside GitHub Actions CI"
Expand Down
2 changes: 1 addition & 1 deletion .ci/set_commit_status.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
#
# SHA: SHA of a commit to set a status on.

set -e
set -e -E -u -o pipefail

if [ -z "$GITHUB_ACTIONS" ]; then
echo "Must be run inside GitHub Actions CI"
Expand Down
7 changes: 6 additions & 1 deletion .ci/setup.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
#!/bin/bash

set -e -E -o pipefail
set -e -E -u -o pipefail

# defaults
AZURE=${AZURE:-"false"}
IN_UBUNTU_BASE_CONTAINER=${IN_UBUNTU_BASE_CONTAINER:-"false"}
SETUP_CONDA=${SETUP_CONDA:-"true"}

ARCH=$(uname -m)

Expand Down
2 changes: 2 additions & 0 deletions .ci/test-python-oldest.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/bin/bash

set -e -E -u -o pipefail

# oldest versions of dependencies published after
# minimum supported Python version's first release
#
Expand Down
12 changes: 9 additions & 3 deletions .ci/test.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
#!/bin/bash

set -e -E -o pipefail
set -e -E -o -u pipefail

# defaults
IN_UBUNTU_BASE_CONTAINER=${IN_UBUNTU_BASE_CONTAINER:-"false"}
METHOD=${METHOD:-""}
PRODUCES_ARTIFACTS=${PRODUCES_ARTIFACTS:-"false"}
SANITIZERS=${SANITIZERS:-""}

ARCH=$(uname -m)

Expand Down Expand Up @@ -83,11 +89,11 @@ if [[ $TASK == "lint" ]]; then
'r-lintr>=3.1'
source activate $CONDA_ENV
echo "Linting Python code"
sh ${BUILD_DIRECTORY}/.ci/lint-python.sh || exit 1
bash ${BUILD_DIRECTORY}/.ci/lint-python.sh || exit 1
echo "Linting R code"
Rscript ${BUILD_DIRECTORY}/.ci/lint_r_code.R ${BUILD_DIRECTORY} || exit 1
echo "Linting C++ code"
sh ${BUILD_DIRECTORY}/.ci/lint-cpp.sh || exit 1
bash ${BUILD_DIRECTORY}/.ci/lint-cpp.sh || exit 1
exit 0
fi

Expand Down
7 changes: 7 additions & 0 deletions .ci/test_r_package.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
#!/bin/bash

set -e -E -u -o pipefail

# defaults
ARCH=$(uname -m)
INSTALL_CMAKE_FROM_RELEASES=${INSTALL_CMAKE_FROM_RELEASES:-"false"}

# set up R environment
CRAN_MIRROR="https://cran.rstudio.com"
Expand Down Expand Up @@ -212,6 +216,9 @@ if [[ $check_succeeded == "no" ]]; then
exit 1
fi

# ensure 'grep --count' doesn't cause failures
set +e

used_correct_r_version=$(
cat $LOG_FILE_NAME \
| grep --count "using R version ${R_VERSION}"
Expand Down
2 changes: 2 additions & 0 deletions .ci/test_r_package_valgrind.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/bin/bash

set -e -E -u -o pipefail

RDscriptvalgrind -e "install.packages(c('R6', 'data.table', 'jsonlite', 'Matrix', 'RhpcBLASctl', 'testthat'), repos = 'https://cran.rstudio.com')" || exit 1
sh build-cran-package.sh \
--r-executable=RDvalgrind \
Expand Down
2 changes: 1 addition & 1 deletion .ci/trigger_dispatch_run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
#
# DISPATCH_NAME: Name of a dispatch to be triggered.

set -e
set -e -E -u -o pipefail

if [ -z "$GITHUB_ACTIONS" ]; then
echo "Must be run inside GitHub Actions CI"
Expand Down
9 changes: 0 additions & 9 deletions .github/workflows/r_package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,15 +134,6 @@ jobs:
r_version: 4.3
build_type: cran
container: null
################
# Other checks #
################
- os: ubuntu-latest
task: r-rchk
compiler: gcc
r_version: 4.3
build_type: cran
container: 'ubuntu:22.04'
Copy link
Collaborator Author

@jameslamb jameslamb Apr 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rchk job has been silently broken for over a year.

I'm working on fixing it in #6332 . See that PR for details.

In this PR, I'm proposing removing it entirely to save some CI time and to not have to introduce complexity to workaround its issues as a part of this PR.

steps:
- name: Prevent conversion of line endings on Windows
if: startsWith(matrix.os, 'windows')
Expand Down
2 changes: 2 additions & 0 deletions R-package/recreate-configure.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/bin/bash

set -e -E -u -o pipefail

# recreates 'configure' from 'configure.ac'
# this script should run on Ubuntu 22.04
AUTOCONF_VERSION=$(cat R-package/AUTOCONF_UBUNTU_VERSION)
Expand Down
2 changes: 1 addition & 1 deletion build-cran-package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
# # skip vignette building
# sh build-cran-package.sh --no-build-vignettes

set -e
set -e -E -u

# Default values of arguments
BUILD_VIGNETTES=true
Expand Down
2 changes: 1 addition & 1 deletion build-python.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
# Install into user-specific instead of global site-packages directory.
# Only used with 'install' command.

set -e -u
set -e -E -u

echo "building lightgbm"

Expand Down
2 changes: 2 additions & 0 deletions docs/build-docs.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/bin/bash

set -e -E -u -o pipefail

rm -f ./_FIRST_RUN.flag

export PATH="${CONDA}/bin:${PATH}"
Expand Down
Loading