Skip to content

Commit

Permalink
ci(check-fixtures): parallelise fixture checking commands
Browse files Browse the repository at this point in the history
Introduces several independent `run-tests.sh` fixture-checking commands
in order to speed up fixture checking by parallelisation.

Removes `pydocstyle` checks since we moved to `black` code formatter.

Renames `run-tests.sh` script options and CI rules to better separate
data checks, formatting checks and linting checks.

Introduces `/run-tests.sh --help` explaining all the checking options.

Updates CI environment to Ubuntu 24.04 and latest actions
(`actions/checkout@v4`, `actions/setup-node@v4`,
`actions/setup-python@v5`).

Amends `.editorconfig` to add rules for shell scripts and remove rules
for ReST files that are no longer needed after switch to Markdown.

BREAKING CHANGE: Refactors `run-tests.sh` script options.
  • Loading branch information
tiborsimko committed Jan 10, 2025
1 parent 952b104 commit d48b632
Show file tree
Hide file tree
Showing 5 changed files with 195 additions and 128 deletions.
6 changes: 3 additions & 3 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
#
# This file is part of CERN Open Data Portal.
# Copyright (C) 2016 CERN.
# Copyright (C) 2016, 2025 CERN.
#
# CERN Open Data Portal is free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public License as
Expand Down Expand Up @@ -40,8 +40,8 @@ known_first_party = cernopendata
multi_line_output = 2
default_section = THIRDPARTY

# RST files (used by sphinx)
[*.rst]
# Shell script files
[*.sh]
indent_size = 4

# CSS, HTML, JS, JSON, YML
Expand Down
159 changes: 100 additions & 59 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# This file is part of CERN Open Data Portal.
# Copyright (C) 2020, 2023, 2024 CERN.
# Copyright (C) 2020, 2023, 2024, 2025 CERN.
#
# CERN Open Data Portal is free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public License as
Expand All @@ -20,117 +20,158 @@
# waive the privileges and immunities granted to it by virtue of its status
# as an Intergovernmental Organization or submit itself to any jurisdiction.

name: CI
name: ci

on: [push, pull_request]

jobs:
lint-shellcheck:
runs-on: ubuntu-20.04
data-dois:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Runs shell script static analysis
run: |
sudo apt-get install shellcheck
./run-tests.sh --check-shellscript
- name: Setup node
uses: actions/setup-node@v4

lint-black:
runs-on: ubuntu-20.04
- name: Check data record DOIs
run: ./run-tests.sh --data-dois

data-licenses:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.9

- name: Check Python code formatting
- name: Check data licenses
run: |
pip install black
./run-tests.sh --check-black
pip install --upgrade pip
./run-tests.sh --data-licenses
lint-pycodestyle:
runs-on: ubuntu-20.04
data-recids:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup node
uses: actions/setup-node@v4

- name: Check data record IDs
run: ./run-tests.sh --data-recids

data-slugs:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup node
uses: actions/setup-node@v4

- name: Check data docs slugs
run: ./run-tests.sh --data-slugs

data-types:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup node
uses: actions/setup-node@v4

- name: Check data types
run: ./run-tests.sh --data-types

format-black:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.9

- name: Check compliance with Python coding style conventions
- name: Check Python code formatting
run: |
pip install --upgrade pip
pip install pycodestyle
./run-tests.sh --check-pycodestyle
pip install black
./run-tests.sh --format-black
lint-pydocstyle:
runs-on: ubuntu-20.04
format-isort:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.9

- name: Check compliance with Python docstring conventions
- name: Check isort formatting
run: |
pip install --upgrade pip
pip install pydocstyle
./run-tests.sh --check-pydocstyle
pip install isort
./run-tests.sh --format-isort
check-fixtures:
runs-on: ubuntu-20.04
format-whitespace:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup node
uses: actions/setup-node@v1
with:
node-version: "14"
uses: actions/setup-node@v4

- name: Install Node dependencies
run: npm install -g jsonlint
- name: Check whitespace formatting
run: ./run-tests.sh --format-whitespace

- name: Check fixtures
run: ./run-tests.sh --check-fixtures

check-isort:
runs-on: ubuntu-20.04
lint-jsonlint:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Setup Node
uses: actions/setup-node@v4

- name: Check isort
- name: Lint JSON files
run: |
pip install --upgrade pip
pip install isort
./run-tests.sh --check-isort
npm install jsonlint --global
./run-tests.sh --lint-jsonlint
check-licenses:
runs-on: ubuntu-20.04
lint-pydocstyle:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.9

- name: Check licenses
- name: Lint Python docstring conventions
run: |
pip install --upgrade pip
./run-tests.sh --check-licenses
pip install pydocstyle
./run-tests.sh --lint-pydocstyle
lint-shellcheck:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Lint shell script static analysis
run: |
sudo apt-get install shellcheck
./run-tests.sh --lint-shellcheck
Loading

0 comments on commit d48b632

Please sign in to comment.