Improving ruff setup (#224)

castelao · web-flow · commit 5dda178e798d · 2024-06-28T15:51:50.000-06:00
* Redefining ruff, starting with target-version = "py38" Let's remove all rules so far, and re-introduce those by following the definitions of pylint. The first rule is target-version infered from the package definition that requires-python >= 3.8. * Excluding CSV and ref.*?py Patterns defined at .python-lint lines 10 & 14. * Disabling rules according to .python-lint Current setup defined between lines 57-105. * max-nested-blocks Defining `max-nested-blocks` as .pyhton-lint line 141. * Confirming with expected-line-ending-format While pylint defines `expected-line-ending-format`, ruff uses `line-ending` instead. Defined in .python-lint line 254. * Defining line-lenght as .python-lint line 267 * indent-width * multiple-statements-on-one-line-colon Equivalent to single-line-if-stmt at line 278 of .python-lint * task-tags In reference to notes, line 291 of .python-lint * In respect t ologging-modules, line 285 of .python-lint We don't need that with ruff, but here is a placeholder if we want to add more than default `logger` in the future. * unused imports Addressing "init-import=no" in line 402 from .python-lint * Defining max-args Interestingly, max-args is defined as 5 in line 434, but PLR0913 rule is ignored. * Defining DESIGN explicit setup To reflect DESIGN section in .python-lint, pages 431-463. * Closest option of satisfy EXCEPTIONS Ruff doesn't implement the exactly the same, but E722 is good enough if not even better. * Re-introducing all (W) Warning rules * Re-introducing (Q) flake8-quotes rules sup3r do not follow: - Q000 bad-quotes-inline-string - Q004 unnecessary-escaped-quote * Re-introducing (I) isort rules sup3r do not follow I001: unsorted-imports. * Re-introducing (NPY) NymPy-specific rules sup3r doesn't follow NPY002, but we might want to re-consider that in the future. * style: Conforming with UP039 * Re-introducing (UP) pyupgrade rules sup3r doesn't follow: - UP009: utf8-encoding-declaration - UP015: redundant-open-modes - UP032: f-string * Re-introducing (A) flake8-builtins sup3r doesn't conform with, but should reconsider that in the future: - A001: builtin-variable-shadowing - A002: builtin-argument-shadowing * Re-introducing (ARG) flake8-unused-arguments rules sup3r doesn't conform with: - ARG002: unused-method-argument - ARG003: unused-class-method-argument - ARG004: unused-static-method-argument - ARG005: unused-lambda-argument * Re-introducing (E) pycodestyle rules * Re-introducing (F) Pyflakes rules * Re-introducing (COM) flake8-commas rules sup3r doesn't conform with: - COM812: missing-trailing-comma * Introducing (N) pep8-naming rules sup3r doesn't conform with: - N802: invalid-function-name - N803: invalid-argument-name - N806: non-lowercase-variable-in-function * Introducing (D) pydocstyle rules sup3r doesn't conform with: - D105: undocumented-magic-method - D200: fits-on-one-line - D202: no-blank-line-after-function - D204: one-blank-line-after-class - D205: blank-line-after-summary - D207: under-indentation - D209: new-line-after-last-paragraph - D400: ends-in-period - D401: non-imperative-mood - D404: docstring-starts-with-this * Introducing (PL) Pylint rules sup3r doesn't conform with: - PLR2004: magic-value-comparison - PLW2901: redefined-loop-name * Running ruff with GA * Nothing is fixable for now * Using ruff with pre-commmit * style: Conforming with NPY003 & NPY201 * style: Conforming with COM819 * Forgot to include: quote-style & indent-style * Updating requirements on ruff * GA outputs ruff with github style * Adding (C4) flake8-comprehensions As suggested by @bnb32 sup3r doesn't conform with: - C408: unnecessary-collection-call - C414: unnecessary-double-cast-or-process * Adding rule (C90) mccabe As suggested by @bnb32 Using max-complexity as previously defined for flake8 (lintesrs/.flake8) as equal to 12. * Adding convention (C) and flake8-logging (LOG) @bnb32 any other rule that you would like to include? * validate all codebase with super-linter There were some misterious situations where super-linter was not properly checking the code. Paul mentioned a setup that limits checks to the very last commit only. I expect that VALIDATE_ALL_CODEBASE should address that. * Adding (SIM) flake8-simplify rules As suggested by @bnb32. sup3r doesn't conform with: - SIM108: if-else-block-instead-of-if-exp - SIM117: multiple-with-statements - SIM118: in-dict-keys - SIM211: if-expr-with-false-true * Adding (PERF) Perflint rules As suggested by @bnb32 sup3r doesn't conform with: - PERF102: incorrect-dict-iterator - PERF203: try-except-in-loop - PERF401: manual-list-comprehension * We don't need to run on the full code base anymore With the pull_request trigger we don't need VALIDATE_ALL_CODEBASE set to true anymore. Otherwise it's running twice. Thanks @ppinchuk! * clean: Removing implicit rules While conforming with the legacy setup, I added a few explicit specific rules but in the final version didn't make sense anymore since those are now implicit. For instance, no need of "E701" is added "E".
diff --git a/.github/workflows/linter.yml b/.github/workflows/linter.yml
@@ -4,6 +4,8 @@ on:
   push:
     branches-ignore:
       - 'gh-pages'
+  pull_request:
+    branches: [main]
 
 jobs:
   build:
@@ -28,3 +30,13 @@ jobs:
           VALIDATE_YAML: false
           DEFAULT_BRANCH: main
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  ruff:
+    name: Ruff
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: chartboost/ruff-action@v1
+        with:
+          version: 0.4.10
+          args: check --output-format=github
+          src: "./sup3r ./tests"
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -12,3 +12,7 @@ repos:
     rev: v3.1.0
     hooks:
     -   id: pylint
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.4.10
+  hooks:
+    - id: ruff
diff --git a/pixi.lock b/pixi.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -71,49 +71,172 @@ line-length = 79
 indent-width = 4
 
 target-version = "py38"
+exclude = [
+  "CSV",
+  "ref.*?py",
+]
 
 [tool.ruff.lint]
-fixable = ["ALL"]
-preview = true
+fixable = []
+# preview = true
+# logger-objects = []
+task-tags = ["TODO", "FIXME", "XXX"]
 select = [
+  "A", # flake8-builtins
+  "ARG", # flake8-unused-arguments
+  "C",
+  "C4", # flake8-comprehensions
+  "C90", # mccabe
+  "COM", # flake8-commas
+  "D", # pydocstyle
   "E", # pycodestyle
   "F", # Pyflakes
+  "G", # flake8-logging-format
+  "I", # isort
+  "LOG", # flake8-logging
+  "N", # pep8-naming
+  "NPY", # numpy-specific
+  "PERF", # Perflint
+  "PL", # Pylint
+  "Q", # flake8-quotes
+  "SIM", # flake8-simplify
+  "UP", # pyupgrade
   "W", # Warning
   ]
 
 ignore = [
-  "B008", # function-call-in-default-argument
-  "B024", # abstract-base-class-without-abstract-method
-  "B028", # no-explicit-stacklevel
-  "B905", # zip-without-explicit-strict
+  # Currently don't conform but we might want to reconsider
+  "A001", # builtin-variable-shadowing
+  # Currently don't conform but we might want to reconsider
+  "A002", # builtin-argument-shadowing
+  "ARG002", # unused-method-argument
+  "ARG003", # unused-class-method-argument
+  "ARG004", # unused-static-method-argument
+  "ARG005", # unused-lambda-argument
+#  "B008", # function-call-in-default-argument
+#  "B024", # abstract-base-class-without-abstract-method
+#  "B028", # no-explicit-stacklevel
+#  "B905", # zip-without-explicit-strict
+  "C408", # unnecessary-collection-call
+  "C414", # unnecessary-double-cast-or-process
+  "COM812", # missing-trailing-comma
   "D105", # undocumented-magic-method
+  "D200", # fits-on-one-line
   "D202", # no-blank-line-after-function
+  "D204", # one-blank-line-after-class
   "D205", # blank-line-after-summary
+  "D207", # under-indentation
   "D209", # new-line-after-last-paragraph
-  "D212", # multi-line-summary-first-line
-  "D213", # multi-line-summary-second-linek
+#  "D212", # multi-line-summary-first-line
+#  "D213", # multi-line-summary-second-linek
   "D400", # ends-in-period
   "D401", # non-imperative-mood
-  "D413", # blank-line-after-last-section
-  "D415", # ends-in-punctuation
-  "E902", # io-error
+  "D404", # docstring-starts-with-this
+#  "D413", # blank-line-after-last-section
+#  "D415", # ends-in-punctuation
+#  "E902", # io-error
+  "FIX001",	# line-contains-fixme
+  # We currently don't conform but we might want to reconsider
+  "G001", # logging-string-format
+  # We currently don't conform but we might want to reconsider
+  "G004", # logging-f-string
+  "I001", # unsorted-imports
+  "N802", # invalid-function-name
+  "N803", # invalid-argument-name
+  "N806", # non-lowercase-variable-in-function
+  # Consider conforming with NPY002
+  "NPY002", # numpy-legacy-random
+  "PERF102", # incorrect-dict-iterator
+  "PERF203", # try-except-in-loop
+  "PERF401", # manual-list-comprehension
+  "PLR0904", # too-many-public-methods
+  "PLR0912", # too-many-branches
   "PLR0913", # too-many-arguments
+  "PLR0914", # too-many-locals
+  "PLR0915", # too-many-statements
+  "PLR1702", # too-many-nested-blocks
+  "PLR1704", # redefined-argument-from-local
+  "PLR2004", # magic-value-comparison
+  "PLW1514", # unspecified-encoding
+  "PLW2901", # redefined-loop-name
+  "Q000", # bad-quotes-inline-string
+  "Q004", # unnecessary-escaped-quote
+  "SIM108", # if-else-block-instead-of-if-exp
+  "SIM117", #multiple-with-statements
+  "SIM118", # in-dict-keys
+  "SIM211", # if-expr-with-false-true
   "UP009", # utf8-encoding-declaration
+  "UP015", # redundant-open-modes
   "UP032", # f-string
-  "UP038" # non-pep604-isinstance
+#  "UP038" # non-pep604-isinstance
   ]
+# Ignored in pylint setup but missing on ruff. We shall delete from the
+# following lines what is not intended to follow anymore.
+# arguments-renamed
+# consider-using-f-string
+# raw-checker-failed
+# bad-inline-option
+# locally-disabled
+# file-ignored
+# suppressed-message
+# useless-suppression
+# deprecated-pragma
+# protected-access
+# redefined-outer-name
+# redefined-builtin
+# broad-except
+# logging-format-interpolation
+# logging-fstring-interpolation
+# wrong-import-order
+# wrong-import-position
+# relative-beyond-top-level
+# too-many-instance-attributes
+# too-few-public-methods
+# invalid-name
+# import-error
+# try-except-raise
+# no-else-raise
+# no-else-return
+# unexpected-keyword-arg
+# no-value-for-parameter
+# too-many-lines
+# arguments-differ
+# import-outside-toplevel
+# super-init-not-called
+# isinstance-second-argument-not-valid-type
+# inconsistent-return-statements
+# no-else-break
+# too-many-function-args
+# redundant-keyword-arg
+# c-extension-no-member
+
+[tool.ruff.lint.mccabe]
+# Flag errors (`C901`) whenever the complexity level exceeds 5.
+max-complexity = 12
 
 [tool.ruff.lint.per-file-ignores]
 "__init__.py" = [
   "F401", # unused-import
   ]
-"docs/source/conf.py" = [
-  "E402", # unused-import
-  ]
+#"docs/source/conf.py" = [
+#  "E402", # unused-import
+#  ]
+
+[tool.ruff.lint.pylint]
+max-args = 5 # (PLR0913) Maximum number of arguments for function / method
+max-bool-expr = 5 # ( PLR0916) Boolean in a single if statement
+max-branches=12 # (PLR0912) branches allowed for a function or method body
+max-locals=15 # (PLR0912) local variables allowed for a function or method body
+max-nested-blocks = 5 # (PLR1702) nested blocks within a function or method body
+max-public-methods=20 # (R0904) public methods allowed for a class
+max-returns=6 # (PLR0911) return statements for a function or method body
+max-statements=50 # (PLR0915) statements allowed for a function or method body
 
 [tool.ruff.format]
 quote-style = "single"
 indent-style = "space"
+# Consider adopting "lf" instead
+line-ending = "auto"
 
 [tool.ruff.lint.pydocstyle]
 convention = "numpy"
@@ -167,4 +290,4 @@ pytest = ">=5.2"
 [tool.pixi.feature.dev.dependencies]
 build = ">=0.6"
 twine = ">=5.0"
-ruff = ">=0.2"
+ruff = ">=0.4"
diff --git a/sup3r/bias/mixins.py b/sup3r/bias/mixins.py
@@ -10,7 +10,7 @@
 logger = logging.getLogger(__name__)
 
 
-class FillAndSmoothMixin():
+class FillAndSmoothMixin:
     """Fill and extend parameters for calibration on missing positions"""
     def fill_and_smooth(self,
                         out,
diff --git a/tests/data_handling/test_data_handling_h5.py b/tests/data_handling/test_data_handling_h5.py
@@ -29,7 +29,7 @@
 val_split = 0.2
 dh_kwargs = {'target': target, 'shape': shape, 'max_delta': 20,
              'sample_shape': sample_shape,
-             'lr_only_features': ('BVF*m', 'topography',),
+             'lr_only_features': ('BVF*m', 'topography'),
              'temporal_slice': slice(None, None, 1),
              'worker_kwargs': {'max_workers': 1}}
 bh_kwargs = {'batch_size': 8, 'n_batches': 20,
diff --git a/tests/data_handling/test_data_handling_nc.py b/tests/data_handling/test_data_handling_nc.py
@@ -29,7 +29,7 @@
 dh_kwargs = dict(target=target,
                  shape=shape,
                  max_delta=20,
-                 lr_only_features=('BVF*m', 'topography',),
+                 lr_only_features=('BVF*m', 'topography'),
                  sample_shape=sample_shape,
                  temporal_slice=slice(None, None, 1),
                  worker_kwargs=dict(max_workers=1),
diff --git a/tests/forward_pass/test_forward_pass_exo.py b/tests/forward_pass/test_forward_pass_exo.py
@@ -190,7 +190,7 @@ def test_fwp_multi_step_spatial_model_topo_noskip():
         max_workers = 1
         fwp_chunk_shape = (4, 4, 8)
         s_enhancements = [2, 2, 1]
-        s_enhance = np.product(s_enhancements)
+        s_enhance = np.prod(s_enhancements)
 
         exo_kwargs = {
             'topography': {
@@ -299,7 +299,7 @@ def test_fwp_multi_step_model_topo_noskip():
         max_workers = 1
         fwp_chunk_shape = (4, 4, 8)
         s_enhancements = [2, 2, 3]
-        s_enhance = np.product(s_enhancements)
+        s_enhance = np.prod(s_enhancements)
         t_enhance = 4
 
         exo_kwargs = {
@@ -955,7 +955,7 @@ def test_fwp_multi_step_model_multi_exo():
         max_workers = 1
         fwp_chunk_shape = (4, 4, 8)
         s_enhancements = [2, 2, 3]
-        s_enhance = np.product(s_enhancements)
+        s_enhance = np.prod(s_enhancements)
         t_enhance = 4
 
         exo_kwargs = {
diff --git a/tests/output/test_output_handling.py b/tests/output/test_output_handling.py
@@ -208,7 +208,7 @@ def test_h5_collect_mask(log=False):
         (out_files, data, _, _, features, _, _, _, _, _, _) = out
 
         CollectorH5.collect(out_files, fp_out, features=features)
-        indices = np.arange(np.product(data.shape[:2]))
+        indices = np.arange(np.prod(data.shape[:2]))
         indices = indices[slice(-len(indices) // 2, None)]
         removed = []
         for _ in range(10):
diff --git a/tests/utilities/test_utilities.py b/tests/utilities/test_utilities.py
@@ -195,7 +195,7 @@ def test_weighted_box_sampler():
                 or chunks[5][0] <= slice_3.start <= chunks[5][-1])
 
     shape = (1, 1)
-    weights = np.zeros(np.product(data.shape))
+    weights = np.zeros(np.prod(data.shape))
     weights_4 = weights.copy()
     weights_4[5] = 1
 
@@ -366,7 +366,7 @@ def test_t_coarsen():
     """Test temporal coarsening of 5D array"""
     t_enhance = 4
     hr_shape = (3, 10, 10, 48, 2)
-    arr = np.arange(np.product(hr_shape)).reshape(hr_shape).astype(float)
+    arr = np.arange(np.prod(hr_shape)).reshape(hr_shape).astype(float)
 
     # test 4x temporal enhancement averaging
     arr_lr = temporal_coarsening(arr, t_enhance=t_enhance, method='average')
