chore(ci): replace flake8 with Ruff as a linter

[leandrodamascena]

Issue number: #2380

Summary

Changes

This PR introduces the incredible project Ruff as the replacement for flake8 and isort as our linter. By adopting Ruff, we gain access to numerous new checks that greatly enhance code maintainability, standards, docstyle, docstring quality, and more, benefitting from over 500 rules provided by Ruff. Also, Ruff is EXTREMELY fast compared to flake8.

1 - Performance
Ruff is fast, Ruff is very fast!! 😳. I did a comparison test running make lint using flake8 and Ruff. Ruff runs in less than 1 second, while flake8 takes around 25s to run.

FLAKE8

❯ time make lint                                                                                                 
poetry run isort aws_lambda_powertools tests examples
poetry run black aws_lambda_powertools tests examples
All done! ✨ 🍰 ✨
307 files left unchanged.
poetry run flake8 aws_lambda_powertools tests examples
make lint  25.74s user 0.38s system 360% cpu 7.253 total

RUFF

❯ time make lint
poetry run black aws_lambda_powertools tests examples
All done! ✨ 🍰 ✨
307 files left unchanged.
poetry run ruff aws_lambda_powertools tests examples
make lint  0.89s user 0.12s system 100% cpu 1.000 total

We still gain performance when running locally on the pre-commit hook, but since pre-commit only runs on modified files, the difference isn't too big. While Ruff takes less than 1s to inspect 10 files, flake8 takes 3s, but ok, we still gain performance and code standards.

2 - Black and isort
I suggest keeping black and removing isort for now. Ruff covers most (if not all) cases that isort covers, but the documentation suggests we integrate Ruff and black

3 - pydocstyle (D)
We are failing most new rules checked by Ruff - https://beta.ruff.rs/docs/rules/#pydocstyle-d. If we enable all rules in pydocstyle, we will have around 5k errors to fix 🥲. I think we can check the most important and necessary, activate and fix the files.

4 - Fixing files for new checks
To provide clear explanations for the additions and removals of features supported by Ruff, I will comment out specific lines in the modified files. Considering the extensive changes required to be compliant with the new checks, it might be good to address them through separate PRs for better organization and review.

User experience

There is no change in the user experience.

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[leandrodamascena]

Hi friends! Before moving forward with this PR, we need to set things straight. I made a lot of comments that we need to address.

[codecov-commenter]

Codecov Report

Patch and project coverage have no change.

Comparison is base (fea3f6a) 97.21% compared to head (cbcb771) 97.21%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #2495   +/-   ##
========================================
  Coverage    97.21%   97.21%           
========================================
  Files          158      158           
  Lines         7370     7370           
  Branches       537      537           
========================================
  Hits          7165     7165           
  Misses         158      158           
  Partials        47       47           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[heitorlessa]

GREAT One. IIRC Ruff also replaces Bandit for SAST - could you check whether it accepts an existing baseline? If it does, you can definitely remove bandit too... if it doesn't, we can investigate in another PR.

Agree on pydoc. There's another big chunk of work for API documentation to be integrated into our User Guide docs, and that requires some cleaning and additions ;) We can do progressively.

https://github.com/aws-powertools/powertools-lambda-python/blob/develop/Makefile#L78

[heitorlessa]

forgot to ask as the diff is huge but you prolly are removing all flake8- deps now, right?

[leandrodamascena]

forgot to ask as the diff is huge but you prolly are removing all flake8- deps now, right?

I almost forgot about that, @heitorlessa! Done.

If you don't have any consideration, I think we can merge this PR and I'll open the others to enable rules and fix files.

[rubenfonseca]

WOOOOOOOOOW THANK YOU FOR TAKING TIME TO DO THIS!! Looks awesome!