Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add linter sql and markdown #316

Merged
merged 60 commits into from
Sep 21, 2022

Conversation

vdelacruzb
Copy link
Contributor

@vdelacruzb vdelacruzb commented Sep 14, 2022

Description

Shortcut

Add linters for sql and md files.

Some changes to take into account:

  • Adding sqlfluff support has been added through a python script to ease the control over the text to lint. Sqlfluff complains when finding @ so the python code helped to these replacements without intermediate temp files and also to control in which scripts it's failing. I had to add support to python in common for some providers as SF and BQ because of this.
  • A space is inserted between markdowns headers and body.
  • Added jest.config.js it was required for BQ and SF tests.
  • Tox.ini was also required to skip some issues with max_length but it was enough with adding some exceptions to the flake8 command to make it work.
  • Variables are forced to lower everything else to upper. Existing camelcase variables and arguments are being replaced by snake case.
  • In case the sql linter is stuck it exits 1 and return the failed file. Sometimes is not trivial to know why it's stuck and it will require some manual fixes before everything runs ok, normally it could be an indentation issue.
  • Everyone should use tab size equal to 4 spaces. Many scripts have been found with 2 spaces, some with 6 and another with 8.

Missing linting on pre-commit. Right now linting could take a while for some modules when we add that feature let's make sure only diff files are linted.

Type of change

  • Feature

Acceptance

  1. Go to any SQL file not added to sqlfluffignore at any provider. Replace a keywork with lower case, example 'CREATE TABLE' by 'create table'.
  2. Use make lint.
  3. See how it gets back to 'CREATE TABLE'.
  4. Remove a space between a header and body at any markdown file.
  5. Use make lint.
  6. See how the space is back.

@shortcut-integration
Copy link

This pull request has been linked to Shortcut Story #250230: Add linter SQL and Markdown.

@vdelacruzb vdelacruzb requested a review from Jesus89 September 19, 2022 17:11
@vdelacruzb vdelacruzb marked this pull request as ready for review September 19, 2022 17:11
@vdelacruzb vdelacruzb requested a review from a team as a code owner September 19, 2022 17:11
- `yarn`: https://yarnpkg.com/ (v1.22)
- `bq`: https://cloud.google.com/bigquery/docs/bq-command-line-tool
- `gsutil`: https://cloud.google.com/storage/docs/gsutil (v5.5)
- `make`: <https://www.gnu.org/software/make/>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add Python3

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vdelacruzb vdelacruzb merged commit 2700874 into master Sep 21, 2022
@vdelacruzb vdelacruzb deleted the chore/sc-250230/add-linter-sql-and-markdown branch September 21, 2022 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants