Skip to content

Add CI pipeline step and script that validates all files in the repo contain ASCII-only bytes and no UTF-8 BOM #4134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ahsonkhan merged 13 commits into from
Jan 26, 2023
Merged

Add CI pipeline step and script that validates all files in the repo contain ASCII-only bytes and no UTF-8 BOM #4134

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a0740e0
Add CI pipeline step and script that validates all files in the repo …
ahsonkhan Nov 28, 2022
e600ed2
Update platform-matrix.json to turn on CHECK_ASCII_CHARACTERS
ahsonkhan Nov 28, 2022
5271408
Fix missing comma in json.
ahsonkhan Nov 28, 2022
b84f4b1
Update indentation in the ci.test.yml for the bash script
ahsonkhan Nov 28, 2022
ba207c1
Exclude the .github folder and test recording json files.
ahsonkhan Nov 28, 2022
c4aae67
Remove "recordings" exclusion since it doesn't exist anymore.
ahsonkhan Jan 17, 2023
a7a7aa0
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-cpp int…
ahsonkhan Jan 17, 2023
ae5083a
Fixed UTF-8 with BOM and other non-ASCII character issues within the
ahsonkhan Jan 17, 2023
7ba5c3e
Exclude nlohmann json tests from ASCII validation check.
ahsonkhan Jan 17, 2023
e77ee6c
Log an intro to the CI console for the validation step.
ahsonkhan Jan 17, 2023
bee644a
Remove codecoverage.cmake exclusion.
ahsonkhan Jan 17, 2023
5e5d9c2
Remove exclusion for docs and tools directories.
ahsonkhan Jan 17, 2023
a654760
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-cpp int…
ahsonkhan Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion eng/pipelines/templates/jobs/ci.tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ jobs:
- template: /eng/pipelines/templates/steps/vcpkg.yml

# Validate all the files are formatted correctly according to the
# .clang-format file. This step runs on linux only only and assumes that
# .clang-format file. This step runs on linux only and assumes that
# clang-format-11 is installed.
- bash: |
# Run clang-format recursively on each source and header file within the repo sdk folder.
Expand All @@ -132,6 +132,29 @@ jobs:
displayName: Validate Clang Format
condition: and(succeededOrFailed(), eq(variables['CHECK_CLANG_FORMAT'], 1))

# Validate all the files are saved as ASCII only without a UTF-8 BOM.
- bash: |
# Run grep recursive excluding git folder and known expected files and save a file with results.
grep -I -P -n "[^\x00-\x7F]" -r --exclude-dir ".git" --exclude-dir ".github" --exclude-dir "recordings" --exclude-dir "vcpkg_installed" --exclude-dir "_deps" --exclude-dir "docs" --exclude-dir "tools" --exclude "*CodeCoverage.cmake*" --exclude "grepResults" . > grepResults

# Display results to console.
cat grepResults

# Each result will produce one line, count how many lines were found.
files_with_non_ascii=($(wc -l < grepResults))

# Show info about the total files that needs attention.
echo Files found with non-ASCII characters: $files_with_non_ascii

# Remove the grepResults file.
rm grepResults

# Return the count. When greater than 0, the step will fail.
exit $files_with_non_ascii

displayName: Validate Characters are ASCII
condition: and(succeededOrFailed(), eq(variables['CHECK_ASCII_CHARACTERS'], 1))

- ${{ each artifact in parameters.Artifacts }}:
- template: /eng/common/pipelines/templates/steps/set-test-pipeline-version.yml
parameters:
Expand Down
3 changes: 2 additions & 1 deletion eng/pipelines/templates/stages/platform-matrix.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@
},
"BuildSettings": {
"clang-11": {
"CHECK_CLANG_FORMAT": "1"
"CHECK_CLANG_FORMAT": "1",
"CHECK_ASCII_CHARACTERS": "1"
},
"included_release": {
"CMAKE_BUILD_TYPE": "Release"
Expand Down