Skip to content

[CI] Port CMake Codecov job from Codebuild to Github Actions. #5809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 12, 2021
Merged

[CI] Port CMake Codecov job from Codebuild to Github Actions. #5809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1e18a11
Port CMake Codecov job from Codebuild to Github Actions.
NlightNFotis Feb 9, 2021
892af6b
Remove codebuild codecov jobs.
NlightNFotis Feb 10, 2021
f02904e
Fix issue with coverage targets not being listed as dependencies
NlightNFotis Feb 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/pull-request-checks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,3 +385,34 @@ jobs:
docker run -v ${PWD}/.github/workflows/smoke_test_assets:/mnt/smoke -t cbmc jbmc --classpath /mnt/smoke Test
- name: Smoke test goto-analyzer
run: docker run -v ${PWD}/.github/workflows/smoke_test_assets:/mnt/smoke -t cbmc goto-analyzer /mnt/smoke/test.goto --unreachable-functions

codecov-coverage-report:
runs-on: ubuntu-20.04
steps:
- name: Clone repository
uses: actions/checkout@v2
with:
submodules: recursive
- name: Download testing and coverage dependencies
env:
# This is needed in addition to -yq to prevent apt-get from asking for
# user input
DEBIAN_FRONTEND: noninteractive
run: |
sudo apt-get update
sudo apt-get install --no-install-recommends -y g++ gcc binutils flex bison cmake maven jq libxml2-utils openjdk-11-jdk-headless lcov
- name: Configure CMake CBMC build with coverage instrumentation parameters
run: cmake -S . -Bbuild -Denable_coverage=1 -Dparallel_tests=2 -DCMAKE_CXX_COMPILER=/usr/bin/g++
- name: Execute CMake CBMC build
run: cmake --build build --target coverage -- -j2
- name: Collect coverage statistics
run: |
lcov --capture --directory build --output-file lcov.info
lcov --remove lcov.info '/usr/*' --output-file lcov.info
Copy link
Contributor

@piotr-grabalski piotr-grabalski Feb 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there anything else that could be removed from trace file (e.g. miniSAT/or other SAT solver files)?

Copy link
Contributor Author

@NlightNFotis NlightNFotis Feb 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I'm not sure to be honest, and I'd rather not play around with the contents of the files for now, as this is supposed to be just a port with functionality parity.

Attempting any changes like this would cause us to need extra time to validate/test those changes. I am happy however to add them on a todo list for later inspection/implementation. Does that work for you, @piotr-grabalski ?

Copy link
Contributor

@piotr-grabalski piotr-grabalski Feb 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is OK for me.

I've been wondering if the report could be more accurate and display the test coverage of the code that is a part of CBMC rather than some third-party stuff that we are not involved in. I don't know how much of external code is going into the statistics so, as you said, more investigation will be required.

- name: Upload coverage statistics to Codecov
uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./lcov.info
fail_ci_if_error: true
Copy link
Contributor

@hannes-steffenhagen-diffblue hannes-steffenhagen-diffblue Feb 12, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Last concern here: Is this a required status check? I don't think it should be, given how codecov has been a bit flaky in the past. IMHO codecov checks should be advisory.

Copy link
Contributor Author

@NlightNFotis NlightNFotis Feb 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've marked it fail_ci_if_error so that less coverage (or an error) will cause the specific check to fail, marking it with a red X, so that it's behaviourally consistent with the previous codecov jobs we had.

However, it's not marked as a required check in Github, and as a result, a possible failure of this job doesn't block the merging of a PR.

verbose: true
2 changes: 2 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,8 @@ if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang" OR
"$<TARGET_FILE:jbmc>"
"$<TARGET_FILE:jdiff>"
"$<TARGET_FILE:smt2_solver>"
"$<TARGET_FILE:symtab2gb>"
Copy link
Contributor

@hannes-steffenhagen-diffblue hannes-steffenhagen-diffblue Feb 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW all this "$<TARGET_FILE:` stuff in here isn't invalid per se, but it also doesn't help. Just listing the target names here would work just as well and look less confusing.

Copy link
Collaborator

@tautschnig tautschnig Feb 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hannes-steffenhagen-diffblue Would it be worth doing a cleanup across the code base? Would you mind driving this as my CMake expertise is, well, not really existent?

"libour_archive.a"
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
)
endif()
Expand Down
40 changes: 0 additions & 40 deletions buildspec-linux-cmake-gcc-cov.yml
View file
Open in desktop

This file was deleted.

47 changes: 0 additions & 47 deletions buildspec-linux-make-gcc-cov.yml
View file
Open in desktop

This file was deleted.