Skip to content

Update coverage tool to manually filter some Fortran90 continuation lines#604

Merged
apcraig merged 2 commits into
CICE-Consortium:masterfrom
apcraig:lcov04
May 27, 2021
Merged

Update coverage tool to manually filter some Fortran90 continuation lines#604
apcraig merged 2 commits into
CICE-Consortium:masterfrom
apcraig:lcov04

Conversation

@apcraig

@apcraig apcraig commented May 26, 2021

Copy link
Copy Markdown
Contributor

PR checklist

  • Short (1 sentence) summary of your PR:
    Update the coverage implementation, manually filter out some lines from the coverage tool
  • Developer(s):
    apcraig
  • Suggest PR reviewers from list in the column to the right.
  • Please copy the PR test results link or provide a summary of testing completed below.
    One source code line was changed to help the filter. Ran a quick suite on cheyenne with 3 compilers, those test pass fine, https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash_forks#a09e51fb161b59af72f63f39c6247c9698a2ace2
  • How much do the PR code changes differ from the unmodified code?
    • bit for bit
    • different at roundoff level
    • more substantial
  • Does this PR create or have dependencies on Icepack or any other models?
    • Yes
    • No
  • Does this PR add any new test cases?
    • Yes
    • No
  • Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/. A test build of the technical docs will be performed as part of the PR testing.)
    • Yes
    • No, does the documentation need to be updated at a later time?
      • Yes
      • No
  • Please provide any additional information or relevant details below:

The gcov/lcov tool occasionally generates false negatives or incorrect
statistics with respect to coverage for Fortran 90 continuation lines (using "&").
In some cases, within a single continuation block, some lines have hits which
are correctly counted while other lines have misses which are incorrect and
also counted. This seems to be partly associate with the lcov -a feature that we
use to aggregate multiple tests, but it also probably is ultimately created by gcov
because it doesn't seem to handle skipped vs missed lines entirely consistently.

With this modification, --coverage creates a temporary sandbox where the tests are
run. Prior to running the tests, lcov_modify_source.sh is invoked.
" ! LCOV_EXCL_LINE" (a special defined string in lcov) is added to the end
of some source code lines to request that gcov exclude them. The lines excluded
are Fortran 90 continuation lines. In all cases the first and last line of a
continuation block and lines that continue after an "if" are never excluded.
This seems to improve the accuracy of the coverage output overall.

Several other attempts were made to modify the lcov and geninfo perl scripts to
handle the Fortran90 continuation line filtering there without success, so this
solution was implemented.

To see a before and after for the same code,

before: https://apcraig.github.io/lcov_cice_210520-185156:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/index.html

after: https://apcraig.github.io/lcov_cice_210526-003153:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/index.html

A good place to see the error/differences is after line 255 of

https://apcraig.github.io/lcov_cice_210520-185156:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/cicecore/cicedynB/dynamics/ice_dyn_evp.F90.gcov.html

and

https://apcraig.github.io/lcov_cice_210526-003153:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/cicecore/cicedynB/dynamics/ice_dyn_evp.F90.gcov.html

In the first case, 20+ lines of code are recorded as missed even though they are hit. In the updated coverage output, most of those lines are skipped and the statistics are cleaned up significantly. Coverage for the routine goes from 73% to 90%, and I think more accurately reflects the true coverage of the routine. In addition, manually searching for missing coverage is easier if some of the false negatives are removed from the output. The total coverage went from 70.3% to 71.6% with this change which suggest this problem is fairly localized.

The gcov/lcov tool occasionally generates false negatives or incorrect
statistics with respect to coverage for Fortran 90 continuation lines (using "&").
In some cases, within a single continuation block, some lines have hits which
are correctly counted while other lines have misses which are incorrect and
also counted.  This seems to be partly associate with the lcov -a feature that we
use to aggregate multiple tests, but it also probably is ultimately created by gcov
because it doesn't seem to handle skipped vs missed lines entirely consistently.

With this modification, --coverage creates a temporary sandbox where the tests are
run.  Prior to running the tests, lcov_modify_source.sh is invoked.
"  ! LCOV_EXCL_LINE" (a special defined string in lcov) is added to the end
of some source code lines to request that gcov exclude them.  The lines excluded
are Fortran 90 continuation lines.  In all cases the first and last line of a
continuation block and lines that continue after an "if" are never excluded.
This seems to improve the accuracy of the coverage output overall.

Several other attempts were made to modify the lcov and geninfo perl scripts to
handle the Fortran90 continuation line filtering there without success, so this
solution was implemented.

@eclare108213 eclare108213 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting!

@apcraig apcraig merged commit bd512d4 into CICE-Consortium:master May 27, 2021
@phil-blain

Copy link
Copy Markdown
Member

Nice approach @apcraig, simple and effective. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants