Three incremental tests failing with the cfg comparison

[stilscher]

Currently three incremental test cases fail when the cfg comparison is activated. The first two throw a Not_found exception, for the third the result is not as expected.

• 03/02 reluctant-int-annotation
• 03/03 reluctant-int-annotation-dyn
• 13/05 race-call-remove

Since the cfg comparison often breaks when new features are introduced and the only the AST comparison is considered when adapting the incremental pre-processing for the solver, I would suggest to run the incremental tests also with the cfg comparison in the regular regression test runs.

Testing 03/02 precision-annotation/reluctant-int-annotation      Status: 2 (exception)

  - Timings:
  - TOTAL                           0.007 s
  -   parse                           0.002 s
  -   convert to CIL                  0.003 s
  -   mergeCIL                        0.002 s
  -   compareCilFiles                 0.001 s
  -     createCFG                       0.001 s
  -       handle                          0.001 s
  -       iter_connect                    0.001 s
  -         computeSCCs                     0.001 s

Testing 03/03 precision-annotation/reluctant-int-annotation-dyn  Status: 2 (exception)

  - Timings:
  - TOTAL                           0.008 s
  -   parse                           0.003 s
  -   convert to CIL                  0.003 s
  -   mergeCIL                        0.002 s
  -   compareCilFiles                 0.001 s
  -     createCFG                       0.001 s
  -       handle                          0.001 s
  -       iter_connect                    0.001 s
  -         computeSCCs                     0.001 s

03/02 failed!

Expected assert, but registered nothing on reluctant-int-annotation-dyn:10
  __goblint_check(in == 17);
Expected fail, but registered nothing on reluctant-int-annotation-dyn:21
  __goblint_check(a); // FAIL!
Expected assert, but registered nothing on reluctant-int-annotation-dyn:23
  __goblint_check(a == 17);
03/03 failed!

Expected norace, but registered race on race-call-remove:7
  g++; // NORACE (unique thread)
Expected norace, but registered race on race-call-remove:12
  g++; // NORACE
13/05 failed!

3 test(s) failed: ["03/02 reluctant-int-annotation", "03/03 reluctant-int-annotation-dyn", "13/05 race-call-remove"]

[sim642]

Some of the incremental tests might have been written in a way that assumes AST comparison: for example, a change being made to a function with the purpose of reanalyzing the entire function from scratch.

[stilscher]

You could still achieve this by setting the configuration option force-reanalyze. I would find that more transparent because I think that there should not be any hidden assumption regarding the kind of comparison.