Skip to content

Expect test failures on stddev under Windows #293

@jmid

Description

@jmid

When experimenting with reenabling dune runtest under Windows (MinGW) in #292 I've noticed that they fail expect tests due to different stddev's being printed.

There's a reported difference with both OCaml 4.14, 5.0, and 5.2:

 stats dist:
-  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218496.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842
+  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218500.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842

Stddev is computed as follows:

    let stddev =
      Hashtbl.fold
        (fun i res m -> m +. (float_of_int i -. !avg) ** 2. *. float_of_int res)
        tbl 0.
      |> (fun s -> if !num>0 then s /. float_of_int !num else s)
      |> sqrt

and printed as:

    Printf.bprintf out
      "  num: %d, avg: %.2f, stddev: %.2f, median %d, min %d, max %d\n"
      !num !avg stddev !median min_idx max_idx;

I've not dug further into the reason at this point, but suspect that one of the involved floating point operations is causing the difference.

Full details from the 5.2 run:
https://github.com/c-cube/qcheck/actions/runs/9796388729/job/27050729649?pr=292

File "test/core/QCheck2_expect_test.expected", line 1, characters 0-0:
"C:\Program Files\Git\bin\git.exe" --no-pager diff --no-index --color=always -u --ignore-cr-at-eol _build/default/test/core/QCheck2_expect_test.expected _build/default/test/core/QCheck2_expect_test.output
diff --git a/_build/default/test/core/QCheck2_expect_test.expected b/_build/default/test/core/QCheck2_expect_test.output
index bce5b4b..47e0420 100644
--- a/_build/default/test/core/QCheck2_expect_test.expected
+++ b/_build/default/test/core/QCheck2_expect_test.output
@@ -1405,7 +1405,7 @@ stats dist:
 +++ Stats for int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218496.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842
+  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218500.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842

   -4611371852367564818..-4150222578233413331: #####################################################          5003
   -4150222578233413330..-3689073304099261843: #######################################################        5106
   -3689073304099261842..-3227924029965110355: ######################################################         5052
@@ -1430,7 +1430,7 @@ stats dist:
 +++ Stats for oneof int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002560.00, median 0, min -4611686018427387904, max 4611686018427387903
+  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002600.00, median 0, min -4611686018427387904, max 4611686018427387903

   -4611686018427387904..-4150517416584649089: #################                                               198
   -4150517416584649088..-3689348814741910273:                                                                   0
   -3689348814741910272..-3228180212899171457:                                                                   0
@@ -1459,7 +1459,7 @@ random seed: 153870556
 +++ Stats for int_dist_empty_bucket ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212928.00, median 9, min -4606614955625884935, max 4611686018427387903
+  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212900.00, median 9, min -4606614955625884935, max 4611686018427387903

   -4606614955625884935..-4145699906923221320: ##                                                               27
   -4145699906923221319..-3684784858220557704: ##                                                               22
   -3684784858220557703..-3223869809517894088: ##                                                               29
File "test/core/QCheck_expect_test.expected", line 1, characters 0-0:
"C:\Program Files\Git\bin\git.exe" --no-pager diff --no-index --color=always -u --ignore-cr-at-eol _build/default/test/core/QCheck_expect_test.expected _build/default/test/core/QCheck_expect_test.output
diff --git a/_build/default/test/core/QCheck_expect_test.expected b/_build/default/test/core/QCheck_expect_test.output
index ed35317..e7f0b76 100644
--- a/_build/default/test/core/QCheck_expect_test.expected
+++ b/_build/default/test/core/QCheck_expect_test.output
@@ -1361,7 +1361,7 @@ stats dist:
 +++ Stats for int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218496.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842
+  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218500.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842

   -4611371852367564818..-4150222578233413331: #####################################################          5003
   -4150222578233413330..-3689073304099261843: #######################################################        5106
   -3689073304099261842..-3227924029965110355: ######################################################         5052
@@ -1386,7 +1386,7 @@ stats dist:
 +++ Stats for oneof int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002560.00, median 0, min -4611686018427387904, max 4611686018427387903
+  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002600.00, median 0, min -4611686018427387904, max 4611686018427387903

   -4611686018427387904..-4150517416584649089: #################                                               198
   -4150517416584649088..-3689348814741910273:                                                                   0
   -3689348814741910272..-3228180212899171457:                                                                   0
@@ -1415,7 +1415,7 @@ random seed: 153870556
 +++ Stats for int_dist_empty_bucket ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212928.00, median 9, min -4606614955625884935, max 4611686018427387903
+  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212900.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions