Support coq-native by passing through COQEXTRAFLAGS

[JasonGross]

Since compcert < 3.13 doesn't install .coq-native files, when coq-native is installed, vcfloat should avoid looking for .coq-native files.

[silene]

What do you mean? CoqInterval does install the .coq-native directories:

$ find .opam/system/lib/coq/user-contrib/Interval/ -name .coq-native
.opam/system/lib/coq/user-contrib/Interval/Real/.coq-native
.opam/system/lib/coq/user-contrib/Interval/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Tactics/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Integral/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Missing/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Eval/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Interval/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Poly/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Language/.coq-native
.opam/system/lib/coq/user-contrib/Interval/Float/.coq-native

[JasonGross]

What do you mean?

find ~/.opam/4.14.1/lib/coq/user-contrib/Interval/ -name .coq-native is empty for me.

opam list

$ opam list
# Packages matching: installed
# Name                       # Installed     # Synopsis
atd                          2.12.0          Parser for the ATD data format description language
atdgen                       2.12.0          Generates efficient JSON serializers, deserializers and validators
atdgen-runtime               2.12.0          Runtime library for code generated by atdgen
atdts                        2.12.0          TypeScript code generation for ATD APIs
base-bigarray                base
base-threads                 base
base-unix                    base
biniou                       1.2.2           Binary data format designed for speed, safety, ease of use and backward compatibility as protocols evolve
camlp-streams                5.0.1           The Stream and Genlex libraries for use with Camlp4 and Camlp5
cmdliner                     1.2.0           Declarative definition of command line interfaces for OCaml
conf-autoconf                0.1             Virtual package relying on autoconf installation
conf-g++                     1.0             Virtual package relying on the g++ compiler (for C++)
conf-gmp                     4               Virtual package relying on a GMP lib system installation
conf-which                   1               Virtual package relying on which
coq                          8.17.1          pinned to version 8.17.1
coq-bignums                  9.0.0+coq8.17   Bignums, the Coq library of arbitrarily large numbers
coq-compcert                 3.12            pinned to version 3.12 at file:///home/ubuntu/neural-net-coq-interp/coq-compcert
coq-coquelicot               3.4.0           A Coq formalization of real analysis compatible with the standard library
coq-core                     8.17.1          The Coq Proof Assistant -- Core Binaries and Tools
coq-elpi                     1.18.0          Elpi extension language for Coq
coq-flocq                    4.1.2           A formalization of floating-point arithmetic for the Coq system
coq-hierarchy-builder        1.5.0           High level commands to declare and evolve a hierarchy based on packed classes
coq-interval                 4.8.0           pinned to version 4.8.0 at file:///home/ubuntu/neural-net-coq-interp/coq-interval
coq-mathcomp-algebra         1.17.0          Mathematical Components Library on Algebra
coq-mathcomp-algebra-tactics 1.1.1           Ring, field, lra, nra, and psatz tactics for Mathematical Components
coq-mathcomp-analysis        0.6.4           An analysis library for mathematical components
coq-mathcomp-bigenough       1.0.1           A small library to do epsilon - N reasoning
coq-mathcomp-classical       0.6.4           A library for classical logic for mathematical components
coq-mathcomp-field           1.17.0          Mathematical Components Library on Fields
coq-mathcomp-fingroup        1.17.0          Mathematical Components Library on finite groups
coq-mathcomp-finmap          1.5.2           Finite sets, finite maps, finitely supported functions
coq-mathcomp-solvable        1.17.0          Mathematical Components Library on finite groups (II)
coq-mathcomp-ssreflect       1.17.0          Small Scale Reflection
coq-mathcomp-zify            1.3.0+1.12+8.13 Micromega tactics for Mathematical Components
coq-menhirlib                20230608        A support library for verified Coq parsers produced by Menhir
coq-native                   1               Package flag enabling coq's native-compiler flag
coq-record-update            0.3.2           Generic support for updating record fields in Coq
coq-stdlib                   8.17.1          The Coq Proof Assistant -- Standard Library
coq-vcfloat                  2.1.1           pinned to version 2.1.1 at file:///home/ubuntu/neural-net-coq-interp/coq-vcfloat
coq-vst                      2.12            pinned to version 2.12 at file:///home/ubuntu/neural-net-coq-interp/coq-vst
coq-vst-lib                  2.12            pinned to version 2.12 at file:///home/ubuntu/neural-net-coq-interp/coq-vst
coq-vst-zlist                2.12            A list library indexed by Z type, with a powerful automatic solver
coqide-server                8.17.1          The Coq Proof Assistant, XML protocol server
cppo                         1.6.9           Code preprocessor like cpp for OCaml
dune                         3.10.0          Fast, portable, and opinionated build system
easy-format                  1.3.4           High-level and functional interface to the Format module of the OCaml standard library
elpi                         1.17.0          ELPI - Embeddable λProlog Interpreter
menhir                       20230608        An LR(1) parser generator
menhirLib                    20230608        Runtime support library for parsers generated by Menhir
menhirSdk                    20230608        Compile-time library for auxiliary tools related to Menhir
ocaml                        4.14.1          The OCaml compiler (virtual package)
ocaml-base-compiler          4.14.1          Official release 4.14.1
ocaml-compiler-libs          v0.12.4         OCaml compiler libraries repackaged
ocaml-config                 2               OCaml Switch Configuration
ocaml-options-vanilla        1               Ensure that OCaml is compiled with no special options enabled
ocamlfind                    1.9.6           A library manager for OCaml
ppx_derivers                 1.2.1           Shared [@@deriving] plugin registry
ppx_deriving                 5.2.1           Type-driven code generation for OCaml
ppxlib                       0.30.0          Standard infrastructure for ppx rewriters
re                           1.11.0          RE is a regular expression library for OCaml
result                       1.5             Compatibility Result module
seq                          base            Compatibility package for OCaml's standard iterator type starting from 4.07.
sexplib0                     v0.16.0         Library containing the definition of S-expressions and some base converters
stdlib-shims                 0.3.0           Backport some of the new stdlib features to older compiler
yojson                       2.1.0           Yojson is an optimized parsing and printing library for the JSON format
zarith                       1.13            Implements arithmetic and logical operations over arbitrary-precision integers

Let's see what GitHub Actions says happens

[silene]

Since you wrote that CoqInterval is incompatible with Flocq 4.1.2, your list of Opam packages presumably means that you are using custom builds of those, which might explain why native compilation failed.

[JasonGross]

Yes, I am using slightly customized builds. But why would the result be a lack of .coq-native files

[JasonGross]

According to the GitHub actions run, vcfloat fails to build with coq-native because CompCert fails to install .coq-native files.

File "./.coq-native/Nvcfloat_Rounding.native", line 19250, characters 3-55:
19250 |    Ncompcert_lib_Maps.const_Ncompcert_lib_Maps_PMap_get
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Unbound module Ncompcert_lib_Maps
Error: Native compiler exited with status 2 (in case of stack overflow,2023-09-08T07:33:18.1032728Z - increasing stack size (typically with "ulimit -s") often helps)

make: *** [Makefile:19: Rounding.vo] Error 1

Let me try again more directly with coq-interval and see if it installs files.

[JasonGross]

Ah, I bet what happened was that I tried setting COQEXTRAFlAGS="-native-compiler on demand" to get some packages installed as if coq-native were not installed, and this resulted in interval not installing .coq-native files because I forgot to disable this setting when I was fixing the incompatibility between interval and flocq. Sorry for pointing the finger at interval here @silene , when (I now believe though am not sure) it's only coq-compcert<3.13 that's the issue

[andrew-appel]

What is .coq-native, anyway?
In my newer projects, I'm generally using coq-makefile nowadays. vcfloat has an old-fashioned Makefile partly for historical reasons, that is, in 2021 I took over some open-source code from Reservoir Labs that hadn't been maintained since 2015, and it didn't use coq-makefile. It ought to be possible to redo its build using coq-makefile.

[andrew-appel]

I'll be happy to merge this, just please explain what it's about. What's .coq-native ?

[JasonGross]

Maxime Dénès (and others?) adapted the OCaml runtime to run reduction under binders, allowing the compilation of Coq ASTs via ocamlopt to compute. In practice, the overhead of ocamlopt means that native_compute is almost always slower than vm_compute except for expensive computations (measured in at least dozens of seconds to minutes) on relatively few lines of code with relatively small output (so not anything consuming nor producing ASTs). To ameliorate the overhead of ocamlopt, Coq has a mode (now deprecated in favor of a separate coq-native binary) where it will precompile all definitions with ocamlopt, so that future native_compute invocations don't have to pay the cost of compiling the whole stdlib and dependency chain, only the cost of whatever code is in the current file. (The constant time overhead of ocamlopt is empirically 0.01--0.2 seconds, so for small computations this doesn't help, but for larger ones it can.) These precompiled files are stored in .coq-native.

Coq has three modes for the native compiler: "off" means "use vm_compute wherever the user writes native_compute", "ondemand" means "compile dependencies on the fly, do not emit .coq-native files", and "on" means "assume all dependencies have .coq-native files installed, and generate and install them". If project dependencies do not install .coq-native files, then the project will fail to compile in "on" mode.

It ought to be possible to redo its build using coq-makefile.

That would be great, though this patch is still needed for compatibility with coq-native + CompCert < 3.13

[andrew-appel]

Ugh. The amount of expertise needed just to write a the build+package instructions for a project that uses Coq (plus typically a few non-Coq components) just keeps growing and growing.