Solver jobs

[patricoferris]

This PR builds on top of #156 (the first commit) and adds the following:

• It vendors https://github.com/patricoferris/solver-service, which is the core logic of OCaml-CI's solver services in a separate library and executable. This is to hopefully future-proof expanding the use of a solver-service to other clients like opam-health-check/opam-repo-ci. The cluster workers are extended to support solving solver requests.
• The scheduler is extended to support submitting custom job specifications.
• The example adds a small solver job to get the dependencies of obuilder. It was also converted from OBuilder to Docker for portability (i.e. so it runs on my mac 🍎)

Here is the example pipeline:

And the log at the time of writing for the solver job:

Solver job lob

2022-01-31 16:36.44: New job: Custom job using linux-x86_64 in
                                [https://github.com/ocurrent/obuilder.git#refs/heads/master (df44de3a865129df9fe17d84c3006967746f3d0a)]

Custom job:

solve

2022-01-31 16:36.44: Using cache hint "https://github.com/ocurrent/obuilder.git"
2022-01-31 16:36.44: Connecting to build cluster...
2022-01-31 16:36.44: Waiting for resource in pool OCluster
2022-01-31 16:36.44: Waiting for worker...
2022-01-31 16:36.44: Got resource from pool OCluster
Building on build
All commits already cached
HEAD is now at df44de3 Merge pull request #97 from mtelvers/master
2022-01-31 16:36.45 [INFO] Solving for obuilder-spec.dev, obuilder.dev
2022-01-31 16:37.22 [INFO] os: found solution in 0.72 s
2022-01-31 16:37.22 [INFO] = os =
2022-01-31 16:37.22 [INFO] -> alcotest.1.5.0 alcotest-lwt.1.5.0 astring.0.8.5
                              base.v0.14.3 base-bigarray.base base-bytes.base
                              base-domains.base base-nnp.base
                              base-threads.base base-unix.base
                              bigarray-compat.1.1.0 biniou.1.2.1
                              cmdliner.1.0.4 conf-pkg-config.2 conf-sqlite3.1
                              cppo.1.6.8 csexp.1.5.1 cstruct.6.0.1
                              cstruct-lwt.6.0.1 dune.2.9.3
                              dune-configurator.2.9.3 easy-format.1.3.2
                              fmt.0.9.0 logs.0.7.0 lwt.5.5.0 mmap.1.1.0
                              num.1.4 obuilder.dev obuilder-spec.dev
                              ocaml.5.00.0 ocaml-compiler-libs.v0.12.4
                              ocaml-config.2 ocaml-syntax-shims.1.0.0
                              ocaml-variants.5.00.0+trunk ocamlbuild.0.14.0
                              ocamlfind.1.9.3 ocplib-endian.1.2
                              parsexp.v0.14.2 ppx_cstruct.6.0.1
                              ppx_derivers.1.2.1 ppx_deriving.5.2.1
                              ppx_sexp_conv.v0.14.3 ppxlib.0.25.0~5.00preview
                              re.1.10.3 result.1.5 seq.base sexplib.v0.14.0
                              sexplib0.v0.14.0 sha.1.15.1 sqlite3.5.1.0
                              stdlib-shims.0.3.0 tar.2.0.0 tar-unix.2.0.0
                              topkg.1.0.5 uchar.0.0.2 uutf.1.0.2 yojson.1.7.0
2022-01-31 16:37.22 [INFO] (valid since opam-repository commit f9f1b91b24a391866ce58902d1d7a588cb8009a6)
["Ok",[{"id":"os","packages":["alcotest.1.5.0","alcotest-lwt.1.5.0","astring.0.8.5","base.v0.14.3","base-bigarray.base","base-bytes.base","base-domains.base","base-nnp.base","base-threads.base","base-unix.base","bigarray-compat.1.1.0","biniou.1.2.1","cmdliner.1.0.4","conf-pkg-config.2","conf-sqlite3.1","cppo.1.6.8","csexp.1.5.1","cstruct.6.0.1","cstruct-lwt.6.0.1","dune.2.9.3","dune-configurator.2.9.3","easy-format.1.3.2","fmt.0.9.0","logs.0.7.0","lwt.5.5.0","mmap.1.1.0","num.1.4","obuilder.dev","obuilder-spec.dev","ocaml.5.00.0","ocaml-compiler-libs.v0.12.4","ocaml-config.2","ocaml-syntax-shims.1.0.0","ocaml-variants.5.00.0+trunk","ocamlbuild.0.14.0","ocamlfind.1.9.3","ocplib-endian.1.2","parsexp.v0.14.2","ppx_cstruct.6.0.1","ppx_derivers.1.2.1","ppx_deriving.5.2.1","ppx_sexp_conv.v0.14.3","ppxlib.0.25.0~5.00preview","re.1.10.3","result.1.5","seq.base","sexplib.v0.14.0","sexplib0.v0.14.0","sha.1.15.1","sqlite3.5.1.0","stdlib-shims.0.3.0","tar.2.0.0","tar-unix.2.0.0","topkg.1.0.5","uchar.0.0.2","uutf.1.0.2","yojson.1.7.0"],"commit":"f9f1b91b24a391866ce58902d1d7a588cb8009a6"}]]Job succeeded
2022-01-31 16:37.22: Job succeeded

[talex5]

Great work!

But I think it would be better to keep OCluster as a generic build service (not specific to OCaml or CI stuff), and instead deploy the solver as its own service, connected to its own pool on the scheduler. Allowing this separation was the reason for using AnyPointer and generic jobs, rather than adding a specific solver job type.

[patricoferris]

Ah okay makes sense, thanks for taking a look 👍

I'll pull out the changes to the scheduler and put them on #156 though because at the moment it doesn't actually support custom job submission... (i.e. https://github.com/ocurrent/ocluster/pull/157/files#diff-bb8011a7ab51aad47afd6d179def81780180a05239312e01bb6c75a1e607f8b3)

[patricoferris]

Closing this PR now in favour of:

• Exposing the custom jobs in the ocurrent plugin here: 0fb3ebd
• Adding a solver-worker to the solver-service repository: Add solver worker solver-service#8

Thanks for the quick clarifications :))