-
Slurmtools for submitting NONMEM runs
-
slurmtools is an R package for interacting with slurm
-(fka Simple Linux
-Utility for Resource
-Management) and submitting NONMEM jobs. You can submit
-a NONMEM job with submit_nonmem_model, you can view current
-jobs with get_slurm_jobs, and you can see the available
-partitions with get_slurm_partitions.
-
-
-
Submitting a NONMEM job with bbi
-
To submit a NONMEM job, we need to supply either the path to a mod
-file or create a model object from bbr, and supply a
-slurm-template.tmpl file. To use bbi we also
-need a bbi.yaml file, which I’ve also supplied in
-/model/nonmem/bbi.yaml (and is also supplied with the R
-project starter).
-
Here is an example of a template file that will call
-bbi:
-
#!/bin/bash
-#SBATCH --job-name="{{job_name}}"
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task={{ncpu}}
-#SBATCH --partition={{partition}}
-#SBATCH --account={{project_name}}
-
-# submit_nonmem_model uses the whisker package to populate template files
-# https://github.com/edwindj/whisker
-
-{{#parallel}}
-{{bbi_exe_path}} nonmem run local {{model_path}}.mod --parallel --threads={{ncpu}} --config {{bbi_config_path}}
-{{/parallel}}
-
-
-{{^parallel}}
-{{bbi_exe_path}} nonmem run local {{model_path}}.mod --config {{bbi_config_path}}
-{{/parallel}}
-
This file will call bbi to run our supplied model
-({{model_path}}.mod) if ncpu > 1 then
-parallel will be true and the code between {{#parallel}}
-and {{/parallel}} will be populated. if
-ncpu = 1 then parallel will be false and the code between
-{{^parallel}} and {{/parallel}} will be
-populated. By default, submit_nonmem_model will inject
-Sys.which("bbi") into the template, so if bbi
-is not on your path we’ll have to supply the bbi_exe_path
-for it to start the NONMEM run.
-
Sys.which("bbi")
- bbi
-"/usr/local/bin/bbi"
-
We will use a few different template files with different
-functionality so we’ll inject those template file paths to
-submit_nonmem_model. However, we’ll use the
-submission-log directory for the output, so we’ll set that
-option as well as bbi_config_path so
-submit_nonmem_model defaults can be used. The slurm
-template files are saved in ~/model/nonmem/ Additionally,
-there is a simple NONMEM control stream in 1001.mod in the
-same directory that we can use for testing.
-
library(bbr)
-library(here)
-
-nonmem = file.path(here::here(), "vignettes", "model", "nonmem")
-
-options('slurmtools.submission_root' = file.path(nonmem, "submission-log"))
-options('slurmtools.bbi_config_path' = file.path(nonmem, "bbi.yaml"))
-
To create the bbr model object, we need to have both
-1001.mod and 1001.yaml which contains metadata
-about the model in the supplied directory
-(./model/nonmem/). We’ll check for mod_number.yaml and if
-it exists, read in the model otherwise create it and then read it.
-
mod_number <- "1001"
-
-if (file.exists(file.path(nonmem, paste0(mod_number, ".yaml")))) {
- mod <- bbr::read_model(file.path(nonmem, mod_number))
-} else {
- mod <- bbr::new_model(file.path(nonmem, mod_number))
-}
-
We can now submit the job and point to the template file in
-model/nonmem/slurm-job-bbi.tmpl.
-
submission <- slurmtools::submit_nonmem_model(
- mod,
- slurm_job_template_path = file.path(nonmem, "slurm-job-bbi.tmpl"),
-)
-
-submission
-$status
-[1] 0
-
-$stdout
-[1] "Submitted batch job 878\n"
-
-$stderr
-[1] ""
-
-$timeout
-[1] FALSE
-
We see a status with an exit code of 0 suggesting a
-successful command, and the stdout gives us the batch job
-number. We can use slurmtools::get_slurm_jobs() to monitor
-the status of the job. Here, we can supply the user = “matthews”
-argument to filter to just the jobs I’ve submitted.
-
slurmtools::get_slurm_jobs(user = 'matthews')
-# A tibble: 11 × 10
- job_id job_state cpus partition standard_input standard_output
- <int> <chr> <int> <chr> <chr> <chr>
- 1 868 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
- 2 869 FAILED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
- 3 870 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
- 4 871 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
- 5 872 COMPLETED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
- 6 873 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
- 7 874 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
- 8 875 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
- 9 876 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-10 877 FAILED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-11 878 PENDING 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-# ℹ 4 more variables: submit_time <dttm>, start_time <dttm>, user_name <chr>,
-# current_working_directory <chr>
-
If we look in the slurmtools.submisstion_root directory
-we can see the shell script that was generated with
-submit_nonmem_model. Here is the whisker replaced call to
-bbi:
-
/usr/local/bin/bbi nonmem run local /cluster-data/user-homes/matthews/Projects/slurmtools_vignette/model/nonmem/1001.mod --config /cluster-data/user-homes/matthews/Projects/slurmtools_vignette/model/nonmem/bbi.yaml
-
-
-
Extending templates
-
Because the templates create a bash shell script there is an almost
-infinite number of things we can do with our template. Anything
-you can do in bash you can do by appropriately updating the template
-file and injecting the needed information!
-
Let’s add a notification feature that will send a notification when
-the job has started and finished. We can use ntfy.sh and add the necessary info to our template to
-achieve this.
-
Here is a modified template file that adds a
-JOBID=$SLURM_JOBID and some ntfy calls. To get a
-notification we can supply submit_nonmem_model with
-ntfy variable to send notifications. I’ll use
-ntfy = ntfy_demo for this.
-
#!/bin/bash
-#SBATCH --job-name="{{job_name}}"
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task={{ncpu}}
-#SBATCH --partition={{partition}}
-#SBATCH --account={{project_name}}
-
-JOBID=$SLURM_JOBID
-
-# submit_nonmem_model uses the whisker package to populate template files
-# https://github.com/edwindj/whisker
-
-{{#ntfy}}
-curl -d "Starting model run: {{job_name}} $JOBID" ntfy.sh/{{ntfy}}
-{{/ntfy}}
-
-{{#parallel}}
-{{bbi_exe_path}} nonmem run local {{model_path}}.mod --parallel --threads={{ncpu}} --config {{bbi_config_path}}
-{{/parallel}}
-
-{{^parallel}}
-{{bbi_exe_path}} nonmem run local {{model_path}}.mod --config {{bbi_config_path}}
-{{/parallel}}
-
-{{#ntfy}}
-curl -d "Finished model run: {{job_name}} $JOBID" ntfy.sh/{{ntfy}}
-{{/ntfy}}
-
Since we’ve already run this model we will provide the
-overwrite = TRUE argument to force a new nonmem run.
-
submission_ntfy <- slurmtools::submit_nonmem_model(
- mod,
- slurm_job_template_path = file.path(nonmem, "slurm-job-bbi-ntfy.tmpl"),
- overwrite = TRUE,
- slurm_template_opts = list(
- ntfy = "ntfy_demo")
-)
-
-submission_ntfy
-$status
-[1] 0
-
-$stdout
-[1] "Submitted batch job 879\n"
-
-$stderr
-[1] ""
-
-$timeout
-[1] FALSE
-
We again get a 0 exit code status and now instead of using
-slurmtools::get_slurm_jobs() to monitor the job, we can
-rely on the new notifications we just set up. 
-
and when the run finished we get another notification: 
-
Note that the run number will match the run specified in
-submission$stdout. We can see the new shell script this
-updated template file generated
-
#!/bin/bash
-#SBATCH --job-name="1001-nonmem-run"
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=1
-#SBATCH --partition=cpu2mem4gb
-#SBATCH --account=slurmtools
-
-JOBID=$SLURM_JOBID
-
-curl -d "Starting model run: 1001-nonmem-run $JOBID" ntfy.sh/ntfy_demo
-
-/usr/local/bin/bbi nonmem run local /cluster-data/user-homes/matthews/Projects/slurmtools_vignette/model/nonmem/1001.mod --config /cluster-data/user-homes/matthews/Projects/slurmtools_vignette/model/nonmem/bbi.yaml
-
-curl -d "Finished model run: 1001-nonmem-run $JOBID" ntfy.sh/ntfy_demo
-
To reiterate, this template file is run as a bash shell script so
-anything you can do in bash you can put into the template and pass the
-needed arguments and customize the behavior to your liking.
-
-
-
Submitting a NONMEM job with nmm
-
Instead of using bbi we can use nmm (NONMEM Monitor) which
-currently has some additional functionality of sending notifications
-about zero gradients, missing -1E9 lines in ext file, and some very
-basic control stream errors. Currently, only slack or ntfy.sh is supported for receiving notifications. To
-use nmm you can install the latest release from the github
-repository linked above.
-
We can update the template file accordingly:
-
#!/bin/bash
-#SBATCH --job-name="{{job_name}}"
-#SBATCH --nodes=1
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task={{ncpu}}
-#SBATCH --partition={{partition}}
-
-{{nmm_exe_path}} -c {{config_toml_path}} run
-
default, submit_nonmem_model will provide
-nmm_exe_path and config_toml_path to the
-template. Just like with bbi_exe_path,
-nmm_exe_path is determined with
-Sys.which("nmm") which may or may not give you the path to
-the nmm binary if it is on your path or not. We can inject the
-nmm_exe_path like we did with bbi_exe_path and
-assume it’s not on our path.
-
The config.toml file controls what nmm will
-monitor and where to look for files and how to alert you. We’ll use
-generate_nmm_config() to create this file. First we can
-look at the documentation to see what type of information we should pass
-to this function.
-
mod_number <- "1001"
-
-if (file.exists(file.path(nonmem, paste0(mod_number, ".yaml")))) {
- mod <- bbr::read_model(file.path(nonmem, mod_number))
-} else {
- mod <- bbr::new_model(file.path(nonmem, mod_number))
-}
-
slurmtools::generate_nmm_config(mod)
-
This generates the following toml file. Notice that alert is set to
-‘None’, and both email and topic are empty. Since we’re in vignettes
-we’ll need to update the watched_dir and
-output_dir.
-
model_number = '1001'
-files_to_track = [ 'lst', 'ext', 'grd' ]
-tmp_dir = '/tmp'
-watched_dir = '/cluster-data/user-homes/matthews/Packages/slurmtools/model/nonmem'
-output_dir = '/cluster-data/user-homes/matthews/Packages/slurmtools/model/nonmem/in_progress'
-poll_duration = 1
-alert = 'None'
-level = 'Debug'
-email = ''
-threads = 1
-topic = ''
-
slurmtools::generate_nmm_config(
- mod,
- watched_dir = "/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem",
- output_dir = "/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/in_progress")
-
This updates the 1001.toml config file to:
-
model_number = '1001'
-files_to_track = [ 'lst', 'ext', 'grd' ]
-tmp_dir = '/tmp'
-watched_dir = '/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem'
-output_dir = '/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/in_progress'
-poll_duration = 1
-alert = 'None'
-level = 'Debug'
-email = ''
-threads = 1
-topic = ''
-
We can now run submit_nonmem_model and get essentially
-the same behavior as running with bbi. On linux
-~/.local/bin/ will be on your path so saving binaries there
-is a good approach.
-
submission_nmm <- slurmtools::submit_nonmem_model(
- mod,
- overwrite = TRUE,
- slurm_job_template_path = file.path(nonmem, "slurm-job-nmm.tmpl"),
- slurm_template_opts = list(
- nmm_exe_path = normalizePath("~/.local/bin/nmm"))
-)
-
-submission_nmm
-#> $status
-#> [1] 0
-#>
-#> $stdout
-#> [1] "Submitted batch job 876\n"
-#>
-#> $stderr
-#> [1] ""
-#>
-#> $timeout
-#> [1] FALSE
-
slurmtools::get_slurm_jobs()
-#> # A tibble: 9 × 10
-#> job_id job_state cpus partition standard_input standard_output
-#> <int> <chr> <int> <chr> <chr> <chr>
-#> 1 868 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/mat…
-#> 2 869 FAILED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/mat…
-#> 3 870 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/mat…
-#> 4 871 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d0…
-#> 5 872 COMPLETED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d0…
-#> 6 873 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d0…
-#> 7 874 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d0…
-#> 8 875 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d0…
-#> 9 876 PENDING 1 cpu2mem4gb /dev/null /cluster-data/user-homes/mat…
-#> # ℹ 4 more variables: submit_time <dttm>, start_time <dttm>, user_name <chr>,
-#> # current_working_directory <chr>
-
The one difference between using nmm compared to
-bbi is that a new directory is created that contains a log
-file that caught some issues with our run. This file is updated as
-nonmem is running and monitors gradient values, parameters that hit
-zero, as well as other errors from bbi. Looking at the first few lines
-we can see that bbi was successfully able to call nonmem.
-We also see an info level log that OMEGA(2,1) has 0 value – in our mod
-file we don’t specify any omega values off the diagonal so these are
-fixed at 0. Finally we see that GRD(6) hit 0 relatively early in the
-run.
-
20:12:36 [INFO] bbi log: time="2024-08-27T20:12:36Z" level=info msg="Successfully loaded default configuration from /cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/bbi.yaml"
-20:12:36 [INFO] bbi log: time="2024-08-27T20:12:36Z" level=info msg="Beginning Local Path"
-20:12:36 [INFO] bbi log: time="2024-08-27T20:12:36Z" level=info msg="A total of 1 models have completed the initial preparation phase"
-20:12:36 [INFO] bbi log: time="2024-08-27T20:12:36Z" level=info msg="[1001] Beginning local work phase"
-20:12:58 [INFO] OMEGA(2,1) has 0 value
-20:12:58 [INFO] SIGMA(2,1) has 0 value
-20:13:00 [INFO] SIGMA(2,1) has 0 value
-20:13:00 [INFO] OMEGA(2,1) has 0 value
-20:13:04 [INFO] SIGMA(2,1) has 0 value
-20:13:04 [INFO] OMEGA(2,1) has 0 value
-20:13:04 [WARN] "/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/1001/1001.grd" has 0 gradient for parameter: GRD(6)
-
After a run has finished several messages are sent to the log after a
-final check of the files listed in the files_to_track field
-of the 1001.toml file.
-
20:13:16 [INFO] Received Exit code: exit status: 0
-20:13:16 [WARN] 1001.ext: Missing ext final output lines. Observed lines were: [-1000000000.0, -1000000004.0, -1000000006.0, -1000000007.0]
-20:13:16 [WARN] "/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/1001/1001.grd": The following parameters hit zero gradient through the run: ["GRD(6)"]
-
We see that GRD(6) hit zero during the run and that only a subset of
-the -1E9 lines were present in the .ext file.
-
-
Getting alerted during a run
-
Like we did with bbi and altering the slurm template
-file to get notifications from ntfy.sh
-nmm has this feature built in! The messages in the log file
-that relate to zero gradients, missing -1E9 lines, and 0 parameter
-values can also be sent to ntfy by altering the 1001.toml
-file. We can get these alerts in real time without having to dig through
-a noisy log file.
-
Let’s update our call to generate_nmm_config to have
-nmm send notifications to the NONMEMmonitor
-topic on ntfy.sh.
-
slurmtools::generate_nmm_config(
- mod,
- alert = "Ntfy",
- topic = "NONMEMmonitor",
- watched_dir = "/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem",
- output_dir = "/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/in_progress")
-
This updates the 1001.toml file to this:
-
model_number = '1001'
-files_to_track = [ 'lst', 'ext', 'grd' ]
-tmp_dir = '/tmp'
-watched_dir = '/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem'
-output_dir = '/cluster-data/user-homes/matthews/Packages/slurmtools/vignettes/model/nonmem/in_progress'
-poll_duration = 1
-alert = 'Ntfy'
-level = 'Debug'
-email = ''
-threads = 1
-topic = 'NONMEMmonitor'
-
When we re-run the submit_nonmem_model call we will now
-get ntfy notifications. One thing to note is that nmm will
-print full paths in the log, but will only send notifications with the
-model_number (or
-model_number.file_extension).
-
submission_nmm <- slurmtools::submit_nonmem_model(
- mod,
- overwrite = TRUE,
- slurm_job_template_path = file.path(nonmem, "slurm-job-nmm.tmpl"),
- slurm_template_opts = list(
- nmm_exe_path = normalizePath("~/.local/bin/nmm-x86_64-unknown-linux-gnu/nmm"))
-)
-#> Warning in normalizePath("~/.local/bin/nmm-x86_64-unknown-linux-gnu/nmm"):
-#> path[1]="/cluster-data/user-homes/matthews/.local/bin/nmm-x86_64-unknown-linux-gnu/nmm":
-#> No such file or directory
-
-submission_nmm
-#> $status
-#> [1] 0
-#>
-#> $stdout
-#> [1] "Submitted batch job 877\n"
-#>
-#> $stderr
-#> [1] ""
-#>
-#> $timeout
-#> [1] FALSE
-
slurmtools::get_slurm_jobs(user = "matthews")
-#> # A tibble: 10 × 10
-#> job_id job_state cpus partition standard_input standard_output
-#> <int> <chr> <int> <chr> <chr> <chr>
-#> 1 868 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-#> 2 869 FAILED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-#> 3 870 CANCELLED 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-#> 4 871 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
-#> 5 872 COMPLETED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
-#> 6 873 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
-#> 7 874 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
-#> 8 875 FAILED 1 cpu2mem4gb /dev/null /tmp/RtmpIBC04D/Rbuild1af3d…
-#> 9 876 PENDING 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-#> 10 877 PENDING 1 cpu2mem4gb /dev/null /cluster-data/user-homes/ma…
-#> # ℹ 4 more variables: submit_time <dttm>, start_time <dttm>, user_name <chr>,
-#> # current_working_directory <chr>
-
This gives us the notifications in a much more digestible format
-
-
-
nmm ntfy.sh alerts
-
-
-
-