Error: Cannot get property on null object

[stevekm]

Bug report

(Please follow this template replacing the text between parentheses with the requested information)

I sporadically get errors such as this:

Error executing process > 'custom_sample_report (3)'

Caused by:
  Cannot get property 'signatures_Rds' on null object

Source block:
  prefix = "${tumorID}"
  html_output = "${prefix}.report.html"
  """
      # convert report file symlinks to copies of original files, because knitr doesnt work well unless all report files are in pwd
      for item in *.Rmd *.css *.bib; do
          if [ -L "\${item}" ]; then
              sourcepath="\$(python -c "import os; print(os.path.realpath('\${item}'))")"
              echo ">>> resolving source file: \${sourcepath}"
              rsync -va "\${sourcepath}" "\${item}"
          fi
      done

      R --vanilla <<E0F
      rmarkdown::render(input = "main.Rmd",
      params = list(
          sampleID = "${tumorID}",
          signatures_Rds = "${filemap['deconstructSigs']['suffix']['signatures_Rds']}",
          signatures_plot_Rds = "${filemap['deconstructSigs']['suffix']['signatures_plot_Rds']}",
          signatures_pieplot_Rds = "${filemap['deconstructSigs']['suffix']['signatures_pieplot_Rds']}",
          paired_normal = "${normalID}"
          ),
      output_format = "html_document",
      output_file = "${html_output}")
      E0F
      """

Expected behavior and actual behavior

In these steps of our pipeline, some values are read from an external JSON file. The values are later used as parameters in the pipeline.

Most of the time, this works fine. However, seemingly at random, these values will fail to be initialized when the Nextflow process tries to execute.

Steps to reproduce the problem

The code surrounding this part of the pipeline looks like this:

https://github.com/NYU-Molecular-Pathology/NGS580-nf/blob/b3bb4b99de58cd4b8ce60bf323c91fbc7b3e42a5/main.nf#L107

// load a mapping dict to use for keeping track of the names and suffixes for some files throughout the pipeline
String filemapJSON = new File("filemap.json").text
def filemap = jsonSlurper.parseText(filemapJSON)

https://github.com/NYU-Molecular-Pathology/NGS580-nf/blob/b3bb4b99de58cd4b8ce60bf323c91fbc7b3e42a5/main.nf#L5065

process custom_sample_report {
    // create per-sample reports
    publishDir "${params.outputDir}/reports", mode: 'copy'

    input:
    set val(tumorID), val(normalID), file(tumorNormalFiles: "*"), file(sampleFiles: "*"), file(report_items: '*') from sample_pairs_output_files2.combine(samples_report_files)

    output:
    file("${html_output}")

    script:
    prefix = "${tumorID}"
    html_output = "${prefix}.report.html"
    """
    # convert report file symlinks to copies of original files, because knitr doesnt work well unless all report files are in pwd
    for item in *.Rmd *.css *.bib; do
        if [ -L "\${item}" ]; then
            sourcepath="\$(python -c "import os; print(os.path.realpath('\${item}'))")"
            echo ">>> resolving source file: \${sourcepath}"
            rsync -va "\${sourcepath}" "\${item}"
        fi
    done
    R --vanilla <<E0F
    rmarkdown::render(input = "main.Rmd",
    params = list(
        sampleID = "${tumorID}",
        signatures_Rds = "${filemap['deconstructSigs']['suffix']['signatures_Rds']}",
        signatures_plot_Rds = "${filemap['deconstructSigs']['suffix']['signatures_plot_Rds']}",
        signatures_pieplot_Rds = "${filemap['deconstructSigs']['suffix']['signatures_pieplot_Rds']}",
        paired_normal = "${normalID}"
        ),
    output_format = "html_document",
    output_file = "${html_output}")
    E0F
    """
}

Program output

I am not able to find the part of the .nextflow.log that describes the error, because I am using the Nextflow Tower interface and the resulting .nextflow.log is far too large to parse...

I am sure that if I run this same pipeline again, the error will vanish temporarily. It has not proven to be possible to reproduce it voluntarily.

Environment

• Nextflow version:

$ ./nextflow -version

      N E X T F L O W
      version 19.09.0-edge build 5167
      created 16-09-2019 21:15 UTC (17:15 EDT)
      cite doi:10.1038/nbt.3820
      http://nextflow.io

• have also gotten the same error in Nextflow 19.07, and most previous versions as well.

• Java version:

$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

• Operating system: RHEL7

[stevekm]

I am running into an issue that I suspect may be related. In that same pipeline, the pipeline is initiated by parsing a .tsv samplesheet and the entries become the input items that get passed through channels to initiate the workflow. However, after updating our samplesheet and running with -resume, it appears that some or none of the new entries are making it through the pipeline. It seems as if resume is not re-initializing all of the parts of the workflow that parse and load items into the channels, the same way that resume in this case is not always re-initializing the objects that hold variables used in the pipeline

[pditommaso]

Sorry, unable to replicate.

[raivivek]

I have faced this issue as well. I am working on getting a reproducible example but looks like it is not an end user configuration error but a subtle bug. Wondering if it's possible to keep this open with a tag like "need example" or something similar.

[kemin711]

I have the same issue:
Caused by:
Cannot get property 'bwaNumthread' on null object

params.bwaNumthread
bwaNumthread is a parameter, that was used in other process without any problem. Only in this process caused trouble. I think I might have another syntax or some sort of error trigger this since I have just added this process to an existing pipeline that had been working well.

[kemin711]

I figured out my own problem: cpus { $params.bwaNumthread }, in the derivative section I have added the $ which is not allowed. The error message may be improved to help. Took me a while to find this out.