Custom database report file bugs

[lam-c]

Describe the bug
Bugs occurred when annotating custom database, that table in report files are EMPTY, although annotiation summary files NOT EMPTY.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'output/report_files/custom_databases'
2. Click on 'report_{customdb}.html'
3. Scroll down to header 'Results'
4. See error: EMPTY table of BLAST results, while the annotation summary file in output/custom_annotations/{customdb}/{sid}_{customdb}_{blast_cmd}.summary.txt are NOT EMPTY at the mean time

Expected behavior
Expected the BLAST table showing the results in annotation summary file.

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu 16.04.6 LTS
• Browser: Edge 105.0.1343.53
• Bacannot: v3.1.5

Additional context
I've check the working directory, it seems no errors thrown out, errors and logs as below:

$cat .command.err 

processing file: report_custom_blast.Rmd
output file: report_custom_blast.knit.md


Output created: report_custom_blast.html

$cat .command.log


processing file: report_custom_blast.Rmd
  |............                                                          |  17%
   inline R code fragments

  |.......................                                               |  33%
label: setup (with options)
List of 1
 $ include: logi FALSE

  |...................................                                   |  50%
   inline R code fragments

  |...............................................                       |  67%
label: unnamed-chunk-1
  |..........................................................            |  83%
   inline R code fragments

  |......................................................................| 100%
label: unnamed-chunk-2

output file: report_custom_blast.knit.md

/usr/bin/pandoc +RTS -K512m -RTS report_custom_blast.knit.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output report_custom_blast.html --lua-filter /usr/local/lib/R/site-library/bookdown/rmarkdown/lua/custom-environment.lua --lua-filter /usr/local/lib/R/site-library/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /usr/local/lib/R/site-library/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --wrap preserve --variable bs3=TRUE --section-divs --template /usr/local/lib/R/site-library/rmarkdown/rmd/h/default.html --no-highlight --variable highlightjs=1 --variable theme=readable --css styles.css --mathjax --variable 'mathjax-url=https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --include-in-header /tmp/RtmpgRmtiY/rmarkdown-str17172e53e1.html --variable code_folding=show --variable code_menu=1

Output created: report_custom_blast.html

[lam-c]

Seems like bugs come from the input.2 file (not exised) that parsed to report markdown parameter blast_gff. Sorry I don't know how to fixed this problem for now, due to unfamiliar with nextflow.

$cat .command.sh
#!/usr/bin/env Rscript

## Copy reports
system("cp /work/reports/* .") ;

## Remove empty files
system("rm -f input.??") ;
system("rm -f input.?") ;  # INPUT.2 BE REMOVED ?

## Generate Report
rmarkdown::render("report_custom_blast.Rmd", params = list(    blast_id = 65 ,     blast_cov = 65,     query = "S41325",     custom_blast = "S41325_resfams_tblastn.summary.txt",     blast_db = "resfams",     blast_gff = "input.2")) ;

# relevant clips from .command.run

286 nxf_stage() {
287     true
288     # stage input files
289     rm -f S41325_resfams_tblastn.summary.txt
290     rm -f input.2
291     ln -s ~/nf/bacannot_custom_db/temp/06/5b7bb5980806ab029e24fd897ea62a/S41325_resfams_tblastn.summary.txt S41325_resfams_tblastn.summary.txt
292     ln -s ~/cy/nf/bacannot_custom_db/temp/tmp/1d/31d33e97b604e691ebf8eb0cd2ef35/input.2 input.2  # NOT A GFF NEEDED
293 }

300 nxf_main() {
...
313     nxf_stage
...
328 }

[fmalmeida]

Hi @lam-c ,

Thanks for opening the issue and making it so informative it really helps A LOT.

Secondly, based on your second comment, I guess you already narrowed it down nicely and found the problem 👌🏼

I will take a look at that later today after work, and expect to have a proper look on the weekend.

😬

[lam-c]

I try to change the CUSTOME_DATDABASE_REPORT process input, CUSTOM_DATABASE.out.gff instead of MERGED_ANNOTATION.out.gff, and it works out, though not sure whether it is a proper solution.

# bacannot/workflows/bacannot.nf
297       // Render reports
298       if (params.custom_db || params.ncbi_proteins) {
299         CUSTOM_DATABASE_REPORT(
300           CUSTOM_DATABASE.out.summary.join( CUSTOM_DATABASE.out.gff, remainder:true )  # CHANGE GFF FILE SOURCE
301         )
302       }

But the jbrowser interface (in shiny app) doesn't include the custom annotation results ...

[fmalmeida]

Nice, so really is input source. Something similar must be happening in jbrowser.

And the errors make sense because these days I changed the named output channels so I may have gotten confused in some.

I will check all the logging and changes you’ve made to see what actually makes sense on the pipeline, I mean, the change you've made is correct in the NF perspective and also reports perspective, I just need to make sure that it is following what I was wanting for those files because they also go to jbrowse as you commented.

Thanks for the changes and the spot-on highlights.

I’ll address all as soon as possible.

😬

[fmalmeida]

Hi @lam-c ,

For the REPORT module, I think now I've fixed on your branch, which has the PR opened, the gff channel that is given as input. Maybe now it is correctly generating the reports. Can you give it a try later?

For JBROWSE, I've yet not had a look on it.

[lam-c]

Hi @lam-c ,

For the REPORT module, I think now I've fixed on your branch, which has the PR opened, the gff channel that is given as input. Maybe now it is correctly generating the reports. Can you give it a try later?

For JBROWSE, I've yet not had a look on it.

Sure, I'm willing to do that and give feedback later.

[lam-c]

Hi @lam-c ,

For the REPORT module, I think now I've fixed on your branch, which has the PR opened, the gff channel that is given as input. Maybe now it is correctly generating the reports. Can you give it a try later?

For JBROWSE, I've yet not had a look on it.

Sorry, the report table looks still empty.

[fmalmeida]

Ok,
Thanks.

Although I fixed the input channel and is now passing the correct gff to it, there may be something wrong on the actual .Rmd file. I will check on that further in the next days :)

[fmalmeida]

Hi @lam-c,

When testing later today you remembered to pull the changes and use the latest version of your master branch? Then one that is bringing the changes to this PR?

Because I just tried to run the code from this PR again, using the available quicktest profile provided and, for the custom dataset with hits, the hits were successfully placed in the report:

[fmalmeida]

Just saw! I now get it. The reports and modules are fine. The local copy of the report that goes on the report_files sub-directory, works fine. It has everything.

The jbrowse is also working fine, populating the custom annotations in it.

However, when you use the shiny app, the tables content from the generated .html reports don´t appear! But at least, the jbrowse data is working in the shiny app.

So ... I was looking at the wrong place haha. The problem is in the shiny app code. I will try to identify what is the problem.

But for now, you can at least run the analysis and use the available local copies of the reports and use the shiny app only for the blast and dataset querying until I fix it.

😄

[lam-c]

@fmalmeida Sorry I might mix the branches when running the test. The reports meet demand, and I will use the pr branch to run local analysis. Thanks again!

[fmalmeida]

Generating a new branch to tackle this issue that seems to be related to the shiny app.

[fmalmeida]

Everything is now working on dev branch. The new branch uses some new scripts and docker images that may not be already available.

I am working to merge the dev to master within this week.