Move the markdown package from Imports to Suggests

[yihui]

For package authors who use R Markdown vignettes based on the vignette engine knitr::knitr (or knitr::docco_linear, knitr::docco_classic, etc.), you should declare the (soft) dependency on markdown, e.g., in the package DESCRIPTION:

Suggests: markdown
VignetteBuilder: knitr

Previously they didn't need to do this because markdown has been a hard dependency of knitr (in Imports), so the availability of markdown is guaranteed by knitr, but I want to make markdown a soft dependency of knitr in the future.

I think most packages use the vignette engine knitr::rmarkdown to write vignettes now, but there may still be a few vignettes using the knitr::knitr engine, which depends on markdown. If I move markdown to Suggests in knitr, users have to make sure markdown is available to R CMD check. To achieve that, they have to add markdown to Suggests to their DESCRIPTION files.

Similarly, if your vignette is based on the vignette engine knitr::rmarkdown, you have to declare the dependency on rmarkdown, e.g.,

Suggests: rmarkdown
VignetteBuilder: knitr

The error message when building the source package via R CMD build should tell you if you need to add markdown or rmarkdown to Suggests, which looks like this:

* checking for file ‘PKG/DESCRIPTION’ ... OK
* preparing ‘PKG’:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘doc.Rmd’ using rmarkdown_notangle
Error: processing vignette 'doc.Rmd' failed with diagnostics:
The 'rmarkdown' package should be declared as a dependency of
the 'PKG' package (e.g., in the  'Suggests' field of DESCRIPTION),
because the latter contains vignette(s) built with the 'rmarkdown'
package. Please see https://github.com/yihui/knitr/issues/1864
for more information.
--- failed re-building ‘doc.Rmd’

SUMMARY: processing the following file failed:
  ‘doc.Rmd’

Error: Vignette re-building failed.
Execution halted

If you are a package author affected by this issue, but not clear about what you need to do, please reply below and I'll be glad to help. I recommend that you skip the replies below. Thanks!

[tdhock]

hi I got this from win-builder:

* checking re-building of vignette outputs ... [1s] WARNING
Error(s) in re-building vignettes:
--- re-building 'examples.Rmd' using knitr
Error: processing vignette 'examples.Rmd' failed with diagnostics:
The 'markdown' package should be declared as a dependency of the 'directlabels' package (e.g., in the  'Suggests' field of DESCRIPTION), because it contains vignette(s) built with the 'markdown' package. Please see https://github.com/yihui/knitr/issues/1864 for more information.
--- failed re-building 'examples.Rmd'

SUMMARY: processing the following file failed:
  'examples.Rmd'

Error: Vignette re-building failed.
Execution halted

my vignette uses

%\VignetteEngine{knitr::knitr}

which I think does not require markdown (even in suggests), if I understand your discussion above correctly. so it seems strange that I would need to add markdown in suggests. do I really have to?

[yihui]

Yes, you have to.

[tdhock]

ok

[mschubert]

I think I'm getting a false positive warning on the CRAN Winbuilder. I've got all my vignettes set to

%\VignetteEngine{knitr::rmarkdown}

If I have not set in DESCRIPTION that

Suggests:
    rmarkdown

I'm getting no warnings locally (R=4.0.2, rmarkdown=2.2, knitr=1.28). On the CRAN Winbuilder, I'm getting:

Error: processing vignette 'quickstart.Rmd' failed with diagnostics:
The 'markdown' package should be declared as a dependency of the 'clustermq' package (e.g., in the 'Suggests' field of DESCRIPTION), because it contains vignette(s) built with the 'markdown' package. Please see #1864 for more information.

The missing package, however, is Suggests: rmarkdown and not markdown.

[yihui]

@mschubert I need a reproducible example. Could you provide a link to your package (presumably on Github?)? Thanks!

[mschubert]

Package was clustermq:3d223af and you can find the Winbuilder output here. I won't have time to track this down further unfortunately (since adding Suggests: rmarkdown solves the issue & a warning for the right package is there), sorry.

[yihui]

@mschubert In your case, you are using the vignette engine knitr::rmarkdown, so you have to add rmarkdown to Suggests (which you did mschubert/clustermq@02e06f8): https://yihui.org/knitr/demo/vignette/ Because you didn't do that, you saw the message about markdown, and that was because knitr fell back to using markdown to build your vignettes. Neither rmarkdown nor markdown was specified in your Suggests, and that's why you saw the error.

[mschubert]

Yes, that's what I tried to say - sorry if that was not clear. My comment was that the error message of "missing markdown package in Suggests" is confusing when the vignette states that it uses rmarkdown (and the markdown package should never be required).

Perhaps a better error message would be to add whatever markdown builder you use to Suggests? Or, maybe do nothing, but I thought it was useful to document that this message is also displayed when rmarkdown is missing (I guess people will google the issue and find this).

[yihui]

@mschubert Yes, I definitely agree the messages were confusing in this case. I'll think more about it and try to make it clearer. Thanks a lot for your feedback!

[nsheff]

I agree with @mschubert. I had the same situation. I was about to add 'markdown' to suggests because of the message, when I really should have added 'rmarkdown' to suggests, because I use knitr::rmarkdown (if I understood correctly).

[yihui]

With knitr 1.30 (the current latest version on CRAN), the message should tell you to add rmarkdown instead of markdown to Suggests when the vignette engine knitr::rmarkdown is used.

[jangorecki]

Any idea when the actual moving to suggests may happen?

[yihui]

@jangorecki It may happen relatively soon. I'm talking to CRAN and if they could forgive me for breaking (potentially a small number of) packages that didn't declare markdown in Suggests, I'll move markdown to Suggests in knitr in the next CRAN release. There might be some BioC packages that will be affected, and I'm not sure how many will still have this problem after six months.

[dnadave]

Is there a way to get around this issue? This error is preventing us from submitting two packages to Bioconductor...

[yihui]

@dnadave Yes, please do what the error message told you to do.

[yihui]

@jangorecki I talked to CRAN maintainers last month and about 470 CRAN packages will be broken if I moved markdown to Suggests. This is not a trivial number, and I definitely don't want these packages to be killed just because of this minor issue. The next step for me is to write to these package maintainers and see how many would be willing to make this small change and update their packages on CRAN.

[lshep]

@yihui this may have been the wrong issue to post on but what I'm concerned with it how to reproduce the ERROR we were seeing on the Bioconductor build report since it has disappeared with knitr1.33.8? As this seemed to be a real ERROR that you wanted corrected and we were able to see the ERROR with knitr 1.33.3. My understanding was the work around was a temporary fix since the original updated was right before the Bioconductor release and breaking numerous packages; now we wanted the ERROR to appear so maintainers would correct?

[yihui]

@lshep Looking at the changes from 1.33.3 to 1.33.8 (ae6ab6e...fad3ead), I can't think of a possible reason why the error could have disappeared in 1.33.8. The only possibly relevant commit is 93720d3, which was @hpages's suggestion from #1864 (comment), but that shouldn't make the error disappear. I don't know why 1.33.3 could cause the error, either...

[hpages]

@yihui Correction: we were using knitr 1.33.5 on our build machines, which was raising the R CMD build error if either markdown or rmarkdown was not declared as a dep. With the latest knitr (1.33.8), we only see the error if markdown is not declared as a dep but we don't see it anymore for rmarkdown. The change is in commit c56fd61 from Aug 25.

So for example, a package like RMassBank (which uses rmarkdown to build its vignette) was getting an R CMD build error with knitr 1.33.5 but not anymore with knitr 1.33.8. Is this intended?

We've contacted many developers to let them know about these errors and ask them to fix their package. Now they're getting back to us telling us they don't know what we're talking about because they don't see the error on our daily build report. What should we tell them? Thanks

[yihui]

@hpages I'm confused now. Neither v1.33.5 nor v1.33.8 should cause an error on Bioc, because both markdown and rmarkdown are available on Bioc. If I understood you correctly last time (93720d3), you have set KNITR_NO_TEST_VIGNETTE_DEP=true on Bioc, right? With that, no Bioc packages should need to do anything, and R CMD build should not throw an error.

That said, I just committed 0dcd462, which completely removed the possibility of build error as long as markdown is available (which is true on Bioc).

[hpages]

@yihui We only use KNITR_NO_TEST_VIGNETTE_DEP=true for our release builds (BioC 3.13). We don't set this variable for our devel builds (BioC 3.14) because we want BioC developers to fix whatever error knitr detects in the devel version of their package.

I'm confused now. Neither v1.33.5 nor v1.33.8 should cause an error on Bioc, because both markdown and rmarkdown are available on Bioc.

Both markdown and rmarkdown have always been available on our build machines and yet we were getting these R CMD build errors. The error message was:

The 'markdown' package should be installed and declared as a dependency of the 'foo' package (e.g., in the 'Suggests' field of DESCRIPTION), because the latter contains vignette(s) built with the 'markdown' package.

The message complains that markdown should be declared as a dependency. AFAIK the fact that a package needs to declare another package as a dependency has nothing to do with what's installed on the system where the package is being tested. So if knitr considers it is a mistake that markdown or rmarkdow are not listed as a dependency, this is a fact that remains true whether these packages are installed or not.

With knitr v1.33.9, R CMD build RMassBank still doesn't give the error we used to get with knitr 1.33.5, despite the fact that RMassBank still uses rmarkdown to build its vignette but does not list it as a dep.

[yihui]

With knitr v1.33.9, R CMD build RMassBank still doesn't give the error we used to get with knitr 1.33.5, despite the fact that RMassBank still uses rmarkdown to build its vignette but does not list it as a dep.

Okay, that's the (new) behavior that I expect on Bioc now. Once CRAN submission is re-opened, I'll try to submit a new version of knitr to CRAN. Thanks!

[hpages]

Okay, that's the (new) behavior that I expect on Bioc now.

Focusing on the behavior on Bioc is not that helpful. When Bioc developers run R CMD build on their package, they don't do it on the Bioconductor build machines. They do it on their own machine and with a stock knitr installed from CRAN (knitr 1.33). This version of knitr gives them an error about markdown or rmarkdown not being in Suggests. This has been the case for months now (since knitr 1.33 was released on CRAN), and during all these months we've been using a setup on our devel build machines that was consistent with what people get on their own machine.

Even though this new check by knitr came as a surprise in April and broke hundreds of Bioconductor packages a few days before the BioC 3.13 release (see #1864 (comment)), we embraced it in BioC 3.14 and have been working all these months with our developers to help them fix their package.

But now it seems that you've changed your mind. Are you saying that knitr will no longer consider that it's an error when a package uses markdown or rmarkdown to build its vignette and doesn't depend on it? Just want to make sure so we can clarify the situation with our developers. Thanks!

[yihui]

Are you saying that knitr will no longer consider that it's an error when a package uses markdown or rmarkdown to build its vignette and doesn't depend on it?

Yes. I'm following your original suggestion in #1864 (comment), i.e., "let things happen naturally." I will no longer throw an error on my side. If the build/check platform (e.g., CRAN) thinks it is an error not to declare the dependency, I'll just let the platform throw the error. The previous error on my side was meant to serve as a precaution and avoid future breakages on CRAN.

Now that I have gotten the permission from CRAN (and was even strongly encouraged by them) that it's time to break packages that still haven't made the correction, I'll just let them break, although personally I really hope to give them more time.

[hpages]

Thanks for clarifying. I know, it was my suggestion to let things happen naturally. More generally I don't think a package should meddle too much in what its reverse deps do with their Suggests field. We just need to know where you are going so we can follow, anticipate, and avoid miscommunication/confusion with our developers. Thanks again.

[stefanoborini]

For the record, I continue getting this message, despite both rmarkdown and markdown being declared in Suggests or Imports of my package, as well as having both of them installed in the environment.

[yihui]

@stefanoborini If you get the message even if you declare them in Imports, then there must be something else wrong. Chances are your .libPaths() is not properly configured on your computer and you installed rmarkdown/markdown to a custom library path that R is not able to find during R CMD build or R CMD check. You may try to configure R_LIBS_USER in ~/.Renviron.

[stefanoborini]

@yihui I use .Rprofile to extend the libPaths indeed, but then I don't understand why devtools::build works and R CMD build does not. Why would they behave differently? Don't they work exactly in the same way?

[yihui]

@stefanoborini I don't know why (I'm not the author of either of them), but my experience is that R commands (such as R CMD build or check) works better with .Renviron in general. If you want to set a custom library path, I recommend that you set it via R_LIBS_USER in .Renviron instead.

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.