`--output-dir` warns about non-empty directories, but deletes contents anyway

[eheinzen]

I have:

• searched the issue tracker for similar issues
• installed the latest version of Quarto CLI
• formatted my issue following the Bug Reports guide

Bug description

I'm running quarto 1.8.24. When I specify --output-dir, it first warns about it not being an empty directory, but deletes files therein anyway:

Steps to reproduce

$ mkdir test_quarto
$ cd test_quarto
$ echo '---
> title: "Untitled"
> format: html
> ---' > test.qmd
$ mkdir important_files
$ echo 'Something important!' > important_files/do_not_delete.txt
$ quarto render test.qmd --output-dir important_files
ERROR: Directory not empty (os error 39): remove '<sensitive path>/test_quarto/.quarto'

Stack trace: <...>

$ cat important_files/do_not_delete.txt # It's not there!
cat: important_files/do_not_delete.txt: No such file or directory

However, it doesn't have this behavior with a _quarto.yml

$ mkdir test_quarto
$ cd test_quarto
$ echo '---
> title: "Untitled"
> format: html
> ---' > test.qmd
$ echo 'project:
>   output-dir: important_files' > _quarto.yml
$ mkdir important_files
$ echo 'Something important!' > important_files/do_not_delete.txt
$ quarto render test.qmd
  
Output created: important_files/test.html

$ cat important_files/do_not_delete.txt # It's there!
Something important!

Actual behavior

Deletes my files.

Expected behavior

Don't delete my files.

Your environment

Linux

[mcanouil]

@eheinzen Could you format your code for it to be usable? Thank you.

Also please provide the output of quarto check as requested in the template.

Note that the behaviour in project is to delete all the output directory (if/when inside the project). Have you consider looking at the help of the CLI? quarto render --help, especially the --no-clean flag?

[eheinzen]

Sorry, was trying to incorporate the output at the same time as the code, which I now realize makes your job harder!

mkdir test_quarto
cd test_quarto
echo '---
title: "Untitled"
format: html
---' > test.qmd
mkdir important_files
echo 'Something important!' > important_files/do_not_delete.txt
quarto render test.qmd --output-dir important_files
cat important_files/do_not_delete.txt # It's not there!

quarto check has a lot of sensitive paths in it that I needed to scrub out:

$ quarto check
Quarto 1.8.24
[✓] Checking environment information...
      Quarto cache location: /home/.../.cache/quarto
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.6.3: OK
      Dart Sass version 1.87.0: OK
      Deno version 2.3.1: OK
      Typst version 0.13.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.8.24
      Path: /.../software/quarto/quarto-1.8.24/bin

[✓] Checking tools....................OK
      TinyTeX: (external install)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /home/.../.TinyTeX/bin/x86_64-linux
      Version: 2023

[✓] Checking Chrome Headless....................OK
      Using: Chrome found on system
      Path: /usr/bin/chromium-browser
      Source: PATH

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.6.8
      Path: /usr/bin/python3
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
      Version: 4.2.2
      Path: /.../software/R/R-4.2.2/lib64/R
      LibPaths:
        - /.../renv/library/R-4.2/x86_64-pc-linux-gnu
        - /.../software/R/R-4.2.2/lib64/R/library
      knitr: 1.45
      rmarkdown: 2.25

[✓] Checking Knitr engine render......OK

I'll take a look at that other documentation. Wasn't aware of that option; I was just surprised when all my files in my output directory got deleted.

[eheinzen]

Update: --no-clean retains the files in the output directory, which is what I want. Should that be the default behavior? It behaves that way when you use the _quarto.yml (that is: no files get deleted).

[mcanouil]

It behaves that way when you use the _quarto.yml (that is: no files get deleted).

That's not true. As I said, all files get deleted when output directory is within the project which is the expected location (at least for all non "default" projec type). Content is not deleted when it's outside of the project because there are risks to delete something not at all tied to the project.

"single documents" are also a "project" which for now does not behave exactly the same way as "default" project.

[mcanouil]

Also, here you are using project.output-dir and --output-dir which are known to not behave exactly the same way as discussed in:

• Output directory in quarto-cli #2171

See a pretty much comprehensive overview of the behaviours in the comment: #2171 (comment)

Output directory cleaning seems to be another discrepancy between "default", "single document" projects, and the other types of project.

Anyhow, because some users might already rely on the current behaviour, a change in the default is unlikely to happen at the moment as it could lead to a lot of breakage. The --no-clean flag is there precisely to avoid cleaning.

[eheinzen]

Thanks for the explanation! That makes more sense now.

Output directory cleaning seems to be another discrepancy between "default", "single document" projects, and the other types of project.

This is where I was getting stuck. To me, deleting all files in an output directory doesn't ever seem like a good idea, and I was getting hung up on it working in some places but not others. For now, I'll stick with --no-clean.

[mcanouil]

In most projects, you don't want outdated files to stick thus the cleaning default.

[cderv]

It is an interesting use case because it reliably recreate on windows the OSERROR 32

ERROR: Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus. (os error 32): remove 'C:\Users\chris\AppData\Local\Temp\quarto\test_quarto\.quarto'

• os error 32? #10775
• windows - wait to file to unlock for removal #10629

I'll have a look in this.

[mcanouil]

I realised I forgot to mention that the error is indeed limited to Windows.