Getting "Executed file not found" error when .py file present along with .ipynb

[bblais]

Bug description

I am trying to compile a book with quarto, sourced by .ipynb files. I have chapters specified in the _quarto.yml and they are all .ipynb files. When doing quarto render I get the error if any of the .ipynb files also have a .py file with the same name, and no error if I remove the .py file. Is this expected? I usually have as a default for my Jupyter to always save a .py with my .ipynb to make it easier for searching through old work. Thanks!

Steps to reproduce

No response

Expected behavior

No response

Actual behavior

No response

Your environment

• Python 3 installation....Version: 3.11.8 (Conda)
• Mac 14.4.1

Quarto check output

Quarto 1.4.554
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.11: OK
      Dart Sass version 1.69.5: OK
      Deno version 1.37.2: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.4.554
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /Library/TeX/texbin
      Version: 2023

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.11.8 (Conda)
      Path: /opt/anaconda3/envs/work/bin/python
      Jupyter: 5.5.0
      Kernels: python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

[cscheid]

We have support for percent scripts: https://quarto.org/docs/computations/render-scripts.html#syntax . I suspect what's happening is that our code for determining which inputs are .qmd files and which inputs are .py files is going wrong.

[mcanouil]

Could you share a small self-contained "working" (reproducible) example to work with, i.e., a complete Quarto document or a Git repository? The goal is to make it as easy as possible for us to recreate your problem so that we can fix it: please help us help you! Thanks.

---

You can share a self-contained "working" (reproducible) Quarto document using the following syntax, i.e., using more backticks than you have in your document (usually four ````).
See https://quarto.org/bug-reports.html#small-is-beautiful-aim-for-a-single-document-with-10-lines.

If you have multiple files (and if it is absolutely required to have multiple files), please share as a Git repository.

R	Python
````qmd --- title: "Reproducible Quarto Document" format: html engine: knitr --- This is a reproducible Quarto document. ```{r} x <- c(1, 2, 3, 4, 5) y <- c(1, 4, 9, 16, 25) plot(x, y) ```  The end. ````	````qmd --- title: "Reproducible Quarto Document" format: html engine: jupyter --- This is a reproducible Quarto document. ```{python} import matplotlib.pyplot as plt x = [1, 2, 3, 4, 5] y = [1, 4, 9, 16, 25] plt.plot(x, y) plt.show() ```  The end. ````

Additionally and if not already given, please share the output of quarto check within a code blocks (i.e., using three backticks ```txt), see https://quarto.org/bug-reports.html#check.

[bblais]

Here is a reproducible example in 4 very small files.
quarto_error.zip

There's _quarto.yml

project:
  type: book

book:
  title: "testing quarto"
  chapters:
    - index.qmd
    - TestChapter.ipynb


format:
  html:
    theme: cosmo
  pdf:
    subject: "hello"

an index.qmd

# Something

and a TestChapter.ipynb with a single cell

print("hello")

and a TestChapter.py which is a python export of that notebook. If the .py is there, the error happens when rendering pdf, but the html render also fails -- only silently.

remove the .py file, everything works.

Does that help?

[cscheid]

It does, thank you.

[cderv]

That is really helpful to help reproduce and investigate.

What happens is that, TestChapter.ipynb is identified somehow as an intermediate and so filter out from project files

quarto-cli/src/project/project-context.ts

Lines 796 to 801 in 76d4ae6

	const inputFiles = ld.difference(
	ld.uniq(files),
	ld.uniq(intermediateFiles),
	) as string[];
	return { files: inputFiles, engines };

And this is indeed seen as intermediates because of the .py file existence with same name

quarto-cli/src/execute/jupyter/jupyter.ts

Lines 594 to 608 in 76d4ae6

	intermediateFiles: (input: string) => {
	const files: string[] = [];
	const [fileDir, fileStem] = dirAndStem(input);
	if (!isJupyterNotebook(input)) {
	files.push(join(fileDir, fileStem + ".ipynb"));
	} else if (
	[...kQmdExtensions, ...kJupyterPercentScriptExtensions].some((ext) => {
	return existsSync(join(fileDir, fileStem + ext));
	})
	) {
	files.push(input);
	}
	return files;
	},

It seems our logic for Jupyter engine is

• If the input is not a jupyter notebook,
  • then add .ipynb to intermediates
• If the input is a jupyter notebook
  • If a .qmd or .py with same file stem exists, then add the ipynb input as intermediates

This does not check that the .py script is a percent script - it will consider all py script with same name.

Maybe just adding this specific check would be working. I don't think we can allow a .py percent script and a ipynb of the same name.

Otherwise, more generally, it seems odd that an input file can be then filtered out as intermediates but I guess any produced ipynb could be seen as input but they could be intermediates leftover from previous run.

Anyhow, these lines are the source of the issue;

Summary of the current process:

• Quarto see the index.qmd and TestChapter.ipynb as input files for the book project
  

  quarto-cli/src/project/types/book/book-config.ts

  Lines 243 to 249 in 76d4ae6

  	// save our own render list (which has more fine grained info about parts,
  	// appendices, numbering, etc.) and popuplate the main config render list
  	const renderItems = await bookRenderItems(project, language, config);
  	book[kBookRender] = renderItems;
  	config.project[kProjectRender] = renderItems
  	.filter((target) => !!target.file)
  	.map((target) => target.file!);

• When computing the project context, the above render target are processed to determine the projectInputFiles
  

  quarto-cli/src/project/project-context.ts

  Lines 776 to 794 in 76d4ae6

  	const renderFiles = metadata?.project[kProjectRender];
  	if (renderFiles) {
  	const exclude = projIgnoreGlobs.concat(outputDir ? [outputDir] : []);
  	const resolved = resolvePathGlobs(dir, renderFiles, exclude, {
  	mode: "auto",
  	});
  	await Promise.all(
  	(ld.difference(resolved.include, resolved.exclude) as string[])
  	.map((file) => {
  	if (Deno.statSync(file).isDirectory) {
  	return addDir(file);
  	} else {
  	return addFile(file);
  	}
  	}),
  	);
  	} else {
  	await addDir(dir);
  	}

• When adding files to the list, we consider intermediates for each engine detected
  

  quarto-cli/src/project/project-context.ts

  Lines 732 to 799 in 76d4ae6

  	const engine = await fileExecutionEngine(file, undefined, project);
  	if (engine) {
  	if (!engines.includes(engine.name)) {
  	engines.push(engine.name);
  	}
  	files.push(file);
  	const engineIntermediates = executionEngineIntermediateFiles(
  	engine,
  	file,
  	);
  	if (engineIntermediates) {
  	intermediateFiles.push(...engineIntermediates);
  	}
  	}
  	}
  	};
  	const addDir = async (dir: string) => {
  	// ignore selected other globs
  	for (
  	const walk of walkSync(
  	dir,
  	{
  	includeDirs: false,
  	// this was done b/c some directories e.g. renv/packrat and potentially python
  	// virtualenvs include symblinks to R or Python libraries that are in turn
  	// circular. much safer to not follow symlinks!
  	followSymlinks: false,
  	skip: [kSkipHidden].concat(
  	engineIgnoreDirs().map((ignore) =>
  	globToRegExp(join(dir, ignore) + SEP)
  	),
  	),
  	},
  	)
  	) {
  	const pathRelative = pathWithForwardSlashes(relative(dir, walk.path));
  	if (!projectIgnores.some((regex) => regex.test(pathRelative))) {
  	await addFile(walk.path);
  	}
  	}
  	};
  	const renderFiles = metadata?.project[kProjectRender];
  	if (renderFiles) {
  	const exclude = projIgnoreGlobs.concat(outputDir ? [outputDir] : []);
  	const resolved = resolvePathGlobs(dir, renderFiles, exclude, {
  	mode: "auto",
  	});
  	await Promise.all(
  	(ld.difference(resolved.include, resolved.exclude) as string[])
  	.map((file) => {
  	if (Deno.statSync(file).isDirectory) {
  	return addDir(file);
  	} else {
  	return addFile(file);
  	}
  	}),
  	);
  	} else {
  	await addDir(dir);
  	}
  	const inputFiles = ld.difference(
  	ld.uniq(files),
  	ld.uniq(intermediateFiles),
  	) as string[];

  
  And the end, it is clear that the intermediates are filtered out from Project input files
• Jupyter engine intermediate function adds the input .ipynb as an intermediate
  

  quarto-cli/src/execute/jupyter/jupyter.ts

  Lines 594 to 608 in 76d4ae6

  	intermediateFiles: (input: string) => {
  	const files: string[] = [];
  	const [fileDir, fileStem] = dirAndStem(input);
  	if (!isJupyterNotebook(input)) {
  	files.push(join(fileDir, fileStem + ".ipynb"));
  	} else if (
  	[...kQmdExtensions, ...kJupyterPercentScriptExtensions].some((ext) => {
  	return existsSync(join(fileDir, fileStem + ext));
  	})
  	) {
  	files.push(input);
  	}
  	return files;
  	},

Possible fix for it:

diff --git a/src/execute/jupyter/jupyter.ts b/src/execute/jupyter/jupyter.ts
index 2f2cdfe04..b4d6374de 100644
--- a/src/execute/jupyter/jupyter.ts
+++ b/src/execute/jupyter/jupyter.ts
@@ -595,13 +595,26 @@ export const jupyterEngine: ExecutionEngine = {
     const files: string[] = [];
     const [fileDir, fileStem] = dirAndStem(input);

+    const isJupyterNotebookProduct = function (input: string): boolean {
+      // Either a qmd file or some Jupyter Percent Script can produce .ipynb as intermediate files
+      return [...kQmdExtensions, ...kJupyterPercentScriptExtensions].some(
+        (ext) => {
+          const file = join(fileDir, fileStem + ext);
+          const exists = existsSync(file);
+          if (exists) {
+            if (kJupyterPercentScriptExtensions.includes(ext)) {
+              return isJupyterPercentScript(file);
+            }
+            return true;
+          }
+          return false;
+        },
+      );
+    };
+
     if (!isJupyterNotebook(input)) {
       files.push(join(fileDir, fileStem + ".ipynb"));
-    } else if (
-      [...kQmdExtensions, ...kJupyterPercentScriptExtensions].some((ext) => {
-        return existsSync(join(fileDir, fileStem + ext));
-      })
-    ) {
+    } else if (isJupyterNotebookProduct(input)) {
       files.push(input);
     }
     return files;

It depends if we want to fix it that way

[cderv]

@cscheid I think your recent fix at

• jupyter - use different name for input .ipynb file #8752

probably makes all this file handling obsolete, or maybe possibly not working as they should...

I understand that now, our intermediates will be .quarto_ipynb
https://github.com/quarto-dev/quarto-cli/blob/4a28c314bfe05210fc4d07ada5b19aad57f57eff/src/execute/jupyter/jupyter.ts#L195-L201

All this file computations for intermediates but also the keep-ipynb are impacted and should consider .quarto_ipynb it seems.