Skip to content

output: remove: catch FileNotFoundError #9574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
efiop merged 3 commits into from
Jun 20, 2023
Merged

output: remove: catch FileNotFoundError #9574

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
542d9df
output: set remove to recursive=false
Jun 9, 2023
e7723a7
try to make test pass on windows
Jun 20, 2023
931b35a
Update dvc/output.py
efiop Jun 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dvc/output.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,7 +965,7 @@ def relative_update(self, inc: int = 1) -> None:
return added, False if added else modified

def remove(self, ignore_remove=False):
self.fs.remove(self.fs_path, recursive=True)
self.fs.remove(self.fs_path, recursive=False)
Copy link
Contributor

@efiop efiop Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've discussed this before, but it loks wrong. E.g. imagine a directory with files as an output, this will error-out (at least unless I'm missing something). Could you elaborate on the problem that you are trying to solve, please?

Copy link
Contributor Author

@dberenbaum dberenbaum Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what the test tries to cover unless I'm misunderstanding you, but we probably do need to test it across other filesystems.

The problem is in #8757 - the recursive call errors out for a nonexistent path in s3fs.

As mentioned above, I don't mind closing if we come up with a different approach to solve it in s3fs or fsspec, but wanted to open it for discussion as a potential quick fix.

Copy link
Contributor

@efiop efiop Jun 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are spot on in your explanation in the issue. localfs is ignoring that flag (looks like our oversight) and since all other filesystems would also raise FileNotFoundError, we should just catch and ignore FileNotFoundError here as well.

Suggested change
self.fs.remove(self.fs_path, recursive=False)
try:
self.fs.remove(self.fs_path, recursive=True)
except FileNotFoundError:
pass

The test you've added could be removed or kept around. Ideally, we would test external outputs in dvc.testing, but that will probably just have to wait for better times.

Copy link
Contributor

@efiop efiop Jun 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test will become relevant once we make our localfs comply with the flag, but IIRC there are a few more places that need to be fixed along the way, so that could wait.

if self.protocol != Schemes.LOCAL:
return

Expand Down
9 changes: 9 additions & 0 deletions tests/func/repro/test_repro.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1273,3 +1273,12 @@ def test_repro_missing_lock_info(tmp_dir, dvc, copy_script):

stages = dvc.reproduce(stage.addressing)
assert len(stages) == 1


def test_repro_rm_recursive(tmp_dir, dvc):
# check that dir output recursively removes files in the dir
tmp_dir.gen({"dir": {"foo": "foo"}})
dvc.stage.add(name="dir", cmd="mkdir dir", outs=["dir"])
dvc.reproduce()
assert (tmp_dir / "dir").exists()
assert not (tmp_dir / "dir" / "foo").exists()