`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

redsun82 · 2024-07-10T10:41:44Z

A detailed description of the bug or problem you are having

When creating the following structure:

x/
├─ y/
│  ├─ y/
│  ├─ test_y.py

then pytest --import-mode=importlib fails with the following cryptic message:

================================= ERRORS =================================
_____________________ ERROR collecting x/y/test_y.py _____________________
<frozen importlib._bootstrap_external>:1448: in find_spec
    ???
<frozen importlib._bootstrap_external>:1222: in __init__
    ???
<frozen importlib._bootstrap_external>:1238: in _get_parent_path
    ???
E   KeyError: 'x'

To trigger this bug:

the directory containing the test needs to be within at least another directory, it can't be directly underneath the root path
the directory containing the test must in turn contain another directory with the same base name

Running with --pdb gives a slightly more useful backtrace:

  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/runner.py(341)from_call()
-> result: Optional[TResult] = func()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/runner.py(389)collect()
-> return list(collector.collect())
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(548)collect()
-> self._register_setup_module_fixture()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(561)_register_setup_module_fixture()
-> self.obj, ("setUpModule", "setup_module")
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(287)obj()
-> self._obj = obj = self._getobj()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(545)_getobj()
-> return importtestmodule(self.path, self.config)
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(492)importtestmodule()
-> mod = import_path(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(565)import_path()
-> mod = _import_module_using_spec(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(660)_import_module_using_spec()
-> parent_module = _import_module_using_spec(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(633)_import_module_using_spec()
-> spec = meta_importer.find_spec(module_name, [str(module_location)])
  <frozen importlib._bootstrap_external>(1448)find_spec()
  <frozen importlib._bootstrap_external>(1222)__init__()
> <frozen importlib._bootstrap_external>(1238)_get_parent_path()

Output of `pip list` from the virtual environment you are using

I tried this on a minimal poetry env with just pytest

-------------- -------
exceptiongroup 1.2.1
iniconfig      2.0.0
packaging      24.1
pip            24.0
pluggy         1.5.0
pytest         8.2.2
setuptools     70.0.0
tomli          2.0.1

pytest and operating system versions

pytest 8.2.2. Encountered this both with python 3.10 and python 3.12, so the python version doesn't seem to be a factor.

Running on Ubuntu 22.04.4 LTS.

cc @criemen who encountered this

The text was updated successfully, but these errors were encountered:

Zac-HD · 2024-07-21T03:53:24Z

Feels like this might be an upstream issue; we should try to get a reproducer using stdlib code only.

…in the namespace package causes a KeyError.（pytest-dev#12592）

Directories inside a namespace package with the same name as the namespace package would cause a `KeyError` with `--import-mode=importlib`. Fixes #12592 Co-authored-by: Bruno Oliveira <[email protected]> (cherry picked from commit 6486c3f)

#12843) Directories inside a namespace package with the same name as the namespace package would cause a `KeyError` with `--import-mode=importlib`. Fixes #12592 Co-authored-by: Bruno Oliveira <[email protected]> (cherry picked from commit 6486c3f) Co-authored-by: dongfangtianyu <[email protected]>

valeriupredoi · 2024-12-02T16:23:15Z

cheers for fixing this, folks! I noticed a side-effect of this though: if pytest is running a test on a module that imports e.g.

from cow.moo.farm import udder

and there is an /cow/moo/moo.py, so module moo.py is in a dir identically called moo and there is no __init__.py in /moo then the test fails with an ImportError - not a biggie since IMHO since one uses package-level imports, then there must be init files all over the place where modules live, it's all good, but just wanted to let you know if yous get some others complaining about it 🍺

michael-christen · 2024-12-03T16:56:16Z

Seeing a similar issue as #12592 (comment)

Perhaps it's my misunderstanding of namespace packages, but I typically don't have any __init__.py files in my repo. And I'm seeing an AttributeError for my module now:

 AttributeError: module 'third_party.my_protobuf_utils.my_protobuf_utils' has no attribute '_read_varint'

Automatically Upgrade Dependencies - by upgrade-dependencies.yml The new version of pytest introduced an issue with files named the same as their module: pytest-dev/pytest#12592 (comment) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: michael-christen <[email protected]>

dongfangtianyu · 2024-12-11T13:41:15Z

Hi, @valeriupredoi @michael-christen
Sorry for seeing these messages so late. Can you please provide a simple reproducible example?

I tried to reproduce an example, Is it the same as this? Please feel free to provide additional examples

https://github.com/dongfangtianyu/pytest_importlib_issue

Regression in #12716 In short: `PathFinder.find_spec` received the argument `/cow/moo` but loaded `/cow/moo/moo.py` instead. **Trigger conditions:** 1. `/cow/moo/moo.py` exists (a file and directory with the same name). 2. `/cow/moo/test_moo.py` exists (test case resides in the directory). When pytest loads test files in `importlib` mode, it continues recursive loading upward: - When loading `cow.moo`, it should return a namespace but unexpectedly returns a module. - When loading `cow.moo.moo`, it should return a module but unexpectedly returns a namespace. **Complete example:** [[GitHub repository](https://github.com/dongfangtianyu/pytest_importlib_issue)](https://github.com/dongfangtianyu/pytest_importlib_issue) - `main.py`: Reproduces the error. - `debug.py`: Demonstrates the behavior of `PathFinder.find_spec`. **Context:** #12592 (comment) #12592 (comment) --------- Co-authored-by: Bruno Oliveira <[email protected]>

Regression in #12716 In short: `PathFinder.find_spec` received the argument `/cow/moo` but loaded `/cow/moo/moo.py` instead. **Trigger conditions:** 1. `/cow/moo/moo.py` exists (a file and directory with the same name). 2. `/cow/moo/test_moo.py` exists (test case resides in the directory). When pytest loads test files in `importlib` mode, it continues recursive loading upward: - When loading `cow.moo`, it should return a namespace but unexpectedly returns a module. - When loading `cow.moo.moo`, it should return a module but unexpectedly returns a namespace. **Complete example:** [[GitHub repository](https://github.com/dongfangtianyu/pytest_importlib_issue)](https://github.com/dongfangtianyu/pytest_importlib_issue) - `main.py`: Reproduces the error. - `debug.py`: Demonstrates the behavior of `PathFinder.find_spec`. **Context:** #12592 (comment) #12592 (comment) --------- Co-authored-by: Bruno Oliveira <[email protected]> (cherry picked from commit 28e1e25)

michael-christen · 2024-12-12T16:03:42Z

Hi, @valeriupredoi @michael-christen Sorry for seeing these messages so late. Can you please provide a simple reproducible example?

I tried to reproduce an example, Is it the same as this? Please feel free to provide additional examples

https://github.com/dongfangtianyu/pytest_importlib_issue

Yep, that's the exact case I have, thanks @dongfangtianyu 🙏

…#13054) Regression in #12716 In short: `PathFinder.find_spec` received the argument `/cow/moo` but loaded `/cow/moo/moo.py` instead. **Trigger conditions:** 1. `/cow/moo/moo.py` exists (a file and directory with the same name). 2. `/cow/moo/test_moo.py` exists (test case resides in the directory). When pytest loads test files in `importlib` mode, it continues recursive loading upward: - When loading `cow.moo`, it should return a namespace but unexpectedly returns a module. - When loading `cow.moo.moo`, it should return a module but unexpectedly returns a namespace. **Complete example:** [[GitHub repository](https://github.com/dongfangtianyu/pytest_importlib_issue)](https://github.com/dongfangtianyu/pytest_importlib_issue) - `main.py`: Reproduces the error. - `debug.py`: Demonstrates the behavior of `PathFinder.find_spec`. **Context:** #12592 (comment) #12592 (comment) --------- Co-authored-by: Bruno Oliveira <[email protected]> (cherry picked from commit 28e1e25) Co-authored-by: dongfangtianyu <[email protected]>

Zac-HD added the topic: collection related to the collection phase label Jul 21, 2024

dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 30, 2024

Fix: Using --import-mode=importlib, a directory with the same name …

f9fef27

…in the namespace package causes a KeyError.（pytest-dev#12592）

dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 30, 2024

Fix: Using --import-mode=importlib, a directory with the same name …

098e5a6

…in the namespace package causes a KeyError.（pytest-dev#12592）

dongfangtianyu mentioned this issue Aug 30, 2024

Fixed the issue of causes KeyError when using the parameter --import-mode=importlib in pytest>=8.2 . (#12592) #12752

Merged

dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Aug 31, 2024

Fix: Using --import-mode=importlib, a directory with the same name …

f741433

…in the namespace package causes a KeyError.（pytest-dev#12592）

dongfangtianyu added a commit to dongfangtianyu/pytest that referenced this issue Sep 3, 2024

Fix: Using --import-mode=importlib, a directory with the same name …

7108bb9

…in the namespace package causes a KeyError.（pytest-dev#12592）

nicoddemus closed this as completed in #12752 Sep 26, 2024

nicoddemus closed this as completed in 6486c3f Sep 26, 2024

patchback bot mentioned this issue Sep 26, 2024

[PR #12752/6486c3f3 backport][8.3.x] Fixed the issue of causes KeyError when using the parameter --import-mode=importlib in pytest>=8.2 . (#12592) #12843

Merged

valeriupredoi mentioned this issue Dec 2, 2024

Add init files in package-like diagnostic directories to comply with pytest >=8.3.4 ESMValGroup/ESMValTool#3834

Merged

4 tasks

michael-christen mentioned this issue Dec 3, 2024

Automatically Upgrade Dependencies michael-christen/toolbox#171

Merged

dongfangtianyu mentioned this issue Dec 11, 2024

Class Paths Have Strange Behavior on 8.3.4 #13050

Open

4 tasks

dongfangtianyu mentioned this issue Dec 12, 2024

fix: Resolved an ImportError crash when using --import-mode=importlib #13053

Merged

patchback bot mentioned this issue Dec 12, 2024

[PR #13053/28e1e25a backport][8.3.x] fix: Resolved an ImportError crash when using --import-mode=importlib #13054

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

redsun82 commented Jul 10, 2024

Zac-HD commented Jul 21, 2024

valeriupredoi commented Dec 2, 2024 •

edited

Loading

michael-christen commented Dec 3, 2024

dongfangtianyu commented Dec 11, 2024 •

edited

Loading

michael-christen commented Dec 12, 2024

KeyError fired from importlib during collection when running with --import-mode=importlib on a directory containing a same-named one #12592

KeyError fired from importlib during collection when running with --import-mode=importlib on a directory containing a same-named one #12592

Comments

redsun82 commented Jul 10, 2024

A detailed description of the bug or problem you are having

Output of pip list from the virtual environment you are using

pytest and operating system versions

Zac-HD commented Jul 21, 2024

valeriupredoi commented Dec 2, 2024 • edited Loading

michael-christen commented Dec 3, 2024

dongfangtianyu commented Dec 11, 2024 • edited Loading

michael-christen commented Dec 12, 2024

`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

Output of `pip list` from the virtual environment you are using

valeriupredoi commented Dec 2, 2024 •

edited

Loading

dongfangtianyu commented Dec 11, 2024 •

edited

Loading