Mocked modules aren't ignored, redux

[rbeyer]

Details

• Read the Docs project URL: https://readthedocs.org/projects/hiproc/
• Build URL (if applicable): https://readthedocs.org/projects/hiproc/builds/13832034/
• Read the Docs username (if applicable): rbeyer

Expected Result

My Python project uses GDAL, which has all kinds of difficulty in installing cleanly via pip, so I used autodoc_mock_imports in my conf.py file to keep it from being imported (and did not include it in my requirements.txt file). I expected the build to complete.

It appears that #6209 had a similar issue, but did get things to work?

Actual Result

Unfortunately, the build failed with a ModuleNotFoundError (which I assumed the autodoc_mock_imports would handle). The unfound module is "osgeo" which is the top level import statement in the modules (why it is "osgeo" and not "gdal" is a historical oddity, but if you install the gdal python package, you import osgeo).

Instead of autodoc_mock_imports I tried using conda (but the solve took too long and the build failed). I tried using apt_packages in the .readthedocs.yaml file to install GDAL (as suggested in #8160), and set environment variables. This allowed GDAL to install, but yielded a failure when it tried to import _gdal_array (My failing Build 13831926 ) which was something that gdal/osgeo was doing internally, so that didn't seem like I could get much farther.

It seems like autodoc_mock_imports should do what I want to safely ignore gdal/osgeo, but it doesn't. Do I have some syntax wrong somewhere?

[stsewd]

Hi, looks like the import error comes from using autoprogram https://github.com/rbeyer/hiproc/blob/2a245d3ed299dc2a6a356fba62740d2432549b28/docs/conf.py#L55

https://github.com/rbeyer/hiproc/blob/main/docs/console.rst

I tried using conda (but the solve took too long and the build failed).

I can increase the resources for your project, so you can try using conda.

[rbeyer]

Thanks. Apparently, I don't understand autodoc_mock_imports because even local tests on my machine didn't make Sphinx mock the gdal/osgeo import the way I expected.

It seems wasteful to me to to use more cycles to have conda do its thing to install a library that only needs to sit there so that "import osgeo" doesn't fail when Sphinx imports "my" modules to read their docstrings. No part of the Sphinx documentation run needs gdal/osgeo. I'll try and dig deeper into how to mock the imports so it isn't needed. I was hoping that I was just missing something about how autodoc_mock_imports works.

I'll spend some more time digging, but may still sheepishly come back to ask for more resources, but I hope I won't have to.

Thanks again.

[astrojuanlu]

I tried using conda (but the solve took too long and the build failed).

@rbeyer Is there a chance that installing libgdal with apt fixes your build issues? You have more information here https://blog.readthedocs.com/apt-packages/

[rbeyer]

@astrojuanlu, I tried that in RTD Build 13831893, but then I got import errors about _gdal_array:

ModuleNotFoundError: No module named '_gdal_array'

I had this in my .readthedocs.yaml:

build:
    apt_packages:
        - gdal-bin
        - libgdal-dev

And gdal==2.2.3 in my docs/requirements.txt file to align with the expected version of the library. I wasn't sure how to progress from the _gdal_array include problem that was being reported. Some light research seemed to indicate that the solution (on a regular ubuntu system, anyway) was to perform some install/uninstall dance with gdal and numpy, but I wasn't sure how to do that via RTD (and it seemed weird anyway). Maybe there's a better way forward to solve that _gdal_array import error (if so, that would be great)?

[astrojuanlu]

Oh, I see. I think I arrived to the same answers as you:

https://gis.stackexchange.com/a/274328/99665

(Also, sorry - I just realized you wrote it in the first comment)

I confirmed locally with a Ubuntu Bionic image that installing NumPy before the GDAL Python bindings indeed fixes the No module named '_gdal_array' error. And indeed, this is not something that can be easily done at the moment on Read the Docs.

The last thing I'd suggest is switching back the configuration to conda, and turn CONDA_USES_MAMBA on for your project, which should solve the environment way faster and therefore avoid the problem you observed.

[rbeyer]

@astrojuanlu, no worries, and thank you.

Hmm, does the requirements.txt file get run serially in order? Maybe I just need to try changing the order (I have gdal first, just because I typically arrange my requirements.txt file alphabetically). When I get a chance, I'll try that. And will also try the mamba approach. Thanks for the suggestions.

[astrojuanlu]

I think pip largely ignores the order, wouldn't trust it that much. The mamba approach would require an action from our side, let us know when you're ready 👍🏽

[rbeyer]

So, the thing that I was missing about how autodoc_mock_imports works was that it applies to the "autodoc" Sphinx extension, and not the "autoprogram" extension that was resulting in the errors (as @stsewd clearly saw early on, and then I distracted them).

So no amount of adding things to autodoc_mock_imports would have done anything for me. Once I realized that, and looked more closely at autoprogram, I filed an Issue about maybe getting something like the autodoc_mock_imports parameter functionality set up for autoprogram.

Then I messed around some more, and got a mocking situation to work (as detailed in that Issue) that didn't require a code change to autoprogram, and also allowed my RTD run to complete, without needing to install gdal/osgeo at all (either via apt_packages, pip, or conda).

Hopefully I've mentioned GDAL enough in this issue that folks will find it, if they end up having similar trouble getting something that just uses gdal/osgeo to work with RTD.

Thanks to all the help in this thread as I lurched myself to understanding and solution!