pydantic installation succeeds, but throws ModuleNotFoundError at runtime

[FilipPasternak]

Describe the bug

I try to use openai library in my project. In briefcase dev, program runs just fine, but when trying to run briefcase run android -r ModuleNotFound error pops up like the package was not installed. In the logs you can clearly see that all required packages are installed without any problem. I tried re-creating my app and building it with -u -r. Nothing seen about this topic resolves my problem.

Steps to reproduce

1. briefcase create
2. briefcase build android -r
3. briefcase run android -r

Expected behavior

App that is seeing all the required modules that are clearly imported and compiled into the project

Screenshots

No response

Environment

• Operating System: Windows 11
• Python version: 3.10.11
• Software versions:
  • Briefcase:0.3.19
  • Toga:0.4.7
  • ...

Logs

briefcase.2024_10_04-16_00_53.run.log

Additional context

No response

[rmartin16]

Calling this out for others since I couldn't replicate it (or particularly understand why this is happening) but wheels for x86-64 are being installed for Android:

           Chaquopy: Installing for arm64-v8a                                                                                                                      subprocess.py:195
[15:59:37] Looking in indexes: https://pypi.org/simple, https://chaquo.com/pypi-13.1                                                                               subprocess.py:195
           Collecting openai                                                                                                                                       subprocess.py:195
             Using cached openai-1.51.0-py3-none-any.whl (383 kB)                                                                                                  subprocess.py:195
[15:59:38] Collecting toga-android~=0.4.5                                                                                                                          subprocess.py:195
             Using cached toga_android-0.4.7-py3-none-any.whl (67 kB)                                                                                              subprocess.py:195
           Collecting sniffio                                                                                                                                      subprocess.py:195
             Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)                                                                                                   subprocess.py:195
           Collecting tqdm>4                                                                                                                                       subprocess.py:195
             Using cached tqdm-4.66.5-py3-none-any.whl (78 kB)                                                                                                     subprocess.py:195
           Collecting anyio<5,>=3.5.0                                                                                                                              subprocess.py:195
             Using cached anyio-4.6.0-py3-none-any.whl (89 kB)                                                                                                     subprocess.py:195
[15:59:39] Collecting httpx<1,>=0.23.0                                                                                                                             subprocess.py:195
             Using cached httpx-0.27.2-py3-none-any.whl (76 kB)                                                                                                    subprocess.py:195
           Collecting distro<2,>=1.7.0                                                                                                                             subprocess.py:195
             Using cached distro-1.9.0-py3-none-any.whl (20 kB)                                                                                                    subprocess.py:195
           Processing                                                                                                                                              subprocess.py:195
           c:\users\revte\appdata\local\chaquopy\pip\cache\wheels\08\e7\a6\65ff68ab347dc90d4eece22790f5ee0617a6a0450ee98d656a\jiter-0.5.0-cp310-none-win_amd64.whl                  
           Collecting typing-extensions<5,>=4.11                                                                                                                   subprocess.py:195
             Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)                                                                                        subprocess.py:195
[15:59:40] Collecting pydantic<3,>=1.9.0                                                                                                                           subprocess.py:195
             Using cached pydantic-2.9.2-py3-none-any.whl (434 kB)                                                                                                 subprocess.py:195
           Collecting toga-core==0.4.7                                                                                                                             subprocess.py:195
             Using cached toga_core-0.4.7-py3-none-any.whl (139 kB)                                                                                                subprocess.py:195
           Collecting exceptiongroup>=1.0.2; python_version < "3.11"                                                                                               subprocess.py:195
             Using cached exceptiongroup-1.2.2-py3-none-any.whl (16 kB)                                                                                            subprocess.py:195
[15:59:41] Collecting idna>=2.8                                                                                                                                    subprocess.py:195
             Using cached idna-3.10-py3-none-any.whl (70 kB)                                                                                                       subprocess.py:195
           Collecting certifi                                                                                                                                      subprocess.py:195
             Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)                                                                                              subprocess.py:195
           Collecting httpcore==1.*                                                                                                                                subprocess.py:195
             Using cached httpcore-1.0.6-py3-none-any.whl (78 kB)                                                                                                  subprocess.py:195
[15:59:44] Processing                                                                                                                                              subprocess.py:195
           c:\users\revte\appdata\local\chaquopy\pip\cache\wheels\e3\2a\c3\87d389b54a9702721219edef00388a7c0f764084defdbb140f\pydantic_core-2.23.4-cp310-none-win_                  
           amd64.whl                                                                                                                                                                
           Collecting annotated-types>=0.6.0                                                                                                                       subprocess.py:195
             Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB)                                                                                           subprocess.py:195
           Collecting travertino>=0.3.0                                                                                                                            subprocess.py:195
             Using cached travertino-0.3.0-py3-none-any.whl (16 kB)                                                                                                subprocess.py:195
           Collecting h11<0.15,>=0.13                                                                                                                              subprocess.py:195
             Using cached h11-0.14.0-py3-none-any.whl (58 kB)                                                                                                      subprocess.py:195
[15:59:45] Installing collected packages: sniffio, tqdm, exceptiongroup, typing-extensions, idna, anyio, certifi, h11, httpcore, httpx, distro, jiter,             subprocess.py:195
           pydantic-core, annotated-types, pydantic, openai, travertino, toga-core, toga-android                                                                                    
[15:59:55] Successfully installed annotated-types-0.7.0 anyio-4.6.0 certifi-2024.8.30 distro-1.9.0 exceptiongroup-1.2.2 h11-0.14.0 httpcore-1.0.6 httpx-0.27.2     subprocess.py:195
           idna-3.10 jiter-0.5.0 openai-1.51.0 pydantic-2.9.2 pydantic-core-2.23.4 sniffio-1.3.1 toga-android-0.4.7 toga-core-0.4.7 tqdm-4.66.5 travertino-0.3.0                    
           typing-extensions-4.12.2 

That aside, we'll need to be a little more judicious in which versions of these packages we try to use. The openai package (and more and more of Python at large) is growing its dependency on Rust. Given that building packages using Rust isn't quite nailed down yet for Android, I was able to avoid this issue using these requirements:

requires = [
    "openai < 1.40",
    "pydantic < 2",
]

[mhsmith]

I think I've seen similar issues before – Chaquopy uses a number of techniques to make the build fail quickly and with a comprehensible message if it attempts to compile native code, but it looks like none of them apply to Rust.

If the user doesn't have the Rust tools installed, then the build will fail similarly to chaquo/chaquopy#1227. But if they do have Rust installed, then it'll build a wheel for the build machine (in this case Windows). So the build appears to succeed, but the module can't be loaded on Android.

[FilipPasternak]

Yes, I have installed rust - app simply does not build when it was not installed. After installation app builded succesfuly, but issue described above by me occured

[FilipPasternak]

The problem was solved!

Calling this out for others since I couldn't replicate it (or particularly understand why this is happening) but wheels for x86-64 are being installed for Android:

           Chaquopy: Installing for arm64-v8a                                                                                                                      subprocess.py:195
[15:59:37] Looking in indexes: https://pypi.org/simple, https://chaquo.com/pypi-13.1                                                                               subprocess.py:195
           Collecting openai                                                                                                                                       subprocess.py:195
             Using cached openai-1.51.0-py3-none-any.whl (383 kB)                                                                                                  subprocess.py:195
[15:59:38] Collecting toga-android~=0.4.5                                                                                                                          subprocess.py:195
             Using cached toga_android-0.4.7-py3-none-any.whl (67 kB)                                                                                              subprocess.py:195
           Collecting sniffio                                                                                                                                      subprocess.py:195
             Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)                                                                                                   subprocess.py:195
           Collecting tqdm>4                                                                                                                                       subprocess.py:195
             Using cached tqdm-4.66.5-py3-none-any.whl (78 kB)                                                                                                     subprocess.py:195
           Collecting anyio<5,>=3.5.0                                                                                                                              subprocess.py:195
             Using cached anyio-4.6.0-py3-none-any.whl (89 kB)                                                                                                     subprocess.py:195
[15:59:39] Collecting httpx<1,>=0.23.0                                                                                                                             subprocess.py:195
             Using cached httpx-0.27.2-py3-none-any.whl (76 kB)                                                                                                    subprocess.py:195
           Collecting distro<2,>=1.7.0                                                                                                                             subprocess.py:195
             Using cached distro-1.9.0-py3-none-any.whl (20 kB)                                                                                                    subprocess.py:195
           Processing                                                                                                                                              subprocess.py:195
           c:\users\revte\appdata\local\chaquopy\pip\cache\wheels\08\e7\a6\65ff68ab347dc90d4eece22790f5ee0617a6a0450ee98d656a\jiter-0.5.0-cp310-none-win_amd64.whl                  
           Collecting typing-extensions<5,>=4.11                                                                                                                   subprocess.py:195
             Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)                                                                                        subprocess.py:195
[15:59:40] Collecting pydantic<3,>=1.9.0                                                                                                                           subprocess.py:195
             Using cached pydantic-2.9.2-py3-none-any.whl (434 kB)                                                                                                 subprocess.py:195
           Collecting toga-core==0.4.7                                                                                                                             subprocess.py:195
             Using cached toga_core-0.4.7-py3-none-any.whl (139 kB)                                                                                                subprocess.py:195
           Collecting exceptiongroup>=1.0.2; python_version < "3.11"                                                                                               subprocess.py:195
             Using cached exceptiongroup-1.2.2-py3-none-any.whl (16 kB)                                                                                            subprocess.py:195
[15:59:41] Collecting idna>=2.8                                                                                                                                    subprocess.py:195
             Using cached idna-3.10-py3-none-any.whl (70 kB)                                                                                                       subprocess.py:195
           Collecting certifi                                                                                                                                      subprocess.py:195
             Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)                                                                                              subprocess.py:195
           Collecting httpcore==1.*                                                                                                                                subprocess.py:195
             Using cached httpcore-1.0.6-py3-none-any.whl (78 kB)                                                                                                  subprocess.py:195
[15:59:44] Processing                                                                                                                                              subprocess.py:195
           c:\users\revte\appdata\local\chaquopy\pip\cache\wheels\e3\2a\c3\87d389b54a9702721219edef00388a7c0f764084defdbb140f\pydantic_core-2.23.4-cp310-none-win_                  
           amd64.whl                                                                                                                                                                
           Collecting annotated-types>=0.6.0                                                                                                                       subprocess.py:195
             Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB)                                                                                           subprocess.py:195
           Collecting travertino>=0.3.0                                                                                                                            subprocess.py:195
             Using cached travertino-0.3.0-py3-none-any.whl (16 kB)                                                                                                subprocess.py:195
           Collecting h11<0.15,>=0.13                                                                                                                              subprocess.py:195
             Using cached h11-0.14.0-py3-none-any.whl (58 kB)                                                                                                      subprocess.py:195
[15:59:45] Installing collected packages: sniffio, tqdm, exceptiongroup, typing-extensions, idna, anyio, certifi, h11, httpcore, httpx, distro, jiter,             subprocess.py:195
           pydantic-core, annotated-types, pydantic, openai, travertino, toga-core, toga-android                                                                                    
[15:59:55] Successfully installed annotated-types-0.7.0 anyio-4.6.0 certifi-2024.8.30 distro-1.9.0 exceptiongroup-1.2.2 h11-0.14.0 httpcore-1.0.6 httpx-0.27.2     subprocess.py:195
           idna-3.10 jiter-0.5.0 openai-1.51.0 pydantic-2.9.2 pydantic-core-2.23.4 sniffio-1.3.1 toga-android-0.4.7 toga-core-0.4.7 tqdm-4.66.5 travertino-0.3.0                    
           typing-extensions-4.12.2 

That aside, we'll need to be a little more judicious in which versions of these packages we try to use. The openai package (and more and more of Python at large) is growing its dependency on Rust. Given that building packages using Rust isn't quite nailed down yet for Android, I was able to avoid this issue using these requirements:

requires = [
    "openai < 1.40",
    "pydantic < 2",
]

Problem was solved! Thank you very much. Changing to older openai library removes the requirement for usage of Rust!