[R][CI]: failing to activate python venv after installation

[jonkeane]

Describe the bug, including details regarding any error messages, version, and platform.

Starting ~2weeks ago, the test-r-offline-maximal job has started to fail when it attempts to activate the venv:

Successfully installed pyarrow-20.0.0.dev120
  'test-python.R:33:3' [success]
ERROR: The requested version of Python
('/home/runner/.virtualenvs/arrow-test/bin/python') cannot be used, as
another version of Python
('/home/runner/.cache/R/reticulate/uv/cache/archive-v0/CVdoePB5wstbsp_1TXeaj/bin/python3')
has already been initialized. Please restart the R session if you need
to attach reticulate to a different version of Python.
Error:   'test-python.R:35:3' [error]
End test: install_pyarrow

This coincides with a reticulate release, which is likely the cause of the behavior (IIRC, they started using uv under the hood by default there and given the message...). But the strange thing is: why isn't this failing elsewhere? Do we not have tests that run into this pyarrow issue?

Component(s)

Continuous Integration, R

[amoeba]

I'm not sure on your question about why this isn't failing but the failure can be fixed by commenting out this line:

arrow/r/tests/testthat/test-python.R

Line 26 in d8d7e79

skip_on_python_older_than("3.9")

and the failure can be reproduced with just reticulate. Our code is basically doing this:

library(reticulate)

# skip_on_python_older_than() basically runs this reticulate code
reticulate::py_available(initialize = TRUE) # <--- this indirectly creates a managed uv environment
reticulate::py_version()

reticulate::virtualenv_create("my-venv")
reticulate::py_install("pyarrow", "my-venv")
reticulate::use_virtualenv("my-venv")
# ERROR: The requested version of Python ('/Users/bryce/.virtualenvs/my-venv/bin/python') cannot be used, as another version of Python
# ('/Users/bryce/Library/Caches/org.R-project.R/R/reticulate/uv/cache/archive-v0/ixMLW89TxW5NZvnI1CqYP/bin/python3') has already been initialized.
# Please restart the R session if you need to attach reticulate to a different version of Python.
# Error in use_python(python, required = required) : 
#  failed to initialize requested version of Python

I'm not sure about a workaround but I guess we could refactor things to just use py_require, as described in https://rstudio.github.io/reticulate/articles/package.html.

[amoeba]

I filed rstudio/reticulate#1781 to report the change in behavior and see if there's a good workaround. I did a bit of testing and I think we could refactor the reticulate stuff to just use py_require but that's a more drastic change than I'd like.

[amoeba]

From the response on rstudio/reticulate#1781, I think we may just need to tweak how we detect the available Python interpreter versions so it doesn't use and initialize reticulate. I'll work on that.

[amoeba]

PR ready for review: #46079