./configure: Make --with-sage-venv=auto the default #32442

mkoeppe · 2021-08-31T19:59:32Z

Since Sage 9.4, it is possible to use an installation tree for Python packages separate from SAGE_LOCAL:
https://wiki.sagemath.org/ReleaseTours/sage-9.4#For_developers:_..2Fconfigure_--prefix.3DSAGE_LOCAL_--with-sage-venv.3DSAGE_VENV

In this ticket we make a new setting --with-sage-venv=auto the default, which means:

if --prefix has not been used (so SAGE_LOCAL=local), then set SAGE_VENV to local/var/lib/sage/venv-PYTHONVERSION
Users can also pass --with-sage-venv or --with-sage-venv=yes, which means to unconditionally set SAGE_VENV to SAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION.

(We keep the layout as is in incremental builds.)

By keying the default name to the Python version, we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJ

Moreover, at config.status time, we create a symbolic link SAGE_ROOT/venv -> SAGE_VENV (overwriting an existing symbolic link)

The link is for easy access by users and is not used otherwise. Instead of local/lib/python3/site-packages/ or local/bin/sage, use venv/local/lib/python3/site-packages/ etc.

For symmetry, we also create a symbolic link SAGE_ROOT/prefix -> SAGE_LOCAL. (This generalizes what is already done by tox local.)

Users will still be able to restore the previous behavior by using --with-sage-venv=no, or use a specific path as in --with-sage-venv=/path/to/venv.

CC: @dimpase @jhpalmieri

Component: build: configure

Author: Matthias Koeppe

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/32442

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-08-31T20:01:29Z

Changed dependencies from #32286, #32387 to #32386, #32387

mkoeppe · 2021-09-11T03:18:22Z

Branch: u/mkoeppe/__configure__make___with_sage_venv_the_default

mkoeppe · 2021-09-11T03:33:11Z

New commits:

`2f5be2d`	`configure.ac, build/pkgs/python3/spkg-configure.m4: New default --with-sage-venv=auto`
`f700640`	`configure.ac: In config.status, create convenience symlinks prefix, venv`

mkoeppe · 2021-09-11T03:33:11Z

Commit: f700640

mkoeppe · 2021-09-11T03:33:11Z

Author: Matthias Koeppe

sagetrac-git · 2021-09-15T23:44:40Z

Changed commit from f700640 to 12cbbda

sagetrac-git · 2021-09-15T23:44:40Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b4a3845`	`m4/sage_spkg_collect.m4: If we install python3 spkg, install it in SAGE_VENV`
`12cbbda`	`configure.ac: Keep old behavior of SAGE_VENV in incremental builds`

sagetrac-git · 2021-09-16T00:30:13Z

Changed commit from 12cbbda to 117fe3a

sagetrac-git · 2021-09-16T00:30:13Z

Branch pushed to git repo; I updated commit sha1. New commits:

`117fe3a`	`build/make/Makefile.in (...-clean): Do not fail trying to uninstall if SAGE_VENV has not been created yet`

mkoeppe · 2021-09-27T04:14:39Z

Changed dependencies from #32386, #32387 to none

sagetrac-git · 2021-09-27T04:15:16Z

Changed commit from 117fe3a to 8af6041

sagetrac-git · 2021-09-27T04:15:16Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8af6041`	`Merge tag '9.5.beta2' into t/32442/__configure__make___with_sage_venv_the_default`

jhpalmieri · 2021-09-27T20:34:16Z

comment:11

After running make distclean && ./configure:

% git status
On branch t/32442/__configure__make___with_sage_venv_the_default
Your branch is up to date with 'trac/u/mkoeppe/__configure__make___with_sage_venv_the_default'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	venv

nothing added to commit but untracked files present (use "git add" to track)

jhpalmieri · 2021-09-27T20:35:09Z

comment:12

Oh, and venv is a symlink pointing to local.

sagetrac-git · 2021-09-27T20:49:27Z

Changed commit from 8af6041 to b3f7d34

sagetrac-git · 2021-09-27T20:49:27Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b3f7d34`	`.gitignore: Add /venv`

mkoeppe · 2021-09-27T20:49:42Z

comment:14

Thanks for catching this!

mkoeppe · 2021-09-27T20:51:14Z

comment:15

Replying to @jhpalmieri:

Oh, and venv is a symlink pointing to local.

Yes, in an incremental build, nothing should change (except that this symlink is created).

After a make distclean, the new behavior will show

mkoeppe · 2021-09-27T20:52:06Z

comment:16

Oh, now I see that you said you did make distclean...

jhpalmieri · 2021-09-28T17:06:57Z

comment:17

Maybe this has been going on with venv builds for a while, but the symlinks look messed up to me: my local directory looks like

% ls -l local
total 0
drwxr-xr-x  148 jpalmier  staff  4736 Sep 27 16:21 bin
drwxr-xr-x    2 jpalmier  staff    64 Sep 27 16:09 etc
drwxr-xr-x   47 jpalmier  staff  1504 Sep 27 16:21 include
drwxr-xr-x   73 jpalmier  staff  2336 Sep 27 16:44 lib
lrwxr-xr-x    1 jpalmier  staff     3 Sep 27 16:09 lib64 -> lib
drwxr-xr-x    4 jpalmier  staff   128 Sep 27 16:21 libexec
lrwxr-xr-x    1 jpalmier  staff     5 Sep 27 16:09 local -> local
drwxr-xr-x   34 jpalmier  staff  1088 Sep 27 16:21 share
drwxr-xr-x    5 jpalmier  staff   160 Sep 27 16:13 var
lrwxr-xr-x    1 jpalmier  staff    33 Sep 27 16:09 venv-python3.9 -> local/var/lib/sage/venv-python3.9

In particular, local/local points to itself, and so venv-python3.9 doesn't point anywhere. venv-python3.9 should probably point to ../local/..... I don't know what the local symlink is supposed to do.

Everything still builds, so this is obviously not crucial, but it doesn't look right.

mkoeppe · 2021-09-28T17:09:48Z

comment:18

Yes, that doesn't look right. Could you post config.log please?

mkoeppe · 2021-10-13T17:24:59Z

comment:51

They are in the Docker container.
At the end of the build, tox will tell you name and/or sha256 hash of the image built.
You can to docker run -it IMAGE bash to enter the container and then it's all in /sage/logs

jhpalmieri · 2021-10-13T19:25:31Z

comment:52

Okay, before this latest change I see the Python problem, as well as:

* package:         python3-3.9.7
  last build time: Oct 13 17:49
  log file:        /sage/logs/pkgs/python3-3.9.7.log

* package:         fplll-5.4.1
  last build time: Oct 13 18:01
  log file:        /sage/logs/pkgs/fplll-5.4.1.log
  build directory: /sage/local/var/tmp/sage/build/fplll-5.4.1

* package:         cmake-3.21.0
  last build time: Oct 13 18:04
  log file:        /sage/logs/pkgs/cmake-3.21.0.log
  build directory: /sage/local/var/tmp/sage/build/cmake-3.21.0

* package:         gfortran-10.3.0
  last build time: Oct 13 18:14
  log file:        /sage/logs/pkgs/gfortran-10.3.0.log
  build directory: /sage/local/var/tmp/sage/build/gfortran-10.3.0

The other failures appear to be unrelated to this ticket: I get them with the develop branch.

mkoeppe · 2021-10-13T19:32:39Z

comment:53

Replying to @jhpalmieri:

The other failures appear to be related to this ticket: I get them with the develop branch.

Did you mean to write "unrelated to this ticket"?

jhpalmieri · 2021-10-13T21:18:19Z

comment:54

Yes, sorry, unrelated. I've edited to make it clearer in case anyone looks at this later. With the most recent branch here, Python builds correctly. I think that's good enough for another positive review.

mkoeppe · 2021-10-13T21:22:13Z

comment:55

Thank you!

By the way, I cannot reproduce the failures with the other packages, and they also built correctly on GH Actions https://github.com/sagemath/sage/runs/3863930628?check_suite_focus=true

jhpalmieri · 2021-10-13T23:20:10Z

comment:56

Maybe the issue was that my Docker was out of date, or maybe it was a memory issue. I upgraded Docker and allocated more RAM to it, and now those packages build.

jhpalmieri · 2021-10-14T00:20:49Z

comment:57

I do get another failure, though; don't know if it's related to this ticket. With tox -e docker-debian-stretch-minimal, I get:

gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
****************************************************
Package 'setuptools' is currently not installed
No legacy uninstaller found for 'setuptools'; nothing to do
python3: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory
********************************************************************************
Error building / installing setuptools for Python
********************************************************************************

real	0m0.021s
user	0m0.011s
sys	0m0.010s
************************************************************************
Error installing package setuptools-58.0.2
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the log files
  /sage/logs/pkgs/setuptools-58.0.2.log
and
  /sage/config.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/setuptools-58.0.2 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/setuptools-58.0.2' && '/sage/sage' --buildsh)
When you are done debugging, you can type "exit" to leave the subshell.

jhpalmieri · 2021-10-14T00:22:52Z

comment:58

There is a file SAGE_ROOT/venv/local/lib/libpython3.9.so.1.0. If I run ./sage --python then I get the same error.

mkoeppe · 2021-10-14T01:56:40Z

comment:59

Yes, I see this error now too.

mkoeppe · 2021-10-14T02:01:39Z

comment:61

Looks like on Linux I need to set the linker flags differently.

sagetrac-git · 2021-10-14T02:30:54Z

Changed commit from 2101b8c to 3bbc5d8

sagetrac-git · 2021-10-14T02:30:54Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3bbc5d8`	`build/pkgs/python3/spkg-build.in: Set rpath`

mkoeppe · 2021-10-14T05:44:13Z

comment:64

This fixes it for me; this time I ran a complete build

mkoeppe · 2021-10-14T22:41:08Z

comment:65

This ticket (at 2101b8c) is already on Volker's branch.
I've opened #32698 for the fix.

jhpalmieri · 2021-10-15T03:22:23Z

comment:66

This works for me with a few different builds (standard build on my machine and a few tox builds via Docker).

mkoeppe · 2021-10-19T22:01:59Z

Changed commit from 3bbc5d8 to none

mkoeppe · 2021-10-19T22:01:59Z

comment:67

This was merged along with #32698

mkoeppe · 2021-10-19T22:01:59Z

Changed branch from u/mkoeppe/__configure__make___with_sage_venv_the_default to none

mkoeppe added this to the sage-9.5 milestone Aug 31, 2021

mkoeppe added c: build: configure labels Aug 31, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~./configure: Make --with-sage-venv the default~~ ./configure: Make --with-sage-venv=auto the default Sep 11, 2021

mkoeppe added the s: needs review label Sep 11, 2021

This comment has been minimized.

Sign in to view

jhpalmieri added s: positive review and removed s: needs review labels Oct 13, 2021

jhpalmieri added s: needs work and removed s: positive review labels Oct 14, 2021

mkoeppe added s: needs review and removed s: needs work labels Oct 14, 2021

jhpalmieri added s: positive review and removed s: needs review labels Oct 15, 2021

mkoeppe removed the s: positive review label Oct 19, 2021

mkoeppe closed this as completed Oct 19, 2021

tobiasdiez mentioned this issue Jan 1, 2022

Add VS Code config for SAGE_ROOT/venv #30677

Closed

mkoeppe added a commit to mkoeppe/sage that referenced this issue Feb 17, 2023

Merge sagemath#32442

9a63841

./configure: Make --with-sage-venv=auto the default #32442

./configure: Make --with-sage-venv=auto the default #32442

Comments

mkoeppe commented Aug 31, 2021

mkoeppe commented Aug 31, 2021

This comment has been minimized.

This comment has been minimized.

mkoeppe commented Sep 11, 2021

mkoeppe commented Sep 11, 2021

mkoeppe commented Sep 11, 2021

mkoeppe commented Sep 11, 2021

sagetrac-git mannequin commented Sep 15, 2021

sagetrac-git mannequin commented Sep 15, 2021

sagetrac-git mannequin commented Sep 16, 2021

sagetrac-git mannequin commented Sep 16, 2021

This comment has been minimized.

mkoeppe commented Sep 27, 2021

sagetrac-git mannequin commented Sep 27, 2021

sagetrac-git mannequin commented Sep 27, 2021

jhpalmieri commented Sep 27, 2021

jhpalmieri commented Sep 27, 2021

sagetrac-git mannequin commented Sep 27, 2021

sagetrac-git mannequin commented Sep 27, 2021

mkoeppe commented Sep 27, 2021

mkoeppe commented Sep 27, 2021

mkoeppe commented Sep 27, 2021

jhpalmieri commented Sep 28, 2021

mkoeppe commented Sep 28, 2021

mkoeppe commented Oct 13, 2021

jhpalmieri commented Oct 13, 2021

mkoeppe commented Oct 13, 2021

jhpalmieri commented Oct 13, 2021

mkoeppe commented Oct 13, 2021

jhpalmieri commented Oct 13, 2021

jhpalmieri commented Oct 14, 2021

jhpalmieri commented Oct 14, 2021

mkoeppe commented Oct 14, 2021

mkoeppe commented Oct 14, 2021

sagetrac-git mannequin commented Oct 14, 2021

sagetrac-git mannequin commented Oct 14, 2021

mkoeppe commented Oct 14, 2021

mkoeppe commented Oct 14, 2021

jhpalmieri commented Oct 15, 2021

mkoeppe commented Oct 19, 2021

mkoeppe commented Oct 19, 2021

mkoeppe commented Oct 19, 2021