Check versions used to create the venv and auto-wipe #3432

humitos · 2017-12-21T18:52:21Z

Initial drawing of ideas on how to implement #3190

ericholscher · 2017-12-21T21:16:41Z

Looks like a good start!

humitos · 2017-12-21T22:45:01Z

This is getting better, I think. Some notes on this:

old venv without environment.json will always return is_obsolete=False to keep compatibility
when an environment is created, we save the python and build version in the environment.json file
if the environment.json file exist we use that versions to compare against the ones that the current version need to be used, if any of them differs, we wipe the environment
the environment.json is saved under each venv version so if one builder has one version and another build has another one, this doesn't interfers

Questions:

do we need to treat conda environment in a special way?
where tests for this should live? I need to see a similar test example to understand how to write them (I will need to mock json.load and other places)
do I need to test something else than is_obsolete or that's enough?

ericholscher · 2017-12-21T23:04:27Z

I think conda shouldn't be effected by this.
Probably here: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/rtd_tests/tests/test_doc_building.py#L33
That's a good start for now.

humitos · 2017-12-22T00:57:13Z

readthedocs/doc_builder/python_environments.py

+        )
+
+    @property
+    def is_obsolete(self):


I wrote this at PythonEnvironment level, but @ericholscher said that conda won't be affected by this. Should I leave this here or move it to Virtualenv and write something customized by Conda?

Honestly not sure. @willingc do you know anything about how conda will be effected by underlying docker image changes? My thought if that it's self-contained, but not really sure.

Hmm... not sure. My understanding is docker images are built in layers so if you change an underlying layer it impacts the layers above.

This would be changing out the underlying system python or conda version, I believe, is that main worry. We have the files on disk from a previous run, but we're hoping to let users change their docker image. We're planning to blow away the virtualenvs, because it breaks, but curious if conda will be effected.

@willingc if a virtualenv is created with python 2.7.13 and we change the python version to 2.7.14, the venv stop working and need to be re-created. In the case of Conda we are not sure about that case but also the case that the conda environment was created with conda 4.x and then it's ran with conda 4.(x+1). What would happen in this case? Do you know?

ccing @juanlu001 who was my Conda teacher :)

humitos · 2017-12-22T00:58:33Z

readthedocs/rtd_tests/tests/test_doc_building.py

+            config=config,
+        )
+        env_json_data = '{"build": {"image": "readthedocs/build:2.0"}, "python": {"version": 3.5}}'
+        with patch('os.path.exists') as exists, patch('readthedocs.doc_builder.python_environments.open', mock_open(read_data=env_json_data)) as _open:  # noqa


These with patch() lines are terrible, but I didn't find a better way to write them :(

It's fine to escape with backslash here according to pep8 https://www.python.org/dev/peps/pep-0008/#maximum-line-length

humitos · 2017-12-22T00:59:11Z

readthedocs/rtd_tests/tests/test_doc_building.py

+            exists.return_value = True
+            self.assertTrue(python_env.is_obsolete)
+
+    @pytest.mark.xfail(reason='build.image is not being considered yet')


to make this test pass, we need the PR that get the image from yaml merged... then we can remove this line and it should pass :)

ericholscher

Looks like a great change to me!

ericholscher · 2017-12-22T19:43:54Z

readthedocs/doc_builder/python_environments.py

+        """Return the path to the ``environment.json`` file for this venv."""
+        return os.path.join(
+            self.venv_path(),
+            'environment.json',


I do wonder if this should be namespaced a bit more (readthedocs-environment.json?) so that it doesn't conflict with anything in the future.

ericholscher · 2017-12-22T19:44:30Z

readthedocs/doc_builder/python_environments.py

+
+        :rtype: bool
+        """
+        # Always return True if we don't have information about what Python


This returns False.

Ha! I did it wrong at first, then I fix the return statement and forgot the comment :)

ericholscher · 2017-12-27T21:37:52Z

Is this now clearing conda envs? If so, I think we're good to go.

humitos · 2017-12-28T01:03:15Z

Is this now clearing conda envs? If so, I think we're good to go.

Yes. It's clearing the conda env if it's was created with a different python/build image version; but it's not checking the conda version.

I think it's should be a good start but maybe we will want to check the conda version also in the future --I'm not sure if that could affect or not, really.

ericholscher · 2017-12-28T01:10:16Z

Great. Have you tested it locally w/ the docker image changes and virtualenv? If that works, it would be great to get this all merged so we can test it locally before deploying it next week.

humitos · 2017-12-28T02:15:51Z

I tested this manually with the python version change from the YAML and docker image from project.container_image since using the YAML for the build image depends on #3339 and readthedocs/readthedocs-build#33

Besides,

I just add a new commit with a change to make it py2 and py3 compatible when saving the json to disk.
I create a new method to get the python_full_version (making a refactor of the existent code) so we get and save the minor and major version to the json file
I added a test for changing the container_image from the Admin (at project level)

Finally, I just realized that the conda env is always deleted if its exist at these lines:

https://github.com/rtfd/readthedocs.org/blob/da1fde31d1756cb3f2790f0f5c753c7c1c0cc595/readthedocs/doc_builder/python_environments.py#L295-L298

Those lines were added at 475cf0b

So, if we are going to leave those lines there it seems useless to add the logic of this PR at PythonEnvironment class level, and should be moved to Virtualenv class I think.

Please, take a look again at these changes and give me your feedback.

astrojuanlu · 2017-12-28T11:21:55Z

Sorry I didn't have the time to review this. Just let me add that be careful with conda 4.4, since it breaks a number of things and it's causing some problems.

…

On Dec 23, 2017 1:20 AM, "Manuel Kaufmann" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In readthedocs/doc_builder/python_environments.py <#3432 (comment)>: > @@ -87,6 +98,70 @@ def venv_bin(self, filename=None): parts.append(filename) return os.path.join(*parts) + def environment_json_path(self): + """Return the path to the ``environment.json`` file for this venv.""" + return os.path.join( + self.venv_path(), + 'environment.json', + ) + + @Property + def is_obsolete(self): @willingc <https://github.com/willingc> if a virtualenv is created with python 2.7.13 and we change the python version to 2.7.14, the venv stop working and need to be re-created. In the case of Conda we are not sure about that case but also the case that the conda environment was created with conda 4.x and then it's ran with conda 4.(x+1). What would happen in this case? Do you know? ccing @Juanlu001 <https://github.com/juanlu001> who was my Conda teacher :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3432 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATUZT2rKaYqCrDHPp_qA241ExsgLKI2ks5tDEcvgaJpZM4RKNru> .

humitos added the PR: work in progress Pull request is not ready for full review label Dec 21, 2017

humitos commented Dec 22, 2017

View reviewed changes

humitos added PR: ready for review and removed PR: work in progress Pull request is not ready for full review labels Dec 22, 2017

ericholscher approved these changes Dec 22, 2017

View reviewed changes

humitos added 9 commits December 27, 2017 20:20

Check versions used to create the venv and auto-wipe

d4d355c

Save environment.json file

1541bf0

Open file properly

85d7603

Make it compatible with current master

05c2aed

Minor refactor

e99673a

Fix linting errors

d38b5c3

Test cases for PythonEnvironment.is_obsolete

b7a4cc1

Comment fix

49ed943

Rename environment file

da1fde3

humitos force-pushed the humitos/wipe/automatically branch from 3e464e2 to da1fde3 Compare December 28, 2017 01:24

humitos added 3 commits December 27, 2017 20:53

Make json.dump compatible with py2 and py3

3b2f538

Use config.python_full_version instead of config.python_version

a151cde

Test for changing container_image at project level

f31144a

ericholscher merged commit 1a4bba1 into master Dec 28, 2017

humitos mentioned this pull request Jan 2, 2018

Support opt in to latest build image, automate wiping of environment #3190

Closed

4 tasks

stsewd deleted the humitos/wipe/automatically branch August 15, 2018 22:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check versions used to create the venv and auto-wipe #3432

Check versions used to create the venv and auto-wipe #3432

humitos commented Dec 21, 2017

ericholscher commented Dec 21, 2017

humitos commented Dec 21, 2017

ericholscher commented Dec 21, 2017

humitos Dec 22, 2017

ericholscher Dec 22, 2017

willingc Dec 22, 2017

ericholscher Dec 22, 2017

humitos Dec 23, 2017

humitos Dec 22, 2017

stsewd Dec 22, 2017

humitos Dec 22, 2017

ericholscher left a comment

ericholscher Dec 22, 2017

humitos Dec 22, 2017

ericholscher Dec 22, 2017

humitos Dec 22, 2017

ericholscher commented Dec 27, 2017

humitos commented Dec 28, 2017

ericholscher commented Dec 28, 2017

humitos commented Dec 28, 2017

astrojuanlu commented Dec 28, 2017 via email

Check versions used to create the venv and auto-wipe #3432

Check versions used to create the venv and auto-wipe #3432

Conversation

humitos commented Dec 21, 2017

ericholscher commented Dec 21, 2017

humitos commented Dec 21, 2017

ericholscher commented Dec 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Dec 27, 2017

humitos commented Dec 28, 2017

ericholscher commented Dec 28, 2017

humitos commented Dec 28, 2017

astrojuanlu commented Dec 28, 2017 via email