Add docker image from the YAML config integration #3339

ericholscher · 2017-11-30T20:54:39Z

This depends on readthedocs/readthedocs-build#33

humitos · 2017-11-30T23:17:23Z

readthedocs/doc_builder/config.py

+    def build_image(self):
+        if self._project.container_image:
+            # Allow us to override per-project still
+            assert 'readthedocs/build' in self._project.container_image, (


How does this assert work?

Shouldn't we raise a proper exception like ConfigError or InvalidConfig here?

seems we shouldn't need to do validation here at all actually -- container_image is admin only already.

Yea, I know I've set it wrong before in the admin, so like having clarity there. Can definitely raise other errors though :)

humitos · 2017-11-30T23:22:17Z

readthedocs/doc_builder/config.py

    try:
-        sphinx_env_config = env_config.copy()
+        sphinx_env_config = {}
        sphinx_env_config.update({


You can just create the dict at this line instead of .update :)

humitos · 2017-11-30T23:27:23Z

readthedocs/doc_builder/config.py

+            assert 'readthedocs/build' in self._project.container_image, (
+                'container image must be fully qualified')
+            return self._project.container_image
+        return 'readthedocs/build:{}'.format(self._yaml_config['build']['image'])


Do we want the user to just put the last part?

Would it be interesting if I can put something like humitos/rtd:latest and install whatever I want in the docker image? Would it be too dangerous? Would it bring a lot of problems?

Just an idea (probably stupid, but...) / comment...

I don't think we want users installing their own images on the server. For private instances, I think the addition of DOCKER_BUILD_IMAGES setting made a lot of sense and serves this purpose.

Also, this shouldn't hard code the build image prefix here. Instead, just do lookups on DOCKER_BUILD_IMAGES setting for full build image keys in validation on readthedocs-build

agjohnson

Definitely a good step to a more complete config. Similar to the readthedocs-build change, there are a few points of regression in configuration and user reporting that i think this needs to address, but this mostly looks to be due to moved logic.

agjohnson · 2017-12-01T01:48:44Z

docs/yaml-config.rst

+
+The docker image to use for specific builds.
+This lets users specify a more experimental builder,
+if they want to be on the cutting edge.


For consistency and clarity:

"docker image" -> "build image"
"builder" -> "build image"

agjohnson · 2017-12-01T01:49:11Z

docs/yaml-config.rst

+if they want to be on the cutting edge.
+
+Certain Python versions require a certain builder,
+as defined here::


Same on "builder" here, builder is an internal term, this refers to the build image instead.

agjohnson · 2017-12-01T01:54:24Z

docs/yaml-config.rst

+
+* '1.0': 2, 2.7, 3, 3.4
+* '2.0': 2, 2.7, 3, 3.5
+* 'latest': 2, 2.7, 3, 3.3, 3.4, 3.5, 3.6


You might be able to pull this from settings directly, similar to settings.rst. Though i think it's only doing key name lookups now.

Literal formatting would make this more clear

agjohnson · 2017-12-01T01:55:48Z

readthedocs/doc_builder/config.py

@@ -59,7 +57,7 @@ def python_interpreter(self):
                list(
                    filter(
                        lambda x: x < ver + 1,
-                        self._yaml_config.get_valid_python_versions(),
+                        self._yaml_config.PYTHON_SUPPORTED_VERSIONS,


Raised issue on similar change in readthedocs-build, will require change here too.

agjohnson · 2017-12-01T01:57:39Z

readthedocs/doc_builder/constants.py

-        'python': {'supported_versions': [2, 2.7, 3, 3.3, 3.4, 3.5, 3.6]},
-    },
-}
-DOCKER_BUILD_IMAGES.update(getattr(settings, 'DOCKER_BUILD_IMAGES', {}))


I think keeping full build image names here on readthedocs-build makes sense, as we don't have to hard code any prefixes.

agjohnson · 2017-12-01T02:00:25Z

readthedocs/doc_builder/config.py

+            assert 'readthedocs/build' in self._project.container_image, (
+                'container image must be fully qualified')
+            return self._project.container_image
+        return 'readthedocs/build:{}'.format(self._yaml_config['build']['image'])


I don't think we want users installing their own images on the server. For private instances, I think the addition of DOCKER_BUILD_IMAGES setting made a lot of sense and serves this purpose.

Also, this shouldn't hard code the build image prefix here. Instead, just do lookups on DOCKER_BUILD_IMAGES setting for full build image keys in validation on readthedocs-build

agjohnson · 2017-12-01T02:37:52Z

docs/yaml-config.rst

+```````````
+
+* Default: ``2.0``
+* Options: ``1.0``, ``2.0``, ``latest``


With settings re-enabled, this can be pulled from settings. See settings.rst and doc_extensions.py in conf/

agjohnson · 2017-12-01T02:51:43Z

readthedocs/doc_builder/config.py

+    def build_image(self):
+        if self._project.container_image:
+            # Allow us to override per-project still
+            assert 'readthedocs/build' in self._project.container_image, (


seems we shouldn't need to do validation here at all actually -- container_image is admin only already.

agjohnson · 2017-12-01T03:02:57Z

readthedocs/rtd_tests/tests/test_config_wrapper.py

-            }),
-        ])
-        self.assertEqual(config.python_version, 2)
-


These are some of the tests I was looking for in readthedocs-build. I think these tests should be repaired or moved to readthedocs-build, and this should make sure this logic is preserved.

ericholscher · 2017-12-05T20:05:27Z

@agjohnson @humitos these should be ready for another review.

humitos · 2017-12-05T22:30:51Z

readthedocs/doc_builder/config.py

@@ -133,8 +133,24 @@ def load_yaml_config(version):
    parsing consistent between projects.
    """
    checkout_path = version.project.checkout_path(version.slug)
+    env_config = {}


This one is not needed. You are re-creating it some lines below.

…tion

agjohnson · 2017-12-11T07:40:07Z

Before this can go out, we need to address #3190. Changing your docker image will break your builds at the moment, as the virtualenv has python 2.7 from the 2.0 image, and when we all of a sudden use the python 2.7 from latest, parts of the virtualenv break.

agjohnson · 2017-12-05T22:39:10Z

readthedocs/doc_builder/constants.py

+old_config = getattr(settings, 'DOCKER_BUILD_IMAGES', None)
+if old_config:
+    log.warning('Old config detected, DOCKER_BUILD_IMAGES->DOCKER_IMAGE_SETTINGS')
+    DOCKER_IMAGE_SETTINGS.update(old_config)


This warning could be a good use of a django system check warning as well.

agjohnson · 2017-12-11T07:46:07Z

Looks good, we can discuss best way to wipe venv, or best way to detect when to wipe the venv.

Reminder before merge here, requirements needs updating.

* ref: readthedocs/readthedocs.org#3339

ericholscher added 2 commits November 30, 2017 12:54

Add docker image from the YAML config integration

442c68a

Fix tests

961c18e

ericholscher mentioned this pull request Nov 30, 2017

Add ability to specify which version of Docker image to use #3338

Closed

ericholscher requested review from agjohnson and humitos November 30, 2017 21:48

Bump build dep for tests

aa15e78

ericholscher force-pushed the add-docker-image-option branch from 7d45b6a to aa15e78 Compare November 30, 2017 22:34

humitos reviewed Nov 30, 2017

View reviewed changes

agjohnson reviewed Dec 1, 2017

View reviewed changes

ericholscher added 6 commits December 5, 2017 09:27

Address review feedback and remove most of my deleted code :)

a705f9d

A bit more clenaup

eb7c4b2

Clean up setting names

8d1a58d

Support backwards compat

c27686e

Fix tests

67d5595

Bump test commit

5708f10

humitos reviewed Dec 5, 2017

View reviewed changes

ericholscher added 3 commits December 8, 2017 14:22

Support passing full setting through

edfca36

Merge remote-tracking branch 'origin/master' into add-docker-image-op…

d0ee0f7

…tion

Fix lint

da7ec2b

agjohnson added the Status: blocked Issue is blocked on another issue label Dec 11, 2017

agjohnson reviewed Dec 11, 2017

View reviewed changes

humitos mentioned this pull request Dec 17, 2017

Support for UML via PlantUML & sphinxcontrib-plantuml #407

Closed

ericholscher mentioned this pull request Dec 21, 2017

Show syntax error with Python 3.6 #2581

Closed

humitos mentioned this pull request Dec 28, 2017

Check versions used to create the venv and auto-wipe #3432

Merged

ericholscher merged commit da7ec2b into master Dec 28, 2017

achimnol added a commit to achimnol/aiotools that referenced this pull request Jan 3, 2018

docs: Add explicit image config for RTD

085d94a

* ref: readthedocs/readthedocs.org#3339

matt-gardner mentioned this pull request Jan 4, 2018

Update README.md allenai/allennlp#662

Merged

funkyfuture mentioned this pull request Jan 6, 2018

Build docs on rtd w/ Python 3.6 funkyfuture/deck-chores#30

Closed

stsewd deleted the add-docker-image-option branch August 15, 2018 22:15

stsewd mentioned this pull request Jan 15, 2019

Use default settings for Config object #5056

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docker image from the YAML config integration #3339

Add docker image from the YAML config integration #3339

ericholscher commented Nov 30, 2017 •

edited by agjohnson

Loading

humitos Nov 30, 2017

agjohnson Dec 1, 2017

ericholscher Dec 5, 2017

humitos Nov 30, 2017

humitos Nov 30, 2017

agjohnson Dec 1, 2017

agjohnson left a comment

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

agjohnson Dec 1, 2017

ericholscher commented Dec 5, 2017

humitos Dec 5, 2017

agjohnson commented Dec 11, 2017

agjohnson Dec 5, 2017

agjohnson commented Dec 11, 2017

Add docker image from the YAML config integration #3339

Add docker image from the YAML config integration #3339

Conversation

ericholscher commented Nov 30, 2017 • edited by agjohnson Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Dec 5, 2017

Choose a reason for hiding this comment

agjohnson commented Dec 11, 2017

Choose a reason for hiding this comment

agjohnson commented Dec 11, 2017

ericholscher commented Nov 30, 2017 •

edited by agjohnson

Loading