CRUD for EnvironmentVariables from Project's admin #4899

humitos · 2018-11-13T11:41:54Z

This PR allows users to add environment variables to be used at build time. This way, they can include secrets (or configs) that should not be in the repository.

Phase II from #3992 | PR branched from #4894

Missing things:

django templates
crud tests
form restriction tests
phase I merged (Expose environment variables from database into build commands #4894)

I only let the user to LIST, CREATE and DELETE environment variables since I don't think it's a good idea to be able to "Edit" and show a sensible secret on the browser. Also, if that's a secret that only one of the owners knows we should not expose that value to all the owners of the project.

codecov · 2018-11-13T11:57:26Z

Codecov Report

Merging #4899 into master will increase coverage by 0.12%.
The diff coverage is 95.12%.

@@            Coverage Diff             @@
##           master    #4899      +/-   ##
==========================================
+ Coverage   76.74%   76.87%   +0.13%     
==========================================
  Files         158      158              
  Lines        9951     9981      +30     
  Branches     1244     1246       +2     
==========================================
+ Hits         7637     7673      +36     
+ Misses       1980     1976       -4     
+ Partials      334      332       -2

Impacted Files	Coverage Δ
readthedocs/projects/models.py	`85.56% <100%> (ø)`	⬆️
readthedocs/projects/urls/private.py	`100% <100%> (ø)`	⬆️
readthedocs/projects/forms.py	`80.63% <100%> (+0.97%)`	⬆️
readthedocs/restapi/serializers.py	`96.25% <50%> (-1.19%)`	⬇️
readthedocs/projects/views/private.py	`80.55% <93.33%> (+0.5%)`	⬆️
readthedocs/gold/views.py	`66.17% <0%> (ø)`	⬆️
readthedocs/doc_builder/environments.py	`87.06% <0%> (+0.78%)`	⬆️
readthedocs/vcs_support/backends/git.py	`82.27% <0%> (+1.14%)`	⬆️
readthedocs/core/signals.py	`86.66% <0%> (+4.66%)`	⬆️

humitos

Questions about adding a paragraph explaining detail and list pages.

humitos · 2018-12-04T18:11:13Z

readthedocs/templates/projects/environmentvariable_detail.html

+{% endblock %}
+
+{% block project_edit_content %}
+  <form method="post" action="{% url 'projects_environmentvariables_delete' project_slug=project.slug environmentvariable_pk=environmentvariable.pk %}">


What do we want to communicate on Environment Variable detail page to the user?

We probably need to explain that "These environment variables are available to all build steps." This is another place to keep UI prose copy short.

humitos · 2018-12-04T18:11:59Z

readthedocs/templates/projects/environmentvariable_list.html

+      <li>
+        <a class="button"
+           href="{% url 'projects_environmentvariables_create' project_slug=project.slug %}">
+          {% trans "Add Environment Variable" %}


What do we want to communicate on Environment Variable list page to the user?

I like Heroku's example. Just a short blob explaining env vars and point to the docs for anything substantial. I'm not big on prose content in UI.

davidfischer

I like the direction this is going. I do think we need a way to reveal and edit the envvar value. Currently once set, it cannot be edited or even viewed! I think there are a couple decent patterns here and I attached some screenshots.

I also think we should have a little bit of verbiage on the envvar list view page along the lines of:

Environment variables are a way to store settings that are used in your documentation builds.
They are a good place to store any private data such as API keys that your build may need.

Some other sites UIs for reference

Azure

By default, Azure hides the values but you can show them.

Heroku

Heroku hides the keys and values until you hit "reveal config vars" and then it shows names and values.

davidfischer · 2018-12-04T18:27:30Z

readthedocs/templates/projects/environmentvariable_form.html

+  <p>
+    {% blocktrans trimmed %}
+      Notice that the values are not escaped when your builds are executed.
+      Special characters (for bash) should be escaped accordingly.


I think we should escape these sorts of things. Think a long random secret key that might have some special characters like $ in it. The user should be able to just copy and paste it in.

Agreed, let's make it easier for the user. This is a technical implementation detail that the user shouldn't have to worry about

Good point.

I don't want to do this by myself. I checked the code and I found that we are already doing something similar at

https://github.com/rtfd/readthedocs.org/blob/089753253397a0399175969e2e028eecc3b74dd2/readthedocs/doc_builder/environments.py#L294-L295

I found that there is shlex.quote in Python3 and pipes.quote in Python2. I will use this accordingly.

NOTE: we may want to replace our bash_escape_re regex also.

I tested this with,

enter: {1}\|\1+-(8)$

results: "'{1}\\|\\1+-(8)$'"

and,

enter: alo'ja

results: '\'alo\'"\'"\'ja\''

are they correct?

humitos · 2018-12-04T18:57:20Z

@davidfischer I like your feedback. At the moment, I just kept this simple and I based it on what Travis does: do not edit and do not scape (in fact, I copied the message from their settings page)

Currently once set, it cannot be edited or even viewed!

This was on purpose since I was not sure how we want to manage this. I put an example on the PR description about viewing some secrets added by another owner of the project that the user that is seeing them may not be able to see them.

I found that Travis behaves in this way. Also, GitHub for example does not allow you to reveal "Personal Access Tokens" once they are created and copied to the clipboard. It's not exactly the same use case, but similar in the sense they are secrets.

It's pretty easy to show a form to update them. I mean, it's not a technical issue. We only need to make a decision about this.

davidfischer · 2018-12-04T19:00:50Z

I put an example on the PR description about viewing some secrets added by another owner of the project that the user that is seeing them may not be able to see them.

I had not considered this but that's a pretty good reason to not make them viewable or editable.
I'm fine if we decide to go that route, but I think we should be up front about it. We should say on the list view and the form view that they cannot be seen or edited once set.

humitos · 2018-12-04T19:07:51Z

I had not considered this but that's a pretty good reason to not make them viewable or editable.

There are more things that could be considered here. Travis, for example, shows a boolean saying "Display value on build log". We could add a boolean to mark a variable as "Secret" and do not display/allow to edit only these ones.

Although, I'm not sure how if this complexity worth in the end.

agjohnson · 2018-12-08T00:37:03Z

Let's keep this simple to start. I like the plan for making the UI read only as the token is created. If we want to make the UI fancier later, we can revisit this as a new feature.

agjohnson

I'll pull this down next week and test, or feel free to attach some screen grabs if you get to this before I can test. I prefer to keep UI copy minimal -- UI with a lot of copy usually indicates your UI is not obivous. We'll link to doc pages with lots of information for users instead.

I like the pattern of keeping this UI as simple as possible for now. Read once on envvar input makes a lot of sense for a few reasons, at least to start.

agjohnson · 2018-12-08T00:39:46Z

readthedocs/templates/projects/environmentvariable_detail.html

+{% endblock %}
+
+{% block project_edit_content %}
+  <form method="post" action="{% url 'projects_environmentvariables_delete' project_slug=project.slug environmentvariable_pk=environmentvariable.pk %}">


We probably need to explain that "These environment variables are available to all build steps." This is another place to keep UI prose copy short.

agjohnson · 2018-12-08T00:40:44Z

readthedocs/templates/projects/environmentvariable_form.html

+  <p>
+    {% blocktrans trimmed %}
+      Notice that the values are not escaped when your builds are executed.
+      Special characters (for bash) should be escaped accordingly.


Agreed, let's make it easier for the user. This is a technical implementation detail that the user shouldn't have to worry about

agjohnson · 2018-12-08T00:41:41Z

readthedocs/templates/projects/environmentvariable_list.html

+      <li>
+        <a class="button"
+           href="{% url 'projects_environmentvariables_create' project_slug=project.slug %}">
+          {% trans "Add Environment Variable" %}


I like Heroku's example. Just a short blob explaining env vars and point to the docs for anything substantial. I'm not big on prose content in UI.

humitos · 2018-12-10T10:14:23Z

Some screenshots,

Empty list view,

Create/Add view,

Non-empty list view,

Validation error,

I think we need more validations on the name, like only allow a limited number of ASCI chars (letters, numbers and underscore).

Delete view,

This one is too poor :/

stsewd · 2018-12-13T03:02:34Z

readthedocs/projects/forms.py

+            raise forms.ValidationError(
+                _("Variable name can't start with READTHEDOCS"),
+            )
+        elif self.project.environmentvariable_set.filter(name=name).exists():


We could add this as a constraint in the model

Yes. I suppose that we still need the validation in the form otherwise I think it will explode when trying to save it and the user will have a 500.

humitos · 2018-12-17T16:56:12Z

I think we need more validations on the name, like only allow a limited number of ASCI chars (letters, numbers and underscore).

This is already done via a regex.

humitos · 2018-12-17T16:57:58Z

The only missing thing here could be to decide what to show in the detail page of the environment variable, otherwise, we could merge this PR.

I'm not really sure what kind of text to add there since there are not too much to say or to show really because we don't want to show the value.

stsewd · 2018-12-17T17:04:20Z

Maybe we can add some user docs? and/or maybe update https://docs.readthedocs.io/en/latest/builds.html

humitos · 2018-12-18T09:55:01Z

Maybe we can add some user docs? and/or maybe update docs.readthedocs.io/en/latest/builds.html

Good idea! I added a simple .. tip:: communicating this in the "Build environment" section.

Add the `environment_variables` field to this serializer that will be returned only when the user is admin.

ericholscher

Looks good pending more docs and a bit more copy.

ericholscher · 2019-01-10T16:36:55Z

readthedocs/templates/projects/environmentvariable_detail.html

+    Environment Variable: {{ name }}
+  {% endblocktrans %}
+{% endblock %}
+


I think we need copy here saying "The value here is not shown for security purposes" or similar. Otherwise it's a weird UI.

ericholscher · 2019-01-10T16:37:46Z

docs/builds.rst

+.. tip::
+
+   In case extra environment variables are needed to the build process (like secrets, tokens, etc),
+   you can add them going to **Admin > Environment Variables** in your project.


I think we need more docs than this. We should have a guide in guides/, something along the lines of "I need secrets or environment variables in my build" that mentions the use case of keeping secrets.

I pushed a guide for this.

It mention only one example for now (using the secret to access to a user/pass protected URL). I'm not sure if we want to mention other cases or something else.

ericholscher

Changes look good. Let's ship it! 👍

ericholscher · 2019-01-15T16:00:03Z

docs/guides/environment-variables.rst

+
+   Values will never be exposed to users, even to owners of the project. Once you create an environment variable you won't be able to see its value anymore because of security purposes.
+
+After adding an environment variable from your project's admin, you can access it from your build process using Python, for example:


💯 addition.

humitos added the PR: work in progress Pull request is not ready for full review label Nov 13, 2018

humitos force-pushed the humitos/admin/crud-env-variables branch from 02b0742 to cdc848d Compare November 13, 2018 11:57

humitos force-pushed the humitos/admin/crud-env-variables branch from cdc848d to 2657a19 Compare December 4, 2018 15:44

humitos commented Dec 4, 2018

View reviewed changes

humitos removed the PR: work in progress Pull request is not ready for full review label Dec 4, 2018

davidfischer reviewed Dec 4, 2018

View reviewed changes

humitos requested a review from a team December 4, 2018 19:15

agjohnson reviewed Dec 8, 2018

View reviewed changes

humitos force-pushed the humitos/admin/crud-env-variables branch from 0d1c4eb to feaba50 Compare December 10, 2018 09:59

humitos requested a review from a team December 10, 2018 10:15

humitos added this to the 2.9 milestone Dec 10, 2018

stsewd reviewed Dec 13, 2018

View reviewed changes

stsewd mentioned this pull request Dec 15, 2018

How to pass private config vars to RTD? #5007

Closed

humitos force-pushed the humitos/admin/crud-env-variables branch 2 times, most recently from f0d37c0 to 8a002b9 Compare December 17, 2018 16:55

humitos requested a review from a team December 17, 2018 16:56

humitos force-pushed the humitos/admin/crud-env-variables branch 2 times, most recently from 65e3022 to ee5bb12 Compare December 26, 2018 10:52

humitos added 2 commits December 27, 2018 13:29

Use project detail endpoint with admin serializer

1884e62

Add the `environment_variables` field to this serializer that will be returned only when the user is admin.

Form and views for EnvironmentVariable CRUD from Project's Admin

ac572ba

humitos added 12 commits December 27, 2018 13:29

Admin tab menu to List, Create and Delete

f01bbbe

Form test for environment variables

9a20328

Add environmentvariable_pk for testing URLs

74274cf

Fix test to delete an environment variable

63b208d

Lint

26eba5a

Escape values for environment variables before saving them

67d29a4

Add copy for list page

88d70b5

Validate that name does not contain spaces on it

74216af

Test for escaped variables

b427496

Validate with a regex for valid chars

3381e85

Coherence in style when mentioning the Admin steps

bd0b9fa

Tip communicating that environment variables can be added manually

dec39ba

humitos force-pushed the humitos/admin/crud-env-variables branch from ee5bb12 to dec39ba Compare December 27, 2018 12:30

Use re.fullmatch for Py3 and custom fullmatch for Py2

e7a2ea9

humitos force-pushed the humitos/admin/crud-env-variables branch from 2beb9e2 to e7a2ea9 Compare January 3, 2019 16:59

ericholscher approved these changes Jan 10, 2019

View reviewed changes

humitos added 2 commits January 14, 2019 12:14

Add message on env variable detail

663f383

Add a document guide with more documentation

ca5cf05

humitos requested a review from ericholscher January 14, 2019 11:47

ericholscher approved these changes Jan 15, 2019

View reviewed changes

humitos merged commit 35856b0 into master Jan 15, 2019

delete-merged-branch bot deleted the humitos/admin/crud-env-variables branch January 15, 2019 17:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRUD for EnvironmentVariables from Project's admin #4899

CRUD for EnvironmentVariables from Project's admin #4899

humitos commented Nov 13, 2018 •

edited

Loading

codecov bot commented Nov 13, 2018 •

edited

Loading

humitos left a comment

humitos Dec 4, 2018

agjohnson Dec 8, 2018

humitos Dec 4, 2018

agjohnson Dec 8, 2018

davidfischer left a comment

davidfischer Dec 4, 2018

agjohnson Dec 8, 2018

humitos Dec 10, 2018

humitos Dec 10, 2018 •

edited

Loading

humitos commented Dec 4, 2018

davidfischer commented Dec 4, 2018

humitos commented Dec 4, 2018

agjohnson commented Dec 8, 2018

agjohnson left a comment

agjohnson Dec 8, 2018

agjohnson Dec 8, 2018

agjohnson Dec 8, 2018

humitos commented Dec 10, 2018

stsewd Dec 13, 2018

humitos Dec 13, 2018

humitos commented Dec 17, 2018

humitos commented Dec 17, 2018

stsewd commented Dec 17, 2018

humitos commented Dec 18, 2018

ericholscher left a comment

ericholscher Jan 10, 2019

ericholscher Jan 10, 2019 •

edited

Loading

humitos Jan 14, 2019

ericholscher left a comment

ericholscher Jan 15, 2019


		Values will never be exposed to users, even to owners of the project. Once you create an environment variable you won't be able to see its value anymore because of security purposes.

		After adding an environment variable from your project's admin, you can access it from your build process using Python, for example:

CRUD for EnvironmentVariables from Project's admin #4899

CRUD for EnvironmentVariables from Project's admin #4899

Conversation

humitos commented Nov 13, 2018 • edited Loading

codecov bot commented Nov 13, 2018 • edited Loading

Codecov Report

humitos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidfischer left a comment

Choose a reason for hiding this comment

Some other sites UIs for reference

Azure

Heroku

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos Dec 10, 2018 • edited Loading

Choose a reason for hiding this comment

humitos commented Dec 4, 2018

davidfischer commented Dec 4, 2018

humitos commented Dec 4, 2018

agjohnson commented Dec 8, 2018

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Dec 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Dec 17, 2018

humitos commented Dec 17, 2018

stsewd commented Dec 17, 2018

humitos commented Dec 18, 2018

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher Jan 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Nov 13, 2018 •

edited

Loading

codecov bot commented Nov 13, 2018 •

edited

Loading

humitos Dec 10, 2018 •

edited

Loading

ericholscher Jan 10, 2019 •

edited

Loading