Make tests extensible from corporate site #4095

humitos · 2018-05-15T18:32:49Z

This PR allows us to import the community test suite under the corporate test suite and extend them as well as skip some test cases that are specific for the community site and we don't need in the corporate project.

Related https://github.com/readthedocs/readthedocs-corporate/pull/329
Related https://github.com/readthedocs/readthedocs-corporate/pull/330

humitos · 2018-05-15T18:33:59Z

readthedocs/rtd_tests/tests/test_core_tags.py

 class CoreTagsTests(TestCase):
    fixtures = ["eric", "test_data"]

+    # Determine if we are under community (http) or corporate site (https)
+    scheme = 'https' if settings.CELERY_APP_NAME == 'readthedocsinc' else 'http'


I don't like this.

We should find a different way to know where we are here to decide the user of http or https.

Same. Perhaps there is a setting we could use or add? This is very fragile and not something we should reference in the .org code (we don't ever reference our commercial hosting code from the community side)

I'm using conftest.py to set these options. Example,

PYTEST_OPTIONS = ( # Include `not community` to parameters so that community tests do not run ('markexpr', 'not community'), # Options to set test environment ('community', False), ('environment', 'readthedocsinc'), ('url_scheme', 'https'), ) def pytest_configure(config): for option, value in PYTEST_OPTIONS: setattr(config.option, option, value)

So, this way in the test case I'm doing pytest.config.option.url_scheme which will be https in the corporate site and http in the community.

This way, we will have all the test options under this file and easy to tweak/modify/understand where they come from.

humitos · 2018-05-15T18:34:27Z

readthedocs/rtd_tests/tests/test_domains.py

@@ -102,6 +105,7 @@ def test_canonical_change(self):
        self.assertEqual(domain.domain, 'example2.com')


+@pytest.mark.only_community


We are running pytest -m "not only_community" from the corporate project.

This is 100% trivial, but only_community is mildly unnatural for english in this context, its clearer to me written in reverse: community_only. Perhaps just community as the tag?

I renamed it to just community

humitos · 2018-05-23T15:06:50Z

Although this is a work in progress PR since there are more work to do, I'd like to have some feedback from the Core Team to know if I'm going in the right direction. There are more comments/ideas in the related/linked PRs.

agjohnson

Most of the changes look good so far. I'm not 100% certain on the use of a tag like only_community to skip these. I think something more explicit or descriptive is better.

For instance, perhaps tests should mark xfail on a condition:

@pytest.mark.xfail(not pytest.config.getvalue("community"),
                    reason="Overridden logic on the queryset can cause this to fail")
def test_case_something_something(...):
    pass

Again, this is all a little weird as we should not be referencing commercial code in any of this. That would be really confusing for contributors, as they will never be able to see this code.

Alternatively, inside the test was can xfail with even more specificity.

A few of the patterns used in this PR can be replaced with more pytest internals -- such as replacing any mention of readthedocsinc and such with a pytest config option.

agjohnson · 2018-05-24T22:22:24Z

readthedocs/rtd_tests/tests/test_core_tags.py

@@ -9,10 +11,26 @@
 from readthedocs.core.templatetags import core_tags


-@override_settings(USE_SUBDOMAIN=False, PUBLIC_DOMAIN='readthedocs.org')
+@override_settings(USE_SUBDOMAIN=False, PRODUCTION_DOMAIN='readthedocs.org')


Why is this change required?

It was wrong. The test wants to use PRODUCTION_DOMAIN not PUBLIC_DOMAIN.

It's using USE_SUBDOMAIN=False, so the URL used for the documentation comes from PRODUCTION_DOMAIN. If you change the value override of PUBLIC_DOMAIN here to something different, all the tests keep passing which is wrong.

agjohnson · 2018-05-24T22:26:36Z

readthedocs/rtd_tests/tests/test_core_tags.py

+    pip_abc_xyz_document = '{scheme}://readthedocs.org/docs/pip/en/abc/index.html#document-xyz'.format(scheme=scheme)
+    pip_abc_xyz_fr_document = '{scheme}://readthedocs.org/docs/pip/fr/abc/index.html#document-xyz'.format(scheme=scheme)
+    pip_latest_document_url = '{scheme}://readthedocs.org/docs/pip/en/latest/document/'.format(scheme=scheme)
+    pip_latest_document_page_url = '{scheme}://readthedocs.org/docs/pip/en/latest/document.html'.format(scheme=scheme)


instead of duplicating a lot of text here, we could reduce this section to just reuse a common base. that is:

url_base = '{scheme}://{production_domain}/docs/pip' def test_something(self): ... self.assertEqual(url, self.url_base + '/en/latest/')

I did something similar than what you are proposing here. Not exactly the same, though.

agjohnson · 2018-05-24T22:33:54Z

readthedocs/rtd_tests/tests/test_domains.py

@@ -102,6 +105,7 @@ def test_canonical_change(self):
        self.assertEqual(domain.domain, 'example2.com')


+@pytest.mark.only_community


This is 100% trivial, but only_community is mildly unnatural for english in this context, its clearer to me written in reverse: community_only. Perhaps just community as the tag?

agjohnson · 2018-05-24T22:36:38Z

readthedocs/rtd_tests/utils.py

+    Return the absolute path of the ``readthedocs`` app.
+    """
+    path = getcwd()
+    if path.endswith('readthedocsinc'):


This is also fragile and references commercial code that won't exist on the community side (other contributors will be puzzled by this). Perhaps there is another flag or something we can use from pytest here? Can we check to see if the tag community_only exists here maybe? Or is there a more general tag we could be using from pytest or settings?

I just used:

import readthedocs readthedocs.__path__[0]

which works not matter where you have the readthedocs module installed.

agjohnson · 2018-06-04T23:38:59Z

readthedocs/rtd_tests/tests/test_core_tags.py

 class CoreTagsTests(TestCase):
    fixtures = ["eric", "test_data"]

+    # Determine if we are under community (http) or corporate site (https)
+    scheme = 'https' if settings.CELERY_APP_NAME == 'readthedocsinc' else 'http'


Same. Perhaps there is a setting we could use or add? This is very fragile and not something we should reference in the .org code (we don't ever reference our commercial hosting code from the community side)

agjohnson · 2018-06-08T15:15:23Z

readthedocs/rtd_tests/utils.py

+    path = getcwd()
+    if path.endswith('readthedocsinc'):
+        # readthedocs-corporate
+        path = pjoin(path, '..', '..', 'readthedocs.org', 'readthedocs')


Also, -1 on this pattern in general. We need to stop making assumptions that these two repos are siblings in a similar path.

These tests won't be ran when imported from the corporate site.

All the translation tests manage different user permissions on each different project, which is more complicated to test them from Corporate. So, they are marked as `only_community`

When loading this test from outside (readthedocsinc) the ``reverse`` can't be calculated because the ``search`` name is not defined in the URLConf.

Accessing ``/`` while logged in redirects to ``/dashboard/`` in the corporate site. This change makes this test compatible with both environments

humitos · 2018-06-28T17:44:27Z

conftest.py

+try:
+    # TODO: this file is read/executed even when called from ``readthedocsinc``,
+    # so it's overriding the options that we are defining in the ``conftest.py``
+    # from the corporate site. We need to find a better way to avoid this.


conftest.py is directory dependent. That's why we can't override these pytest's options here. I'd like to find a better way to do this.

Does it make sense to define these PYTEST_OPTIONS in the settings.py file?

Perhaps this would be a good addition. Alternatively, maybe something in setup.cfg instead? I'm not sure if the load order would allow us to use django settings, given pytest-django likely sets up the application later in the process. We can make an issue to track updating this pattern.

I opened #4317 for this

humitos · 2018-06-28T17:48:58Z

readthedocs/rtd_tests/tests/test_project.py

+
+# Most of these tests don't handle Corporate permissions (Organizations, Teams,
+# etc) in a proper way
+@pytest.mark.community


Do we want to include these comments as "reason"? Do we prefer the reason= argument? Should we remove all of these comments from this repo?

I like to explain the why. I did that in the conftest.py from the corporate when I'm ignoring a whole .py file, but I'd like to do something similar when I'm excluding only one/several tests from a file that it's not completely ignored.

These marks are not needed anymore :D

humitos · 2018-06-28T17:49:52Z

At this point, I don't want to make the whole suite to be executed in both repositories and pass. I think that marking all community test as xfail does not add value since they are not thought to fail for any specific reason. I'd say that we can think a better way for those tests and later and improve what we have.

I had to modify a couple of tests to be compatible with both repositories and I found some patterns that we will need to follow when writing new tests in the community site:

do not base our tests in default value of settings. If we need a setting with a specific value, we should use override_setting for the setting's name and value we want (example, PRODUCTION_DOMAIN)
be careful when defining class level variables. There are some values that can't be calculated because the configs differs between different projects (example, reverse('urlname'))

Since this is executed even when the test is skipped, we need to use the ``_lazy`` function.

This Feature is created inside a db migration. As we depend strictly on this Feature we can't rely on data from the migration since it's ran only once when the db is created and not re-populated on each test. We need to be sure that the data we rely on is in the database in a valid state and that it wasn't modified/deleted by any other test case. Because of this, we need to use a fixture or delete and create the data we rely on inside the ``setUp`` for this test.

These tests performs a call to our API from a signal hook in our corporate site and the API call in that file is not mocked, so they fail.

These tests are excluded from the corporate site using pytest internals.

agjohnson

I like this pattern, besides overriding PYTEST_OPTIONS, which we'll find a better way to do later, this is all really clean on both sets of code. 👍

agjohnson · 2018-07-02T16:07:00Z

conftest.py

+try:
+    # TODO: this file is read/executed even when called from ``readthedocsinc``,
+    # so it's overriding the options that we are defining in the ``conftest.py``
+    # from the corporate site. We need to find a better way to avoid this.


Perhaps this would be a good addition. Alternatively, maybe something in setup.cfg instead? I'm not sure if the load order would allow us to use django settings, given pytest-django likely sets up the application later in the process. We can make an issue to track updating this pattern.

humitos · 2018-07-02T19:14:41Z

I'm merging this. We can follow that TODO in the issue I created.

ericholscher · 2018-07-03T06:52:17Z

🎆

humitos commented May 15, 2018

View reviewed changes

humitos added the PR: work in progress Pull request is not ready for full review label May 15, 2018

humitos requested a review from a team May 23, 2018 15:05

humitos force-pushed the humitos/corporate/tests branch from 10b6d53 to 65a9ebb Compare June 6, 2018 14:08

agjohnson requested changes Jun 8, 2018

View reviewed changes

agjohnson added this to the 2.7 milestone Jun 8, 2018

humitos added 11 commits June 28, 2018 11:23

Get the proper path for sample repo under corporate

cd82565

Use files existing in community and corporate site for tests

8eaf66d

Use http for Community site and https for Corporate site on tests

4623873

Mark some OAuth tests as "only_community"

7ce8015

These tests won't be ran when imported from the corporate site.

Extend MiddlewareTests class to set the urlconf from outsite

c58de84

Mark TestAPI for domain as "only community"

be48ae2

Restore the cwd by using a decorator helper

ec133fb

Generalize tests naming

d55a15e

Generalize names

9b06d61

Force ALLOW_PRIVATE_REPOS=False for test case

f17e79e

Split project tests in classes by topic

e6c879d

All the translation tests manage different user permissions on each different project, which is more complicated to test them from Corporate. So, they are marked as `only_community`

humitos force-pushed the humitos/corporate/tests branch from a8ca72b to e6c879d Compare June 28, 2018 14:25

humitos added 10 commits June 28, 2018 12:12

Rename pytest.mark only_community to community

02b3236

Use pytest option to create valid URL depending the environment

f98fb39

Get 'readthedocs' module path in the right way

d4222e5

Minor fix to URL base

821f3c9

Define the URL inside setUp because it fails when loading the test

af4e010

When loading this test from outside (readthedocsinc) the ``reverse`` can't be calculated because the ``search`` name is not defined in the URLConf.

Use setting for subdomain URL conf instead of hardcoded value

10fd674

Mark test case as community only

e10e2d4

Use fixed PRODUCTION_DOMAIN for tests

7e96031

Avoid overriding conftest settings when called from readthedocsinc

ab39aef

Use /dashboard/ instead of / to avoid redirects

33813b5

Accessing ``/`` while logged in redirects to ``/dashboard/`` in the corporate site. This change makes this test compatible with both environments

humitos commented Jun 28, 2018

View reviewed changes

humitos removed the PR: work in progress Pull request is not ready for full review label Jun 28, 2018

humitos requested a review from agjohnson June 28, 2018 17:50

Extend pytest's markexpr conf in case it already exists

7c2ec2e

humitos force-pushed the humitos/corporate/tests branch from 16728be to 7c2ec2e Compare June 28, 2018 17:59

humitos added 6 commits June 28, 2018 15:07

Fix test fixture loading

638b740

Use reverse_lazy to avoid issues

86e2844

Since this is executed even when the test is skipped, we need to use the ``_lazy`` function.

Fix inheritance order

74c2094

Mark building test as community only

7317ee9

These tests performs a call to our API from a signal hook in our corporate site and the API call in that file is not mocked, so they fail.

Remove all community marks from pytests

178f4ab

These tests are excluded from the corporate site using pytest internals.

agjohnson approved these changes Jul 2, 2018

View reviewed changes

humitos mentioned this pull request Jul 2, 2018

Better pattern to use PYTEST_OPTIONS #4317

Closed

humitos merged commit f895af5 into master Jul 2, 2018

humitos deleted the humitos/corporate/tests branch July 2, 2018 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make tests extensible from corporate site #4095

Make tests extensible from corporate site #4095

humitos commented May 15, 2018 •

edited

Loading

humitos May 15, 2018

agjohnson Jun 4, 2018

humitos Jun 28, 2018

humitos May 15, 2018 •

edited

Loading

agjohnson May 24, 2018

humitos Jun 28, 2018

humitos commented May 23, 2018

agjohnson left a comment

agjohnson May 24, 2018

humitos Jun 18, 2018

agjohnson May 24, 2018

humitos Jun 28, 2018

agjohnson May 24, 2018

agjohnson May 24, 2018

humitos Jun 28, 2018

agjohnson Jun 4, 2018

agjohnson Jun 8, 2018

humitos Jun 28, 2018

humitos Jun 28, 2018

agjohnson Jul 2, 2018

humitos Jul 2, 2018

humitos Jun 28, 2018

humitos Jul 2, 2018

humitos commented Jun 28, 2018

agjohnson left a comment

agjohnson Jul 2, 2018

humitos commented Jul 2, 2018

ericholscher commented Jul 3, 2018

		@@ -102,6 +105,7 @@ def test_canonical_change(self):
		self.assertEqual(domain.domain, 'example2.com')


		@pytest.mark.only_community

Make tests extensible from corporate site #4095

Make tests extensible from corporate site #4095

Conversation

humitos commented May 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos May 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented May 23, 2018

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Jun 28, 2018

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Jul 2, 2018

ericholscher commented Jul 3, 2018

humitos commented May 15, 2018 •

edited

Loading

humitos May 15, 2018 •

edited

Loading