Update tests

[medecau]

This pull-request:

• Drops test support for python 2.6.
  • too many errors related to pylint.
  • decided to keep pylint and drop python 2.6.
• Adds requests version variations to tox config similar to travis.
  • tox -l for a list of all environments included
  • detox to run tests in parallel
• Moves test dependencies into tox.
• Adds a new environment for code style conformance validation.
  • pycodestyle
  • fixed some code style errors
• Changes travis configuration to use tox.
• Changes travis configuration to use containers.

[john-kurkowski]

Thank you for your contribution. Can I ask, what prompted this? How is this better?

[john-kurkowski]

This is Travis's default

[medecau]

True.
According to the docs this is the default for all repos activated since 2015. And was already the default behaviour for tldextract.

Should I remove it?

[john-kurkowski]

Eh, you're right, explicit > implicit

[john-kurkowski]

Sorry for the huge delay in response.

Can I ask, what prompted this? How is this better?

Bump. I'm still wondering what prompted the major changes in this PR.

I've been happy with the test setup and developer ergonomics of the project. Aren't the instructions in the README's Contribute section pretty standard for Python developers? I.e. the use of requirements.txt? A single command to run the test suite for the current Python version?

Do developers need to run all environment permutations all the time? Is Tox the new hotness?

[medecau]

What prompted this?

There wasn’t a single cause for the changes or I can no longer remember what it was.
But I made the changes. I think they are good and stand by them.

How is this better?

Before remote tests were more exhaustive than local tests. i.e.: local tests could pass while remote tests failed. With these changes the steps for local tests are the same as for remote tests.
Devs are able to test on all relevant environments before pushing to remote. And during development it is possible to run only one set of tests for the specific environment the changes affect.
When Python 3.6 is released it will be added to tox.ini along with one line to .travis.yml.
There is an environment solely for code style. In the future you can add environments for doc style and security.

Do developers need to run all environment permutations all the time?

No. Developers can run tests against only one environment or a small set of environments. e.g.: tox -e py35-requests-2.1.0
More info on tox docs.

Is Tox the new hotness?

Probably. Depends on the definition of ‘new’ and ‘hotness’. I’ll have to leave this one up to you.
Here are a few commits of tox being added to some influential projects:

• 2016 django
• 2015 ansible
• 2012 httpie
• 2010 flask

[medecau]

After posting the previous comment I noticed it included an appeal to authority.

Let me try with my personal experience. Anecdotal as it may be.

I've been using tox for about the same time as I've been using py.test. A few years now. I don’t consider it new. I find it extremely usefull. Even if it may be hot I find it to be usefully hot. Using tox on a project gives me the guarantee that whatever changes I make do not brake the project at any level that tox is able to test. That is, the project builds, it installs, dependencies do not brake, and tests pass. I have not explored running the test on different OSs. For now I leave that to travis-ci.

I was also going to list a bunch of tox features and guarantees that tox provides but decided against it because the tox wiki does a much better job than I could possibly do. See tox docs.

I also remembered that a similar clarification was requested on a different project. See transitions issue #123

[john-kurkowski]

Minor: does Travis have any syntax for syncing this list of envs to run with tox.ini? Too bad it has to be in 2 places. But it's not too bad.

[medecau]

You could run tox without any TOXENV vars but that would collapse all the environments into a single one. Tox would then run all permutations but it would be harder to which tests failed from the travis interface.

By defining the different environments it forces travis to display the results for the different environments.

[medecau]

What the above means in usability is that currently if a test fails in one of the environments travis will display the failed environment from the UI. e.g.: https://travis-ci.org/john-kurkowski/tldextract/builds/93708810

If no TOXENV vars are defined then tox will run all permutations under a single build, hiding the details of which environment failed deep in the logs. Tox makes it easy to see which envs failed but not as nice as travis web UI.

[john-kurkowski]

Thank you very much for the explanation. I like seeing the other examples and discussions. From this PR's description alone, I wasn't sure what to make of the increased reliance on Tox.

Before remote tests were more exhaustive than local tests.

I hadn't realized that. The way this PR smooths that over is nice.

And no argument against PEP-8 compliance from me. I was looking for that anyway.

So, this new direction sounds good. I'll bite.

[john-kurkowski]

I'm trying to check out your branch locally, and try the new test suite, as if I were a newb. The README's Contribute section doesn't seem to apply any more. There's no requirements.txt. I get tox: command not found. Can you update README.md?

Hopefully that section can remain as succinct as it is. It doesn't need to be nearly as detailed as Django's addition of tox.

[medecau]

You may have received a notification for an update on this PR. Please ignore that.

I went back to the README and am trying to figure out the steps for contribution.
The current steps for installation are not the steps for contribution.

For contribution I would simply mention pull-requests as the way to go. And request that contributors validate that all tests pass before publishing.

[john-kurkowski]

Oh, I'm not asking you to add more info to that section than what's already there. Just to update the incorrect info.

In particular, I'm trying to run your PR locally, but the instructions in that section don't work any more.

[john-kurkowski]

Whew, very simple instructions. Tried it out locally. Works great. Thank you for your contribution and for seeing this through!

[medecau]

You are welcome!

Thanks for taking the time to review the pull-request.