Update dependency versions

[Ewen Cheslack-Postava (ewencp)]

I also added a couple of comments where we're constrained by py27 support.

[Ewen Cheslack-Postava (ewencp)]

Also, wrt testing, system tests passed with these updates.

[Andrew Egelhofer (andrewegel)]

Also, wrt testing, system tests passed with these updates.

The bump to paramiko 2.6, that was then released to kafka here: apache/kafka#8879 has since seen SSHExceptions being raised randomly in Kafka System Tests. While maybe a bump to 2.7 might fix that, I think we need some large amounts of data comparing pre-2.6 to 2.7 test results in order to feel safe about this change.

[Ewen Cheslack-Postava (ewencp)]

Also, wrt testing, system tests passed with these updates.

The bump to paramiko 2.6, that was then released to kafka here: apache/kafka#8879 has since seen SSHExceptions being raised randomly in Kafka System Tests. While maybe a bump to 2.7 might fix that, I think we need some large amounts of data comparing pre-2.6 to 2.7 test results in order to feel safe about this change.

Yeah, system tests here are pretty minimal and not anywhere close to, e.g., length of test in kafkatest (which might affect these types of exceptions). Happy to downgrade, though we will have to upgrade eventually.

I don't have much context on that specific issue, but I can do branch builds of kafka system tests with this branch of ducktape pulled in to verify. Probably a good idea anyway given all the other dependency upgrades.

All that said, do you have links to the specific issues? SSHException is very generic, so hard to understand what the issue might be. It could be legitimate and tests (or the fmwk) just need to handle them more gracefully, or they might be real issues with the commands being executed.

[Stanislav Vodetskyi (stan-is-hate)]

I can do branch builds of kafka system tests with this branch of ducktape pulled in to verify

I'm also doing this in the background (specifically between paramiko 2.3 and 2.6), but right now a bit swamped with other work, so the progress is slow there.

[Andrew Egelhofer (andrewegel)]

I'm fine with shipping this change into master while we figure out what we want to do with AK and 0.7.x

[Ewen Cheslack-Postava (ewencp)]

Andrew Egelhofer (@andrewegel) Do we have data (or a sense) for how frequent the SSHExceptions are? I'm happy to kick off some kafkatest runs with this same change and check results for SSHException, but would need a baseline number to compare to.

[Magnus Edenhill (edenhill)]

LGTM. Let's get this merged.

[Stanislav Vodetskyi (stan-is-hate)]

+1 to Andrew's comment - let's get it in master, but keep 0.7.x as it is.
For context, our team is working on updating both AK as well as internal testing tools to py3 this quarter, so we'll be able to stop supporting py2 at some point over the next couple of months.

[Magnus Edenhill (edenhill)]

stan-confluent We want to push out a new release of ducktape to unblock the py3 changes to AK system tests, and I think this PR should be part of the new release.
Should we bump ducktape to 0.8.0 so you can stay on <0.8?

[Stanislav Vodetskyi (stan-is-hate)]

cc Ian McDonald (@imcdo) who's working on updating muckrake to py3
cc Nikesh Shettipalli (@nikesh0822) for visibility

Magnus Edenhill (@edenhill) what's blocking you from upgrading AK system tests to py3 first?
Ducktape is already on version 0.8.0 but we haven't cut a release branch yet. I don't see why not cut a 0.8.x branch if you need it - we haven't solved SSHExceptions issue, but we cannot forever stay on paramiko 2.3.2.
One feature in 0.8.0 that I'd ideally like to complete before doing a proper release is a better test suite support (we've added support for running test suites but didn't update the reporting) - but we can do it in 0.8.1 or 0.9.0.
Ewen Cheslack-Postava (@ewencp) what's your thoughts on this?

[Magnus Edenhill (edenhill)]

stan-confluent AK system tests should use a released version of ducktape, not off master, for consistency and reproducability.

So it sounds like we can merge this PR, cut the 0.8.x release branch and do a 0.8.0 release, I'll get on it.

[Stanislav Vodetskyi (stan-is-hate)]

Magnus Edenhill (@edenhill) yeah, if Ewen Cheslack-Postava (@ewencp) is also fine with it, sure. I was just curious what was stopping you from using py3 with ducktape 0.7.x?

[Ivan Daschinskiy (ivandasch)]

Hi! I would like to remind that there is no need of pysistence anymore. It should be removed from requirements.

[Ivan Daschinskiy (ivandasch)]

stan-confluent 0.7.x is not fully compatible with python3. I contributed few patches to fix this

[Ewen Cheslack-Postava (ewencp)]

Ivan Daschinskiy (@ivandasch) just because i filed this before that change was made, adding an update now