Skip to content

Conversation

@mrburke
Copy link
Contributor

@mrburke mrburke commented Oct 27, 2021

CP-6007 Futurize vSDK - Stage 2
CP-6009 Add to_str and to_byte helpers
CP-6010 Add Github Action(s) to run vSDK unit tests in py3
CP-6250 Remove tools from py2.7 unit tests in github actions

Context:

This diff introduces Python3.8 support for the Delphix vSDK.

What does this mean for plugin authors?

  1. The Delphix Engine will continue to accept Python2.7-based plugin uploads.
  2. The Delphix Engine will also accept Python3.8-based plugin uploads
  3. dvp may only be installed and used with Python 3.8

What exactly is in this diff?

  1. futurize --stage2 was run and I then manually cleaned up some of its changes. For example it wraps any instance of <dict object>.[keys|values|items]() with list(), which is not always necessary and should in some situations have the potential to be slower (though in truth it shouldn't be a significant slowdown for our use cases)
  2. We've added common to_bytes and to_str functions which work with python 2 and 3 and handles a number of data types.
    1. These helpers are used throughout the SDK to ensure data is converted properly just after entering and before leaving the program.
  3. Updated our pre-commit actions to run the following 4 jobs against the same matrix of platforms
    1. Python 2.7 unit tests
      1. Packages: common, libs, platform
    2. Python 3.8 unit tests
      1. Packages: common, libs, platform, tools
    3. Python 2.7 linting
      1. Packages: common, libs, platform
    4. Python 3.8 linting
      1. Packages: common, libs, platform, tools
    5. Note: I kept both 2.7 and 3.8 linting with the thought that it could help us catch code that is not both py2 and py3 compatible, when it should be
    6. Note: common, libs, and platform were not being linted prior to this diff, so there are a handful of simple formatting fixes throughout the diff as well.
  4. Fixed a lot of unit tests to account for moderate changes in many messages/exceptions that come with the switch to python3. When we eventually deprecate py2 support we can simply go through and remove all the if six.PY2: ... else: lines from the relevant test files.
  5. Updated dvp/setup.cfg to require Python 3.8

Testing

qa-gate and qa-toolkits changes must be made before we can automate py3 plugin testing. These changes are in progress. However py2 still looks good, and manual testing with py3 looks good so far as well.

Testing with new PY38 qa-appdata-toolkits plugins looks good. Continuing on automation.

All the pre-commit actions have passed.

@mrburke mrburke force-pushed the develop branch 27 times, most recently from cbbb83f to d5d088c Compare October 28, 2021 00:15
@mrburke mrburke force-pushed the develop branch 8 times, most recently from 3b49c11 to 51d0a0d Compare November 9, 2021 18:30
Copy link
Contributor

@nhlien93 nhlien93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good just a couple more nits! Thanks for getting this done.

@mrburke mrburke force-pushed the develop branch 7 times, most recently from 14a0c2e to 73f528f Compare November 18, 2021 17:58
@mrburke mrburke closed this Nov 18, 2021
@mrburke mrburke reopened this Nov 18, 2021
@mrburke mrburke merged commit 568d0da into delphix:develop Nov 18, 2021
mrburke pushed a commit that referenced this pull request Jan 12, 2022
* DOC-1074 Document password vault support for VSDK plugins (#347)

* Correct Typo in Example, it should be UNIX not WINDOWS (#360)

* Add documentation for gRPC message and memory limits #390 (#391)

* Fix version number in doc header (#358) (#397)

* Update Delphix Engine version required to 6.0.7.0. (#386) (#398)

* Update SDK Docs deployment to use a deploy_key (#392) (#399)

* CP-6005 Ensure vSDK test coverage (#403)

* CP-6006 Futurize vSDK - Stage 1 (#404)

* Add support for virtualCleanup decorator (#400) (#405)

Co-authored-by: Kurt Hutchison <[email protected]>

* Fixes #401 Networking capabilities should be documented (#402)

Fixes #401 Networking capabilities should be documented

* CP-6009 Add to_str and to_byte helpers (#407)

* CP-6621 bump sdk version to 4.0.0 (#409)

* CP-6694 Update docs for SDK 4.0.0 (#410)

* CP-7022 vSDK 4.0.0 release tasks

* Update VirtualDelete flow diagram to include the virtual.cleanup call. (#414)

Co-authored-by: Kurt Hutchison <[email protected]>

* Update documentation with new gRPC message limits (#415)

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>
mrburke pushed a commit to mrburke/virtualization-sdk that referenced this pull request Jan 13, 2022
* DOC-1074 Document password vault support for VSDK plugins (delphix#347)

* Correct Typo in Example, it should be UNIX not WINDOWS (delphix#360)

* Add documentation for gRPC message and memory limits delphix#390 (delphix#391)

* Fix version number in doc header (delphix#358) (delphix#397)

* Update Delphix Engine version required to 6.0.7.0. (delphix#386) (delphix#398)

* Update SDK Docs deployment to use a deploy_key (delphix#392) (delphix#399)

* CP-6005 Ensure vSDK test coverage (delphix#403)

* CP-6006 Futurize vSDK - Stage 1 (delphix#404)

* Add support for virtualCleanup decorator (delphix#400) (delphix#405)

Co-authored-by: Kurt Hutchison <[email protected]>

* Fixes delphix#401 Networking capabilities should be documented (delphix#402)

Fixes delphix#401 Networking capabilities should be documented

* CP-6009 Add to_str and to_byte helpers (delphix#407)

* CP-6621 bump sdk version to 4.0.0 (delphix#409)

* CP-6694 Update docs for SDK 4.0.0 (delphix#410)

* CP-7022 vSDK 4.0.0 release tasks

* Update VirtualDelete flow diagram to include the virtual.cleanup call. (delphix#414)

Co-authored-by: Kurt Hutchison <[email protected]>

* Update documentation with new gRPC message limits (delphix#415)

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>
mrburke pushed a commit that referenced this pull request Jan 13, 2022
* DOC-1074 Document password vault support for VSDK plugins (#347)

* Correct Typo in Example, it should be UNIX not WINDOWS (#360)

* Add documentation for gRPC message and memory limits #390 (#391)

* Fix version number in doc header (#358) (#397)

* Update Delphix Engine version required to 6.0.7.0. (#386) (#398)

* Update SDK Docs deployment to use a deploy_key (#392) (#399)

* CP-6005 Ensure vSDK test coverage (#403)

* CP-6006 Futurize vSDK - Stage 1 (#404)

* Add support for virtualCleanup decorator (#400) (#405)

Co-authored-by: Kurt Hutchison <[email protected]>

* Fixes #401 Networking capabilities should be documented (#402)

Fixes #401 Networking capabilities should be documented

* CP-6009 Add to_str and to_byte helpers (#407)

* CP-6621 bump sdk version to 4.0.0 (#409)

* CP-6694 Update docs for SDK 4.0.0 (#410)

* CP-7022 vSDK 4.0.0 release tasks

* Update VirtualDelete flow diagram to include the virtual.cleanup call. (#414)

Co-authored-by: Kurt Hutchison <[email protected]>

* Update documentation with new gRPC message limits (#415)

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>
mrburke pushed a commit that referenced this pull request Jan 20, 2022
* CP-7022 vSDK 4.0.0 release tasks (#416)

* Bumping sdk to 4.0.2 and dvp-api to 1.6.3 due to bad upload of dvp-api 1.6.1 to pypi (#419)

* Release (#418)

* DOC-1074 Document password vault support for VSDK plugins (#347)

* Correct Typo in Example, it should be UNIX not WINDOWS (#360)

* Add documentation for gRPC message and memory limits #390 (#391)

* Fix version number in doc header (#358) (#397)

* Update Delphix Engine version required to 6.0.7.0. (#386) (#398)

* Update SDK Docs deployment to use a deploy_key (#392) (#399)

* CP-6005 Ensure vSDK test coverage (#403)

* CP-6006 Futurize vSDK - Stage 1 (#404)

* Add support for virtualCleanup decorator (#400) (#405)

Co-authored-by: Kurt Hutchison <[email protected]>

* Fixes #401 Networking capabilities should be documented (#402)

Fixes #401 Networking capabilities should be documented

* CP-6009 Add to_str and to_byte helpers (#407)

* CP-6621 bump sdk version to 4.0.0 (#409)

* CP-6694 Update docs for SDK 4.0.0 (#410)

* CP-7022 vSDK 4.0.0 release tasks

* Update VirtualDelete flow diagram to include the virtual.cleanup call. (#414)

Co-authored-by: Kurt Hutchison <[email protected]>

* Update documentation with new gRPC message limits (#415)

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>

* Release (#418) (#421)

* DOC-1074 Document password vault support for VSDK plugins (#347)

* Correct Typo in Example, it should be UNIX not WINDOWS (#360)

* Add documentation for gRPC message and memory limits #390 (#391)

* Fix version number in doc header (#358) (#397)

* Update Delphix Engine version required to 6.0.7.0. (#386) (#398)

* Update SDK Docs deployment to use a deploy_key (#392) (#399)

* CP-6005 Ensure vSDK test coverage (#403)

* CP-6006 Futurize vSDK - Stage 1 (#404)

* Add support for virtualCleanup decorator (#400) (#405)

Co-authored-by: Kurt Hutchison <[email protected]>

* Fixes #401 Networking capabilities should be documented (#402)

Fixes #401 Networking capabilities should be documented

* CP-6009 Add to_str and to_byte helpers (#407)

* CP-6621 bump sdk version to 4.0.0 (#409)

* CP-6694 Update docs for SDK 4.0.0 (#410)

* CP-7022 vSDK 4.0.0 release tasks

* Update VirtualDelete flow diagram to include the virtual.cleanup call. (#414)

Co-authored-by: Kurt Hutchison <[email protected]>

* Update documentation with new gRPC message limits (#415)

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>

Co-authored-by: Raul Santelices <[email protected]>
Co-authored-by: Ranzo J Taylor, III <[email protected]>
Co-authored-by: Jeff Ngo <[email protected]>
Co-authored-by: Ankur <[email protected]>
Co-authored-by: kurthdelphix <[email protected]>
Co-authored-by: Kurt Hutchison <[email protected]>
Co-authored-by: Tom Walsh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants