CP-6694 Update docs for SDK 4.0.0

.github/workflows/pre-commit.yml

@@ -10,7 +10,7 @@ jobs:
       max-parallel: 4
       matrix:
         python-version: [2.7]
-        os: [ubuntu-latest, macos-latest, windows-latest]
+        os: [ubuntu-latest, macos-10.15, windows-latest]
         package: [common, libs, platform]
 
     steps:

README.md

@@ -10,9 +10,10 @@ The latest user documentation can be found [here](https://developer.delphix.com)
 ### Prerequisites
 
 - macOS 10.14+, Ubuntu 16.04+, or Windows 10
-- Python 2.7 (Python 3 is not supported)
+- Python 2.7 (vSDK 3.1.0 and earlier)
+- Python 3.8 (vSDK 4.0.0 and later)
 - Java 7+
-- Delphix Engine 5.3.5.0 or above
+- A Delphix Engine of an [appropriate version](/References/Version_Compatibility.md)
 
 ### Installing
 

docs/CONTRIBUTING.md

@@ -70,7 +70,7 @@ We have two goals: Provide the best documentation we can for our customers, and
 
 1. Learn Markdown or use a really good IDE. It's easy to use, but there are complex topics like tables, admonishments, links, and images that you may need some practice with. Look at the other docs in the repo for inspiration and tutelage.
 2. Test everything in mkdocs locally. Best practice is to always have mkdocs running in one terminal tab. It auto-refreshes when you make changes, so you can make sure that nothing breaks, and that your content looks good.
-3. Do not create new directories (nav categories) in /docs without working with Jaspal Sumal (jaspal.sumal@delphix.com)
+3. Do not create new directories (nav categories) in /docs without working with Ryan Fowler (ryan.fowler@delphix.com)
 4. Place all screenshots in the local media/ directory of the category you're editing in. For example, if you're editing a page in docs/Getting_Started, put any screenshots you're going to use in docs/Getting_Started/media
 5. Use relative links to reference screenshots (./media/image.png) and other pages (../Getting_Started/pagename/)
 6. Beware the .pages file. .pages is a hidden file in every folder that provides page order. Any pages not listed in .pages will be alphabetically ordered _after_ the pages that have been listed. If you have a typo in this file or specify a renamed/deleted page, it will break mkdocs.
@@ -84,16 +84,16 @@ We have two goals: Provide the best documentation we can for our customers, and
 
 1. The diff can usually provide what you need for reviewing changes. However, use mkdocs to review locally whenever possible to ensure good formatting and no breaks to mkdocs.
 2. For minor corrections, leave a general comment on the review and vote to ship it so the author can fix it and push.
-3. For major docs projects (e.g. whole new sections of docs or large batches of changes), coordinate with Jas. It is possible we'd be better off using another approach to review (e.g. track notes via Google Sheets)
-4. If you're a reviewer that is not hooked into reviewboard, and unable to get set up to use it, work with Jas on an alternative approach (e.g. track notes via Google Sheets)
+3. For major docs projects (e.g. whole new sections of docs or large batches of changes), coordinate with Ryan. It is possible we'd be better off using another approach to review (e.g. track notes via Google Sheets)
+4. If you're a reviewer that is not hooked into reviewboard, and unable to get set up to use it, work with Ryan on an alternative approach (e.g. track notes via Google Sheets)
 5. If there are issues in production docs, the current procedure is to post the issue in the #docs channel.
 
 ## Publishing
 
 Publishing is currently a manual process that will be automated into the release process at a future point in time. The publishing workflow follows these steps:
 
-1. After the git repo is frozen, Jas begins review and adjustments.
-2. If there are technical questions or issues, Jas will take back to engineering for review.
+1. After the git repo is frozen, Ryan begins review and adjustments.
+2. If there are technical questions or issues, Ryan will take back to engineering for review.
 3. The publish process will run. This process will:
   * Pull the appropriate branch to a build machine
   * Run "mkdocs build clean" to compile documentation to HTML

docs/docs/Best_Practices/Code_Sharing.md

@@ -2,7 +2,7 @@
 
 All Python modules inside of `srcDir` can be imported just as they would be if the plugin was executing locally. When a plugin operation is executed `srcDir` is the current working directory so all imports need to be relative to `srcDir` regardless of the path of the module doing the import.
 
-Please refer to Python's [documentation on modules](https://docs.python.org/2/tutorial/modules.html#modules) to learn more about modules and imports.  For more information about using others' code inside your plugin, see [Available Modules](/Best_Practices/Runtime_Environment.md#Available_Modules)
+Please refer to Python's [documentation on modules](https://docs.python.org/3.8/tutorial/modules.html#modules) to learn more about modules and imports.
 
 ## Example
 
@@ -30,10 +30,12 @@ postgres
 Any module in the plugin could import `execution_util.py` with `from utils import execution_util`.
 
 !!! warning "Gotcha"
-	Since the platform uses Python 2.7, every directory needs to have an `__init__.py` file in it otherwise the modules and resources in the folder will not be found at runtime. For more information on `__init__.py` files refer to Python's [documentation on packages](https://docs.python.org/2/tutorial/modules.html#packages).
+	When using a vSDK version that was built on Python 2.7, every directory needs to have an `__init__.py` file in it otherwise the modules and resources in the folder will not be found at runtime. For more information on `__init__.py` files refer to Python's [documentation on packages](https://docs.python.org/2/tutorial/modules.html#packages).
 
 	Note that the `srcDir` in the plugin config file (`src` in this example) does _not_ need an `__init__.py` file.
 
+	For information on which vSDK versions run on Python 2.7, visit the [Version Compatibility Page](/References/Version_Compatibility/#virtualization-sdk-and-python-compatibility-map).
+
 Assume `schema.json` contains:
 
 ```
@@ -124,14 +126,14 @@ def find_schemas(source_connection, repository):
 `execution_util.py ` has two methods `execute_sql` and `execute_shell`. `execute_sql` takes the name of a SQL script in `resources/` and executes it with `resources/execute_sql.sh`. `execute_shell` takes the name of a shell script in `resources/` and executes it.
 
 ```python
-import pkgutil
+from importlib import resources
 
 from dlpx.virtualization import libs
 
 
 def execute_sql(source_connection, install_name, script_name):
-    psql_script = pkgutil.get_data("resources", "execute_sql.sh")
-    sql_script = pkgutil.get_data("resources", script_name)
+    psql_script = resources.read_text("resources", "execute_sql.sh")
+    sql_script = resources.read_text("resources", script_name)
 
     result = libs.run_bash(
         source_connection, psql_script, variables={"SCRIPT": sql_script}, check=True
@@ -140,11 +142,16 @@ def execute_sql(source_connection, install_name, script_name):
 
 
 def execute_shell(source_connection, script_name):
-    script = pkgutil.get_data("resources", script_name)
+    script = resources.read_text("resources", script_name)
 
     result = libs.run_bash(source_connection, script, check=True)
     return result.stdout
 ```
 
+!!! warning
+    If developing a plugin in Python 2.7, you will need to use `pkgutil.get_data` rather than `importlib.resources.read_text`.
+
+    See [Managing Scripts For Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) for more info.
+
 !!! note
 	Both `execute_sql` and `execute_shell` use the `check` parameter which will cause an error to be raised if the exit code is non-zero. For more information refer to the `run_bash` [documentation](/References/Platform_Libraries.md#run_bash).

docs/docs/Best_Practices/Managing_Scripts_For_Remote_Execution.md

@@ -1,9 +1,11 @@
 # Managing Scripts for Remote Execution
 
-To execute a PowerShell or Bash script or Expect script on a remote host, you must provide the script as a string to `run_powershell` or `run_bash` or `run_expect`. While you can keep these strings as literals in your Python code, best practice is to keep them as resource files in your source directory and access them with `pkgutil`.
+To execute a PowerShell or Bash script or Expect script on a remote host, you must provide the script as a string to `run_powershell` or `run_bash` or `run_expect`. While you can keep these strings as literals in your Python code, best practice is to keep them as resource files in your source directory and access them with `pkgutil` or `importlib`, depending on your plugin language.
 
 [pkgutil](https://docs.python.org/2/library/pkgutil.html) is part of the standard Python library. The method that is applicable to resources is [pkgutil.get_data](https://docs.python.org/2/library/pkgutil.html#pkgutil.get_data).
 
+When developing a plugin in Python3, it is instead suggested to use the newer `importlib.resources`. This package is part of the standard Python 3 library. The method that is applicable to resources is [resources.read_text](https://docs.python.org/3.8/library/importlib.html#importlib.resources.read_text), which accepts the same arguments as `pkgutil.get_data`.
+
 ### Basic Usage
 
 Given the following plugin structure:
@@ -66,7 +68,7 @@ def post_snapshot(direct_source, repository, source_config):
 		raise UserError(
 		'Failed to get date',
 		'Make sure the user has the required permissions',
-		'{}\n{}'.format(response.stdout, rsponse.stderr))
+		'{}\n{}'.format(response.stdout, response.stderr))
 
 	return SnapshotDefinition(name='Snapshot', date=response.stdout)
 ```
@@ -75,7 +77,9 @@ def post_snapshot(direct_source, repository, source_config):
 	This assumes that `src/` is Python's current working directory. This is the behavior of the Virtualization Platform.
 
 !!! warning "Resources need to be in a Python module"
-	`pkgutil.get_data` cannot retrieve the contents of a resource that is not in a Python package. This means that a resource that is in the first level of your source directory will not be retrievable with `pkgutil`. Resources must be in a subdirectory of your source directory, and that subdirectory must contain an `__init__.py` file.
+	`pkgutil.get_data` cannot retrieve the contents of a resource that is not in a Python package. When developing with Python 2.7, this means that a resource that is in the first level of your source directory will not be retrievable with `pkgutil`. Resources must be in a subdirectory of your source directory, and that subdirectory must contain an `__init__.py` file.
+
+	Python 3 does not have this requirement.
 
 ### Multi-level Packages
 
@@ -101,3 +105,9 @@ The contents of `src/resources/platform/get_date.sh` can be retrieved with:
 ```python
 script_content = pkgutil.get_data('resources.platform', 'get_date.sh')
 ```
+
+In a Python 3.8 plugin, the suggested approach is:
+
+```python
+script_content = resources.read_text('resources.platform', 'get_date.sh')
+```

docs/docs/Best_Practices/User_Visible_Errors.md

@@ -16,7 +16,7 @@ output | String | **Optional**. Output or stack trace from the failure to give t
 ## Example
 
 ```python
-import pkgutil
+from importlib import resources
 from dlpx.virtualization.platform import Plugin
 from generated.definitions import SourceConfigDefinition
 from dlpx.virtualization.platform.exceptions import UserError
@@ -25,7 +25,7 @@ plugin = Plugin()
 
 @plugin.virtual.start()
 def start(virtual_source, repository, source_config):
-  script_content = pkgutil.get_data('resources', 'start_database.sh')
+  script_content = resources.read_text('resources', 'start_database.sh')
 
   response = libs.run_bash(virtual_source.connection, script_content)
 
@@ -37,6 +37,11 @@ def start(virtual_source, repository, source_config):
     '{}\n{}'.format(response.stdout, response.stderr))
 ```
 
+!!! warning
+    If developing a plugin in Python 2.7, you will need to use `pkgutil.get_data` rather than `importlib.resources.read_text`.
+
+    See [Managing Scripts For Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) for more info.
+
 The UI would show the end user if the plugin operation above fails:
 
 ![Screenshot](images/UserError_Start.png)

docs/docs/Getting_Started.md

@@ -12,9 +12,10 @@ The platform and libs modules expose objects and methods needed to develop a plu
 ## Requirements
 
 - macOS 10.14+, Ubuntu 16.04+, or Windows 10
-- Python 2.7 (Python 3 is not supported)
+- Python 2.7 (vSDK 3.1.0 and earlier)
+- Python 3.8 (vSDK 4.0.0 and later)
 - Java 7+
-- Delphix Engine 6.0.7.0 or above
+- A Delphix Engine of an [appropriate version](/References/Version_Compatibility.md)
 
 ## Installation
 To install the latest version of the SDK run:
@@ -26,9 +27,13 @@ $ pip install dvp
 !!! tip "Use a Virtual Environment"
 	 We highly recommended that you develop plugins inside of a virtual environment. To learn more about virtual environments, refer to [Virtualenv's documentation](https://virtualenv.pypa.io/en/latest/).
 
-	 The virtual environment needs to use Python 2.7. This is configured when creating the virtualenv:
+     If using vSDK 3.1.0 or earlier, the virtual environment needs to use Python 2.7.
 
-	 ```$ virtualenv -p /path/to/python2.7/binary ENV```
+     If using vSDK 4.0.0 or earlier, the virtual environment needs to use Python 3.8.
+
+	 This is configured when creating the virtualenv:
+
+	 ```$ virtualenv -p /path/to/python2.7/binary ENV``` or ```$ virtualenv -p /path/to/python3.8/binary ENV```
 
 To install a specific version of the SDK run:
 

docs/docs/References/Decorators.md

@@ -13,7 +13,7 @@ plugin = Plugin()
 # Use the decorator to annotate the function that corresponds to the "Virtual Source Start" Plugin Operation
 @plugin.virtual_source.start()
 def my_start(virtual_source, repository, source_config):
-  print "running start"
+  print("running start")
 ```
 
 !!! info

docs/docs/References/Logging.md

@@ -6,10 +6,10 @@ The Virtualization Platform keeps plugin-specific log files. A plugin can, at an
 
 ## Overview
 
-The Virtualization Platform integrates with Python's built-in [logging framework](https://docs.python.org/2/library/logging.html). A special [Handler](https://docs.python.org/2/library/logging.html#handler-objects) is exposed by the platform at `dlpx.virtualization.libs.PlatformHandler`. This handler needs to be added to the Python logger your plugin creates. Logging statements made through Python's logging framework will then be routed to the platform.
+The Virtualization Platform integrates with Python's built-in [logging framework](https://docs.python.org/3.8/library/logging.html). A special [Handler](https://docs.python.org/3.8/library/logging.html#handler-objects) is exposed by the platform at `dlpx.virtualization.libs.PlatformHandler`. This handler needs to be added to the Python logger your plugin creates. Logging statements made through Python's logging framework will then be routed to the platform.
 
 ## Basic Setup
- Below is the absolute minimum needed to setup logging for the platform. Please refer to Python's [logging documentation](https://docs.python.org/2/library/logging.html) and the [example below](#customized-example) to better understand how it can be customized.
+ Below is the absolute minimum needed to setup logging for the platform. Please refer to Python's [logging documentation](https://docs.python.org/3.8/library/logging.html) and the [example below](#customized-example) to better understand how it can be customized.
 
 ```python
 import logging
@@ -40,7 +40,7 @@ logger.setLevel(logging.DEBUG)
     There is a limit to how much data can be stored within a log message. See [Message Limits](/Best_Practices/Message_Limits.md) for details.
 
 ## Usage
-Once the `PlatformHandler` has been added to the logger, logging is done with Python's [Logger](https://docs.python.org/2/library/logging.html#logger-objects) object. Below is a simple example including the basic setup code used above:
+Once the `PlatformHandler` has been added to the logger, logging is done with Python's [Logger](https://docs.python.org/3.8/library/logging.html#logger-objects) object. Below is a simple example including the basic setup code used above:
 
 ```python
 import logging
@@ -68,7 +68,7 @@ Imagine you notice that your plugin is taking a very long time to do discovery.
 
 Suppose your plugin has a source config discovery operation that looks like this (code is abbreviated to be easier to follow):
 ```python
-import pkgutil
+from importlib import resources
 
 from dlpx.virtualization import libs
 from dlpx.virtualization.platform import Plugin
@@ -83,21 +83,26 @@ def repository_discovery(source_connection):
 
 @plugin.discovery.source_config()
 def source_config_discovery(source_connection, repository):
-  version_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_version.sh'))
-  users_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_users.sh'))
-  db_results = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_databases.sh'))
-  status_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_database_statuses.sh'))
+  version_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_version.sh'))
+  users_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_users.sh'))
+  db_results = libs.run_bash(source_connection, resources.read_text('resources', 'get_databases.sh'))
+  status_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_database_statuses.sh'))
 
   # Return an empty list for simplicity. In reality
   # something would be done with the results above.
   return []
 
 ```
 
+!!! warning
+    If developing a plugin in Python 2.7, you will need to use `pkgutil.get_data` rather than `importlib.resources.read_text`.
+
+    See [Managing Scripts For Remote Execution](/Best_Practices/Managing_Scripts_For_Remote_Execution.md) for more info.
+
 Now, imagine that you notice that it's taking a long time to do discovery, and you'd like to try to figure out why. One thing that might help is to add logging, like this:
 ```python
 import logging
-import pkgutil
+from importlib import resources
 
 from dlpx.virtualization import libs
 from dlpx.virtualization.platform import Plugin
@@ -143,13 +148,13 @@ def repository_discovery(source_connection):
 @plugin.discovery.source_config()
 def source_config_discovery(source_connection, repository):
   logger.debug('About to get DB version')
-  version_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_version.sh'))
+  version_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_version.sh'))
   logger.debug('About to get DB users')
-  users_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_db_users.sh'))
+  users_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_db_users.sh'))
   logger.debug('About to get databases')
-  db_results = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_databases.sh'))
+  db_results = libs.run_bash(source_connection, resources.read_text('resources', 'get_databases.sh'))
   logger.debug('About to get DB statuses')
-  status_result = libs.run_bash(source_connection, pkgutil.get_data('resources', 'get_database_statuses.sh'))
+  status_result = libs.run_bash(source_connection, resources.read_text('resources', 'get_database_statuses.sh'))
   logger.debug('Done collecting data')
 
   # Return an empty list for simplicity. In reality
@@ -176,7 +181,7 @@ Download a support bundle by going to **Help** > **Support Logs**  and select **
 
 ## Logging Levels
 
-Python has a number of [preset logging levels](https://docs.python.org/2/library/logging.html#logging-levels) and allows for custom ones as well. Since logging on the Virtualization Platform uses the `logging` framework, log statements of all levels are supported.
+Python has a number of [preset logging levels](https://docs.python.org/3.8/library/logging.html#logging-levels) and allows for custom ones as well. Since logging on the Virtualization Platform uses the `logging` framework, log statements of all levels are supported.
 
 However, the Virtualization Platform will map all logging levels into three files: `debug.log`, `info.log`, and `error.log` in the following way: