Fixes #401 Networking capabilities should be documented

.bumpversion.cfg

@@ -3,14 +3,14 @@ current_version = 3.2.0.dev0
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\.(?P<release>[a-z]+)(?P<dev>\d+))?
-serialize = 
+serialize =
 	{major}.{minor}.{patch}.{release}{dev}
 	{major}.{minor}.{patch}
 
 [bumpversion:part:release]
 optional_value = prod
 first_value = dev
-values = 
+values =
 	dev
 	prod
 
@@ -25,4 +25,3 @@ values =
 [bumpversion:file:./libs/src/main/python/dlpx/virtualization/libs/VERSION]
 
 [bumpversion:file:./tools/src/main/python/dlpx/virtualization/_internal/VERSION]
-

docs/docs/Best_Practices/.pages

@@ -4,7 +4,8 @@ arrange:
   - Managing_Scripts_For_Remote_Execution.md
   - User_Visible_Errors.md
   - Sensitive_Data.md
-  - Unicode_Data.md
+  - Strings.md
+  - Runtime_Environment.md
   - Working_with_Powershell.md
   - Scratch_Paths.md
   - Message_Limits.md

docs/docs/Best_Practices/Code_Sharing.md

@@ -2,7 +2,7 @@
 
 All Python modules inside of `srcDir` can be imported just as they would be if the plugin was executing locally. When a plugin operation is executed `srcDir` is the current working directory so all imports need to be relative to `srcDir` regardless of the path of the module doing the import.
 
-Please refer to Python's [documentation on modules](https://docs.python.org/2/tutorial/modules.html#modules) to learn more about modules and imports.
+Please refer to Python's [documentation on modules](https://docs.python.org/2/tutorial/modules.html#modules) to learn more about modules and imports.  For more information about using others' code inside your plugin, see [Available Modules](/Best_Practices/Runtime_Environment.md#Available_Modules)
 
 ## Example
 
@@ -31,7 +31,7 @@ Any module in the plugin could import `execution_util.py` with `from utils impor
 
 !!! warning "Gotcha"
 	Since the platform uses Python 2.7, every directory needs to have an `__init__.py` file in it otherwise the modules and resources in the folder will not be found at runtime. For more information on `__init__.py` files refer to Python's [documentation on packages](https://docs.python.org/2/tutorial/modules.html#packages).
-	
+
 	Note that the `srcDir` in the plugin config file (`src` in this example) does _not_ need an `__init__.py` file.
 
 Assume `schema.json` contains:
@@ -117,8 +117,8 @@ def find_schemas(source_connection, repository):
     return [SourceConfigDefinition(name=name) for name in schema_names]
 ```
 !!! note
-	Even though `discovery.py` is in the `operations` package, the import for `execution_util` is still relative to the `srcDir` specified in the plugin config file. `execution_util` is in the `utils` package so it is imported with `from utils import execution_util`. 
-	
+	Even though `discovery.py` is in the `operations` package, the import for `execution_util` is still relative to the `srcDir` specified in the plugin config file. `execution_util` is in the `utils` package so it is imported with `from utils import execution_util`.
+
 ### execution_util.py
 
 `execution_util.py ` has two methods `execute_sql` and `execute_shell`. `execute_sql` takes the name of a SQL script in `resources/` and executes it with `resources/execute_sql.sh`. `execute_shell` takes the name of a shell script in `resources/` and executes it.
@@ -147,4 +147,4 @@ def execute_shell(source_connection, script_name):
 ```
 
 !!! note
-	Both `execute_sql` and `execute_shell` use the `check` parameter which will cause an error to be raised if the exit code is non-zero. For more information refer to the `run_bash` [documentation](/References/Platform_Libraries.md#run_bash).
+	Both `execute_sql` and `execute_shell` use the `check` parameter which will cause an error to be raised if the exit code is non-zero. For more information refer to the `run_bash` [documentation](/References/Platform_Libraries.md#run_bash).

docs/docs/Best_Practices/Runtime_Environment.md

@@ -0,0 +1,63 @@
+# Plugin Runtime Environment
+
+## Process Lifetime
+Plugin code runs inside of a Python interpreter process on the Delphix Engine.
+
+A fair question to ask is "What is the lifetime of this interpreter process?"  After all, if the interpreter
+process runs for a long time, then the plugin might be able to store things in memory for later access.
+
+Unfortunately, **there are no guarantees about process lifetime**. Your interpreter process could last two years, or it could last 400 microseconds. There is no way to know or predict this ahead of time.
+
+So, do not make any assumptions about interpreter process lifetime in your plugin code.
+
+
+## Available Modules
+Our Python 2.7 runtime environment only contains the [Python Standard Library](https://docs.python.org/2/library/). No additional Python modules/libraries are available.
+
+If you want to use some Python module that is not part of the standard library, you might be able to do so.
+You would need to include that library as part of your plugin. That would involve downloading the source
+code for that module, and copying it into your source directory. For more information on how to lay out code in your source directory, see [Code Sharing](/Best_Practices/Code_Sharing.md).
+
+### Warnings
+There are two major things to watch out for if you decide to incorporate a 3rd-party library.
+
+1) Make sure you're legally allowed to do so! The licensing agreement on the module will decide if, and
+under what circumstances, you're allowed to make copies of, and redistribute the module. Some modules will
+allow this, some will disallow this, and some will allow this for a fee.
+
+2) Some Python libraries include native code (often written in C or C++). There is no support for using
+such libraries with plugin code.  The reason for this is that native code needs to be
+specially compiled and built for the machine that it the library will be running on. And, unfortunately,
+the machine your plugin runs on (the Delphix Engine) is likely very different from the machine you use
+to develop and build your plugin.
+
+## Network Access
+As of Delphix Engine version 6.0.11.0, plugin code is able to use the network directly. No network access is
+possible in earlier versions.
+
+For example, suppose your plugin wants to talk to some DBMS running on some remote host.
+If the DBMS supports it, your plugin code might be able to connect to the DBMS server and talk to the
+DBMS directly. This can avoid the need to do DBMS operations via running Bash/Powershell code on the remote host.
+
+
+### Example
+```python
+import httplib
+import json
+
+dbms_port = 5432
+
+# Directly contact our DBMS's REST server to get a list of databases
+def list_databases(remote_ip):
+    cx = httplib.HTTPConnection(remote_ip, dbms_port)
+    cx.request("GET", "/databases")
+    response = cx.getresponse()
+    return json.loads(response.read())
+```
+
+What your plugin can access depends entirely on the customer. Some customers will set up their Delphix
+Engines such that plugins have full access to the entire internet. Some will completely restrict the network
+so that the plugin can only access a small handful of remote hosts.
+
+If your plugin has any specific network requirements, it's recommended to try, in your code, to confirm that these requirements are met. For example, the plugin could make such a check in the
+`discovery.repository()` operation, and throw an error if the check fails. Like any other requirement, this should of course be documented.

docs/docs/Best_Practices/Strings.md

@@ -0,0 +1,90 @@
+# Working With Strings
+
+Unfortunately, Python 2.7 makes it very easy to accidentally write string-related code that will sometimes work, but sometimes fail (especially for people who are not using English). Read on for some tips for how to avoid this.
+
+## The Two String Types
+Python 2.7 has two different types that are both called "strings". One represents
+a sequence of **bytes**, and the other represents a sequence of **characters**.
+
+```python
+# The default string (aka 'str object') represents bytes
+my_bytes = "This string is a sequence of bytes"
+
+# A 'Unicode object' represents characters (note the u just before the quote)
+my_characters = u"This string is a sequence of characters"
+```
+
+## Unicode Strings Are Preferred
+
+There are a couple of reasons to prefer the "unicode object" over the "str object".
+
+First, in most cases, we care about characters, and we're not particularly interested in which bytes
+are used to represent those characters.  That is, we might care that we have a "letter H" followed by a "letter I", but it's usually irrelevant to us what byte values happen to be used.
+
+Second, there are lots of different schemes available which give rules for how to represent characters as bytes. These schemes are called "encodings"; some examples include "ASCII", "UTF-8", "Shift-JIS", and "UCS-2".  Each encoding uses different rules about which characters are represented by which bytes.
+
+A "str object" doesn't know anything about encodings... it is just a sequence of bytes. So, when a programmer is working with one of these byte strings, they have to know which encoding rules are in play.
+
+In order to avoid problems, **we recommend using Unicode strings everywhere** in your plugin code.
+
+## Delphix I/O
+
+Your plugin will sometimes need to send strings back and forth to Delphix code. There are two supported formats for doing this.  Any time you receive a string from Delphix, it will be in one of the two following forms. This includes arguments to your plugin operations, and return values from "Delphix Libs" functions. Likewise, any time you send a string to Delphix, it must be in one of these two forms.
+
+Acceptable forms:
+
+1. A Unicode string (recommended)
+2. A "str object" (byte string) that uses the UTF-8 encoding
+
+## Converting Between Types
+
+Sometimes (hopefully rarely!), you might find yourself needing to convert back and forth between byte strings and character strings. For example, you might need to read or write a file on a remote system that is required to use some specific encoding. Here's how to do that:
+
+```python
+# Converting from a character string ("unicode") to a byte string ("str")
+my_utf8_byte_string = my_character_string.encode("utf-8")
+my_utf16_byte_string = my_character_string.encode("utf-16")
+
+# Converting from a byte string to a character string
+my_character_string1 = my_utf8_byte_string.decode("utf-8")
+my_character_string2 = my_utf16_byte_string.decode("utf-16")
+my_character_string3 = my_ascii_byte_string.decode("ascii")
+```
+
+Things to note:
+
+- `encode` goes from characters to bytes. `decode` goes from bytes to characters.
+- If you try to `encode` a character string using the `ascii` encoding, but your character string contains non-ascii characters, you'll get an error. More generally: some encodings will error out with some characters.
+- If you don't specify an encoding, Python will supply a default. But, there's a good chance the default will be wrong for your use case. So, always specify the encoding!
+- Don't try to `encode` a byte string. If you do this, Python will "helpfully" insert an implicit `decode` first, which tends to cause very confusing error messages. Likewise, don't try to `decode` a character string.
+- `utf-8` is likely the best encoding to use for most situations. It accepts all characters, does not have issues with byte ordering, and is understood by most systems. This is not true of most other encodings.
+
+## Using Non-ASCII characters in Python files
+
+Python 2.7 source code files are assumed to use the "ASCII" encoding, unless told otherwise. Unfortunately, ASCII is an obsolete encoding that only knows how to deal with a small number of characters, and only really supports American English.
+
+In order to include non-ASCII characters in your source code, you need to use a different encoding than ASCII, and you need to tell the Python interpreter which encoding you're using.  In Python 2.7, this is done with a "magic" comment at the very top of each file.
+
+Here is an example of the first line of a Python file that uses the UTF-8 encoding:
+```python
+# -*- coding: utf-8 -*-
+```
+
+If you do not specify an encoding, and the source code contains any non-ASCII characters, you will get errors
+ when building the plugin using [dvp build](/References/CLI.md#build) or during the execution of a plugin operation.
+
+### Example
+
+```python
+# -*- coding: utf-8 -*-
+from dlpx.virtualization.platform import Plugin
+from dlpx.virtualization import libs
+from generated.definitions import RepositoryDefinition
+
+plugin = Plugin()
+
+@plugin.discovery.repository()
+def repository_discovery(source_connection):
+    # Create a repository with name that uses non-ASCII characters
+    return [RepositoryDefinition(name=u"Théâtre")]
+```

docs/docs/References/Platform_Libraries.md

@@ -1,5 +1,5 @@
 # Platform Libraries
-Set of functions that plugins can use these for executing remote commands, etc.
+Delphix provides a set of functions that plugins can use for executing remote commands, etc.
 
 ## retrieve_credentials
 
@@ -101,7 +101,7 @@ response = libs.run_bash(connection, command)
 Running a bash script that is saved in a directory.
 
 ```python
- 
+
  import pkgutil
  from dlpx.virtualization import libs
 

docs/readme.md

@@ -2,6 +2,13 @@
 
 This is the Markdown-based documentation for the Virtualization SDK.
 
+## Important Note On Building Docs
+
+As of this writing, the rest of the Virtualization SDK codebase is based on Python 2.
+However, our docs infrastructure is based on Python 3! So, **all of the below commands
+must be run in a Python 3 environment**.  It's recommended to use a totally separate
+virtual environment for docs work than the one you use in the rest of the SDK codebase.
+
 ## Local Testing
 Install dependencies for building documentation and run `pipenv run mkdocs serve`
 
@@ -13,8 +20,8 @@ To activate this project's virtualenv, run pipenv shell.
 Alternatively, run a command inside the virtualenv with pipenv run.
 
 $ pipenv run mkdocs serve
-INFO    -  Building documentation... 
-INFO    -  Cleaning site directory 
+INFO    -  Building documentation...
+INFO    -  Cleaning site directory
 [I 200424 15:54:06 server:292] Serving on http://127.0.0.1:8000
 [I 200424 15:54:06 handlers:59] Start watching changes
 [I 200424 15:54:06 handlers:61] Start detecting changes
@@ -59,7 +66,7 @@ Install `setuptools==45` to get around a deprecated API in version 46.
 $ pip install setuptools==45
 Collecting setuptools==45
   Downloading setuptools-45.0.0-py2.py3-none-any.whl (583 kB)
-     |████████████████████████████████| 583 kB 2.7 MB/s 
+     |████████████████████████████████| 583 kB 2.7 MB/s
 Installing collected packages: setuptools
   Attempting uninstall: setuptools
     Found existing installation: setuptools 46.1.3
@@ -85,13 +92,13 @@ This will generate the `site` directory which will contain all the gererated doc
 5. Go to your individual virtualization-sdk repo's settings, scroll to the bottom and verify under the GitHub Pages section the `Source` is set to `gh-pages branch`.
 6. Right above this will be a link explaining where your docs are published.
 
-You can also utilize the GitHub workflow for publishing docs (`.github/workflows/publish-docs.yml`) associated with a pull request. 
+You can also utilize the GitHub workflow for publishing docs (`.github/workflows/publish-docs.yml`) associated with a pull request.
 The workflow is present on the `develop` branch. Create a branch called `docs/x.y.z` off `develop` on your fork of the repository
 to ensure that your docs branch triggers the workflow. If you have more than one `docs/x.y.z` branch in your fork,
 you have to push your doc changes to the docs branch with the latest `x.y.z` version. Otherwise, the workflow won't run.
 You also have to make sure to choose `gh-pages` branch on your fork as the [publishing source](https://help.github.com/en/github/working-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#choosing-a-publishing-source).
 Once you push doc changes to the `docs/.x.y.z` branch, the docs site should be available under
-`<your-github-username>.github.io/virtualization-sdk` shortly after. You can see the status of publishing under 
+`<your-github-username>.github.io/virtualization-sdk` shortly after. You can see the status of publishing under
 `https://github.com/<your-github-username>/virtualization-sdk/actions`. This is a fast way to give a preview of your
 changes in a pull request.