Merge delphix branch

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.2.0.dev1
+current_version = 3.0.0
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\.(?P<release>[a-z]+)(?P<dev>\d+))?

.github/CODEOWNERS

@@ -1 +1 @@
-*       @ankursarin @fdrozdowski @nhlien93 @crystalplumage
+*       @ankursarin @nhlien93 @mothslaw

.github/workflows/publish-docs.yml

@@ -33,8 +33,9 @@ jobs:
         run: |
           CURRENT_BRANCH_VAR=${GITHUB_REF#refs/heads/}
           echo "Current branch: $CURRENT_BRANCH_VAR"
-          # Set CURRENT_BRANCH environment variable to the current branch name.
-          echo "::set-env name=CURRENT_BRANCH::$(echo $CURRENT_BRANCH_VAR)"
+          # Set CURRENT_BRANCH environment variable to the current branch name. Refrain from using 'set-env' as
+          # GitHub has identified the command as a moderate security vulnerability.
+          echo "CURRENT_BRANCH=$(echo $CURRENT_BRANCH_VAR)" >> $GITHUB_ENV
       - name: Display all remote branches
         working-directory: ${{ matrix.package }}
         run: |
@@ -46,8 +47,9 @@ jobs:
           # Get only docs branches, extract the "docs/x.y.z" part, sort them in descending order, and get the first one.
           LATEST_DOCS_BRANCH_VAR=$(git branch -r | grep -e ".*\/*docs\/[0-9].[0-9].[0-9]" | sed -n "s/.*\/*\(docs\/[0-9].[0-9].[0-9]\).*/\1/p" | sort -r | head -n 1)
           echo "Latest docs branch: $LATEST_DOCS_BRANCH_VAR"
-          # Set the LATEST_DOCS_BRANCH environment variable.
-          echo "::set-env name=LATEST_DOCS_BRANCH::$(echo $LATEST_DOCS_BRANCH_VAR)"
+          # Set the LATEST_DOCS_BRANCH environment variable. Refrain from using 'set-env' as GitHub has identified the
+          # command as a moderate security vulnerability.
+          echo "LATEST_DOCS_BRANCH=$(echo $LATEST_DOCS_BRANCH_VAR)" >> $GITHUB_ENV
       - name: Check that the current branch is the latest docs branch, fail otherwise
         working-directory: ${{ matrix.package }}
         run: |

README-dev.md

@@ -125,17 +125,17 @@ To run blackbox tests, follow these steps:
 2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use,
 see the next sections.
 
-At a minimum, each pull request should pass `appdata_python_samples` and `appdata_sanity` tests with a direct or staged plugin.
+At a minimum, each pull request should pass `appdata_python_samples` and `appdata_basic` tests with a direct or staged plugin.
 See the section below for the description of each test suite.
 
 #### Blackbox tests targeting wrappers (mostly Delphix Engine workflows)
 * appdata_python_samples (sample plugins from the app-gate):
 `git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
-* appdata_sanity with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
-* appdata_sanity with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.
+* appdata_basic with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_basic -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
+* appdata_basic with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_basic -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.
 
 #### Blackbox tests targeting the CLI (~80% CLI tests)
 * virtualization_sdk (installs and tests a direct Python plugin on Ubuntu 18): 
 `git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_DIRECT_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
 * virtualization_sdk (installs and tests a staged Python plugin on Ubuntu 18): 
-`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.
+`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.

common/setup.py

@@ -4,7 +4,7 @@
 PYTHON_SRC = 'src/main/python'
 
 install_requires = [
-  "dvp-api == 1.4.0.dev10",
+  "dvp-api == 1.4.0",
 ]
 
 with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file:

common/src/main/python/dlpx/virtualization/common/VERSION

@@ -1 +1 @@
-2.2.0.dev1
+3.0.0

docs/docs/Building_Your_First_Plugin/Data_Ingestion.md

@@ -21,7 +21,7 @@ quite limiting.
 
 For our first plugin, we will be using the more flexible [staging](/References/Glossary.md#staged-linkingsyncing) strategy. With this strategy, the Delphix Engine uses NFS for Unix environments (or iSCSI on Windows environments) to mount storage onto a [staging environment](/References/Glossary.md#staging-environment). Our plugin will then be in full control of how to get data from the source environment onto this storage mount.
 
-With the staging strategy, there are two types of syncs: sync and resync. A `sync` is used to ingestion incremental changes while a `resync` is used to re-ingest all the data for the dSource. For databases, this could mean re-ingesting from a full database backup to reset the dSource. A `sync` and a `resync` execute the same plugin operations and are differentiated by a boolean flag in the [snapshot_parameters](/References/Classes.md#snapshotparametersdefinition) argument passed into [linked.pre_snapshot](/References/Plugin_Operations.md#staged-linked-source-pre-snapshot) and [linked.post_snapshot](/References/Plugin_Operations.md#staged-linked-source-post-snapshot).
+With the staging strategy, there are two types of syncs: sync and resync. A `sync` is used to ingest incremental changes while a `resync` is used to re-ingest all the data for the dSource. For databases, this could mean re-ingesting from a full database backup to reset the dSource. A `sync` and a `resync` will execute the same plugin operations. To differentiate a `sync` from a `resync`, simply add a boolean property (i.e. `resync`) in the plugin's [snapshot parameters definition](References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema). Once `sync` or `resync` is selected, the property will be passed into [linked.pre_snapshot](/References/Plugin_Operations.md#staged-linked-source-pre-snapshot) and [linked.post_snapshot](/References/Plugin_Operations.md#staged-linked-source-post-snapshot) as a [snapshot parameter](/References/Glossary.md#snapshot-parameters).
 
 A regular `sync` is the default and is executed as part of policy driven syncs. A `resync` is only executed during initial ingestion or if the Delphix user manually starts one. The customer can manually trigger a `resync` via the UI by selecting the dSource, going to more options and selecting **Resynchronize dSource**. ![Screenshot](images/Resync.png)
 
@@ -168,7 +168,7 @@ Next, we'll add a new function:
 
 ```python
 @plugin.linked.pre_snapshot()
-def copy_data_from_source(staged_source, repository, source_config, snapshot_parameters):
+def copy_data_from_source(staged_source, repository, source_config, optional_snapshot_parameters):
     stage_mount_path = staged_source.mount.mount_path
     data_location = "{}@{}:{}".format(staged_source.parameters.username,
         staged_source.parameters.source_address,
@@ -249,7 +249,7 @@ In fact, the default implementation that was generated by `dvp init` will work j
 def linked_post_snapshot(staged_source,
                          repository,
                          source_config,
-                         snapshot_parameters):
+                         optional_snapshot_parameters):
     return SnapshotDefinition()
 ```
 

docs/docs/References/Classes.md

@@ -144,29 +144,6 @@ Field | Type | Description
 mounts | list[[Mount](#mount)] | The list of mounts to export the data sets to.
 ownership_specification | [OwnershipSpecification](#ownershipspecification) | **Optional.** Control the ownership attributes for the data set. It defaults to the environment user of the remote environment if it is not specified.
 
-## SnapshotParametersDefinition
-
-User provided parameters for the snapshot operation. It includes a boolean property named `resync` that can be used to indicate to the plugin whether or not to initiate a full ingestion of the dSource. The parameters are only set during a manual snapshot. When using a sync policy, `resync` defaults to `false`.
-
-```python
-from dlpx.virtualization.platform import Plugin
-
-plugin = Plugin()
-
-@plugin.linked.pre_snapshot()
-def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters):
-  if snapshot_parameter.resync:
-      print(snapshot_parameter.resync)
-```
-
-> This class will be generated during build and is located with the autogenerated classes. As it is passed into the operation, importing it is not neccessary.
-
-### Fields
-
-Field | Type | Description
------ | ---- | -----------
-resync | Boolean | Determines if this snapshot should ingest the dSource from scratch.
-
 ## RemoteEnvironment
 
 Represents a remote environment.

docs/docs/References/Glossary.md

@@ -88,10 +88,10 @@ Information that represents a set of dependencies that a dataset requires in ord
 A formal description of a data type. Plugins use JSON format for their [schemas](Schemas_and_Autogenerated_Classes.md#schemas-and-autogenerated-classes).
 
 ## Snapshot
-A point-in-time read-only copy of a dataset. A snapshot includes associated metadata represented by the [SnapshotDefinition Schema](Schemas_and_Autogenerated_Classes.md#snapshotdefinition)
+A point-in-time read-only copy of a dataset. A snapshot includes associated metadata represented by the [SnapshotDefinition Schema](Schemas_and_Autogenerated_Classes.md#snapshotdefinition).
 
-## Snapshot Parameter
-User provided parameters for the snapshot operation. Currently the only properties the parameter has is resync.
+## Snapshot Parameters
+User provided parameters for the snapshot operation which can be defined in a [Snapshot Parameters Definition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition).
 
 ## Source Config
 A collection of information that the Delphix Engine needs to interact with a dataset (whether [linked](#linked-dataset) or [virtual](#virtual-dataset) on an [environment](#environment).

docs/docs/References/Plugin_Operations.md

@@ -177,7 +177,7 @@ Sets up a [dSource](Glossary.md#dsource) to ingest data. Only applies when using
 
 ### Signature
 
-`def linked_pre_snapshot(direct_source, repository, source_config)`
+`def linked_pre_snapshot(direct_source, repository, source_config, optional_snapshot_parameters)`
 
 ### Decorator
 
@@ -190,6 +190,7 @@ Argument | Type | Description
 direct_source | [DirectSource](Classes.md#directsource) | The source associated with this operation.
 repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source.
 source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source.
+optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy.
 
 ### Returns
 None
@@ -220,7 +221,7 @@ Captures metadata from a [dSource](Glossary.md#dsource) once data has been inges
 
 ### Signature
 
-`def linked_post_snapshot(direct_source, repository, source_config)`
+`def linked_post_snapshot(direct_source, repository, source_config, optional_snapshot_parameters)`
 
 ### Decorator
 
@@ -233,6 +234,7 @@ Argument | Type | Description
 direct_source | [DirectSource](Classes.md#directsource) | The source associated with this operation.
 repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source.
 source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source.
+optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy.
 
 ### Returns
 [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class)
@@ -246,7 +248,7 @@ from generated.definitions import SnapshotDefinition
 plugin = Plugin()
 
 @plugin.linked.post_snapshot()
-def linked_post_snapshot(direct_source, repository, source_config):
+def linked_post_snapshot(direct_source, repository, source_config, optional_snapshot_parameters):
   snapshot = SnapshotDefinition()
   snapshot.transaction_id = 1000
   return snapshot
@@ -277,7 +279,7 @@ Sets up a [dSource](Glossary.md#dsource) to ingest data. Only applies when using
 
 ### Signature
 
-`def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters)`
+`def linked_pre_snapshot(staged_source, repository, source_config, optional_snapshot_parameters)`
 
 ### Decorator
 
@@ -290,7 +292,7 @@ Argument | Type | Description
 staged_source | [StagedSource](Classes.md#stagedsource) | The source associated with this operation.
 repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source.
 source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source.
-snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters.
+optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy.
 
 ### Returns
 None
@@ -303,7 +305,7 @@ from dlpx.virtualization.platform import Plugin
 plugin = Plugin()
 
 @plugin.linked.pre_snapshot()
-def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters):
+def linked_pre_snapshot(staged_source, repository, source_config, optional_snapshot_parameters):
   pass
 ```
 
@@ -320,7 +322,7 @@ Captures metadata from a [dSource](Glossary.md#dsource) once data has been inges
 
 ### Signature
 
-`def linked_post_snapshot(staged_source, repository, source_config, snapshot_parameters)`
+`def linked_post_snapshot(staged_source, repository, source_config, optional_snapshot_parameters)`
 
 ### Decorator
 
@@ -333,7 +335,7 @@ Argument | Type | Description
 staged_source | [StagedSource](Classes.md#stagedsource) | The source associated with this operation.
 repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source.
 source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source.
-snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters.
+optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy.
 
 ### Returns
 [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class)
@@ -347,9 +349,9 @@ from generated.definitions import SnapshotDefinition
 plugin = Plugin()
 
 @plugin.linked.post_snapshot()
-def linked_post_snapshot(staged_source, repository, source_config, snapshot_parameters):
+def linked_post_snapshot(staged_source, repository, source_config, optional_snapshot_parameters):
   snapshot = SnapshotDefinition()
-  if snapshot_parameters.resync:
+  if optional_snapshot_parameters is not None and optional_snapshot_parameters.resync:
     snapshot.transaction_id = 1000
   else:
     snapshot.transaction_id = 10

docs/docs/References/Schemas.md

@@ -22,6 +22,7 @@ Delphix Object | Schema | Autogenerated Class
 [Linked Source](Glossary.md#linked-source) | [LinkedSourceDefinition](Schemas_and_Autogenerated_Classes.md#linkedsourcedefinition-schema) | [LinkedSourceDefinition](Schemas_and_Autogenerated_Classes.md#linkedsourcedefinition-class)
 [Virtual Source](Glossary.md#virtual-source) | [VirtualSourceDefinition](Schemas_and_Autogenerated_Classes.md#virtualsourcedefinition-schema)| [VirtualSourceDefinition](Schemas_and_Autogenerated_Classes.md#virtualsourcedefinition-class)
 [Snapshot](Glossary.md#linked-source) | [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-schema) | [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class)
+[Snapshot Parameters](Glossary.md#snapshot-parameters) | [SnapshotParametersDefinition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema) | [SnapshotParametersDefinition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-class)
 
 
 ## JSON Schemas

docs/docs/References/Schemas_and_Autogenerated_Classes.md

@@ -230,3 +230,40 @@ snapshot2.transaction_id = 1500
 # A snapshot that omits the optional "transaction_id" property
 snapshot3 = SnapshotDefinition(version="1.0.0")
 ```
+
+## SnapshotParametersDefinition
+
+Defines [Snapshot Parameters](Glossary.md#snapshot-parameters) for the snapshot operation.
+
+### SnapshotParametersDefinition Schema
+
+```json
+{
+  "type": "object",
+  "required": ["resync"],
+  "additionalProperties": false,
+  "properties": {
+    "resync": { "type": "boolean" }
+  }
+}
+```
+
+### SnapshotParametersDefinition Class
+
+Autogenerated based on the [Snapshot Parameters Definition Schema](#snapshotparametersdefinition-schema).
+
+```python
+class SnapshotParametersDefinition:
+
+  def __init__(self, resync):
+    self._inner_dict = { "resync": resync }
+```
+
+> To use the class:
+
+```python
+from generated.defintions import SnapshotParametersDefinition
+
+# Since "resync" is required, it must be specified when constructing the object
+snapshot_parameter_definition = SnapshotParametersDefinition(resync=True)
+```

docs/docs/Release_Notes/.pages

@@ -1,4 +1,5 @@
 arrange:
+  - 3.0.0
   - 2.1.0
   - 2.0.0
   - 1.0.0

docs/docs/Release_Notes/3.0.0/.pages

@@ -0,0 +1,3 @@
+arrange:
+  - 3.0.0.md
+  - 3.0.0_Breaking_Changes.md