delphix · nhlien93 · Jun 26, 2020 · May 21, 2020 · May 22, 2020 · Jun 4, 2020
@@ -14,7 +14,7 @@ The platform and libs modules expose objects and methods needed to develop a plu
 - macOS 10.14+, Ubuntu 16.04+, or Windows 10
 - Python 2.7 (Python 3 is not supported)
 - Java 7+
-- Delphix Engine 6.0.2.0 or above
+- Delphix Engine 6.0.3.0 or above
 
 ## Installation
 To install the latest version of the SDK run:
@@ -67,4 +67,4 @@ You can also use a [CLI Configuration File](/Best_Practices/CLI_Configuration_Fi
 
 ## Questions?
 
-If you have questions, bugs or feature requests reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/).
+If you have questions, bugs or feature requests reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/).
@@ -5,3 +5,4 @@ arrange:
   - Compatibility.md
   - Backports_And_Hotfixes.md
   - Replication.md
+  - Lua_Toolkit_To_SDK_Plugin_Migration
@@ -0,0 +1,6 @@
+arrange:
+  - Overview.md
+  - Plugin_Config.md
+  - Decorators.md
+  - Plugin_Operations.md
+  - Converting_Migration_Scripts.md
@@ -0,0 +1,56 @@
+# Converting Lua Upgrade Scripts to Python Data Migrations
+To convert migrations (a.k.a. "upgrade scripts") that were originally written in lua, we need to get the version that the migration upgrades from, the object type the migration is written for, and lastly convert the code into python code using the [decorators](Decorators.md) described previously.
+
+## Example
+Assume there are two versions of a lua toolkit, 1.0.0 and 1.1.0 where the 1.1.0 version is following the basic toolkit directory structure (actually containing all operations):
+
+```
+├── main.json
+├── discovery 
+│   ├── repositoryDiscovery.lua
+│   └── sourceConfigDiscovery.lua
+├── staged 
+│   ├── mountSpec.lua
+│   ├── ...
+│   └── worker.lua
+├── virtual 
+│   ├── configure.lua
+│   ├── ...
+│   └── unconfigure.lua
+├── upgrade 
+│   └── 1.0
+│       ├── upgradeLinkedSource.lua
+│       ├── ...
+│       └── upgradeVirtualSource.lua
+├── resources 
+└── ├── log.sh
+    ├── ...
+    └── stop.sh
+```
+
+`upgradeLinkedSource.lua` contains:
+
+```lua
+parameters.dsOldValue = "remove"
+parameters.dsUpdateValue = 1
+parameters.dsLanguage = "LUA"
+return parameters
+```
+
+This can be equalivalently converted into the python code:
+
+```python
+from dlpx.virtualization.platform import Plugin
+
+plugin = Plugin()
+
+@plugin.upgrade.linked_source("1.0", MigrationType.LUA)
+def upgrade_linked_source(old_linked_source):
+  new_linked_source = dict(old_linked_source)
+  new_linked_source["dsOldValue"] = "remove"
+  new_linked_source["dsUpdateValue"] = 1
+  new_linked_source["dsLanguage"] = "LUA"
+  return new_linked_source
+```
+
+You will need to determine how far back in the Lua upgrade chain you want to support multi-step upgrade from, and convert all of those upgrade scripts accordingly. Remember that the execution of these scripts relies on there not being any missing migrations from the `minimumLuaVersion` defined in the plugin config to the last toolkit version written. Lua migrations will be executed from the lowest to highest version that exists. When executing, these migrations are run to the highest Lua toolkit version only. Any migrations needed to get from that toolkit to the Python plugin would need to be written as a regular Python migration.
@@ -0,0 +1,15 @@
+# Decorators
+
+The Virtualization SDK exposes [decorators](/References/Decorators.md) as mentioned in the regular documentation. Below we list the additional operations added to suport lua to python migrations. This assumes the name of the `Plugin()` object is `plugin`:
+
+Plugin Operation | Decorator
+---------------- |  --------
+[Lua Repository Data Migration](Plugin_Operations.md#lua-repository-data-migration) | `@plugin.upgrade.repository(lua_version, MigrationType.LUA)`
+[Lua Source Config Data Migration](Plugin_Operations.md#lua-source-config-data-migration) | `@plugin.upgrade.source_config(lua_version, MigrationType.LUA)`
+[Lua Linked Source Data Migration](Plugin_Operations.md#lua-linked-source-data-migration) | `@plugin.upgrade.linked_source(lua_version, MigrationType.LUA)`
+[Lua Virtual Source Data Migration](Plugin_Operations.md#lua-virtual-source-data-migration) | `@plugin.upgrade.virtual_source(lua_version, MigrationType.LUA)`
+[Lua Snapshot Data Migration](Plugin_Operations.md#lua-snapshot-data-migration) | `@plugin.upgrade.snapshot(lua_version, MigrationType.LUA)`
+
+!!! info "lua_version format"
+    The lua_version field in this decorator should be the major minor string of the lua toolkit. This means if in the main.json file the version is set to "1.1.HOTFIX123" the lua_version passed into this decorator should be "1.1".
+
@@ -0,0 +1,20 @@
+# Overview
+
+Before the Virtualization SDK was written, Delphix only supported toolkits written in Lua. There was only limited documentation to help people write, build, and upload toolkits. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there must be a way to get customers off of Lua toolkits and onto SDK plugins.
+
+If you are reading this and have no idea what a Lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to convert objects created using Lua toolkits to use the newly uploaded Python plugin.
+
+In the next few pages, we also make the assuption that you've written both a Lua toolkit and a python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from Lua toolkits into the mix as described below.
+
+## Basic no-schema Migration
+One way to migrate from a Lua toolkit to a plugin is to write an exactly equivalent plugin that does not make any [schema](/References/Schemas.md) changes to the objects that were defined originally in the Lua toolkit. If this is the scenario you are in, then you only need to update the [plugin config](Plugin_Config.md) with a couple of new Lua migration specific fields.
+
+
+## Migration with schema changes
+The other way to migrate from a Lua toolkit to a plugin is to wait and write a python plugin only once you have new features you want to release. These new features may include schema changes to any of the objects. In this case you will need to update both the [plugin config](Plugin_Config.md) and write new [Lua upgrade operations](Plugin_Operations.md) for each of the objects that needs to be modified during the upgrade.
+
+## Supporting migrations with older versions of Lua
+Having the ability to define Lua upgrade operations in the new plugin code means that older Lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older Lua versions to migrate and become plugins.
+
+!!! warning "New versions of a Lua toolkit is strongly discouraged after Python Plugin is written"
+	After having written a Plugin to migrate a specific Lua toolkit, while possible, you should avoid writing new major/minor versions of the toolkit in Lua. Patch releases with no schema changes can still be done. If you need to write a new Lua toolkit version please contact Delphix Support to get help on updating migrations accordingly.
@@ -0,0 +1,143 @@
+# Plugin Config
+For all regular fields in a plugin config go [here](/References/Plugin_Config.md). The following fields described are the ones needed to migrate lua toolkits to python plugins.
+
+## Fields
+
+|Field Name|Required|Type|Description|
+|----------|:------:|:--:|-----------|
+|luaName|N|string|The name of the lua toolkit this plugin should upgrade from. This field is required if the minimumLuaVersion is defined.|
+|minimumLuaVersion|N|string|The lowest major minor version of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.|
+
+## Example
+Assume a lua toolkit with the following `main.json` file:
+
+```json
+{
+	"type": "Toolkit",
+	"name": "delphixdb",
+	"prettyName": "DelphixDB",
+	"version": "1.0.0",
+	"defaultLocale": "en-us",
+	"hostTypes": ["UNIX"],
+	"discoveryDefinition": {
+		"type": "ToolkitDiscoveryDefinition",
+		"repositorySchema": {
+			"type": "object",
+			"properties": {
+				"installPath": {
+					"type": "string",
+					"prettyName": "Delphix DB Binary Installation Path",
+					"description": "The path to the Delphix DB installation binaries"
+				},
+				"version": {
+					"type": "string",
+					"prettyName": "Version",
+					"description": "The version of the Delphix DB binaries"
+				}
+			}
+		},
+		"repositoryIdentityFields": ["installPath"],
+		"repositoryNameField": "installPath",
+		"sourceConfigSchema": {
+			"type": "object",
+			"properties": {
+				"dataPath": {
+					"type": "string",
+					"prettyName": "Data Path",
+					"description": "The path to the Delphix DB instance's data"
+				},
+				"port": {
+					"type": "integer",
+					"prettyName": "Port",
+					"description": "The port of the Delphix DB"
+				},
+				"dbName": {
+					"type": "string",
+					"prettyName": "Delphix DB Name",
+					"description": "The name of the Delphix DB instance."
+				}
+			}
+		},
+		"sourceConfigIdentityFields": ["dataPath"],
+		"sourceConfigNameField": "dataPath"
+	},
+	"linkedSourceDefinition": {
+		"type": "ToolkitLinkedStagedSource",
+		"parameters": {
+			"type": "object",
+			"additionalProperties": false,
+			"properties": {
+				"primaryDbName": {
+					"type": "string",
+					"prettyName": "Primary DB Name",
+					"description": "The name of the primary database to link.",
+					"default": "primaryDB"
+				},
+				"stagingDbName": {
+				"type": "string",
+					"prettyName": "Staging DB Name",
+					"description": "The name of the staging database to create."
+				},
+				"stagingPort": {
+					"type": "integer",
+					"prettyName": "Staging Port",
+					"description": "The port of the staging database to create.",
+					"default": 1234
+				}
+			}
+		}
+	},
+	"virtualSourceDefinition": {
+		"type": "ToolkitVirtualSource",
+		"parameters": {
+			"type": "object",
+			"additionalProperties": false,
+			"properties": {
+				"port": {
+					"type": "integer",
+					"prettyName": "Port",
+					"description": "Port that provisioned database should use.",
+					"default": 1234
+				},
+				"dbName": {
+					"type": "string",
+					"prettyName": "Database Name",
+					"description": "Name to use for newly provisioned database.",
+					"default": "vdb"
+				}
+			}
+		}
+	},
+	"snapshotSchema": {
+		"type": "object",
+		"properties": {
+			"snapshotID": {
+				"type": "string",
+				"prettyName": "Snapshot ID",
+				"description": "A unique ID for this snapshot"
+			}
+		}
+	}
+}
+```
+
+Here is a valid plugin config for a plugin that wants to be upgradable from the toolkit:
+
+```yaml
+id: ea009cb4-f76b-46dc-bbb6-689e7acecce4
+name: DelphixDB
+luaName: delphixdb
+minimumLuaVersion: "1.0"
+language: PYTHON27
+hostTypes:
+- UNIX
+pluginType: STAGED
+entryPoint: plugin_runner:plugin
+srcDir: src
+schemaFile: schema.json
+buildNumber: 2.0.0
+```
+
+!!! info "`id` and `luaName` fields in plugins versus `name` field in toolkits"
+    * If the `id` of the plugin being uploaded happens to match the `name` in the toolkit already installed on the engine, the upload will fail regardless of what the `luaName` is. Otherwise, the `luaName` will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the `luaName` is not set then no lua toolkit will be upgraded.
+    * When uploading a plugin with the `luaName` set, that `luaName` and `id` pair will be the only pair uploaded successfully. Uploading a new plugin with the same `luaName` but different `id` will fail.