From 32dd9101bb88f1d371977f113a6c45e2274770db Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Wed, 20 May 2020 19:02:57 -0700 Subject: [PATCH 1/7] fixes #110 Write SDK docs with examples on how to add lua upgrades --- docs/docs/Getting_Started.md | 4 +- docs/docs/Versioning_And_Upgrade/.pages | 1 + .../.pages | 6 + .../Converting_Migration_Scripts.md | 56 ++++ .../Decorators.md | 14 + .../Overview.md | 17 ++ .../Plugin_Config.md | 143 +++++++++ .../Plugin_Operations.md | 286 ++++++++++++++++++ docs/mkdocs.yml | 2 +- docs/readme.md | 18 +- 10 files changed, 539 insertions(+), 8 deletions(-) create mode 100644 docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/.pages create mode 100644 docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md create mode 100644 docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md create mode 100644 docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md create mode 100644 docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md create mode 100644 docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md diff --git a/docs/docs/Getting_Started.md b/docs/docs/Getting_Started.md index 0879a0e8..83d08838 100644 --- a/docs/docs/Getting_Started.md +++ b/docs/docs/Getting_Started.md @@ -14,7 +14,7 @@ The platform and libs modules expose objects and methods needed to develop a plu - macOS 10.14+, Ubuntu 16.04+, or Windows 10 - Python 2.7 (Python 3 is not supported) - Java 7+ -- Delphix Engine 6.0.2.0 or above +- Delphix Engine 6.0.3.0 or above ## Installation To install the latest version of the SDK run: @@ -67,4 +67,4 @@ You can also use a [CLI Configuration File](/Best_Practices/CLI_Configuration_Fi ## Questions? -If you have questions, bugs or feature requests reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/). \ No newline at end of file +If you have questions, bugs or feature requests reach out to us via the [Virtualization SDK GitHub repository](https://github.com/delphix/virtualization-sdk/). diff --git a/docs/docs/Versioning_And_Upgrade/.pages b/docs/docs/Versioning_And_Upgrade/.pages index c8648003..335bc1b3 100644 --- a/docs/docs/Versioning_And_Upgrade/.pages +++ b/docs/docs/Versioning_And_Upgrade/.pages @@ -5,3 +5,4 @@ arrange: - Compatibility.md - Backports_And_Hotfixes.md - Replication.md + - Lua_Toolkit_To_SDK_Plugin_Migration diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/.pages b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/.pages new file mode 100644 index 00000000..9306b009 --- /dev/null +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/.pages @@ -0,0 +1,6 @@ +arrange: + - Overview.md + - Plugin_Config.md + - Decorators.md + - Plugin_Operations.md + - Converting_Migration_Scripts.md diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md new file mode 100644 index 00000000..cae20ee6 --- /dev/null +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md @@ -0,0 +1,56 @@ +# Converting Migration Scripts +To convert migrations that were originally written in lua we need to determine which from version the migration is from, which object type the migration is written for, and lastely convert the code into python code using the [decorators](Decorators.md) described previously. + +## Example +Assume there are two versions of a lua toolkit, 1.0.0 and 1.1.0 where the 1.1.0 version is following the basic toolkit directory structure (actually containing all operations): + +``` +├── main.json +├── discovery +│ ├── repositoryDiscovery.lua +│ └── sourceConfigDiscovery.lua +├── staged +│ ├── mountSpec.lua +│ ├── ... +│ └── worker.lua +├── virtual +│ ├── configure.lua +│ ├── ... +│ └── unconfigure.lua +├── upgrade +│ └── 1.0 +│ ├── upgradeLinkedSource.lua +│ ├── ... +│ └── upgradeVirtualSource.lua +├── resources +└── ├── log.sh + ├── ... + └── stop.sh +``` + +`upgradeLinkedSource.lua` contains: + +```lua +parameters.dsOldValue = "remove" +parameters.dsUpdateValue = 1 +parameters.dsLanguage = "LUA" +return parameters +``` + +This can be equalivalently converted into the python code: + +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.upgrade.linked_source("1.0", MigrationType.LUA) +def upgrade_linked_source(old_linked_source): + new_linked_source = dict(old_linked_source) + new_linked_source["dsOldValue"] = "remove" + new_linked_source["dsUpdateValue"] = 1 + new_linked_source["dsLanguage"] = "LUA" + return new_linked_source +``` + +You will need to determine how far back in the lua upgrade chain you want to support multi-step upgrade from, and convert all of them accordingly. Remember that the execution of these scripts rely on there not being any missing migrations, and will be executed from the lowest version that exists to the highest version. diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md new file mode 100644 index 00000000..1c1676d0 --- /dev/null +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md @@ -0,0 +1,14 @@ +# Decorators + +The Virtualization SDK exposes [decorators](/References/Decorators.md) as mentioned in the regular documentation. Below we list the additional operations added to suport lua to python migrations assuming the name of the `Plugin()` object is `plugin`: + +Plugin Operation | Decorator +---------------- | -------- +[Lua Repository Data Migration](Plugin_Operations.md#lua-repository-data-migration) | `@plugin.upgrade.repository(lua_version, MigrationType.LUA)` +[Lua Source Config Data Migration](Plugin_Operations.md#lua-source-config-data-migration) | `@plugin.upgrade.source_config(lua_version, MigrationType.LUA)` +[Lua Linked Source Data Migration](Plugin_Operations.md#lua-linked-source-data-migration) | `@plugin.upgrade.linked_source(lua_version, MigrationType.LUA)` +[Lua Virtual Source Data Migration](Plugin_Operations.md#lua-virtual-source-data-migration) | `@plugin.upgrade.virtual_source(lua_version, MigrationType.LUA)` +[Lua Snapshot Data Migration](Plugin_Operations.md#lua-snapshot-data-migration) | `@plugin.upgrade.snapshot(lua_version, MigrationType.LUA)` + +!!! info "lua_version format" + The lua_version field in this decorator should be the major minor string of the lua toolkit. \ No newline at end of file diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md new file mode 100644 index 00000000..7defb258 --- /dev/null +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md @@ -0,0 +1,17 @@ +# Overview + +Before the Virtualization SDK was written, Delphix use to support toolkits written only in Lua. The toolkit was written, built, and uploaded to the Delphix engine using scripts with little documentation. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there must be a way to get customers off of lua toolkits and onto SDK plugins. + +If you are reading this and have no idea what a lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to be able to upgrade objects that were created using old toolkits into objects in the Delphix Engine pointing to a plugin. + +Reading the next few pages we also make the assuption that you've written both a lua toolkit and a python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from lua toolkits into the mix. + +## Basic no-schema Migration +One way to migrate from a lua toolkit to a plugin is to write an exactly equivalent plugin that does not make any [schema](/References/Schemas.md) changes to the objects that were defined originally in the lua toolkit. If this is the scenario you are in, then you only need to update the [plugin config](Plugin_Config.md) with a couple new lua migration specific fields. + + +## Migration with schema changes +The other way to migrate from a lua toolkit to a plugin is to wait and write a python plugin only once you have new features you want to release. These new features may include schema changes to any of the object. In this case you will need to update both the [plugin config](Plugin_Config.md) and write new [lua upgrade operations](Plugin_Operations.md) for each of the objects that needs to be modified during the upgrade. + +## Supporting migrations with older versions of lua +Having the ability to define lua upgrade operations in the new plugin code means that older lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older lua versions to migrate and become plugins. \ No newline at end of file diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md new file mode 100644 index 00000000..41104686 --- /dev/null +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md @@ -0,0 +1,143 @@ +# Plugin Config +For all regular fields in a plugin config go [here](/References/Plugin_Config.md). The following fields are only want needs to be added to get lua to python upgrades working. + +## Fields + +|Field Name|Required|Type|Description| +|----------|:------:|:--:|-----------| +|luaName|N|string|The name of the lua toolkit this plugin should upgrade from. This field is required if the luaMinimumVersion is defined.| +| luaMinimumVersion |N|string|The lowest major minor vesrion of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.| + +## Example +Assume a lua toolkit with the following `main.json` file: + +```json +{ + "type": "Toolkit", + "name": "delphixdb", + "prettyName": "DelphixDB", + "version": "1.0.0", + "defaultLocale": "en-us", + "hostTypes": ["UNIX"], + "discoveryDefinition": { + "type": "ToolkitDiscoveryDefinition", + "repositorySchema": { + "type": "object", + "properties": { + "installPath": { + "type": "string", + "prettyName": "Delphix DB Binary Installation Path", + "description": "The path to the Delphix DB installation binaries" + }, + "version": { + "type": "string", + "prettyName": "Version", + "description": "The version of the Delphix DB binaries" + } + } + }, + "repositoryIdentityFields": ["installPath"], + "repositoryNameField": "installPath", + "sourceConfigSchema": { + "type": "object", + "properties": { + "dataPath": { + "type": "string", + "prettyName": "Data Path", + "description": "The path to the Delphix DB instance's data" + }, + "port": { + "type": "integer", + "prettyName": "Port", + "description": "The port of the Delphix DB" + }, + "dbName": { + "type": "string", + "prettyName": "Delphix DB Name", + "description": "The name of the Delphix DB instance." + } + } + }, + "sourceConfigIdentityFields": ["dataPath"], + "sourceConfigNameField": "dataPath" + }, + "linkedSourceDefinition": { + "type": "ToolkitLinkedStagedSource", + "parameters": { + "type": "object", + "additionalProperties": false, + "properties": { + "primaryDbName": { + "type": "string", + "prettyName": "Primary DB Name", + "description": "The name of the primary database to link.", + "default": "primaryDB" + }, + "stagingDbName": { + "type": "string", + "prettyName": "Staging DB Name", + "description": "The name of the staging database to create." + }, + "stagingPort": { + "type": "integer", + "prettyName": "Staging Port", + "description": "The port of the staging database to create.", + "default": 1234 + } + } + } + }, + "virtualSourceDefinition": { + "type": "ToolkitVirtualSource", + "parameters": { + "type": "object", + "additionalProperties": false, + "properties": { + "port": { + "type": "integer", + "prettyName": "Port", + "description": "Port that provisioned database should use.", + "default": 1234 + }, + "dbName": { + "type": "string", + "prettyName": "Database Name", + "description": "Name to use for newly provisioned database.", + "default": "vdb" + } + } + } + }, + "snapshotSchema": { + "type": "object", + "properties": { + "snapshotID": { + "type": "string", + "prettyName": "Snapshot ID", + "description": "A unique ID for this snapshot" + } + } + } +} +``` + +Here is a valid plugin config for a plugin that wants to be upgradable from the toolkit: + +```yaml +id: ea009cb4-f76b-46dc-bbb6-689e7acecce4 +name: DelphixDB +luaName: delphixdb +luaMinimumVersion: "1.0" +language: PYTHON27 +hostTypes: +- UNIX +pluginType: STAGED +entryPoint: plugin_runner:plugin +srcDir: src +schemaFile: schema.json +buildNumber: 2.0.0 +``` + +!!! info "id and luaName fields in plugins versus vs name field in toolkits" + * If the id field of the plugin being uploaded happans to match the name field in the toolkit already installed on the engine, the upload will fail regardless of what the luaName is. Otherwise, the luaName will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the luaName is not set then no lua toolkit will be upgraded. + * Also when uploading a plugin with a luaName, that luaName and id pair will be the only one accepted. If a new plugin with the same luaName but different id is uploaded, this will fail. diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md new file mode 100644 index 00000000..296d67b1 --- /dev/null +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md @@ -0,0 +1,286 @@ +# Plugin Operations + +## Summary +Included below are all plugin operations that have to do with lua migrations. For regular operation information go [here](/References/Plugin_Operations.md). + + +Plugin Operation | **Required** | Decorator | Delphix Engine Operations +---------------- | -------- | --------- | ------------------------- +[Lua Repository Data Migration](#lua-repository-data-migration) | **No** | `upgrade.repository(lua_version, MigrationType.LUA)` | [Upgrade](/References/Workflows.md#upgrade) +[Lua Source Config Data Migration](#lua-source-config-data-migration) | **No** | `upgrade.source_config(lua_version, MigrationType.LUA)` | [Upgrade](/References/Workflows.md#upgrade) +[Lua Linked Source Data Migration](#lua-linked-source-data-migration) | **No** | `upgrade.linked_source(lua_version, MigrationType.LUA)` | [Upgrade](/References/Workflows.md#upgrade) +[Lua Virtual Source Data Migration](#lua-virtual-source-data-migration) | **No** | `upgrade.virtual_source(lua_version, MigrationType.LUA)` | [Upgrade](/References/Workflows.md#upgrade) +[Lua Snapshot Data Migration](#snapshot-data-migration) | **No** | `upgrade.snapshot(lua_version, MigrationType.LUA)` | [Upgrade](/References/Workflows.md#upgrade) + + +## Lua Repository Data Migration + +A Lua Repository [Data Migration](/References/Glossary.md#data-migration) migrates repository data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. + +### Required / Optional +**Optional.**
+ +!!! warning + You must ensure that all repository data will match your updated repository schema after an upgrade operation. + +### Delphix Engine Operations + +* [Upgrade](/References/Workflows.md#upgrade) + +### Signature + +`def migrate_repository(old_repository)` + +### Decorator + +`upgrade.repository(lua_version, MigrationType.LUA)` + +### Decorator Arguments + +Argument | Type | Description +-------- | ---- | ----------- +lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per repository data migration must be unique. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. + +### Function Arguments +Argument | Type | Description +-------- | ---- | ----------- +old_repository | Dictionary | The plugin-specific data associated with a repository, that conforms to the previous schema defined in lua. + +!!! warning + The function argument `old_repository` is a Python dictionary, where each property name appears exactly as described in the previous repository schema. This differs from non-upgrade-related operations, where the function arguments are [autogenerated classes](/References/Schemas_and_Autogenerated_Classes.md) based on the schema. + + +### Returns +Dictionary
+A migrated version of the `old_repository` input that must conform to the updated repository schema. + +### Example +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.upgrade.repository("1.1", MigrationType.LUA) +def add_new_flag_to_repo(old_repository): + new_repository = dict(old_repository) + new_repository["useNewFeature"] = False + return new_repository +``` + +## Lua Source Config Data Migration + +A Lua Source Config [Data Migration](/References/Glossary.md#data-migration) migrates source config data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. + +### Required / Optional +**Optional.**
+ +!!! warning + You must ensure that all source config data will match your source config schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more source config data migrations. + +### Delphix Engine Operations + +* [Upgrade](/References/Workflows.md#upgrade) + +### Signature + +`def migrate_source_config(old_source_config)` + +### Decorator + +`upgrade.source_config(lua_version, MigrationType.LUA)` + +### Decorator Arguments + +Argument | Type | Description +-------- | ---- | ----------- +lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per source config data migration must be unique. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. + +### Function Arguments +Argument | Type | Description +-------- | ---- | ----------- +old_source_config | Dictionary | The plugin-specific data associated with a source config, that conforms to the previous schema. + +!!! warning + The function argument `old_source_config` is a Python dictionary, where each property name appears exactly as described in the previous source config schema. This differs from non-upgrade-related operations, where the function arguments are [autogenerated classes](/References/Schemas_and_Autogenerated_Classes.md) based on the schema. + + +### Returns +Dictionary
+A migrated version of the `old_source_config` input that must conform to the updated source config schema. + +### Example +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.upgrade.source_config("1.1", MigrationType.LUA) +def add_new_flag_to_source_config(old_source_config): + new_source_config = dict(old_source_config) + new_source_config["useNewFeature"] = False + return new_source_config +``` +## Lua Linked Source Data Migration + +A Lua Linked Source [Data Migration](/References/Glossary.md#data-migration) migrates linked source data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. + +### Required / Optional +**Optional.**
+ +!!! warning + You must ensure that all linked source data will match your linked source schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more linked source data migrations. + +### Delphix Engine Operations + +* [Upgrade](/References/Workflows.md#upgrade) + +### Signature + +`def migrate_linked_source(old_linked_source)` + +### Decorator + +`upgrade.linked_source(lua_version, MigrationType.LUA)` + +### Decorator Arguments + +Argument | Type | Description +-------- | ---- | ----------- +lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per linked source data migration must be unique. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. + +### Function Arguments +Argument | Type | Description +-------- | ---- | ----------- +old_linked_source | Dictionary | The plugin-specific data associated with a linked source, that conforms to the previous schema. + +!!! warning + The function argument `old_linked_source` is a Python dictionary, where each property name appears exactly as described in the previous linked source schema. This differs from non-upgrade-related operations, where the function arguments are [autogenerated classes](/References/Schemas_and_Autogenerated_Classes.md) based on the schema. + + +### Returns +Dictionary
+A migrated version of the `old_linked_source` input that must conform to the updated linked source schema. + +### Example +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.upgrade.linked_source("1.1", MigrationType.LUA) +def add_new_flag_to_dsource(old_linked_source): + new_linked_source = dict(old_linked_source) + new_linked_source["useNewFeature"] = False + return new_linked_source +``` +## Lua Virtual Source Data Migration + +A Lua Virtual Source [Data Migration](/References/Glossary.md#data-migration) migrates virtual source data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. + +### Required / Optional +**Optional.**
+ +!!! warning + You must ensure that all virtual source data will match your virtual source schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more virtual source data migrations. + +### Delphix Engine Operations + +* [Upgrade](/References/Workflows.md#upgrade) + +### Signature + +`def migrate_virtual_source(old_virtual_source)` + +### Decorator + +`upgrade.virtual_source(lua_version, MigrationType.LUA)` + +### Decorator Arguments + +Argument | Type | Description +-------- | ---- | ----------- +lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per virtual source data migration must be unique. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. + +### Function Arguments +Argument | Type | Description +-------- | ---- | ----------- +old_virtual_source | Dictionary | The plugin-specific data associated with a virtual source, that conforms to the previous schema. + +!!! warning + The function argument `old_virtual_source` is a Python dictionary, where each property name appears exactly as described in the previous virtual source schema. This differs from non-upgrade-related operations, where the function arguments are [autogenerated classes](/References/Schemas_and_Autogenerated_Classes.md) based on the schema. + + +### Returns +Dictionary
+A migrated version of the `old_virtual_source` input that must conform to the updated virtual source schema. + +### Example +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.upgrade.virtual_source("1.1", MigrationType.LUA) +def add_new_flag_to_vdb(old_virtual_source): + new_virtual_source = dict(old_virtual_source) + new_virtual_source["useNewFeature"] = False + return new_virtual_source +``` +## Lua Snapshot Data Migration + +A Lua Snapshot [Data Migration](/References/Glossary.md#data-migration) migrates snapshot data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. + +### Required / Optional +**Optional.**
+ +!!! warning + You must ensure that all snapshot data will match your snapshot schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more snapshot migrations. + +### Delphix Engine Operations + +* [Upgrade](/References/Workflows.md#upgrade) + +### Signature + +`def migrate_snapshot(old_snapshot)` + +### Decorator + +`upgrade.snapshot(lua_version, MigrationType.LUA)` + +### Decorator Arguments + +Argument | Type | Description +-------- | ---- | ----------- +lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per snapshot data migration must be unique. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. + +### Function Arguments +Argument | Type | Description +-------- | ---- | ----------- +old_snapshot | Dictionary | The plugin-specific data associated with a snapshot, that conforms to the previous schema. + +!!! warning + The function argument `old_snapshot` is a Python dictionary, where each property name appears exactly as described in the previous snapshot schema. This differs from non-upgrade-related operations, where the function arguments are [autogenerated classes](/References/Schemas_and_Autogenerated_Classes.md) based on the schema. + + +### Returns +Dictionary
+A migrated version of the `old_snapshot` input that must conform to the updated snapshot schema. + +### Example +```python +from dlpx.virtualization.platform import Plugin + +plugin = Plugin() + +@plugin.upgrade.snapshot("1.1", MigrationType.LUA) +def add_new_flag_to_snapshot(old_snapshot): + new_snapshot = dict(old_snapshot) + new_snapshot["useNewFeature"] = False + return new_snapshot +``` diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index e7caaffc..75fd7ad4 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -1,4 +1,4 @@ -site_name: Delphix Virtualization SDK 2.0.0 +site_name: Delphix Virtualization SDK 2.1.0 theme: name: material custom_dir: 'material/' diff --git a/docs/readme.md b/docs/readme.md index 6049ebc3..8c1e5de4 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -75,12 +75,20 @@ Installing dependencies from Pipfile.lock (65135d)… 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 14/14 — 00:00:03 ``` -## Live Testing and Reviews -The command `git docsdev-review` will handle publishing reviews, and putting your changes on a live docs server. For example, you can clone the `docsdev-server` image on DCOA, and then run `git docsdev-review -m `. This will: +## Live Testing via Github Pages +To publish doc change to your individual fork for review, we use github pages. To set this up follow these following steps. + +1. Create a new local branch named `gh-pages`. +2. Using the same virtual environment above run: +``` +pipenv run mkdocs build --clean +``` +This will generate the `site` directory which will contain all the gererated docs. +3. Copy all these files to the root directory of the virtualization-sdk repo and delete all other files. +4. Commit and push these changes to your individual fork. +5. Go to your individual virtualization-sdk repo's settings, scroll to the bottom and verify under the GitHub Pages section the `Source` is set to `gh-pages branch`. +6. Right above this will be a link exaplining where your docs are published. -- Push your doc changes to your VM -- Give you a link to the docdev server so you can test your changes live in a browser -- Publish a review ## Workflow diagrams We create workflow diagrams using a tool called `draw.io` which allows us to import/export diagrams in html format. If you want to add a diagram or edit an existing one, simply create or import the html file in `docs/References/html` into `draw.io` and make your desired changes. When you are done, select your diagram and export it as a png file. You can think of the html files as source code, and the png files as build artifacts. After this step, you will be prompted to crop what was selected. You'll want this box checked to trim the whitespace around the diagram. After the diagrams are exported, check in the updated html file to `docs/References/html` and png file to `docs/References/images`. From 9c0646b8f307822792164510ac10052e8e282cdf Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Fri, 22 May 2020 16:54:38 -0700 Subject: [PATCH 2/7] update with tom's review --- .../Converting_Migration_Scripts.md | 6 +++--- .../Decorators.md | 5 +++-- .../Overview.md | 15 ++++++++------- .../Plugin_Config.md | 2 +- .../Plugin_Operations.md | 2 +- 5 files changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md index cae20ee6..fe14ddeb 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md @@ -1,5 +1,5 @@ -# Converting Migration Scripts -To convert migrations that were originally written in lua we need to determine which from version the migration is from, which object type the migration is written for, and lastely convert the code into python code using the [decorators](Decorators.md) described previously. +# Converting Lua Upgrade Scripts to Python Data Migrations +To convert migrations (a.k.a. "upgrade scripts") that were originally written in lua we need to get the version that the migration upgrades from, the object type the migration is written for, and lastly convert the code into python code using the [decorators](Decorators.md) described previously. ## Example Assume there are two versions of a lua toolkit, 1.0.0 and 1.1.0 where the 1.1.0 version is following the basic toolkit directory structure (actually containing all operations): @@ -53,4 +53,4 @@ def upgrade_linked_source(old_linked_source): return new_linked_source ``` -You will need to determine how far back in the lua upgrade chain you want to support multi-step upgrade from, and convert all of them accordingly. Remember that the execution of these scripts rely on there not being any missing migrations, and will be executed from the lowest version that exists to the highest version. +You will need to determine how far back in the lua upgrade chain you want to support multi-step upgrade from, and convert all of the upgrade scripts accordingly. Remember that the execution of these scripts rely on there not being any missing migrations, and will be executed from the lowest version that exists to the highest version. diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md index 1c1676d0..df82e3d0 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md @@ -1,6 +1,6 @@ # Decorators -The Virtualization SDK exposes [decorators](/References/Decorators.md) as mentioned in the regular documentation. Below we list the additional operations added to suport lua to python migrations assuming the name of the `Plugin()` object is `plugin`: +The Virtualization SDK exposes [decorators](/References/Decorators.md) as mentioned in the regular documentation. Below we list the additional operations added to suport lua to python migrations. This assumes the name of the `Plugin()` object is `plugin`: Plugin Operation | Decorator ---------------- | -------- @@ -11,4 +11,5 @@ Plugin Operation | Decorator [Lua Snapshot Data Migration](Plugin_Operations.md#lua-snapshot-data-migration) | `@plugin.upgrade.snapshot(lua_version, MigrationType.LUA)` !!! info "lua_version format" - The lua_version field in this decorator should be the major minor string of the lua toolkit. \ No newline at end of file + The lua_version field in this decorator should be the major minor string of the lua toolkit. This means if in the main.json file the version is set to "1.1.HOTFIX123" the lua_version passed into this decorator should be "1.1". + diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md index 7defb258..f070e753 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md @@ -1,17 +1,18 @@ # Overview -Before the Virtualization SDK was written, Delphix use to support toolkits written only in Lua. The toolkit was written, built, and uploaded to the Delphix engine using scripts with little documentation. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there must be a way to get customers off of lua toolkits and onto SDK plugins. +Before the Virtualization SDK was written, Delphix only supported toolkits written in Lua. There was only limited documentation to help people write, build, and upload toolkits. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there must be a way to get customers off of Lua toolkits and onto SDK plugins. -If you are reading this and have no idea what a lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to be able to upgrade objects that were created using old toolkits into objects in the Delphix Engine pointing to a plugin. +If you are reading this and have no idea what a Lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to be able to upgrade objects that were created using old toolkits into objects in the Delphix Engine pointing to a plugin. -Reading the next few pages we also make the assuption that you've written both a lua toolkit and a python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from lua toolkits into the mix. +In the next few pages, we also make the assuption that you've written both a Lua toolkit and a python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from Lua toolkits into the mix as described below. ## Basic no-schema Migration -One way to migrate from a lua toolkit to a plugin is to write an exactly equivalent plugin that does not make any [schema](/References/Schemas.md) changes to the objects that were defined originally in the lua toolkit. If this is the scenario you are in, then you only need to update the [plugin config](Plugin_Config.md) with a couple new lua migration specific fields. +One way to migrate from a Lua toolkit to a plugin is to write an exactly equivalent plugin that does not make any [schema](/References/Schemas.md) changes to the objects that were defined originally in the Lua toolkit. If this is the scenario you are in, then you only need to update the [plugin config](Plugin_Config.md) with a couple of new Lua migration specific fields. ## Migration with schema changes -The other way to migrate from a lua toolkit to a plugin is to wait and write a python plugin only once you have new features you want to release. These new features may include schema changes to any of the object. In this case you will need to update both the [plugin config](Plugin_Config.md) and write new [lua upgrade operations](Plugin_Operations.md) for each of the objects that needs to be modified during the upgrade. +The other way to migrate from a Lua toolkit to a plugin is to wait and write a python plugin only once you have new features you want to release. These new features may include schema changes to any of the objects. In this case you will need to update both the [plugin config](Plugin_Config.md) and write new [Lua upgrade operations](Plugin_Operations.md) for each of the objects that needs to be modified during the upgrade. + +## Supporting migrations with older versions of Lua +Having the ability to define Lua upgrade operations in the new plugin code means that older Lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older Lua versions to migrate and become plugins. -## Supporting migrations with older versions of lua -Having the ability to define lua upgrade operations in the new plugin code means that older lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older lua versions to migrate and become plugins. \ No newline at end of file diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md index 41104686..12e7c329 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md @@ -1,5 +1,5 @@ # Plugin Config -For all regular fields in a plugin config go [here](/References/Plugin_Config.md). The following fields are only want needs to be added to get lua to python upgrades working. +For all regular fields in a plugin config go [here](/References/Plugin_Config.md). The following fields described are the ones needed to migrate lua toolkits to python plugins. ## Fields diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md index 296d67b1..5acacb7f 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md @@ -21,7 +21,7 @@ A Lua Repository [Data Migration](/References/Glossary.md#data-migration) migrat **Optional.**
!!! warning - You must ensure that all repository data will match your updated repository schema after an upgrade operation. + You must ensure that all repository data will match your updated repository schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more repository data migrations. ### Delphix Engine Operations From 71961fabb5b819a6694740cbeb8780de9d829daa Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Thu, 4 Jun 2020 00:18:26 -0700 Subject: [PATCH 3/7] update with comments from girish and vidya --- .../Plugin_Config.md | 6 +++--- .../Plugin_Operations.md | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md index 12e7c329..ef960b22 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md @@ -5,8 +5,8 @@ For all regular fields in a plugin config go [here](/References/Plugin_Config.md |Field Name|Required|Type|Description| |----------|:------:|:--:|-----------| -|luaName|N|string|The name of the lua toolkit this plugin should upgrade from. This field is required if the luaMinimumVersion is defined.| -| luaMinimumVersion |N|string|The lowest major minor vesrion of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.| +|luaName|N|string|The name of the lua toolkit this plugin should upgrade from. This field is required if the minimumLuaVersion is defined.| +|minimumLuaVersion|N|string|The lowest major minor vesrion of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.| ## Example Assume a lua toolkit with the following `main.json` file: @@ -127,7 +127,7 @@ Here is a valid plugin config for a plugin that wants to be upgradable from the id: ea009cb4-f76b-46dc-bbb6-689e7acecce4 name: DelphixDB luaName: delphixdb -luaMinimumVersion: "1.0" +minimumLuaVersion: "1.0" language: PYTHON27 hostTypes: - UNIX diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md index 5acacb7f..79cd3bea 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md @@ -40,7 +40,7 @@ A Lua Repository [Data Migration](/References/Glossary.md#data-migration) migrat Argument | Type | Description -------- | ---- | ----------- lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per repository data migration must be unique. -migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this operation will default to a regular [repository data migration](/References/Plugin_Operations.md#repository-data-migration). ### Function Arguments Argument | Type | Description @@ -95,7 +95,7 @@ A Lua Source Config [Data Migration](/References/Glossary.md#data-migration) mig Argument | Type | Description -------- | ---- | ----------- lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per source config data migration must be unique. -migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this operation will default to a regular [source config data migration](/References/Plugin_Operations.md#source-config-data-migration). ### Function Arguments Argument | Type | Description @@ -149,7 +149,7 @@ A Lua Linked Source [Data Migration](/References/Glossary.md#data-migration) mig Argument | Type | Description -------- | ---- | ----------- lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per linked source data migration must be unique. -migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this operation will default to a regular [linked source data migration](/References/Plugin_Operations.md#linked-source-data-migration). ### Function Arguments Argument | Type | Description @@ -203,7 +203,7 @@ A Lua Virtual Source [Data Migration](/References/Glossary.md#data-migration) mi Argument | Type | Description -------- | ---- | ----------- lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per virtual source data migration must be unique. -migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this operation will default to a regular [virtual source data migration](/References/Plugin_Operations.md#virtual-source-data-migration). ### Function Arguments Argument | Type | Description @@ -257,7 +257,7 @@ A Lua Snapshot [Data Migration](/References/Glossary.md#data-migration) migrates Argument | Type | Description -------- | ---- | ----------- lua_version | String | The lua version of the migration that the upgrade would be migrating from. This is the ID of this migration. The version here is actually just the major and minor version of the lua toolkit therefore each defined migration's lua_version per snapshot data migration must be unique. -migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this field will default to MigrationType.PLATFORM. +migration_type | String | This migration type field indicates whether the operation is lua or just a regular data migration. Signify this as LUA so that the major minor version can be used. If not defined, this operation will default to a regular [snapshot data migration](/References/Plugin_Operations.md#snapshot-data-migration). ### Function Arguments Argument | Type | Description From 80fb3b5d7a2407fe2e5b60996ffd5d68930b5be4 Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Mon, 22 Jun 2020 16:15:55 -0700 Subject: [PATCH 4/7] address jeff's comment + add statement about discouraging new versions of lua once a python plugin is written --- .../Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md | 2 ++ .../Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md | 8 ++++---- .../Plugin_Operations.md | 2 +- docs/readme.md | 2 +- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md index f070e753..f79732a4 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md @@ -16,3 +16,5 @@ The other way to migrate from a Lua toolkit to a plugin is to wait and write a p ## Supporting migrations with older versions of Lua Having the ability to define Lua upgrade operations in the new plugin code means that older Lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older Lua versions to migrate and become plugins. +!!! warning "New versions of a Lua toolkit is strongly discouraged after Python Plugin is written" + After having written a Plugin to migrate a specific Lua toolkit, while possible, you should avoid writing new major/minor versions of the toolkit in Lua. Patch releases with no schema changes can still be done. If you need to write a new Lua toolkit version please contact Delphix Support to get help on updating migrations accordingly. \ No newline at end of file diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md index ef960b22..063ceaed 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md @@ -6,7 +6,7 @@ For all regular fields in a plugin config go [here](/References/Plugin_Config.md |Field Name|Required|Type|Description| |----------|:------:|:--:|-----------| |luaName|N|string|The name of the lua toolkit this plugin should upgrade from. This field is required if the minimumLuaVersion is defined.| -|minimumLuaVersion|N|string|The lowest major minor vesrion of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.| +|minimumLuaVersion|N|string|The lowest major minor version of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.| ## Example Assume a lua toolkit with the following `main.json` file: @@ -138,6 +138,6 @@ schemaFile: schema.json buildNumber: 2.0.0 ``` -!!! info "id and luaName fields in plugins versus vs name field in toolkits" - * If the id field of the plugin being uploaded happans to match the name field in the toolkit already installed on the engine, the upload will fail regardless of what the luaName is. Otherwise, the luaName will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the luaName is not set then no lua toolkit will be upgraded. - * Also when uploading a plugin with a luaName, that luaName and id pair will be the only one accepted. If a new plugin with the same luaName but different id is uploaded, this will fail. +!!! info "ID and luaName fields in plugins versus vs name field in toolkits" + * If the ID field of the plugin being uploaded happens to match the name field in the toolkit already installed on the engine, the upload will fail regardless of what the luaName is. Otherwise, the luaName will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the luaName is not set then no lua toolkit will be upgraded. + * When uploading a plugin with the luaName set, that luaName and ID pair will be the only pair uploaded successfully. Uploading a new plugin with the same luaName but different ID will fail. diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md index 79cd3bea..2f1cf41c 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md @@ -21,7 +21,7 @@ A Lua Repository [Data Migration](/References/Glossary.md#data-migration) migrat **Optional.**
!!! warning - You must ensure that all repository data will match your updated repository schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more repository data migrations. + You must ensure that all repository data will match your updated repository schema after an upgrade operation. Depending on how your schema has changed, this might imply that you need to write one or more repository data migrations. ### Delphix Engine Operations diff --git a/docs/readme.md b/docs/readme.md index 8c1e5de4..d74a142a 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -87,7 +87,7 @@ This will generate the `site` directory which will contain all the gererated doc 3. Copy all these files to the root directory of the virtualization-sdk repo and delete all other files. 4. Commit and push these changes to your individual fork. 5. Go to your individual virtualization-sdk repo's settings, scroll to the bottom and verify under the GitHub Pages section the `Source` is set to `gh-pages branch`. -6. Right above this will be a link exaplining where your docs are published. +6. Right above this will be a link explaining where your docs are published. ## Workflow diagrams From ec45d19af7eaad85af300b7f5f2209f2e71e92d9 Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Tue, 23 Jun 2020 11:52:58 -0700 Subject: [PATCH 5/7] add backticks around field names --- .../Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md index 063ceaed..9b3499ca 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md @@ -138,6 +138,6 @@ schemaFile: schema.json buildNumber: 2.0.0 ``` -!!! info "ID and luaName fields in plugins versus vs name field in toolkits" - * If the ID field of the plugin being uploaded happens to match the name field in the toolkit already installed on the engine, the upload will fail regardless of what the luaName is. Otherwise, the luaName will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the luaName is not set then no lua toolkit will be upgraded. - * When uploading a plugin with the luaName set, that luaName and ID pair will be the only pair uploaded successfully. Uploading a new plugin with the same luaName but different ID will fail. +!!! info "`id` and `luaName` fields in plugins versus `name` field in toolkits" + * If the `id` of the plugin being uploaded happens to match the `name` in the toolkit already installed on the engine, the upload will fail regardless of what the `luaName` is. Otherwise, the `luaName` will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the `luaName` is not set then no lua toolkit will be upgraded. + * When uploading a plugin with the `luaName` set, that `luaName` and `id` pair will be the only pair uploaded successfully. Uploading a new plugin with the same `luaName` but different `id` will fail. From 3361bd7e1831c11d38bddc7a60814b49a24bf453 Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Thu, 25 Jun 2020 17:56:20 -0700 Subject: [PATCH 6/7] update with Ravi and Alex comments --- .../Converting_Migration_Scripts.md | 4 ++-- .../Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md | 4 ++-- .../Plugin_Operations.md | 10 +++++----- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md index fe14ddeb..0787a76c 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md @@ -1,5 +1,5 @@ # Converting Lua Upgrade Scripts to Python Data Migrations -To convert migrations (a.k.a. "upgrade scripts") that were originally written in lua we need to get the version that the migration upgrades from, the object type the migration is written for, and lastly convert the code into python code using the [decorators](Decorators.md) described previously. +To convert migrations (a.k.a. "upgrade scripts") that were originally written in lua, we need to get the version that the migration upgrades from, the object type the migration is written for, and lastly convert the code into python code using the [decorators](Decorators.md) described previously. ## Example Assume there are two versions of a lua toolkit, 1.0.0 and 1.1.0 where the 1.1.0 version is following the basic toolkit directory structure (actually containing all operations): @@ -53,4 +53,4 @@ def upgrade_linked_source(old_linked_source): return new_linked_source ``` -You will need to determine how far back in the lua upgrade chain you want to support multi-step upgrade from, and convert all of the upgrade scripts accordingly. Remember that the execution of these scripts rely on there not being any missing migrations, and will be executed from the lowest version that exists to the highest version. +You will need to determine how far back in the Lua upgrade chain you want to support multi-step upgrade from, and convert all of those upgrade scripts accordingly. Remember that the execution of these scripts relies on there not being any missing migrations from the `minimumLuaVersion` defined in the plugin config to the last toolkit version written. Lua migrations will be executed migrations from the lowest to highest version that exists. When executing, these migrations are run to the highest Lua toolkit version only. Any migrations needed to get from that toolkit to the Python plugin would need to be written as a regular Python migration. diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md index f79732a4..a48470b0 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md @@ -2,7 +2,7 @@ Before the Virtualization SDK was written, Delphix only supported toolkits written in Lua. There was only limited documentation to help people write, build, and upload toolkits. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there must be a way to get customers off of Lua toolkits and onto SDK plugins. -If you are reading this and have no idea what a Lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to be able to upgrade objects that were created using old toolkits into objects in the Delphix Engine pointing to a plugin. +If you are reading this and have no idea what a Lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to convert objects created using Lua toolkits to use the newly uploaded Python plugin. In the next few pages, we also make the assuption that you've written both a Lua toolkit and a python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from Lua toolkits into the mix as described below. @@ -17,4 +17,4 @@ The other way to migrate from a Lua toolkit to a plugin is to wait and write a p Having the ability to define Lua upgrade operations in the new plugin code means that older Lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older Lua versions to migrate and become plugins. !!! warning "New versions of a Lua toolkit is strongly discouraged after Python Plugin is written" - After having written a Plugin to migrate a specific Lua toolkit, while possible, you should avoid writing new major/minor versions of the toolkit in Lua. Patch releases with no schema changes can still be done. If you need to write a new Lua toolkit version please contact Delphix Support to get help on updating migrations accordingly. \ No newline at end of file + After having written a Plugin to migrate a specific Lua toolkit, while possible, you should avoid writing new major/minor versions of the toolkit in Lua. Patch releases with no schema changes can still be done. If you need to write a new Lua toolkit version please contact Delphix Support to get help on updating migrations accordingly. diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md index 2f1cf41c..99f56cee 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Operations.md @@ -15,7 +15,7 @@ Plugin Operation | **Required** | Decorator | Delphix Engine Operations ## Lua Repository Data Migration -A Lua Repository [Data Migration](/References/Glossary.md#data-migration) migrates repository data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. +A Lua Repository [Data Migration](/References/Glossary.md#data-migration) migrates repository data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format defined in the Python plugin. ### Required / Optional **Optional.**
@@ -70,7 +70,7 @@ def add_new_flag_to_repo(old_repository): ## Lua Source Config Data Migration -A Lua Source Config [Data Migration](/References/Glossary.md#data-migration) migrates source config data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. +A Lua Source Config [Data Migration](/References/Glossary.md#data-migration) migrates source config data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format defined in the Python plugin. ### Required / Optional **Optional.**
@@ -124,7 +124,7 @@ def add_new_flag_to_source_config(old_source_config): ``` ## Lua Linked Source Data Migration -A Lua Linked Source [Data Migration](/References/Glossary.md#data-migration) migrates linked source data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. +A Lua Linked Source [Data Migration](/References/Glossary.md#data-migration) migrates linked source data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format defined in the Python plugin. ### Required / Optional **Optional.**
@@ -178,7 +178,7 @@ def add_new_flag_to_dsource(old_linked_source): ``` ## Lua Virtual Source Data Migration -A Lua Virtual Source [Data Migration](/References/Glossary.md#data-migration) migrates virtual source data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. +A Lua Virtual Source [Data Migration](/References/Glossary.md#data-migration) migrates virtual source data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format defined in the Python plugin. ### Required / Optional **Optional.**
@@ -232,7 +232,7 @@ def add_new_flag_to_vdb(old_virtual_source): ``` ## Lua Snapshot Data Migration -A Lua Snapshot [Data Migration](/References/Glossary.md#data-migration) migrates snapshot data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format. +A Lua Snapshot [Data Migration](/References/Glossary.md#data-migration) migrates snapshot data from an older [schema](/References/Glossary.md#schema) format defined originally from a lua toolkit to an updated schema format defined in the Python plugin. ### Required / Optional **Optional.**
From 2662c1aed74f023a3ae0eb8a38bfca8dd560b752 Mon Sep 17 00:00:00 2001 From: Lindsey Nguyen Date: Thu, 25 Jun 2020 18:34:51 -0700 Subject: [PATCH 7/7] last update --- .../Converting_Migration_Scripts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md index 0787a76c..8f77955e 100644 --- a/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md +++ b/docs/docs/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md @@ -53,4 +53,4 @@ def upgrade_linked_source(old_linked_source): return new_linked_source ``` -You will need to determine how far back in the Lua upgrade chain you want to support multi-step upgrade from, and convert all of those upgrade scripts accordingly. Remember that the execution of these scripts relies on there not being any missing migrations from the `minimumLuaVersion` defined in the plugin config to the last toolkit version written. Lua migrations will be executed migrations from the lowest to highest version that exists. When executing, these migrations are run to the highest Lua toolkit version only. Any migrations needed to get from that toolkit to the Python plugin would need to be written as a regular Python migration. +You will need to determine how far back in the Lua upgrade chain you want to support multi-step upgrade from, and convert all of those upgrade scripts accordingly. Remember that the execution of these scripts relies on there not being any missing migrations from the `minimumLuaVersion` defined in the plugin config to the last toolkit version written. Lua migrations will be executed from the lowest to highest version that exists. When executing, these migrations are run to the highest Lua toolkit version only. Any migrations needed to get from that toolkit to the Python plugin would need to be written as a regular Python migration.