Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/Best_Practices/Sensitive_Data.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ This tells the Delphix Engine to take special precautions with this password pro
4. Clients of the Delphix Engine's public API will not be able to access the password.

!!! note
Removing a previously added password property from a field and running a [Data Migration](Glossary.md#data-migration) will expose the password in plaintext. If this is intentional, write a migration to ensure that the new property conforms to the new schema.
Removing a previously added password property from a field and running a [Data Migration](/References/Glossary.md#data-migration) will expose the password in plaintext. If this is intentional, write a migration to ensure that the new property conforms to the new schema.

# Using Environment Variables For Remote Data Passing

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/Building_Your_First_Plugin/Discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ The Delphix Engine will pass us information about the source environment in an a
return [RepositoryDefinition(name='1e87dc30-3cdb-4f0a-9634-07ce017d20d1')]
```

This creates and returns a Python object that corresponds to the format defined by our repository schema. Because out repository has exactly one string property called `name`, therefore this Python object has one property called `name`.
This creates and returns a Python object that corresponds to the format defined by our repository schema. Because our repository has exactly one string property called `name`, therefore this Python object has one property called `name`.

Notice that the code generator has filled in the value of `name` with a random string. This results in a plugin operation that works, but which will not be very helpful for the user. We'll change this later.

Expand Down
6 changes: 3 additions & 3 deletions docs/docs/Building_Your_First_Plugin/Initial_Setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Before we begin to start writing plugin code, we will need to do some setup work

The quoted examples in this section assume you're working on a Unix-like system.

## Sanity check
## Quick Check

First a reminder that it's highly recommended that you develop your plugin in a [virtual environment](https://virtualenv.pypa.io/en/latest/).

Expand Down Expand Up @@ -53,14 +53,14 @@ To start, we will create a new directory where our new plugin code will live.
Now that we are in our new plugin directory, we can use the `dvp` tool to create a plugin for us. This plugin will be a mere skeleton -- it will not do anything useful until we modify it in the subsequent pages.

```
(venv) first_plugin$ dvp init -n first_plugin -s STAGED -p WINDOWS
(venv) first_plugin$ dvp init -n first_plugin -s STAGED -t WINDOWS
```

The `-n` argument here means "plugin name." We are using the name `first_plugin`.

The `-s` argument tells which syncing strategy we want to use.

The `-p` argument tells which host platform our plugin supports.
The `-t` argument tells which host platform our plugin supports.

You can type `dvp init -h` for more information about the options available.

Expand Down
3 changes: 3 additions & 0 deletions docs/docs/References/Glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ The process by which the Delphix Engine ingests data from a dataset on a [source
## Target Environment
An [environment](#environment) on which Delphix-provided virtualized datasets can be used.

## Lua Toolkit
Legacy model for writing "plugins" in Lua, with limited documentation and support for writing, building and uploading toolkits. This was the predecessor to the Virtualization SDK.

## Upgrade Operation
A special plugin operation that takes data produced by an older version of a plugin, and transforms it into the format expected by the new version of the plugin.

Expand Down
12 changes: 8 additions & 4 deletions docs/docs/Release_Notes/2.1.0/2.1.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,12 @@ To install or upgrade the SDK, refer to instructions [here](/Getting_Started.md#

## New & Improved

* Added the ability to migrate existing Lua toolkits to plugins.
* [Convert](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md) any existing Lua upgrade scripts to Python migrations by using our new Lua upgrade [decorators](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md).
* Added new fields `luaName` and `minimumLuaVersion` to the [Plugin Config](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md). These properties allow a plugin to specify which Lua toolkit(s) the plugin can upgrade.
* Added the ability to migrate existing [Lua toolkits](/References/Glossary.md#lua-toolkit) to SDK plugins.
* [Convert](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Converting_Migration_Scripts.md) any existing Lua upgrade scripts to Python migrations by using new Lua upgrade [decorators](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Decorators.md).
* Added new optional fields `luaName` and `minimumLuaVersion` to the [Plugin Config](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Plugin_Config.md). These properties allow a plugin to specify which Lua toolkit(s) the plugin can migrate.

More details about Lua Upgrade can be found [here](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md).
More details about Lua toolkit migration can be found [here](/Versioning_And_Upgrade/Lua_Toolkit_To_SDK_Plugin_Migration/Overview.md).

## Breaking Changes

No breaking changes in this release!
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Converting Lua Upgrade Scripts to Python Data Migrations
To convert migrations (a.k.a. "upgrade scripts") that were originally written in lua, we need to get the version that the migration upgrades from, the object type the migration is written for, and lastly convert the code into python code using the [decorators](Decorators.md) described previously.
To convert migrations (a.k.a. "upgrade scripts") that were originally written in Lua, we need to get the version that the migration upgrades from, the object type the migration is written for, and lastly convert the code into Python code using the [decorators](Decorators.md) described previously.

## Example
Assume there are two versions of a lua toolkit, 1.0.0 and 1.1.0 where the 1.1.0 version is following the basic toolkit directory structure (actually containing all operations):
Assume there are two versions of a lua toolkit, `1.0.0` and `1.1.0` where the `1.1.0` version is following the basic toolkit directory structure (actually containing all operations):

```
├── main.json
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Decorators

The Virtualization SDK exposes [decorators](/References/Decorators.md) as mentioned in the regular documentation. Below we list the additional operations added to suport lua to python migrations. This assumes the name of the `Plugin()` object is `plugin`:
The Virtualization SDK exposes [decorators](/References/Decorators.md) as mentioned in the regular documentation. Below we list the additional operations added to suport Lua to Python migrations. This assumes `Plugin()` is instantiated as `plugin`:

Plugin Operation | Decorator
---------------- | --------
Expand All @@ -11,5 +11,4 @@ Plugin Operation | Decorator
[Lua Snapshot Data Migration](Plugin_Operations.md#lua-snapshot-data-migration) | `@plugin.upgrade.snapshot(lua_version, MigrationType.LUA)`

!!! info "lua_version format"
The lua_version field in this decorator should be the major minor string of the lua toolkit. This means if in the main.json file the version is set to "1.1.HOTFIX123" the lua_version passed into this decorator should be "1.1".

The `lua_version` field in this decorator should be the (major,minor) version of the Lua toolkit. This means if the version is set to `1.1.HOTFIX123` in the `main.json` file for the Lua toolkit, the `lua_version` passed into this decorator should be `1.1`.
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Overview

Before the Virtualization SDK was written, Delphix only supported toolkits written in Lua. There was only limited documentation to help people write, build, and upload toolkits. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there must be a way to get customers off of Lua toolkits and onto SDK plugins.
Before the Virtualization SDK was written, Delphix only supported [toolkits](/References/Glossary.md#lua-toolkit) written in Lua. There was only limited documentation to help people write, build, and upload toolkits. Multiple toolkits were released and are still used by customers today, so as we move towards only supporting SDK Plugins, there needs to be a way to get customers off of Lua toolkits and onto SDK plugins.

If you are reading this and have no idea what a Lua toolkit is, there is no reason to read further into this section. Everything written in these pages will assume the goal is to write specific code as part of a plugin to convert objects created using Lua toolkits to use the newly uploaded Python plugin.

In the next few pages, we also make the assuption that you've written both a Lua toolkit and a python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from Lua toolkits into the mix as described below.
In the next few pages, we also make the assumption that you've written both a Lua toolkit and a Python plugin before and know some of the terminology already established. If this is not true, please try [building a plugin](/Building_Your_First_Plugin/Overview.md) and [writing some upgrade migrations](/Versioning_And_Upgrade/Upgrade.md) first before coming back here to learn how to add upgrading from Lua toolkits into the mix as described below.

## Basic no-schema Migration
One way to migrate from a Lua toolkit to a plugin is to write an exactly equivalent plugin that does not make any [schema](/References/Schemas.md) changes to the objects that were defined originally in the Lua toolkit. If this is the scenario you are in, then you only need to update the [plugin config](Plugin_Config.md) with a couple of new Lua migration specific fields.
Expand All @@ -17,4 +17,4 @@ The other way to migrate from a Lua toolkit to a plugin is to wait and write a p
Having the ability to define Lua upgrade operations in the new plugin code means that older Lua version migration scripts can be [converted](Converting_Migration_Scripts.md), enabling multi-step upgrades from older Lua versions to migrate and become plugins.

!!! warning "New versions of a Lua toolkit is strongly discouraged after Python Plugin is written"
After having written a Plugin to migrate a specific Lua toolkit, while possible, you should avoid writing new major/minor versions of the toolkit in Lua. Patch releases with no schema changes can still be done. If you need to write a new Lua toolkit version please contact Delphix Support to get help on updating migrations accordingly.
After having written a Plugin to migrate a specific Lua toolkit, while possible, you should avoid writing new major/minor versions of the toolkit in Lua. Patch releases with no schema changes can still be done. If you need to write a new Lua toolkit version please contact the Delphix Virtualization SDK Engineering team to get help on updating migrations accordingly.
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Plugin Config
For all regular fields in a plugin config go [here](/References/Plugin_Config.md). The following fields described are the ones needed to migrate lua toolkits to python plugins.
For all regular fields in a plugin config go [here](/References/Plugin_Config.md). The following fields described are the ones needed to migrate Lua toolkits to Python plugins.

## Fields

|Field Name|Required|Type|Description|
|----------|:------:|:--:|-----------|
|luaName|N|string|The name of the lua toolkit this plugin should upgrade from. This field is required if the minimumLuaVersion is defined.|
|minimumLuaVersion|N|string|The lowest major minor version of the lua toolkit that upgrade is supported from. This field is required if the luaName is defined.|
|luaName|N|string|The name of the Lua toolkit this plugin should upgrade from. This field is required if the minimumLuaVersion is defined.|
|minimumLuaVersion|N|string|The lowest major minor version of the Lua toolkit that upgrade is supported from. This field is required if the luaName is defined.|

## Example
Assume a lua toolkit with the following `main.json` file:
Expand Down Expand Up @@ -139,5 +139,7 @@ buildNumber: 2.0.0
```

!!! info "`id` and `luaName` fields in plugins versus `name` field in toolkits"
* If the `id` of the plugin being uploaded happens to match the `name` in the toolkit already installed on the engine, the upload will fail regardless of what the `luaName` is. Otherwise, the `luaName` will be used to determine if an already uploaded lua toolkit is considered a lower version of the plugin being uploaded. If the `luaName` is not set then no lua toolkit will be upgraded.
* The `luaName` will be used to determine if an already uploaded Lua toolkit is considered a lower version of the Pyhon plugin being uploaded.
* If the `luaName` is not set then no Lua toolkit will be upgraded.
* If the `id` of the plugin being uploaded happens to match the `name` in the Lua toolkit already installed on the Delphix Engine, the upload will fail regardless of what the `luaName` is.
* When uploading a plugin with the `luaName` set, that `luaName` and `id` pair will be the only pair uploaded successfully. Uploading a new plugin with the same `luaName` but different `id` will fail.
Loading