CP-3829 Upgrade plugin passwords to new credentials object

[rasantel]

Problem

As discussed on https://docs.google.com/document/d/11qMuv755UquN8a_D3PDukJDOacvNCOv_TK4bbbvkBos/edit#heading=h.enbe66druy0e, plugin writers should have an option to upgrade their existing password fields to the new, more general credentials provider type of field introduced in CP-3163 which adds the option to use password vaults.

Solution

App gate companion review: http://reviews.delphix.com/r/63226/

Added a new callback upgrade_password that plugin migrations can call to upgrade a payload's password fields to credentials supplier. Specifically, upgrade_password returns a credentials supplier that "directly" supplies the password (i.e. a supplier that itself contains the password, rather than using a password vault).

From a Delphix user's perspective, the field continues to provide the same password, so nothing changes immediately. The only difference is that, after the upgrade, the user can now modify the way in which the field supplies a password (e.g. fetch it from a password vault).

Two other optional parameters can be passed to upgrade_password: username (a string) and expectedSecretType (either ANY or PASSWORD, used at runtime to validate the type of credentials returned by the credentials supplier).

The credentials supplier returned by the new callback is a protobuf Struct of this form:

{
  "type": "NamedPasswordCredential",
  "password": "<password passed to upgrade_password>",
  "username": "<username passed to upgrade_password (optional)>",
  "expectedType": "<the string 'password' (optional)>"
}

This Struct conforms to Virtualization's JSON Schema definition of a directPasswordCredentialsSupplier which is one of the possible forms that credentialsSupplier can take: https://gitlab.delphix.com/app/dlpx-app-gate/blob/projects/password-vault-plugins/appliance/server/core/src/main/resources/json-schema/definitions.json

Update

I had to patch protobuf's json_format.py because Jython, unlike Python, cannot read Unicode's surrogate codes if they are not paired. See //github.com/protocolbuffers/protobuf/issues/7776 .

Manual testing

1. Created a test plugin with a password-formatted string property.
2. In a test engine, uploaded the plugin and created a new linked source that sets a test value for that password.
3. Updated the plugin to a new version that in its schema replaces the type of the password property with a reference to the new credentialsSupplier definition.
4. Added a migration to the new plugin that calls upgrade_password with the password value and stores in that field the returned credentials supplier.
5. Verified that, after upgrading the plugin in the engine, the same password is now being supplied at runtime by this new credentials supplier.
6. Verified that a user can now change that supplier so that the password can be provided by a password vault.

Automated testing

Added unit tests.

dx-test: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/dx-integration-tests/21976/
blackbox failed with dc timeout. Re-run: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/65826/consoleFull
appdata_basic APPDATA_PYTHON_DIRECT_CENTOS73: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/65689/
appdata_basic APPDATA_PYTHON_STAGED_CENTOS73: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/65690/
appdata_basic APPDATA_PYTHON_DIRECT_WIN2012: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/65691/
appdata_basic APPDATA_PYTHON_STAGED_WIN2012: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/65692/
appdata_plugin_upgrade from 1.0.0 to 2.1.0 using the VSDK change from PR 269: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/65687/

[mothslaw]

Looks great. Sorry you had to deal with that Jython/unicode nonsense.

[nhlien93]

This looks good to me!