VTOrc Cleanup - Configs, APIs and old UI

doc/releasenotes/15_0_0_summary.md

@@ -266,63 +266,93 @@ With this new `explain` format, you can get an output that is very similar to th
 
 ### VTOrc
 
-#### Configuration Renames
-
-VTOrc configurations that had `Orchestrator` as a substring have been renamed to have `VTOrc` instead. The old configuration won't 
-work in this release. So if backward compatibility is desired, then before upgrading it is suggested to duplicate the old configurations
-and also set them for the new configurations.
-
-VTOrc ignores the configurations that it doesn't understand. So new configurations can be added while running the previous release.
-After the upgrade, the old configurations can be dropped.
-
-|           Old Configuration            |        New Configuration        |
-|:--------------------------------------:|:-------------------------------:|
-|         MySQLOrchestratorHost          |         MySQLVTOrcHost          |
-|  MySQLOrchestratorMaxPoolConnections   |  MySQLVTOrcMaxPoolConnections   |
-|         MySQLOrchestratorPort          |         MySQLVTOrcPort          |
-|       MySQLOrchestratorDatabase        |       MySQLVTOrcDatabase        |
-|         MySQLOrchestratorUser          |         MySQLVTOrcUser          |
-|       MySQLOrchestratorPassword        |       MySQLVTOrcPassword        |
-| MySQLOrchestratorCredentialsConfigFile | MySQLVTOrcCredentialsConfigFile |
-|   MySQLOrchestratorSSLPrivateKeyFile   |   MySQLVTOrcSSLPrivateKeyFile   |
-|      MySQLOrchestratorSSLCertFile      |      MySQLVTOrcSSLCertFile      |
-|       MySQLOrchestratorSSLCAFile       |       MySQLVTOrcSSLCAFile       |
-|     MySQLOrchestratorSSLSkipVerify     |     MySQLVTOrcSSLSkipVerify     |
-|     MySQLOrchestratorUseMutualTLS      |     MySQLVTOrcUseMutualTLS      |
-|  MySQLOrchestratorReadTimeoutSeconds   |  MySQLVTOrcReadTimeoutSeconds   |
-|    MySQLOrchestratorRejectReadOnly     |    MySQLVTOrcRejectReadOnly     |
-
-
-For example, if you have the following configuration -
+#### Old UI Removal and Replacement
+
+The old UI that VTOrc inherited from `Orchestrator` has been removed. A replacement UI, more consistent with the other Vitess binaries has been created.
+In order to use the new UI, `--port` flag has to be provided. 
+
+Along with the UI, the old APIs have also been deprecated. However, some of them have been ported over to the new UI - 
+
+| Old API                          | New API                          | Additional notes                                                      |
+|----------------------------------|----------------------------------|-----------------------------------------------------------------------|
+| `/api/problems`                  | `/api/problems`                  | The new API also supports filtering using the keyspace and shard name |
+| `/api/disable-global-recoveries` | `/api/disable-global-recoveries` | Functionally remains the same                                         |
+| `/api/enable-global-recoveries`  | `/api/enable-global-recoveries`  | Functionally remains the same                                         |
+| `/api/health`                    | `/debug/health`                  | Functionally remains the same                                         |
+| `/api/replication-analysis`      | `/api/replication-analysis`      | Functionally remains the same. Output is now JSON format.             |
+
+Apart from these APIs, we also now have `/debug/status`, `/debug/vars` and `/debug/liveness` available in the new UI.
+
+#### Configuration Refactor and New Flags 
+
+Since VTOrc was forked from `Orchestrator`, it inherited a lot of configurations that don't make sense for the Vitess use-case.
+All of such configurations have been removed.
+
+VTOrc ignores the configurations that it doesn't understand. So old configurations can be kept around on upgrading and won't cause any issues.
+They will just be ignored.
+
+For all the configurations that are kept, flags have been added for them and the flags are the desired way to pass these configurations going forward.
+The config file will be deprecated and removed in upcoming releases. The following is a list of all the configurations that are kept and the associated flags added.
+
+|          Configurations Kept          |           Flags Introduced            |
+|:-------------------------------------:|:-------------------------------------:|
+|            SQLite3DataFile            |         `--sqlite-data-file`          |
+|          InstancePollSeconds          |        `--instance-poll-time`         |
+|    SnapshotTopologiesIntervalHours    |    `--snapshot-topology-interval`     |
+|    ReasonableReplicationLagSeconds    |    `--reasonable-replication-lag`     |
+|             AuditLogFile              |        `--audit-file-location`        |
+|             AuditToSyslog             |         `--audit-to-backend`          |
+|           AuditToBackendDB            |          `--audit-to-syslog`          |
+|            AuditPurgeDays             |       `--audit-purge-duration`        |
+|      RecoveryPeriodBlockSeconds       |  `--recovery-period-block-duration`   |
+| PreventCrossDataCenterPrimaryFailover |    `--prevent-cross-cell-failover`    |
+|        LockShardTimeoutSeconds        |        `--lock-shard-timeout`         |
+|      WaitReplicasTimeoutSeconds       |       `--wait-replicas-timeout`       |
+|     TopoInformationRefreshSeconds     | `--topo-information-refresh-duration` |
+|          RecoveryPollSeconds          |      `--recovery-poll-duration`       |
+
+Apart from configurations, some flags from VTOrc have also been removed -
+- `sibling`
+- `destination`
+- `discovery`
+- `skip-unresolve`
+- `skip-unresolve-check`
+- `noop`
+- `binlog`
+- `statement`
+- `grab-election`
+- `promotion-rule`
+- `skip-continuous-registration`
+- `enable-database-update`
+- `ignore-raft-setup`
+- `tag`
+
+The ideal way to ensure backward compatibility is to remove the flags listed above while on the previous release. Then upgrade VTOrc.
+After upgrading, remove the config file and instead pass the flags that are introduced.
+
+#### Example Upgrade
+
+If you are running VTOrc with the flags `--ignore-raft-setup --clusters_to_watch="ks/0" --config="path/to/config"` and the following configuration
 ```json
 {
-  "MySQLOrchestratorHost": "host"
-}
-```
-then, you should change it to
-```json
-{
-  "MySQLOrchestratorHost": "host",
-  "MySQLVTOrcHost": "host"
-}
-```
-while still on the old release. After changing the configuration, you can upgrade vitess.
-After upgrading, the old configurations can be dropped - 
-```json
-{
-  "MySQLVTOrcHost": "host"
+  "Debug": true,
+  "ListenAddress": ":6922",
+  "MySQLTopologyUser": "orc_client_user",
+  "MySQLTopologyPassword": "orc_client_user_password",
+  "MySQLReplicaUser": "vt_repl",
+  "MySQLReplicaPassword": "",
+  "RecoveryPeriodBlockSeconds": 1,
+  "InstancePollSeconds": 1,
+  "PreventCrossDataCenterPrimaryFailover": true
 }
 ```
+First drop the flag `--ignore-raft-setup` while on the previous release. So, you'll be running VTOrc with `--clusters_to_watch="ks/0" --config="path/to/config"` and the same configuration listed above.
+
+Now you can upgrade your VTOrc version continuing to use the same flags and configurations, and it will continue to work just the same. If you wish to use the new UI, then you can add the `--port` flag as well.
+
+After upgrading, you can drop the configuration entirely and use the new flags like `--clusters_to_watch="ks/0" --recovery-period-block-duration=1s --instance-poll-time=1s --prevent-cross-cell-failover`
 
 #### Default Configuration Files
 
 The default files that VTOrc searches for configurations in have also changed from `"/etc/orchestrator.conf.json", "conf/orchestrator.conf.json", "orchestrator.conf.json"` to
 `"/etc/vtorc.conf.json", "conf/vtorc.conf.json", "vtorc.conf.json"`.
-
-#### Debug Pages in VTOrc
-
-Like the other vitess binaries (`vtgate`, `vttablet`), now `vtorc` also takes a `--port` flag, on which it 
-displays the `/debug` pages including `/debug/status` and variables it tracks on `/debug/vars`.
-
-This change is backward compatible and opt-in by default. Not specifying the flag works like it used to with
-VTOrc running without displaying these pages.

docker/lite/Dockerfile.alpine

@@ -51,8 +51,6 @@ ENV MYSQL_FLAVOR MariaDB103
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.mariadb

@@ -51,8 +51,6 @@ ENV MYSQL_FLAVOR MariaDB
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.mariadb103

@@ -51,8 +51,6 @@ ENV MYSQL_FLAVOR MariaDB103
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.mysql57

@@ -50,8 +50,6 @@ ENV PATH $VTROOT/bin:$PATH
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.mysql80

@@ -51,8 +51,6 @@ ENV MYSQL_FLAVOR MySQL80
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.percona57

@@ -50,8 +50,6 @@ ENV PATH $VTROOT/bin:$PATH
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.percona80

@@ -51,8 +51,6 @@ ENV MYSQL_FLAVOR MySQL80
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.testing

@@ -50,8 +50,6 @@ ENV PATH $VTROOT/bin:$PATH
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 # Create mount point for actual data (e.g. MySQL data dir)

docker/lite/Dockerfile.ubi7.mysql57

@@ -78,8 +78,6 @@ ENV PATH $VTROOT/bin:$PATH
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 RUN mkdir -p /licenses

docker/lite/Dockerfile.ubi7.mysql80

@@ -79,8 +79,6 @@ ENV MYSQL_FLAVOR MySQL80
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 RUN mkdir -p /licenses

docker/lite/Dockerfile.ubi7.percona57

@@ -69,8 +69,6 @@ ENV PATH $VTROOT/bin:$PATH
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 RUN mkdir -p /licenses

docker/lite/Dockerfile.ubi7.percona80

@@ -74,8 +74,6 @@ ENV MYSQL_FLAVOR MySQL80
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 RUN mkdir -p /licenses

docker/lite/Dockerfile.ubi8.arm64.mysql80

@@ -86,8 +86,6 @@ ENV MYSQL_FLAVOR MySQL80
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 RUN mkdir -p /licenses

docker/lite/Dockerfile.ubi8.mysql80

@@ -84,8 +84,6 @@ ENV MYSQL_FLAVOR MySQL80
 
 # Copy artifacts from builder layer.
 COPY --from=builder --chown=vitess:vitess /vt/install /vt
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/orchestrator /vt/web/orchestrator
-COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtorc /vt/web/vtorc
 COPY --from=builder --chown=vitess:vitess /vt/src/vitess.io/vitess/web/vtadmin /vt/web/vtadmin
 
 RUN mkdir -p /licenses

examples/compose/README.md

@@ -150,11 +150,8 @@ vitess/examples/compose$ ./lvtctl.sh Help
 - vtgate web ui:
   http://localhost:15099/debug/status
 
-- vtorc web ui:
+- vtorc ui:
   http://localhost:13000
-
-- vtorc debug ui:
-  http://localhost:13200
 
 - Stream querylog
   `curl -S localhost:15099/debug/querylog`

examples/compose/docker-compose.beginners.yml

@@ -285,8 +285,7 @@ services:
       - vtctld
       - set_keyspace_durability_policy
     ports:
-      - "13200:8080"
-      - "13000:3000"
+      - "13000:8080"
     volumes:
       - ".:/script"
     environment:
@@ -304,7 +303,7 @@ services:
       - DB_PASS
       - DB_CHARSET
     healthcheck:
-      test: ["CMD-SHELL","curl -s --fail --show-error localhost:3000/api/status"]
+      test: ["CMD-SHELL","curl -s --fail --show-error localhost:8080/debug/health"]
       interval: 5s
       timeout: 10s
       retries: 15

examples/compose/docker-compose.yml

@@ -184,8 +184,7 @@ services:
     - DB_PASS=
     image: vitess/lite:${VITESS_TAG:-latest}
     ports:
-    - 13200:8080
-    - 13000:3000
+    - 13000:8080
     volumes:
     - .:/script
   vttablet101:

examples/compose/vtcompose/docker-compose.test.yml

@@ -201,8 +201,7 @@ services:
       - DB_PASS=
     image: vitess/lite:${VITESS_TAG:-latest}
     ports:
-      - 13000:3000
-      - 13200:8080
+      - 13000:8080
     volumes:
       - .:/script
   vttablet101:

examples/compose/vtcompose/vtcompose.go

@@ -750,8 +750,7 @@ func generateVTOrc(dbInfo externalDbInfo, keyspaceInfoMap map[string]keyspaceInf
       - DB_USER=%[4]s
       - DB_PASS=%[5]s
     ports:
-      - "13000:3000"
-      - "13200:%[1]d"
+      - "13000:%[1]d"
     command: ["sh", "-c", "/script/vtorc-up.sh"]
     %[6]s
 `, opts.webPort, opts.topologyFlags, externalDb, dbInfo.dbUser, dbInfo.dbPass, dependsOn)

examples/compose/vtorc-up.sh

@@ -42,5 +42,4 @@ exec /vt/bin/vtorc \
 $TOPOLOGY_FLAGS \
 --logtostderr=true \
 --port $web_port \
---orc_web_dir=/vt/web/vtorc \
 --config $config

examples/compose/vtorc/default.json

@@ -1,7 +1,6 @@
 {
   "Debug": true,
   "EnableSyslog": false,
-  "ListenAddress": ":3000",
   "MySQLTopologyUser": "orc_client_user",
   "MySQLTopologyPassword": "orc_client_user_password",
   "MySQLReplicaUser": "vt_repl",

examples/local/scripts/vtorc-up.sh

@@ -3,13 +3,10 @@
 source ./env.sh
 
 log_dir="${VTDATAROOT}/tmp"
-web_dir="../../web/vtorc"
-vtorc_web_port=16000
-port=16001
+port=16000
 
 vtorc \
   $TOPOLOGY_FLAGS \
-  --orc_web_dir "${web_dir}" \
   --logtostderr \
   --alsologtostderr \
   --config="./vtorc/config.json" \
@@ -21,8 +18,7 @@ echo ${vtorc_pid} > "${log_dir}/vtorc.pid"
 
 echo "\
 vtorc is running!
-  - UI: http://localhost:${vtorc_web_port}
-  - Debug UI: http://localhost:${port}
+  - UI: http://localhost:${port}
   - Logs: ${log_dir}/vtorc.out
   - PID: ${vtorc_pid}
 "

examples/local/vtorc/config.json

@@ -1,5 +1,4 @@
 {
-  "ListenAddress": ":16000",
   "MySQLTopologyUser": "orc_client_user",
   "MySQLTopologyPassword": "orc_client_user_password",
   "MySQLReplicaUser": "vt_repl",