VTOrc Standardisation and Cleanup

[GuptaManan100]

Description

This PR removes the storage of the external dependencies of VTOrc locally.

• The external zk and util packages were unused after the cleanup in VTOrc Cleanup - Configs, APIs and old UI #11356
• The usage of external test package has been changed with the require package.
• The sqlutils package, instead of being stored in the repo is now stored as a go-mod dependency.

Related Issue(s)

• RFC: Orchestrator as a native Vitess component #6612

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

• Is it really necessary to add this flag?
• Flag names should be clear and intuitive (as far as possible)
• Help text should be descriptive.
• Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow should be required, the maintainer team should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from VTop, if used there.

[shlomi-noach]

Looks good -- only thing is to verify that github.com/openark/golib/sqlutils is equal to the local golib/sqlutils package. I have a vague memory that it is -- I believe I synced it a couple years ago, but I'm not entirely sure.

[rsajwani]

he sqlutils package, instead of being stored in the repo is not stored as a go-mod dependency.
--> he sqlutils package, instead of being stored in the repo is "now" stored as a go-mod dependency.

[rsajwani]

LGTM .. same concern that you have pointed out .. is there a way to verify if openark/sqlutils is similar functionally to golib/sqlutils ?

[GuptaManan100]

I did a diff between our sqlutils and sqlutils from openark -

diff sqlutils/sqlite_dialect.go sqlutils-openark/sqlite_dialect.go
20c20
< // queries issued by vtorc. There are known limitations to this design.
---
> // queries issued by orchestrator. There are known limitations to this design.
diff sqlutils/sqlite_dialect_test.go sqlutils-openark/sqlite_dialect_test.go
24c24
<       test "vitess.io/vitess/go/vt/vtorc/external/golib/tests"
---
>       test "github.com/openark/golib/tests"
87c87
<                               ADD COLUMN sql_delay INT UNSIGNED NOT NULL AFTER replica_lag_seconds
---
>                               ADD COLUMN sql_delay INT UNSIGNED NOT NULL AFTER slave_lag_seconds
100c100
<                               ADD INDEX source_host_port_idx (source_host, source_port)
---
>                               ADD INDEX master_host_port_idx (master_host, master_port)
105,106c105,106
<                               source_host_port_idx_database_instance
<                               on database_instance (source_host, source_port)
---
>                               master_host_port_idx_database_instance
>                               on database_instance (master_host, master_port)
129,130c129,130
<                               source_host_port_idx_database_instance
<                               on database_instance (source_host(128), source_port)
---
>                               master_host_port_idx_database_instance
>                               on database_instance (master_host(128), master_port)
135,136c135,136
<                               source_host_port_idx_database_instance
<                               on database_instance (source_host, source_port)
---
>                               master_host_port_idx_database_instance
>                               on database_instance (master_host, master_port)
208c208
<               statement := "AND primary_instance.last_attempted_check <= primary_instance.last_seen + interval ? minute"
---
>               statement := "AND master_instance.last_attempted_check <= master_instance.last_seen + interval ? minute"
210c210
<               test.S(t).ExpectEquals(result, "AND primary_instance.last_attempted_check <= datetime(primary_instance.last_seen, printf('+%d minute', ?))")
---
>               test.S(t).ExpectEquals(result, "AND master_instance.last_attempted_check <= datetime(master_instance.last_seen, printf('+%d minute', ?))")
213c213
<               statement := "select concat(primary_instance.port, '') as port"
---
>               statement := "select concat(master_instance.port, '') as port"
215c215
<               test.S(t).ExpectEquals(result, "select (primary_instance.port || '') as port")
---
>               test.S(t).ExpectEquals(result, "select (master_instance.port || '') as port")
diff sqlutils/sqlutils.go sqlutils-openark/sqlutils.go
21a22
>       "errors"
28c29
<       "vitess.io/vitess/go/vt/log"
---
>       "github.com/openark/golib/log"
69c70
<       cells := make([](*CellData), len(*this))
---
>       cells := make([](*CellData), len(*this), len(*this))
77,78c78,79
< func (this *RowData) Args() []any {
<       result := make([]any, len(*this))
---
> func (this *RowData) Args() []interface{} {
>       result := make([]interface{}, len(*this))
208c209
<       buff := make([]any, len(columns))
---
>       buff := make([]interface{}, len(columns))
210c211
<       for i := range buff {
---
>       for i, _ := range buff {
213c214
<       _ = rows.Scan(buff...)
---
>       rows.Scan(buff...)
256c257
< func QueryRowsMap(db *sql.DB, query string, on_row func(RowMap) error, args ...any) (err error) {
---
> func QueryRowsMap(db *sql.DB, query string, on_row func(RowMap) error, args ...interface{}) (err error) {
269,270c270
<               log.Error(err)
<               return err
---
>               return log.Errore(err)
277c277
< func queryResultData(db *sql.DB, query string, retrieveColumns bool, args ...any) (resultData ResultData, columns []string, err error) {
---
> func queryResultData(db *sql.DB, query string, retrieveColumns bool, args ...interface{}) (resultData ResultData, columns []string, err error) {
280c280
<                       err = fmt.Errorf("QueryRowsMap unexpected error: %+v", derr)
---
>                       err = errors.New(fmt.Sprintf("QueryRowsMap unexpected error: %+v", derr))
285a286
>       defer rows.Close()
287d287
<               log.Error(err)
290,291d289
<       defer rows.Close()
< 
305c303
< func QueryResultData(db *sql.DB, query string, args ...any) (ResultData, error) {
---
> func QueryResultData(db *sql.DB, query string, args ...interface{}) (ResultData, error) {
311c309
< func QueryNamedResultData(db *sql.DB, query string, args ...any) (NamedResultData, error) {
---
> func QueryNamedResultData(db *sql.DB, query string, args ...interface{}) (NamedResultData, error) {
319c317
< func QueryRowsMapBuffered(db *sql.DB, query string, on_row func(RowMap) error, args ...any) error {
---
> func QueryRowsMapBuffered(db *sql.DB, query string, on_row func(RowMap) error, args ...interface{}) error {
335c333
< func ExecNoPrepare(db *sql.DB, query string, args ...any) (res sql.Result, err error) {
---
> func ExecNoPrepare(db *sql.DB, query string, args ...interface{}) (res sql.Result, err error) {
338c336
<                       err = fmt.Errorf("ExecNoPrepare unexpected error: %+v", derr)
---
>                       err = errors.New(fmt.Sprintf("ExecNoPrepare unexpected error: %+v", derr))
344c342
<               log.Error(err)
---
>               log.Errore(err)
351c349
< func execInternal(silent bool, db *sql.DB, query string, args ...any) (res sql.Result, err error) {
---
> func execInternal(silent bool, db *sql.DB, query string, args ...interface{}) (res sql.Result, err error) {
354c352
<                       err = fmt.Errorf("execInternal unexpected error: %+v", derr)
---
>                       err = errors.New(fmt.Sprintf("execInternal unexpected error: %+v", derr))
365c363
<               log.Error(err)
---
>               log.Errore(err)
372c370
< func Exec(db *sql.DB, query string, args ...any) (sql.Result, error) {
---
> func Exec(db *sql.DB, query string, args ...interface{}) (sql.Result, error) {
377c375
< func ExecSilently(db *sql.DB, query string, args ...any) (sql.Result, error) {
---
> func ExecSilently(db *sql.DB, query string, args ...interface{}) (sql.Result, error) {
390c388
< func Args(args ...any) []any {
---
> func Args(args ...interface{}) []interface{} {
394c392
< func NilIfZero(i int64) any {
---
> func NilIfZero(i int64) interface{} {

The only differences are the interface{} to any change and the error messages and logging because we started using vitess's internal logging package. Also the test file differ slightly in master -> source sort of changes. Functionally the packages are still the same.

[GuptaManan100]

Given that the packages are the same, I am going ahead and merging this PR.