vitess Online DDL atomic cut-over #11460

shlomi-noach · 2022-10-11T08:32:56Z

Description

This is work in progress for an atomic RENAME (not two-step gaping hole) in the cut-over process of a vitess/vreplication Online DDL migration.

Currently, we run a two-step rename, when the original table is renamed away, and then a 2nd rename moves the vreplication table in its place. This is protected under a buffering rule on the primary, but replicas see two distinct renames and there is a point in time on the replica, where the table does not exist. Of course this means queries are failing.
There is also a scenario depicted in #11226, where even queries against the primary may fail.

This PR introduces an algorithm similar to the gh-ost cut-over, described in http://code.openark.org/blog/mysql/solving-the-non-atomic-table-swap-take-iii-making-it-atomic and in github/gh-ost#82

We still retain our query buffering logic, which protects is against existing logic failures in the gh-ost implementation, per github/gh-ost#887.

the new logic to be described in detail once the PR is validated and considered safe. Some noteworthy details in the meantime:

We introduce a sentry table, which means VReplication needs to know about it, even though its schema is immaterial. We therefore need to run VReplicationWaitForPos after creating the sentry table, which increases the overall cut-over process time. This does not affect incoming traffic, as the creation of the table comes before the critical section of locking/blocking/renaming. It does, however, lock the executor's mutex, which potentially adds a few more seconds to an already a few seconds worth of process. The impact is internal and of no real concern.

We remove the stowaway_table logic. The swap is atomic, and there is no need to recover a "stowaway scenario". We will keep the recovery logic for backwards compatibility for one full release, then remove it. The recovery logic is found here:

vitess/go/vt/vttablet/onlineddl/executor.go

Lines 3258 to 3280 in 893cc9d

    
           if stowawayTable := row.AsString("stowaway_table", ""); stowawayTable != "" { 
        
           	// whoa 
        
           	// stowawayTable is an original table stowed away while cutting over a vrepl migration, see call to cutOverVReplMigration() down below in this function. 
        
           	// In a normal operation, the table should not exist outside the scope of cutOverVReplMigration 
        
           	// If it exists, that means a tablet crashed while running a cut-over, and left the database in a bad state, where the migrated table does not exist. 
        
           	// thankfully, we have tracked this situation and just realized what happened. Now, first thing to do is to restore the original table. 
        
           	log.Infof("found stowaway table %s journal in migration %s for table %s", stowawayTable, uuid, onlineDDL.Table) 
        
           	attemptMade, err := e.renameTableIfApplicable(ctx, stowawayTable, onlineDDL.Table) 
        
           	if err != nil { 
        
           		// unable to restore table; we bail out, and we will try again next round. 
        
           		return countRunnning, cancellable, err 
        
           	} 
        
           	// success 
        
           	if attemptMade { 
        
           		log.Infof("stowaway table %s restored back into %s", stowawayTable, onlineDDL.Table) 
        
           	} else { 
        
           		log.Infof("stowaway table %s did not exist and there was no need to restore it", stowawayTable) 
        
           	} 
        
           	// OK good, table restored. We can remove the record. 
        
           	if err := e.updateMigrationStowawayTable(ctx, uuid, ""); err != nil { 
        
           		return countRunnning, cancellable, err 
        
           	} 
        
           }

We introduce a stage column to _vt.schema_migrations. It serves multiple purposes:
- we populate the stage column in every step of the cut-over logic, so that we can easily see where a failure took place
- we audit every such change to stage in the logs, so we have a full history
- most importantly, we use it to generate a GTID entry post LOCK TABLES and RENAME TABLES: the LOCK TABLES query does not generate a GTID entry. The RENAME TABLES query is blocked due to LOCK TABLES and of course does not generate a GTID entry. We update stage column immediately after issuing both, and thus ensure we have a post-lock/rename position for which we can wait for in VReplicationWaitForPos.

Related Issue(s)

Checklist

"Backport me!" label has been added if this change should be backported
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <[email protected]>

…-stage Signed-off-by: Shlomi Noach <[email protected]>

Signed-off-by: Shlomi Noach <[email protected]>

vitess-bot · 2022-10-11T08:32:58Z

Signed-off-by: Shlomi Noach <[email protected]>

…-stage Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2022-10-12T08:58:07Z

A walkthrough to explain the new cut-over logic and why it is implemented the way it is.

First, to explain the current logic, also discussed in this blog post.

Current cut-over logic

Assume we modify table t. The changes actually take place on t_vrepl:

                 t             t_vrepl

When t_vrepl is in near sync with t, we place a quey buffering rule on table t, such that new queries on t are buffered. While buffering takes place, we do a two-step rename:

rename table t to t_placeholder, leaving a puncture in our database: t does not exist

t_placeholder    [ ]            t_vrepl

complete any final changes to t_vrepl and then rename t_vrepl to t, t_placeholder to t_vrepl

                  t             t_vrepl

In essence, we have swapped t and t_vrepl. But in the process, there is a point in time where t does not exist. Queries should not be aware of that thanks to the query buffering. However, there is no buffering on the replicas, and inconspicuous queries running on replicas, such as select * from t limit 1 might return with a table t does not exist eror.

Renaming t away serves two purposes:

An implicit block while any pending queries on t complete execution
An absolute guarantee that no new queries operate on t afterwards

New cut-over logic in this PR

We want to avoid the puncture, such that replicas always have t available. Essentially, this means we must run something like rename table t to t_placeholder, t_vrepl to t (we in fact run something more elaborate). However, since vreplication online DDL is asynchronous, we can't just run this query. We have to ensure we apply any remaining binary log queries that operate on t onto t_vrepl.

The query buffering should help us: activating the buffering means no queries should operate on t. However, it's not a full guarantee. The buffering only works for queries coming through vtgate, and they do not apply to queries coming from elsewhere. "elsewhere" could mean any local automation users may place, any monitoring/introspection queries, etc.

We therefore employ a variation of gh-ost's cut-over algorithm, which, together with the query buffering, offers stronger guarantees. We first describe a naive implementation, then point out where it can fail.

In the naive implementation

We place a lock on t via lock tables t write
We apply any remaining entries from the binary log onto t_vrepl
We follow by a rename table t to t_placeholder, t_vrepl to t.

Problem is this won't work as expected. (1) and (3) can't both run from the same connection. If they were, (3) would fail because the connection did not lock t_vrepl. But if the connection were to lock t_vrepl in (1) then vreplication would not be able to apply anything to that table. And we can't let vreplication own that connection because the Online DDL executor operates on a different place than vreplication, even if both run on the same server.

So (1) and (3) must run by different connections. But this poses a risk. When we run (3), how do we ensure that (1) is still holding the lock? Maybe (1)'s connections was terminated, releasing the lock? Meaning queries are actually able to run against t, and so running (3) would cause corruption?

This is how gh-ost cut-over logic came to be. It places protection mechanims so that the rename can't take place unless everything is in perfect order.

Non-naive, simplified implementation

Create a sentry table t_sentry
A connection, call it "lock conn", issues lock tables t_sentry write, t write
A 2nd connection, call it "rename conn" issues rename table t to t_sentry, t_vrepl to t. This query blocks due to the lock query. However, if the lock query is accidentally killed, the rename fails, because t_sentry exists.
The rename does not block writes to t_vrepl because it's locked on renaming t. vreplication is able to complete processing the backlog.
We issue a VReplicationWaitForPos: the two tables are now in sync.
The lock conn issues a drop table t_sentry.
lock conn issues a unlock tables
This releases the rename query, which is able to swap the tables
done

Nuances

We create a sentry table. This confuses vreplication because it does not know the table. It must be told about the table because we lock it. We therefore make an initial VReplicationWaitForPos right after creating the table and before placing locks.
in step (3), we wait until the rename is found to be running in MySQL. The rename takes precedence over any DML that may happen to be queued on the LOCK.
in step (5) we issue a VReplicationWaitForPos to ensure the tables are in sync. But, to which GTID do we wait for? The statement LOCK TABLES ... does not generate a GTID entry. The rename ... is blocked and therefore also does not generate a GTID entry. We therefore inject an intentional transaction after the rename is blocked. We update the _vt.schema_migrations stage column. Doesn't matter exactly what we update, as long as we generate a GTID. We know that this GTID is made after t is completely blocked to writes.
The rename is actually a full swap: rename table t to t_sentry, t_vrepl to t, t_sentry to t_vrepl

Signed-off-by: Shlomi Noach <[email protected]>

…-stage Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2022-10-18T13:14:21Z

Seeing CI errors in the form

target: ks.1.primary: primary is not serving, there is a reparent operation in progress (errno 1105) (sqlstate HY000) during query: revert vitess_migration '5a4c0f1b_4ee5_11ed_8e91_000d3a36ac92'

I suspect something crashes the primary tablet. Of course this only happens in GitHub CI and not on local machine, so it's a bit of a chase to find out what happened.

shlomi-noach · 2022-10-19T05:35:29Z

Not even sure the CI issue is related to this PR. Still looking.

shlomi-noach · 2022-10-20T05:37:48Z

From an internal discussion it seems the aforementioned CI problem is colossal, and not specifically related to this PR.

shlomi-noach · 2022-10-20T05:39:20Z

Ready to review! Please make sure to read the original comment and then also #11460 (comment), if you want to have a good grasp of what's going on. Hint: it's complex!

…-stage Signed-off-by: Shlomi Noach <[email protected]>

deepthi

This sounds good to me.
Do we have a test that reproduces the problem that is described in the PR, or is that too difficult (needs to be timed precisely)?
Let's get a review from @mattlord as well.

shlomi-noach · 2022-11-07T06:29:11Z

Do we have a test that reproduces the problem that is described in the PR, or is that too difficult (needs to be timed precisely)?

Yeah, it's too difficult I think; and on top of this, because we do use query buffering on top, I'm not sure I can reproduce it.

Signed-off-by: Shlomi Noach <[email protected]>

mattlord

Looks good! This approach makes sense to me and seems sound.

I only had some minor comments/questions/suggestions. Thanks!

go/vt/vttablet/onlineddl/executor.go

mattlord · 2022-11-07T18:50:52Z

go/vt/vttablet/onlineddl/executor.go

@@ -739,31 +765,99 @@ func (e *Executor) cutOverVReplMigration(ctx context.Context, s *VReplStream) er
 		return err
 	}
 	isVreplicationTestSuite := onlineDDL.StrategySetting().IsVreplicationTestSuite()
+	e.updateMigrationStage(ctx, onlineDDL.UUID, "starting cut-over")


This can fail for a variety of reasons, e.g. deadlock. We should have some error handling and perhaps retry logic. Or am I missing something?

this is nice-to-have but not strictly critical for the cut-over. It adds a level of auditing/logging. Not sure we should fail the cut-over over failure of this action.

go/vt/vttablet/onlineddl/executor.go

mattlord · 2022-11-07T19:56:30Z

go/vt/vttablet/onlineddl/executor.go

+			e.updateMigrationStage(ctx, onlineDDL.UUID, "dropping sentry table")
+			dropTableQuery := sqlparser.BuildParsedQuery(sqlDropTable, sentryTableName)


Shouldn't these be in the following code block? Or we could just remove the code block. I personally think it's less readable in this case.

mattlord · 2022-11-07T19:58:20Z

go/vt/vttablet/onlineddl/executor.go

+	if err != nil {
+		return err
+	}
+	_, err = e.execQuery(ctx, query)


This is the error that we're not handling.

to summarize, we're ignoring it where it doesn't have a true effect on the cut-over, as we consider it more of a logging/debuggability feature

mattlord · 2022-11-07T20:01:07Z

go/vt/vttablet/onlineddl/schema.go

@@ -81,6 +81,8 @@ const (
 	alterSchemaMigrationsComponentThrottled            = "ALTER TABLE _vt.schema_migrations add column component_throttled tinytext NOT NULL"
 	alterSchemaMigrationsCancelledTimestamp            = "ALTER TABLE _vt.schema_migrations add column cancelled_timestamp timestamp NULL DEFAULT NULL"
 	alterSchemaMigrationsTablePostponeLaunch           = "ALTER TABLE _vt.schema_migrations add column postpone_launch tinyint unsigned NOT NULL DEFAULT 0"
+	alterSchemaMigrationsStage                         = "ALTER TABLE _vt.schema_migrations add column stage text not null"


text seems a little excessive, no? In case we want to use this in query predicates and index it. Not a problem, just struck me.

I believe we're running out of varchar space for 5.7. I may be mistaken but I have this recollection from a few months ago when I tried adding a varchar column.

Ah, we're probably hitting the [on] page size limit (text can be moved off page to secondary storage).

This is a good example of why we should probably start leveraging JSON columns more, otherwise we'll end up with hundreds of columns which may or may not be used at different times. We're using them now in VDiff2 FWIW.

go/vt/vttablet/onlineddl/executor.go

Signed-off-by: Shlomi Noach <[email protected]>

mattlord

LGTM. Nice work on this!

go/vt/vttablet/onlineddl/executor.go

shlomi-noach · 2022-11-09T07:52:41Z

force merging: only failing test is vttablet_prscomplex is is irrelevant to this PR (and resolved in an as yet unmerged PR)

earayu · 2023-02-03T03:40:36Z

Hi, @shlomi-noach

1. We place a lock on t via lock tables t write （lock conn）
2. We apply any remaining entries from the binary log onto t_vrepl
3. We follow by a rename table t to t_placeholder, t_vrepl to t.

(1) and (3) can't both run from the same connection. If they were, (3) would fail because the connection did not lock t_vrepl
I don't quiet understand this statement:
I think step 3 can be executed by lock conn. Won't step 2 release all locks of t_vrepl after finishing applying all the remaining entries? therefore, lock conn in step 3 will be free to acquire t_vrepl's lock.

shlomi-noach · 2023-02-03T10:53:27Z

I think step 3 can be executed by lock conn.

Unfortunately not. Try this at home:

create table t0 (id int primary key);
create table t1 (id int primary key);
lock tables t0 write;
rename table t0 to told, t1 to t0;

output:

ERROR 1100 (HY000): Table 't1' was not locked with LOCK TABLES

rename will only work if:

you have not locked any tables involved in the statement,
or, you have locked all tables involved in the statement.

But we really want to lock just the one table.

For this reason we have to use a third connection, which complicates the entire logic.

See my suggestion for an upstream change: mysql/mysql-server#426, https://bugs.mysql.com/bug.php?id=108864

* cut-over with sentry table Signed-off-by: Shlomi Noach <[email protected]> * wait for rename via channel; write extra transaction post LOCK Signed-off-by: Shlomi Noach <[email protected]> * add stage info Signed-off-by: Shlomi Noach <[email protected]> * reduced wait-for-pos timeout. Improved stage logic Signed-off-by: Shlomi Noach <[email protected]> * cleanup Signed-off-by: Shlomi Noach <[email protected]> * more cleanup Signed-off-by: Shlomi Noach <[email protected]> * even more cleanup Signed-off-by: Shlomi Noach <[email protected]> * context timeout Signed-off-by: Shlomi Noach <[email protected]> * preserve stage by disabling deferred stage changes Signed-off-by: Shlomi Noach <[email protected]> * killing rename query upon error Signed-off-by: Shlomi Noach <[email protected]> * increase test timeout Signed-off-by: Shlomi Noach <[email protected]> * fix defer ordering Signed-off-by: Shlomi Noach <[email protected]> * log.info Signed-off-by: Shlomi Noach <[email protected]> * add and populate cutover_attempts column Signed-off-by: Shlomi Noach <[email protected]> * search PROCESSLIST with LIKE Signed-off-by: Shlomi Noach <[email protected]> * code comment Signed-off-by: Shlomi Noach <[email protected]> * making a variable more local Signed-off-by: Shlomi Noach <[email protected]> * literal string Signed-off-by: Shlomi Noach <[email protected]> Signed-off-by: Shlomi Noach <[email protected]>

* cut-over with sentry table * wait for rename via channel; write extra transaction post LOCK * add stage info * reduced wait-for-pos timeout. Improved stage logic * cleanup * more cleanup * even more cleanup * context timeout * preserve stage by disabling deferred stage changes * killing rename query upon error * increase test timeout * fix defer ordering * log.info * add and populate cutover_attempts column * search PROCESSLIST with LIKE * code comment * making a variable more local * literal string Signed-off-by: Shlomi Noach <[email protected]> Co-authored-by: Shlomi Noach <[email protected]>

shlomi-noach added 8 commits October 1, 2022 09:40

cut-over with sentry table

cb02fd6

Signed-off-by: Shlomi Noach <[email protected]>

wait for rename via channel; write extra transaction post LOCK

908631b

Signed-off-by: Shlomi Noach <[email protected]>

add stage info

cc818db

Signed-off-by: Shlomi Noach <[email protected]>

reduced wait-for-pos timeout. Improved stage logic

288007a

Signed-off-by: Shlomi Noach <[email protected]>

Merge branch 'main' into vrepl-online-ddl-cut-over-atomic-with-sentry…

474c95f

…-stage Signed-off-by: Shlomi Noach <[email protected]>

cleanup

b0c7862

Signed-off-by: Shlomi Noach <[email protected]>

more cleanup

c0bfe63

Signed-off-by: Shlomi Noach <[email protected]>

even more cleanup

6abb9c0

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving labels Oct 11, 2022

shlomi-noach mentioned this pull request Oct 11, 2022

WIP: vitess Online DDL atomic cut-over #11418

Closed

3 tasks

shlomi-noach added 7 commits October 11, 2022 12:39

context timeout

faad587

Signed-off-by: Shlomi Noach <[email protected]>

preserve stage by disabling deferred stage changes

76dfcd5

Signed-off-by: Shlomi Noach <[email protected]>

killing rename query upon error

e15fa21

Signed-off-by: Shlomi Noach <[email protected]>

increase test timeout

795c27c

Signed-off-by: Shlomi Noach <[email protected]>

fix defer ordering

d3ef3b1

Signed-off-by: Shlomi Noach <[email protected]>

log.info

d4aed59

Signed-off-by: Shlomi Noach <[email protected]>

Merge branch 'main' into vrepl-online-ddl-cut-over-atomic-with-sentry…

d2f4bdf

…-stage Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach mentioned this pull request Oct 12, 2022

Bug Report: query error Table doesn't exist during DDL #11226

Closed

shlomi-noach added 3 commits October 12, 2022 12:11

add and populate cutover_attempts column

769507e

Signed-off-by: Shlomi Noach <[email protected]>

Merge branch 'main' into vrepl-online-ddl-cut-over-atomic-with-sentry…

7bf25ff

…-stage Signed-off-by: Shlomi Noach <[email protected]>

Merge branch 'main' into vrepl-online-ddl-cut-over-atomic-with-sentry…

aa2a538

…-stage Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach changed the title ~~WIP: vitess Online DDL atomic cut-over~~ vitess Online DDL atomic cut-over Oct 20, 2022

shlomi-noach marked this pull request as ready for review October 20, 2022 05:38

shlomi-noach requested a review from deepthi as a code owner October 20, 2022 05:38

Merge branch 'main' into vrepl-online-ddl-cut-over-atomic-with-sentry…

9f8d31b

…-stage Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach requested a review from a team October 20, 2022 09:02

shlomi-noach mentioned this pull request Oct 23, 2022

New 'RENAME TABLE ... NO LOCK CHECK' optional clause. mysql/mysql-server#426

Closed

deepthi reviewed Nov 7, 2022

View reviewed changes

deepthi requested a review from mattlord November 7, 2022 01:40

merge main, resolve conflict

450eeb3

Signed-off-by: Shlomi Noach <[email protected]>

mattlord reviewed Nov 7, 2022

View reviewed changes

go/vt/vttablet/onlineddl/executor.go Outdated Show resolved Hide resolved

shlomi-noach added 4 commits November 8, 2022 08:25

search PROCESSLIST with LIKE

e53f030

Signed-off-by: Shlomi Noach <[email protected]>

code comment

a41a940

Signed-off-by: Shlomi Noach <[email protected]>

making a variable more local

123a376

Signed-off-by: Shlomi Noach <[email protected]>

literal string

ad548fe

Signed-off-by: Shlomi Noach <[email protected]>

mattlord approved these changes Nov 8, 2022

View reviewed changes

go/vt/vttablet/onlineddl/executor.go Show resolved Hide resolved

shlomi-noach merged commit a2b75c7 into vitessio:main Nov 9, 2022

shlomi-noach deleted the vrepl-online-ddl-cut-over-atomic-with-sentry-stage branch November 9, 2022 07:52

shlomi-noach mentioned this pull request Feb 9, 2023

Online DDL: remove legacy "stowaway table" logic #12288

Merged

3 tasks

austenLacy mentioned this pull request Jun 23, 2023

Backport fix for Vitess Online DDL atomic cut-over Shopify/vitess#106

Merged

shlomi-noach mentioned this pull request Jun 25, 2023

v15 backport: vitess Online DDL atomic cut-over #13376

Merged

4 tasks

austenLacy mentioned this pull request Aug 1, 2023

Custom jwt http ACL policy Shopify/vitess#114

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vitess Online DDL atomic cut-over #11460

vitess Online DDL atomic cut-over #11460

shlomi-noach commented Oct 11, 2022 •

edited by deepthi

Loading

vitess-bot bot commented Oct 11, 2022 •

edited by mattlord

Loading

shlomi-noach commented Oct 12, 2022

shlomi-noach commented Oct 18, 2022

shlomi-noach commented Oct 19, 2022

shlomi-noach commented Oct 20, 2022

shlomi-noach commented Oct 20, 2022

deepthi left a comment

shlomi-noach commented Nov 7, 2022

mattlord left a comment

mattlord Nov 7, 2022

shlomi-noach Nov 8, 2022

mattlord Nov 7, 2022

shlomi-noach Nov 8, 2022

mattlord Nov 7, 2022

shlomi-noach Nov 8, 2022

mattlord Nov 7, 2022

shlomi-noach Nov 8, 2022

mattlord Nov 8, 2022

mattlord left a comment

shlomi-noach commented Nov 9, 2022

earayu commented Feb 3, 2023 •

edited

Loading

shlomi-noach commented Feb 3, 2023

	if stowawayTable := row.AsString("stowaway_table", ""); stowawayTable != "" {
	// whoa
	// stowawayTable is an original table stowed away while cutting over a vrepl migration, see call to cutOverVReplMigration() down below in this function.
	// In a normal operation, the table should not exist outside the scope of cutOverVReplMigration
	// If it exists, that means a tablet crashed while running a cut-over, and left the database in a bad state, where the migrated table does not exist.
	// thankfully, we have tracked this situation and just realized what happened. Now, first thing to do is to restore the original table.
	log.Infof("found stowaway table %s journal in migration %s for table %s", stowawayTable, uuid, onlineDDL.Table)
	attemptMade, err := e.renameTableIfApplicable(ctx, stowawayTable, onlineDDL.Table)
	if err != nil {
	// unable to restore table; we bail out, and we will try again next round.
	return countRunnning, cancellable, err
	}
	// success
	if attemptMade {
	log.Infof("stowaway table %s restored back into %s", stowawayTable, onlineDDL.Table)
	} else {
	log.Infof("stowaway table %s did not exist and there was no need to restore it", stowawayTable)
	}
	// OK good, table restored. We can remove the record.
	if err := e.updateMigrationStowawayTable(ctx, uuid, ""); err != nil {
	return countRunnning, cancellable, err
	}
	}

		e.updateMigrationStage(ctx, onlineDDL.UUID, "dropping sentry table")
		dropTableQuery := sqlparser.BuildParsedQuery(sqlDropTable, sentryTableName)

vitess Online DDL atomic cut-over #11460

vitess Online DDL atomic cut-over #11460

Conversation

shlomi-noach commented Oct 11, 2022 • edited by deepthi Loading

Description

Related Issue(s)

Checklist

Deployment Notes

vitess-bot bot commented Oct 11, 2022 • edited by mattlord Loading

Review Checklist

General

If a new flag is being introduced:

If a workflow is added or modified:

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

shlomi-noach commented Oct 12, 2022

A walkthrough to explain the new cut-over logic and why it is implemented the way it is.

Current cut-over logic

New cut-over logic in this PR

In the naive implementation

Non-naive, simplified implementation

Nuances

shlomi-noach commented Oct 18, 2022

shlomi-noach commented Oct 19, 2022

shlomi-noach commented Oct 20, 2022

shlomi-noach commented Oct 20, 2022

deepthi left a comment

Choose a reason for hiding this comment

shlomi-noach commented Nov 7, 2022

mattlord left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord left a comment

Choose a reason for hiding this comment

shlomi-noach commented Nov 9, 2022

earayu commented Feb 3, 2023 • edited Loading

shlomi-noach commented Feb 3, 2023

shlomi-noach commented Oct 11, 2022 •

edited by deepthi

Loading

vitess-bot bot commented Oct 11, 2022 •

edited by mattlord

Loading

earayu commented Feb 3, 2023 •

edited

Loading