Tablet init super read only

[rsajwani]

Description

Please review https://github.com/vitessio/vitess/pull/12206/files instead.

This change is build on top of vitessio#11520, where instead of using withddl we use declarative approach. Using declarative approach helps us condense all our schema changes to one place. This provide us an opportunity to apply super-read-only changes to our DB. As of withddl is used across schema engines with make it hard to do super-read-only change.

What is super-read-only change

As of today all replicas comes up in read-only mode, however this doesn't prevent user's like root and vt_dba who have SUPER privileges to change the database anytime anywhere. We want to leverage GLOBAL super_read_only configuration in order to protect against errant GTIDs. This will make sure that apart from primary no other component or offline system can mutate DB resulting in errant GTIDs that are then lying in wait to cause later failures, as you can see in vitessio#9312 and vitessio#10094

Furthermore not all the uses-cases are ensuring that we end up replica in read-only mode. During ERS there are cases where we don't set the read-only. With super-read-only change we will make sure that we cover those cases as well.

Details

This PR usese vitessio#11520 as a building block. I am listing major changes below to help you review the PR.

• my.cnf contains super-read-only, to ensure all Mysql instances comes up in super-read-only mode.
• During init_db we switch super-read-only off temporarily in order to perform some mutations like creating necessary users and permission.
• Since MariaDB doesn't have super-read-only, therefore we need to come up with a separate init_db for it.
• Adding # add custom sql here inside init-db.sql file. This is because in our test we modify the file and add custom SQL like (change password of DBA user etc). Now that we expect super_read_only to execute at the end, we need these custom sql to be added before that last line. See example go/test/endtoend/backup/vtbackup/main_test.go
• During PRS & ERS. We make sure that only primary turn-off super-read-only and rest of replica are strictly in super-read-only mode.
• For unit test I introduced a separate init_db.sql so that we don't run into complexity of super-read-only.
• In some of tests I have to execute commands with super-read-only privileges, given the way the test are written.

For Reference

Some related work items done in the past

vitessio#10363
vitessio#11706
vitessio#10094
vitessio#9312
vitessio#10448

Unresolved Issued during code changes:

I have filed few issues which I will need to resolve once I check-in this PR.

• Remove read_only usage from code base vitessio/vitess#12143
• Deprecate and delete use_super_read_only flag from VTTtablet vitessio/vitess#12140
• MySqld has some MariaDB initialization code, that might need some cleanup. vitessio/vitess#12166

Related Issue(s)

vitessio#12180

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[rsajwani]

vttablet + mysqlctl + mysqlctld needs to be always at the same version during upgrades and downgrades. I am taking this assumption after talking to different folks and it look logical to me as well.

[rohit-nayak-ps]

I have done a quick read through of the changes without really following the new flow. Hope to do that tomorrow. I will also look at all the tests where we are expecting different results now.

Suggested some minor typos/stylistic changes and asked some clarifications.

[rohit-nayak-ps]

Did you try checking if the underlying database is mysql or mariadb inside init_db.sql?

[rsajwani]

I discussed it initially with other team members .. and I concluded it will make code complex if we put that logic in .sql file .. or we might need to create templates. Furthermore Maria DB is going to be deprecated anyways in future release.

[rohit-nayak-ps]

Why do we have conditional code for a tablet with version less than 16?

[rsajwani]

For downgrade / upgrade test, tablet version < 16 will not have super read only code handling. Therefore we are explicitly setting super-read-only to false here.

[rohit-nayak-ps]

The fakesqldb condition is commented out: intended?

[mattlord]

There's a lot of stuff commented out (mixing of multi-line and single line comments, etc) and it's not clear what is intended and why.

[rsajwani]

@Rohit, this was commented out to show you, these changes from you PR is not needed .. But I have removed it now... You don't need to add these specific checks here instead I use existing structure of fakesqldb..

[rohit-nayak-ps]

why do we need super readonly handling here: does the test run DDLs?

[rsajwani]

Yeah these test does not have vttablets. They just create mysql instance and run test against them...

[rohit-nayak-ps]

instead of adding this, can't we just look for the line in the file that sets the readonly and add our custom commands above that?

[rsajwani]

I thought that would be more risky ... we don't know how would it react if client provide their own int_db.sql and it has some statements which does turns out true for our condition of readonly... Hence I thought this will be much cleaner approach...

[rohit-nayak-ps]

we should extract this code which is repeated in multiple functions here, into a utility function

[mattlord]

We can remove the MariaDB specific stuff in v16+ vitessio#9518. We still want to support importing/migrating from MariaDB into Vitess, but in that case the MariaDB instance would be external with an unmanaged tablet so init would not apply.

[rsajwani]

The import/migration test is using init_db.sql which now have super-read-only and therefore it fails. This is added to make sure our import/migrate test uses init_db specific to maria db...

[mattlord]

What is this file for? Is it for vttestserver/vtcombo? What's unique about that compared to standard and then below, unit test cases? We can add a comment at the top to explain. It feels like this is part of a larger problem in that we're not testing super_read_only in many tests.

[rsajwani]

We are testing super-read-only for all the cases.

• super-read-only does not make sense for unit test .. therefore I have come up with new config/init_unit_test_db.sql
• testserver/vtcombo does not operate in same manner.. it uses same sql instance for all the vttablet under the hood so we have to get rid of super-read-only in this case as well , hence config/init_testserver_db.sql
• Every other test e2e and backup/restore etc uses init_db.sql file, which has super-read-only.

[mattlord]

This change can have wide impacts. IMO, we should test it in as many ways as possible. i.e. we should use super-read-only in all of the tests, if possible.

[rsajwani]

test-suite is used only in vttestserver. VtTestserver uses vtcombo which has one sql instance for all vttablet. We cannot turn 'on/'off' super-read-only because it will make vtcombo confuse due to one instance of mysql for all vttablets.

[rsajwani]

Are we using vttestserver in our test/e2e , i don't see instances where we are...

[mattlord]

This is another case where it feels like we're avoiding testing the fact that Vitess works properly when super_read_only is enabled. No?

[rsajwani]

if you closely look at I have introduced a flag 'withSuperReadOnly' ... during test setup we introduce errantGTID (https://github.com/planetscale/vitess/pull/77/files#diff-9162b364b991658ff1575922d4a008aecc15e44fd27e64678f24cce330033d9fR108) thats the only place where we pass true to this flag. For all other places we keep it false. Hence the tests are running with super-read-only false.

[mattlord]

Better to cast it to a MySQLError and check the error code.

[rsajwani]

this is how we are doing check for something not part of specific mysql version... we want to warn if super-read-only is not supported in case of db like mariadb and erorr in all other cases. We are not checking any specific mysql erorr.

[mattlord]

How is this function directly/exclusively tied to restores? If someone runs the equivalent of mysqldump foo.10 | mysql, it would be bad if mysqld automatically disabled super_read_only rather than protecting you as it should.

[rsajwani]

please review vitessio#12206 instead.