Skip to content

Backport: Set fully parsed to false when ignoring errors #9955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GuptaManan100 merged 3 commits into from
Mar 24, 2022
Merged

Backport: Set fully parsed to false when ignoring errors #9955

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bea8ac4
set fully parsed false when ignoring errors
GuptaManan100 May 11, 2021
607283a
ci: fix for mysql 80 installation
GuptaManan100 Mar 24, 2022
d001394
ci: remove installation of mysql-server and client from ubuntu-20 sin…
GuptaManan100 Mar 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 2 additions & 22 deletions .github/workflows/cluster_endtoend_mysql80.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,33 +29,13 @@ jobs:

- name: Get dependencies
run: |

sudo DEBIAN_FRONTEND="noninteractive" apt-get update

# Uninstall any previously installed MySQL first
sudo systemctl stop apparmor
sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-server mysql-client mysql-common
sudo apt-get -y autoremove
sudo apt-get -y autoclean
sudo deluser mysql
sudo rm -rf /var/lib/mysql
sudo rm -rf /etc/mysql

# Install mysql80
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
sudo apt-get update
sudo DEBIAN_FRONTEND="noninteractive" apt-get install -y mysql-server mysql-client

# Install everything else we need, and configure
sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo bash -c "echo '/usr/sbin/mysqld { }' > /etc/apparmor.d/usr.sbin.mysqld" # https://bugs.launchpad.net/ubuntu/+source/mariadb-10.1/+bug/1806263
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld || echo "could not remove mysqld profile"

sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download

wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/cluster_endtoend_upgrade.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ jobs:
sudo rm -rf /etc/mysql

# Install mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/e2e_race.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ jobs:
sudo rm -rf /etc/mysql

# Install mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/legacy_local_example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
sudo rm -rf /etc/mysql

# Install mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/local_example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
sudo rm -rf /etc/mysql

# Install mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/region_example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
sudo rm -rf /etc/mysql

# Install mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/unit_test_mysql80.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ jobs:
sudo rm -rf /etc/mysql

# mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.14-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down
43 changes: 43 additions & 0 deletions go/vt/sqlparser/ast.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ type (
AffectedTables() TableNames
SetTable(qualifier string, name string)
SetFromTables(tables TableNames)
SetFullyParsed(fullyParsed bool)
Statement
}

Expand All @@ -72,6 +73,7 @@ type (
iDBDDLStatement()
IsFullyParsed() bool
GetDatabaseName() string
SetFullyParsed(bool)
Statement
}

Expand Down Expand Up @@ -658,41 +660,69 @@ func (*TruncateTable) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DDLStatement interface
func (*TruncateTable) SetFullyParsed(bool) {}

// IsFullyParsed implements the DDLStatement interface
func (*RenameTable) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DDLStatement interface
func (node *RenameTable) SetFullyParsed(fullyParsed bool) {}

// IsFullyParsed implements the DDLStatement interface
func (node *CreateTable) IsFullyParsed() bool {
return node.FullyParsed
}

// SetFullyParsed implements the DDLStatement interface
func (node *CreateTable) SetFullyParsed(fullyParsed bool) {
node.FullyParsed = fullyParsed
}

// IsFullyParsed implements the DDLStatement interface
func (node *AlterTable) IsFullyParsed() bool {
return node.FullyParsed
}

// SetFullyParsed implements the DDLStatement interface
func (node *AlterTable) SetFullyParsed(fullyParsed bool) {
node.FullyParsed = fullyParsed
}

// IsFullyParsed implements the DDLStatement interface
func (node *CreateView) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DDLStatement interface
func (node *CreateView) SetFullyParsed(fullyParsed bool) {}

// IsFullyParsed implements the DDLStatement interface
func (node *DropView) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DDLStatement interface
func (node *DropView) SetFullyParsed(fullyParsed bool) {}

// IsFullyParsed implements the DDLStatement interface
func (node *DropTable) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DDLStatement interface
func (node *DropTable) SetFullyParsed(fullyParsed bool) {}

// IsFullyParsed implements the DDLStatement interface
func (node *AlterView) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DDLStatement interface
func (node *AlterView) SetFullyParsed(fullyParsed bool) {}

// IsTemporary implements the DDLStatement interface
func (*TruncateTable) IsTemporary() bool {
return false
Expand Down Expand Up @@ -1212,16 +1242,29 @@ func (node *DropDatabase) IsFullyParsed() bool {
return true
}

// SetFullyParsed implements the DBDDLStatement interface
func (node *DropDatabase) SetFullyParsed(fullyParsed bool) {}

// IsFullyParsed implements the DBDDLStatement interface
func (node *CreateDatabase) IsFullyParsed() bool {
return node.FullyParsed
}

// SetFullyParsed implements the DBDDLStatement interface
func (node *CreateDatabase) SetFullyParsed(fullyParsed bool) {
node.FullyParsed = fullyParsed
}

// IsFullyParsed implements the DBDDLStatement interface
func (node *AlterDatabase) IsFullyParsed() bool {
return node.FullyParsed
}

// SetFullyParsed implements the DBDDLStatement interface
func (node *AlterDatabase) SetFullyParsed(fullyParsed bool) {
node.FullyParsed = fullyParsed
}

// GetDatabaseName implements the DBDDLStatement interface
func (node *DropDatabase) GetDatabaseName() string {
return node.DBName.String()
Expand Down
50 changes: 33 additions & 17 deletions go/vt/sqlparser/parse_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1016,8 +1016,13 @@ var (
}, {
input: "alter table a upgrade partitioning",
}, {
input: "alter table a partition by range (id) (partition p0 values less than (10), partition p1 values less than (maxvalue))",
output: "alter table a",
input: "alter table a partition by range (id) (partition p0 values less than (10), partition p1 values less than (maxvalue))",
output: "alter table a",
partialDDL: true,
}, {
input: "create database a garbage values",
output: "create database a",
partialDDL: true,
}, {
input: "alter table `Post With Space` drop foreign key `Post With Space_ibfk_1`",
}, {
Expand Down Expand Up @@ -1063,15 +1068,17 @@ var (
}, {
input: "alter table a add check (ch_1) not enforced",
}, {
input: "alter table a drop check ch_1",
output: "alter table a",
input: "alter table a drop check ch_1",
output: "alter table a",
partialDDL: true,
}, {
input: "alter table a drop foreign key kx",
}, {
input: "alter table a drop primary key",
}, {
input: "alter table a drop constraint",
output: "alter table a",
input: "alter table a drop constraint",
output: "alter table a",
partialDDL: true,
}, {
input: "alter table a drop id",
output: "alter table a drop column id",
Expand Down Expand Up @@ -1107,13 +1114,16 @@ var (
input: "alter schema d collate = 'utf8_bin' character set = geostd8 character set = geostd8",
output: "alter database d collate 'utf8_bin' character set geostd8 character set geostd8",
}, {
input: "create table a",
input: "create table a",
partialDDL: true,
}, {
input: "CREATE TABLE a",
output: "create table a",
input: "CREATE TABLE a",
output: "create table a",
partialDDL: true,
}, {
input: "create table `a`",
output: "create table a",
input: "create table `a`",
output: "create table a",
partialDDL: true,
}, {
input: "create table function_default (x varchar(25) default (trim(' check ')))",
output: "create table function_default (\n\tx varchar(25) default (trim(' check '))\n)",
Expand Down Expand Up @@ -1146,11 +1156,13 @@ var (
input: "create table if not exists a (\n\t`a` int\n)",
output: "create table if not exists a (\n\ta int\n)",
}, {
input: "create table a ignore me this is garbage",
output: "create table a",
input: "create table a ignore me this is garbage",
output: "create table a",
partialDDL: true,
}, {
input: "create table a (a int, b char, c garbage)",
output: "create table a",
input: "create table a (a int, b char, c garbage)",
output: "create table a",
partialDDL: true,
}, {
input: "create table a (b1 bool not null primary key, b2 boolean not null)",
output: "create table a (\n\tb1 bool not null primary key,\n\tb2 boolean not null\n)",
Expand Down Expand Up @@ -1818,6 +1830,10 @@ var (
input: "create database test_db default collate 'utf8mb4_general_ci' collate utf8mb4_general_ci",
}, {
input: "create database test_db character set geostd8",
}, {
input: "alter table corder zzzz zzzz zzzz",
output: "alter table corder",
partialDDL: true,
}, {
input: "create database test_db character set * unparsable",
output: "create database test_db",
Expand Down Expand Up @@ -1956,9 +1972,9 @@ func TestValid(t *testing.T) {
// For mysql 8.0 syntax, the query is not entirely parsed.
// Add more structs as we go on adding full parsing support for DDL constructs for 5.7 syntax.
switch x := tree.(type) {
case *CreateDatabase:
case DBDDLStatement:
assert.Equal(t, !tcase.partialDDL, x.IsFullyParsed())
case *AlterDatabase:
case DDLStatement:
assert.Equal(t, !tcase.partialDDL, x.IsFullyParsed())
}
// This test just exercises the tree walking functionality.
Expand Down
6 changes: 6 additions & 0 deletions go/vt/sqlparser/parser.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,12 @@ func Parse2(sql string) (Statement, BindVars, error) {
return nil, nil, fmt.Errorf("extra characters encountered after end of DDL: '%s'", string(val))
}
log.Warningf("ignoring error parsing DDL '%s': %v", sql, tokenizer.LastError)
switch x := tokenizer.partialDDL.(type) {
case DBDDLStatement:
x.SetFullyParsed(false)
case DDLStatement:
x.SetFullyParsed(false)
}
tokenizer.ParseTree = tokenizer.partialDDL
return tokenizer.ParseTree, tokenizer.BindVars, nil
}
Expand Down
4 changes: 2 additions & 2 deletions go/vt/sqlparser/sql.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions go/vt/sqlparser/sql.y
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2331,9 +2331,9 @@ drop_statement:
{
// Change this to an alter statement
if $3.Lowered() == "primary" {
$$ = &AlterTable{Table: $5,AlterOptions: append([]AlterOption{&DropKey{Type:PrimaryKeyType}},$6...)}
$$ = &AlterTable{FullyParsed: true, Table: $5,AlterOptions: append([]AlterOption{&DropKey{Type:PrimaryKeyType}},$6...)}
} else {
$$ = &AlterTable{Table: $5,AlterOptions: append([]AlterOption{&DropKey{Type:NormalKeyType, Name:$3}},$6...)}
$$ = &AlterTable{FullyParsed: true, Table: $5,AlterOptions: append([]AlterOption{&DropKey{Type:NormalKeyType, Name:$3}},$6...)}
}
}
| DROP VIEW exists_opt view_name_list restrict_or_cascade_opt
Expand Down
7 changes: 7 additions & 0 deletions test/templates/cluster_endtoend_test.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,15 @@ jobs:

- name: Get dependencies
run: |
{{- if .Ubuntu20}}
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
{{- end}}
sudo apt-get update
{{- if .Ubuntu20}}
sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata
{{- else }}
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
{{- end}}
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
Expand Down
1 change: 1 addition & 0 deletions test/templates/unit_test.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ jobs:
{{if (eq .Platform "mysql80")}}

# mysql80
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.14-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
Expand Down