Skip to content

Commit 23cc1d4

Browse files
author
LeopPro
authored
DDL Checker Tool (pingcap#110)
add ddl checker tool
1 parent d57120b commit 23cc1d4

File tree

9 files changed

+610
-4
lines changed

9 files changed

+610
-4
lines changed

Makefile

+6-3
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: build importer checker dump_region binlogctl sync_diff_inspector test check deps
1+
.PHONY: build importer checker dump_region binlogctl sync_diff_inspector ddl_checker test check deps
22

33
# Ensure GOPATH is set before running build process.
44
ifeq "$(GOPATH)" ""
@@ -23,7 +23,7 @@ FILES := $$(find . -name '*.go' -type f | grep -vE 'vendor')
2323
VENDOR_TIDB := vendor/github.com/pingcap/tidb
2424

2525

26-
build: prepare check test importer checker dump_region binlogctl sync_diff_inspector finish
26+
build: prepare check test importer checker dump_region binlogctl sync_diff_inspector ddl_checker finish
2727

2828
prepare:
2929
cp go.mod1 go.mod
@@ -44,6 +44,9 @@ binlogctl:
4444
sync_diff_inspector:
4545
$(GO) build -ldflags '$(LDFLAGS)' -o bin/sync_diff_inspector ./sync_diff_inspector
4646

47+
ddl_checker:
48+
$(GO) build -ldflags '$(LDFLAGS)' -o bin/ddl_checker ./ddl_checker
49+
4750
test:
4851
@export log_level=error; \
4952
$(GOTEST) -cover $(PACKAGES)
@@ -65,4 +68,4 @@ check:
6568

6669
finish:
6770
cp go.mod go.mod1
68-
cp go.sum go.sum1
71+
cp go.sum go.sum1

README.md

+5
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ make checker # build checker
1414
make sync_diff_inspector # build sync_diff_inspector
1515
1616
make binlogctl # build binlogctl
17+
18+
make ddl_checker # build ddl_checker
1719
```
1820

1921
When tidb-tools are built successfully, you can find the binary in the `bin` directory.
@@ -36,6 +38,9 @@ When tidb-tools are built successfully, you can find the binary in the `bin` dir
3638

3739
A tool for performing some tidb-binlog related operations, like querying the status of Pump/Drainer and pause/offline some Pump/Drainer.
3840

41+
- [ddl_checker](./ddl_checker)
42+
43+
A tool for checking if DDL SQL can be successfully executed by TiDB.
3944

4045
## License
4146

ddl_checker/README.md

+58
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## DDLChecker
2+
3+
DDLChecker is a tool for checking if DDL SQL can be successfully executed by TiDB.
4+
5+
## How to use
6+
7+
```
8+
Usage of ./ddl_checker:
9+
-host string
10+
MySQL host (default "127.0.0.1")
11+
-password string
12+
Password
13+
-port int
14+
MySQL port (default 3306)
15+
-schema string
16+
Schema
17+
-user string
18+
User name (default "root")
19+
20+
21+
cd ddl_checker
22+
go build
23+
./ddl_checker --host [host] --port [port] --user [user] --password [password] --schema [schema]
24+
```
25+
26+
## Modes
27+
28+
You can switch modes using the `SETMOD` command.
29+
Auto mode: The program will automatically synchronize the dependent table structure from MySQL and delete the conflict table
30+
Prompt mode: The program will ask you before synchronizing the dependent table structure from MYSQL
31+
Offline mode: This program doesn't need to connect to MySQL, and doesn't perform anything other than executing the input SQL.
32+
33+
SETMOD usage: SETMOD <MODCODE>; MODCODE = ["Auto", "Prompt", "Offline"] (case insensitive).
34+
35+
## Example
36+
37+
```
38+
# ./bin/checker --host 127.0.0.1 --port 3306 --user root --password 123 --schema test
39+
40+
[Auto] > ALTER TABLE table_name MODIFY column_1 int(11) NOT NULL;
41+
[DDLChecker] Syncing Table table_name
42+
[DDLChecker] SQL execution succeeded
43+
44+
[Auto] > SETMOD PROMPT;
45+
[Query] > ALTER TABLE table_name MODIFY column_1 int(11) NOT NULL;
46+
[DDLChecker] Do you want to synchronize table [table_name] from MySQL and drop table [] in ExecutableChecker?(Y/N)y
47+
[DDLChecker] Syncing Table table_name
48+
[DDLChecker] SQL execution succeeded
49+
50+
[Query] > setmod manual;
51+
[Manual] > ALTER TABLE table_name MODIFY column_1 int(11) NOT NULL;
52+
[DDLChecker] SQL execution failed: [schema:1146]Table 'test.table_name' doesn't exist
53+
54+
```
55+
56+
## License
57+
Apache 2.0 license. See the [LICENSE](../LICENSE) file for details.
58+

ddl_checker/main.go

+212
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
// Copyright 2018 PingCAP, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// See the License for the specific language governing permissions and
12+
// limitations under the License.
13+
14+
package main
15+
16+
import (
17+
"bufio"
18+
"context"
19+
"flag"
20+
"fmt"
21+
_ "github.com/go-sql-driver/mysql"
22+
"github.com/pingcap/errors"
23+
"github.com/pingcap/tidb-tools/pkg/dbutil"
24+
"github.com/pingcap/tidb-tools/pkg/ddl-checker"
25+
"os"
26+
"strings"
27+
"unicode"
28+
)
29+
30+
var (
31+
mode = auto
32+
executableChecker *checker.ExecutableChecker
33+
ddlSyncer *checker.DDLSyncer
34+
reader *bufio.Reader
35+
tidbContext = context.Background()
36+
37+
host = flag.String("host", "127.0.0.1", "MySQL host")
38+
port = flag.Int("port", 3306, "MySQL port")
39+
username = flag.String("user", "root", "User name")
40+
password = flag.String("password", "", "Password")
41+
schema = flag.String("schema", "", "Schema")
42+
)
43+
44+
const (
45+
welcomeInfo = "ExecutableChecker: Check if SQL can be successfully executed by TiDB\n" +
46+
"Copyright 2018 PingCAP, Inc.\n\n" +
47+
"You can switch modes using the `SETMOD` command.\n" +
48+
"Auto mode: The program will automatically synchronize the dependent table structure from MySQL " +
49+
"and delete the conflict table\n" +
50+
"Prompt mode: The program will ask you before synchronizing the dependent table structure from MYSQL\n" +
51+
"Offline mode: This program doesn't need to connect to MySQL, and doesn't perform anything other than executing the input SQL.\n\n" +
52+
setmodUsage + "\n"
53+
54+
setmodUsage = "SETMOD usage: SETMOD <MODCODE>; MODCODE = [\"Auto\", \"Prompt\", \"Offline\"] (case insensitive).\n"
55+
56+
auto = "auto"
57+
prompt = "prompt"
58+
offline = "offline"
59+
)
60+
61+
func main() {
62+
fmt.Print(welcomeInfo)
63+
initialise()
64+
mainLoop()
65+
destroy()
66+
}
67+
68+
func initialise() {
69+
flag.Parse()
70+
var err error
71+
reader = bufio.NewReader(os.Stdin)
72+
executableChecker, err = checker.NewExecutableChecker()
73+
if err != nil {
74+
fmt.Printf("[DDLChecker] Init failed, can't create ExecutableChecker: %s\n", err.Error())
75+
os.Exit(1)
76+
}
77+
executableChecker.Execute(tidbContext, "use test;")
78+
dbInfo := &dbutil.DBConfig{
79+
User: *username,
80+
Password: *password,
81+
Host: *host,
82+
Port: *port,
83+
Schema: *schema,
84+
}
85+
ddlSyncer, err = checker.NewDDLSyncer(dbInfo, executableChecker)
86+
if err != nil {
87+
fmt.Printf("[DDLChecker] Init failed, can't open mysql database: %s\n", err.Error())
88+
os.Exit(1)
89+
}
90+
}
91+
92+
func destroy() {
93+
executableChecker.Close()
94+
ddlSyncer.Close()
95+
}
96+
97+
func mainLoop() {
98+
var input string
99+
var err error
100+
for isContinue := true; isContinue; isContinue = handler(input) {
101+
fmt.Printf("[%s] > ", strings.ToTitle(mode))
102+
input, err = reader.ReadString(';')
103+
if err != nil {
104+
fmt.Printf("[DDLChecker] Read stdin error: %s\n", err.Error())
105+
os.Exit(1)
106+
}
107+
}
108+
}
109+
func handler(input string) bool {
110+
lowerTrimInput := strings.ToLower(strings.TrimFunc(input, func(r rune) bool {
111+
return unicode.IsSpace(r) || r == ';'
112+
}))
113+
// cmd exit
114+
if lowerTrimInput == "exit" {
115+
return false
116+
}
117+
// cmd setmod
118+
if strings.HasPrefix(lowerTrimInput, "setmod") {
119+
x := strings.TrimSpace(lowerTrimInput[6:])
120+
switch x {
121+
case auto, prompt, offline:
122+
mode = x
123+
default:
124+
fmt.Print(setmodUsage)
125+
}
126+
return true
127+
}
128+
stmt, err := executableChecker.Parse(input)
129+
if err != nil {
130+
fmt.Println("[DDLChecker] SQL parse error: ", err.Error())
131+
return true
132+
}
133+
if !checker.IsDDL(stmt) {
134+
fmt.Println("[DDLChecker] Warning: The input SQL isn't a DDL")
135+
}
136+
if mode != offline {
137+
// auto and query mod
138+
neededTables, _ := checker.GetTablesNeededExist(stmt)
139+
nonNeededTables, err := checker.GetTablesNeededNonExist(stmt)
140+
// skip when stmt isn't a DDLNode
141+
if err == nil && (mode == auto || (mode == prompt && promptAutoSync(neededTables, nonNeededTables))) {
142+
err := syncTablesFromMysql(neededTables)
143+
if err != nil {
144+
return true
145+
}
146+
err = dropTables(nonNeededTables)
147+
if err != nil {
148+
return true
149+
}
150+
}
151+
}
152+
err = executableChecker.Execute(tidbContext, input)
153+
if err == nil {
154+
fmt.Println("[DDLChecker] SQL execution succeeded")
155+
} else {
156+
fmt.Println("[DDLChecker] SQL execution failed:", err.Error())
157+
}
158+
return true
159+
}
160+
161+
func syncTablesFromMysql(tableNames []string) error {
162+
for _, tableName := range tableNames {
163+
fmt.Println("[DDLChecker] Syncing Table", tableName)
164+
err := ddlSyncer.SyncTable(tidbContext, *schema, tableName)
165+
if err != nil {
166+
fmt.Println("[DDLChecker] Sync table failure:", err.Error())
167+
return errors.Trace(err)
168+
}
169+
}
170+
return nil
171+
}
172+
173+
func dropTables(tableNames []string) error {
174+
for _, tableName := range tableNames {
175+
fmt.Println("[DDLChecker] Dropping table", tableName)
176+
err := executableChecker.DropTable(tidbContext, tableName)
177+
if err != nil {
178+
fmt.Println("[DDLChecker] Drop table", tableName, "Error:", err.Error())
179+
return errors.Trace(err)
180+
}
181+
}
182+
return nil
183+
}
184+
185+
func promptAutoSync(neededTable []string, nonNeededTable []string) bool {
186+
return promptYorN("[DDLChecker] Do you want to synchronize table %v from MySQL "+
187+
"and drop table %v in DDLChecker?(Y/N)", neededTable, nonNeededTable)
188+
}
189+
190+
func promptYorN(format string, a ...interface{}) bool {
191+
for {
192+
fmt.Printf(format, a...)
193+
innerLoop:
194+
for {
195+
result, err := reader.ReadString('\n')
196+
if err != nil {
197+
fmt.Printf("[DDLChecker] Read stdin error: %s\n", err.Error())
198+
return false
199+
}
200+
switch strings.ToLower(strings.TrimSpace(result)) {
201+
case "y", "yes":
202+
return true
203+
case "n", "no":
204+
return false
205+
case "":
206+
continue innerLoop
207+
default:
208+
break innerLoop
209+
}
210+
}
211+
}
212+
}

go.mod1

+5-1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ require (
1212
github.com/coreos/go-systemd v0.0.0-20181031085051-9002847aa142 // indirect
1313
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
1414
github.com/cznic/mathutil v0.0.0-20181021201202-eba54fb065b7 // indirect
15+
github.com/cznic/sortutil v0.0.0-20150617083342-4c7342852e65 // indirect
1516
github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect
1617
github.com/dustin/go-humanize v1.0.0 // indirect
1718
github.com/eapache/go-resiliency v1.1.0 // indirect
@@ -39,6 +40,9 @@ require (
3940
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
4041
github.com/montanaflynn/stats v0.0.0-20180911141734-db72e6cae808 // indirect
4142
github.com/ngaut/log v0.0.0-20180314031856-b8e36e7ba5ac
43+
github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 // indirect
44+
github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef // indirect
45+
github.com/opentracing/basictracer-go v1.0.0 // indirect
4246
github.com/opentracing/opentracing-go v1.0.2 // indirect
4347
github.com/pierrec/lz4 v2.0.5+incompatible // indirect
4448
github.com/pingcap/check v0.0.0-20171206051426-1c287c953996
@@ -67,7 +71,7 @@ require (
6771
github.com/ugorji/go v1.1.1 // indirect
6872
github.com/unrolled/render v0.0.0-20180914162206-b9786414de4d // indirect
6973
github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18 // indirect
70-
go.uber.org/atomic v1.3.2 // indirect
74+
go.uber.org/atomic v1.3.2
7175
go.uber.org/multierr v1.1.0 // indirect
7276
go.uber.org/zap v1.8.0 // indirect
7377
golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16 // indirect

go.sum1

+8
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f h1:lBNOc5arjvs8E5mO2tbp
2121
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
2222
github.com/cznic/mathutil v0.0.0-20181021201202-eba54fb065b7 h1:y+DH9ARrWiiNBV+6waYP2IPcsRbxdU1qsnycPfShF4c=
2323
github.com/cznic/mathutil v0.0.0-20181021201202-eba54fb065b7/go.mod h1:e6NPNENfs9mPDVNRekM7lKScauxd5kXTr1Mfyig6TDM=
24+
github.com/cznic/sortutil v0.0.0-20150617083342-4c7342852e65 h1:hxuZop6tSoOi0sxFzoGGYdRqNrPubyaIf9KoBG9tPiE=
25+
github.com/cznic/sortutil v0.0.0-20150617083342-4c7342852e65/go.mod h1:q2w6Bg5jeox1B+QkJ6Wp/+Vn0G/bo3f1uY7Fn3vivIQ=
2426
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
2527
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
2628
github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM=
@@ -77,6 +79,12 @@ github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5
7779
github.com/montanaflynn/stats v0.0.0-20180911141734-db72e6cae808/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
7880
github.com/ngaut/log v0.0.0-20180314031856-b8e36e7ba5ac h1:wyheT2lPXRQqYPWY2IVW5BTLrbqCsnhL61zK2R5goLA=
7981
github.com/ngaut/log v0.0.0-20180314031856-b8e36e7ba5ac/go.mod h1:ueVCjKQllPmX7uEvCYnZD5b8qjidGf1TCH61arVe4SU=
82+
github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 h1:7KAv7KMGTTqSmYZtNdcNTgsos+vFzULLwyElndwn+5c=
83+
github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7/go.mod h1:iWMfgwqYW+e8n5lC/jjNEhwcjbRDpl5NT7n2h+4UNcI=
84+
github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef h1:K0Fn+DoFqNqktdZtdV3bPQ/0cuYh2H4rkg0tytX/07k=
85+
github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef/go.mod h1:7WjlapSfwQyo6LNmIvEWzsW1hbBQfpUO4JWnuQRmva8=
86+
github.com/opentracing/basictracer-go v1.0.0 h1:YyUAhaEfjoWXclZVJ9sGoNct7j4TVk7lZWlQw5UXuoo=
87+
github.com/opentracing/basictracer-go v1.0.0/go.mod h1:QfBfYuafItcjQuMwinw9GhYKwFXS9KnPs5lxoYwgW74=
8088
github.com/opentracing/opentracing-go v1.0.2 h1:3jA2P6O1F9UOrWVpwrIo17pu01KWvNWg4X946/Y5Zwg=
8189
github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
8290
github.com/pierrec/lz4 v2.0.5+incompatible h1:2xWsjqPFWcplujydGg4WmhC/6fZqK42wMM8aXeqhl0I=

0 commit comments

Comments
 (0)