Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streams and BLOB escapes for large databases #14

Open
wants to merge 73 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
57f1b81
Fork and get a direct stream to modify
BrandonRoehl Jan 23, 2019
c8118b2
Spring cleaning
BrandonRoehl Jan 23, 2019
cd7bc12
Stream based writer
BrandonRoehl Jan 23, 2019
846720a
Update to MySQL 8
BrandonRoehl Jan 23, 2019
6112651
readme update
BrandonRoehl Jan 23, 2019
47914c6
Go concurrency and backticks
BrandonRoehl Jan 24, 2019
011beef
Breaking things
BrandonRoehl Jan 24, 2019
9a9da92
No need to panic
BrandonRoehl Jan 24, 2019
3542d95
0.3.0 for tag
BrandonRoehl Jan 24, 2019
523f634
update readme
BrandonRoehl Jan 24, 2019
90fdf91
Sanitize and blob values
BrandonRoehl Jan 24, 2019
46db301
Version bump for the blob sanitization
BrandonRoehl Jan 24, 2019
30b13cd
Correct escaping
BrandonRoehl Jan 24, 2019
f253cf7
travis to newer go
BrandonRoehl Jan 24, 2019
c9037df
Failing tests on mock
BrandonRoehl Jan 25, 2019
fd3ebcf
Ability to ignore tables
BrandonRoehl Jan 25, 2019
79fd69d
Update docs
BrandonRoehl Jan 25, 2019
36aed1d
Remove vscode artifacts
BrandonRoehl Jan 25, 2019
219da40
0.3.3 release for ignoring tables
BrandonRoehl Jan 25, 2019
53d026e
Update docs
BrandonRoehl Jan 25, 2019
5671c80
Update urls
BrandonRoehl Jan 25, 2019
ac1de70
Missed an upcase
BrandonRoehl Jan 25, 2019
530393c
Collect error from goroutine
BrandonRoehl Jan 28, 2019
06c8b9c
Clean up comments
BrandonRoehl Jan 28, 2019
c87ced3
io.Closer will close any closable connection
BrandonRoehl Jan 28, 2019
fdc6a20
Merge branch 'master' of github.com:jamf/go-mysqldump
BrandonRoehl Jan 29, 2019
87c70ac
Data error to return out of execute
BrandonRoehl Jan 29, 2019
03b8ebd
Lock on err
BrandonRoehl Jan 30, 2019
d5d8630
Update readme
BrandonRoehl Jan 31, 2019
123b8ce
Sanitize query from createTable
BrandonRoehl Feb 1, 2019
ac1637a
Merge branch 'master' into jamf-james
BrandonRoehl Feb 1, 2019
d22e068
Test pass
BrandonRoehl Feb 1, 2019
127baa7
Merge branch 'master' into jamf-james
BrandonRoehl Feb 1, 2019
c41f2ea
File replace only works with the file
BrandonRoehl Feb 9, 2019
0937960
Switch statment when reading
BrandonRoehl Mar 18, 2019
9823f56
All in the templates
BrandonRoehl Mar 18, 2019
63aca4f
Fix test type
BrandonRoehl Mar 18, 2019
6cb5948
Version bump
BrandonRoehl Mar 18, 2019
f61ba94
Merge pull request #1 from BrandonRoehl/master
BrandonRoehl Mar 28, 2019
34666eb
WIP 1
BrandonRoehl Mar 28, 2019
c7cef00
Major refactor
BrandonRoehl Mar 29, 2019
bb5c4f9
Data streams
BrandonRoehl Mar 29, 2019
dabcd57
With data streams
BrandonRoehl Mar 29, 2019
db86117
Bump the version
BrandonRoehl Mar 29, 2019
298b70a
Merge pull request #2 from BrandonRoehl/master
BrandonRoehl Apr 1, 2019
70aca04
inset spliting
BrandonRoehl Apr 4, 2019
1cff840
The MySQL way
BrandonRoehl Apr 4, 2019
4a93eb2
Remove debug
BrandonRoehl Apr 4, 2019
de45383
Fix the test
BrandonRoehl Apr 4, 2019
e58e36d
Update export
BrandonRoehl Apr 4, 2019
d66f8c4
Merge pull request #3 from BrandonRoehl/master
BrandonRoehl Apr 5, 2019
c2fe4af
Update README.md
Greg0 Jun 26, 2019
3ee6198
Merge pull request #4 from Greg0/patch-1
BrandonRoehl Jun 27, 2019
53765f2
Add the ability to lock all tables
BrandonRoehl Sep 24, 2019
19bd136
Merge branch 'master' into lock-tables-first
BrandonRoehl Sep 24, 2019
89292d2
Bump the version number
BrandonRoehl Sep 24, 2019
f00cfb1
Starting to transaction wrap
BrandonRoehl Sep 25, 2019
fdb7b50
Have all statments run on the same transaction
BrandonRoehl Sep 25, 2019
4517c0a
There is no locking by default just a read transaction
BrandonRoehl Sep 25, 2019
adf372a
Change the version down
BrandonRoehl Sep 25, 2019
ca1be9d
Convert to go mod
BrandonRoehl Sep 26, 2019
55278b1
Update comments and pointers
BrandonRoehl Sep 26, 2019
cfdd697
Fix the comment
BrandonRoehl Sep 30, 2019
cd198d0
Merge pull request #5 from BrandonRoehl/lock-tables-first
BrandonRoehl Oct 1, 2019
ecd436b
Merge branch 'master' into single-transaction
BrandonRoehl Oct 1, 2019
d725434
Consolidate the for loops
BrandonRoehl Oct 1, 2019
5c7d255
Merge branch 'single-transaction' of github.com:BrandonRoehl/go-mysql…
BrandonRoehl Oct 1, 2019
f25e26f
Merge pull request #6 from BrandonRoehl/single-transaction
BrandonRoehl Oct 1, 2019
b6b4ed7
Don't need to store the reflection type
BrandonRoehl Oct 8, 2019
f5bcf36
Don't need to store the reflection type
BrandonRoehl Oct 8, 2019
4f247b0
Don't need DEP
BrandonRoehl Oct 9, 2019
38ef509
Merge pull request #7 from BrandonRoehl/dont-need-to-store-the-reflec…
BrandonRoehl Oct 9, 2019
020d6e4
Merge pull request #1 from jamf/master
BrandonRoehl Jan 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
language: go

go:
- 1.6
- 1.7.x
- 1.13.x
- master

script:
Expand Down
15 changes: 0 additions & 15 deletions Gopkg.lock

This file was deleted.

26 changes: 0 additions & 26 deletions Gopkg.toml

This file was deleted.

72 changes: 36 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,49 @@ Create MYSQL dumps in Go without the `mysqldump` CLI as a dependancy.
package main

import (
"database/sql"
"fmt"
"database/sql"
"fmt"

"github.com/JamesStewy/go-mysqldump"
_ "github.com/go-sql-driver/mysql"
"github.com/JamesStewy/go-mysqldump"
"github.com/go-sql-driver/mysql"
)

func main() {
// Open connection to database
username := "your-user"
password := "your-pw"
hostname := "your-hostname"
port := "your-port"
dbname := "your-db"
// Open connection to database
config := mysql.NewConfig()
config.User = "your-user"
config.Passwd = "your-pw"
config.DBName = "your-db"
config.Net = "tcp"
config.Addr = "your-hostname:your-port"

dumpDir := "dumps" // you should create this directory
dumpFilenameFormat := fmt.Sprintf("%s-20060102T150405", dbname) // accepts time layout string and add .sql at the end of file

db, err := sql.Open("mysql", fmt.Sprintf("%s:%s@tcp(%s:%s)/%s", username, password, hostname, port, dbname))
if err != nil {
fmt.Println("Error opening database: ", err)
return
}

// Register database with mysqldump
dumper, err := mysqldump.Register(db, dumpDir, dumpFilenameFormat)
if err != nil {
fmt.Println("Error registering databse:", err)
return
}

// Dump database to file
resultFilename, err := dumper.Dump()
if err != nil {
fmt.Println("Error dumping:", err)
return
}
fmt.Printf("File is saved to %s", resultFilename)

// Close dumper and connected database
dumper.Close()
dumpFilenameFormat := fmt.Sprintf("%s-20060102T150405", config.DBName) // accepts time layout string and add .sql at the end of file

db, err := sql.Open("mysql", config.FormatDSN())
if err != nil {
fmt.Println("Error opening database: ", err)
return
}

// Register database with mysqldump
dumper, err := mysqldump.Register(db, dumpDir, dumpFilenameFormat)
if err != nil {
fmt.Println("Error registering databse:", err)
return
}

// Dump database to file
err := dumper.Dump()
if err != nil {
fmt.Println("Error dumping:", err)
return
}
fmt.Printf("File is saved to %s", dumpFilenameFormat)

// Close dumper, connected database and file stream.
dumper.Close()
}

```

[![GoDoc](https://godoc.org/github.com/JamesStewy/go-mysqldump?status.svg)](https://godoc.org/github.com/JamesStewy/go-mysqldump)
Expand Down
46 changes: 27 additions & 19 deletions doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,44 +3,52 @@ Create MYSQL dumps in Go without the 'mysqldump' CLI as a dependancy.

Example

This example uses the mymysql driver (example 7 https://github.com/ziutek/mymysql) to connect to a mysql instance.
This example uses the mysql driver (https://github.com/go-sql-driver/mysql) to connect to a mysql instance.

package main

import (
"database/sql"
"fmt"
"github.com/JamesStewy/go-mysqldump"
"github.com/ziutek/mymysql/godrv"
"time"
"database/sql"
"fmt"

"github.com/JamesStewy/go-mysqldump"
"github.com/go-sql-driver/mysql"
)

func main() {
// Register the mymysql driver
godrv.Register("SET NAMES utf8")

// Open connection to database
db, err := sql.Open("mymysql", "tcp:host:port*database/user/password")
if err != nil {
fmt.Println("Error opening databse:", err)
config := mysql.NewConfig()
config.User = "your-user"
config.Passwd = "your-pw"
config.DBName = "your-db"
config.Net = "tcp"
config.Addr = "your-hostname:your-port"

dumpDir := "dumps" // you should create this directory
dumpFilenameFormat := fmt.Sprintf("%s-20060102T150405", dbname) // accepts time layout string and add .sql at the end of file

db, err := sql.Open("mysql", config.FormatDNS())
if err != nil {
fmt.Println("Error opening database: ", err)
return
}

// Register database with mysqldump
dumper, err := mysqldump.Register(db, "dumps", time.ANSIC)
dumper, err := mysqldump.Register(db, dumpDir, dumpFilenameFormat)
if err != nil {
fmt.Println("Error registering databse:", err)
return
fmt.Println("Error registering databse:", err)
return
}

// Dump database to file
err = dumper.Dump()
resultFilename, err := dumper.Dump()
if err != nil {
fmt.Println("Error dumping:", err)
return
fmt.Println("Error dumping:", err)
return
}
fmt.Printf("File is saved to %s", resultFilename)

// Close dumper and connected database
// Close dumper, connected database and file stream.
dumper.Close()
}
*/
Expand Down
Loading