Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to use go channels #2

Merged
merged 5 commits into from
Apr 1, 2019
Merged

Conversation

BrandonRoehl
Copy link
Member

@BrandonRoehl BrandonRoehl commented Mar 29, 2019

Switch from goroutines storing rows in memory to use a channel with a buffer of 1 to hold single rows so we can dump tables that far exceed your system memory. Now we have a goroutine blocking and then filling a buffer for the dump.

The buffer of 1 is what maintains the speed that this has had with the memory improvements. This way we can read from MySQL and write to the file in parallel and not block either. Making this a typical consumer producer problem for memory. The table rows are dynamically allocated so usage is usually fairly small.

NOTE: MySQL rows have a max of 1GB so we can theoretically use 3GB of heap. If the buffer is full and there is one waiting to get put into the buffer. The io.Writter is writing to a memory buffer for the previous in the channel.
I have gotten no where near this limit being only able to put rows of 128MB in MySQL before I got weird server unavailable errors and sometimes corrupted MySQL databases entirely.

Copy link

@g-r-a-n-t g-r-a-n-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice.

@BrandonRoehl BrandonRoehl merged commit 298b70a into jamf:master Apr 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants