Skip to content

pass in correct filesize in ceph backup#5369

Merged
sougou merged 2 commits intovitessio:masterfrom
planetscale:ds-ceph-oom-fix
Nov 23, 2019
Merged

pass in correct filesize in ceph backup#5369
sougou merged 2 commits intovitessio:masterfrom
planetscale:ds-ceph-oom-fix

Conversation

@deepthi
Copy link
Copy Markdown
Collaborator

@deepthi deepthi commented Oct 28, 2019

This is a fix for #4804
Any ceph users out there who can review?

Signed-off-by: deepthi deepthi@planetscale.com

@deepthi deepthi requested a review from sougou as a code owner October 28, 2019 18:19
@harshit-gangal
Copy link
Copy Markdown
Member

Just saw it is assigned to me. I will look into it.

Copy link
Copy Markdown
Member

@harshit-gangal harshit-gangal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change ceph backup client will know the size of the object taken for backup upload.

There is a possibility that backup can fail based on Code Path, as maxMultipartPutObjectSize is 5TiB though unlikely as it is very high to be stored on a single host though possible.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this check is not required. -1 was given when file size was not known i.e. not passed to this method. But, now the file size is known this will work irrespective if file size is actually 0 Bytes.

Copy link
Copy Markdown
Collaborator Author

@deepthi deepthi Nov 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The calling code passes 0 when the file size is actually unknown. This happens when we write the manifest file. We call AddFile to get a WriteCloser and we actually write to the file using that.
I can add this as a comment so that it is clearer to anyone reading the code.
The other option is to have the calling code pass -1 when file size is unknown. It doesn't look like that will break any of the other plugins. That will cleanly separate actual 0 size files from files where the size is really unknown.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the other option of passing -1 for unknown file size so no special check is required in this code block and would be clean.

@deepthi
Copy link
Copy Markdown
Collaborator Author

deepthi commented Nov 6, 2019

With this change ceph backup client will know the size of the object taken for backup upload.

There is a possibility that backup can fail based on Code Path, as maxMultipartPutObjectSize is 5TiB though unlikely as it is very high to be stored on a single host though possible.

Yeah, we won't worry about this.

Signed-off-by: deepthi <deepthi@planetscale.com>
Signed-off-by: deepthi <deepthi@planetscale.com>
@sougou sougou merged commit 69309e6 into vitessio:master Nov 23, 2019
@deepthi deepthi deleted the ds-ceph-oom-fix branch May 14, 2020 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants