Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open all vlog files in RDWR mode #923

Merged
merged 7 commits into from
Jul 16, 2019
Merged

Conversation

jarifibrahim
Copy link
Contributor

@jarifibrahim jarifibrahim commented Jul 8, 2019

With this commit, all log files are opened in read-write mode. Earlier,
we would open files in read-write mode for writing and then reopen them
again in read only mode after writing was completed. This approach lead
to unnecessary complexity and issues (see issue #769).


This change is Reviewable

With this commit, all log files are opened in read-write mode. Earlier,
we would open files in read-write mode for writing and then reopen them
again in read only mode after writing was completed. This approach lead
to unnecessary complexity and issues (see
issue #769).
Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 files reviewed, 2 unresolved discussions (waiting on @ashish-goswami, @jarifibrahim, and @manishrjain)


value.go, line 135 at r1 (raw file):

	// Acquire lock because fd will become invalid for a bit.
	// Acquiring the lock is bad because, while we don't hold the lock for a long time, it forces
	// one batch of readers wait for the preceding batch of readers to finish.

Can you rephrase this comment a bit? I find it confusing. First, it says the lock will be acquired but then it says that's bad. Is the reason for it that the fd will become invalid? If so, the comment should say first that acquiring the lock is bad, then explain the reason why we are doing it anyways.


value.go, line 800 at r1 (raw file):

	return nil
}
func (vlog *valueLog) openLogFile(lf *logFile) error {

minor: add a blank line above this line

Copy link
Contributor Author

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion (waiting on @ashish-goswami, @manishrjain, and @martinmr)


value.go, line 135 at r1 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…
	// Acquire lock because fd will become invalid for a bit.
	// Acquiring the lock is bad because, while we don't hold the lock for a long time, it forces
	// one batch of readers wait for the preceding batch of readers to finish.

Can you rephrase this comment a bit? I find it confusing. First, it says the lock will be acquired but then it says that's bad. Is the reason for it that the fd will become invalid? If so, the comment should say first that acquiring the lock is bad, then explain the reason why we are doing it anyways.

I think the comment was added a while ago and since we're no longer opening and closing the file we don't need the lock. I've removed the lock and the comment.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: Got a comment.

Reviewed 1 of 1 files at r2.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @jarifibrahim and @martinmr)


value.go, line 138 at r2 (raw file):

	if err := lf.fd.Truncate(int64(offset)); err != nil {
		return errors.Wrapf(err, "Unable to truncate file: %q", lf.path)
	}

Add a comment here saying that we previously used to reopen files in read only mode, but no longer do so. We keep the log files in RW mode.


value.go, line 740 at r2 (raw file):

		// Open log file "lf" in read-write mode
		if err := vlog.openLogFile(lf); err != nil {

Could be lf.open(vlog.opt)?


value.go, line 796 at r2 (raw file):

}

func (vlog *valueLog) openLogFile(lf *logFile) error {

This method can be on the logfile struct, like the openReadOnly was.


value.go, line 803 at r2 (raw file):

		flags |= y.ReadOnly
		// Set sync flag only for the last vlog file.
	case vlog.opt.SyncWrites && lf.fid == vlog.maxFid:

Now that you're not re-opening the files in read only mode, then all files (even the penultimate log files) can be opened with y.Sync flag, if SyncWrites is true, right?

Copy link
Contributor Author

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 files reviewed, 5 unresolved discussions (waiting on @manishrjain and @martinmr)


value.go, line 138 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Add a comment here saying that we previously used to reopen files in read only mode, but no longer do so. We keep the log files in RW mode.

Added the comment.


value.go, line 740 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Could be lf.open(vlog.opt)?

I'll need to file path also. Changed the function signature to lf.Open(path, flags)


value.go, line 796 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

This method can be on the logfile struct, like the openReadOnly was.

Done.


value.go, line 803 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Now that you're not re-opening the files in read only mode, then all files (even the penultimate log files) can be opened with y.Sync flag, if SyncWrites is true, right?

We won't be writing to any other file except the last one so I thought it we shouldn't unnecessarily open files with sync flag.

Do you think we should open all files in sync mode? Even when we won't be writing to them?

Copy link
Contributor Author

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 files reviewed, 5 unresolved discussions (waiting on @manishrjain and @martinmr)


value.go, line 803 at r2 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

We won't be writing to any other file except the last one so I thought it we shouldn't unnecessarily open files with sync flag.

Do you think we should open all files in sync mode? Even when we won't be writing to them?

Changed the code. Now we open all files with sync mode.

@jarifibrahim jarifibrahim dismissed martinmr’s stale review July 16, 2019 15:56

Addressed all the comments.

@jarifibrahim jarifibrahim merged commit afa8fae into master Jul 16, 2019
@jarifibrahim jarifibrahim deleted the ibrahim/vlog-readonly branch July 16, 2019 15:57
jarifibrahim pushed a commit that referenced this pull request Mar 12, 2020
* Open all vlog files in RDWR mode

With this commit, all log files are opened in read-write mode. Earlier,
we would open files in read-write mode for writing and then reopen them
again in read only mode after writing was completed. This approach lead
to unnecessary complexity and issues (see
issue #769).

(cherry picked from commit afa8fae)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants