Prospector and Harvester Cleanups #2020
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ruflin
added
in progress
ruflin
force-pushed
the
filebeat-todo-cleanup
branch
from
99dd8cc
to
921504c
Compare
ruflin
changed the title
ruflin
added
review
and removed
in progress
| @@ -148,11 +160,12 @@ func (h *Harvester) shouldExportLine(line string) bool { | |||
| // the file system is scanned | |||
| func (h *Harvester) openFile() error { | |||
|
|
|||
| f, err := file.ReadOpen(h.path) | |||
| f, err := file.ReadOpen(h.state.Source) | |||
There was a problem hiding this comment.
After doing file.ReadOpen a number of checks that might return an error are executed. Question: in case of openFile returning an error, who is closing the file handle?
There was a problem hiding this comment.
You are right :-( This could be a reason we leak some file descriptors. We must backport this fix to 1.3 @tsg FYI
This was referenced Jul 13, 2016
ruflin
force-pushed
the
filebeat-todo-cleanup
branch
2 times, most recently
from
f94045f
to
6ff1beb
Compare
| "harvest: %q position:%d (offset snapshot:%d)", h.path, h.getOffset(), offset) | ||
| _, err = file.Seek(h.getOffset(), os.SEEK_SET) | ||
| logp.Debug("harvester", "harvest: %q position:%d (offset snapshot:%d)", h.state.Source, h.state.Offset, offset) | ||
| _, err = file.Seek(h.state.Offset, os.SEEK_SET) |
There was a problem hiding this comment.
Oh, glad you spotted this as otherwise it gets overwritten below :-(
ruflin
force-pushed
the
filebeat-todo-cleanup
branch
from
6ff1beb
to
72f41b5
Compare
| @@ -186,52 +216,38 @@ func (h *Harvester) openFile() error { | |||
| return err | |||
| } | |||
|
|
|||
| // yay, open file | |||
| h.file = source.File{f} | |||
| return nil | |||
| } | |||
|
|
|||
| func (h *Harvester) initFileOffset(file *os.File) error { | |||
There was a problem hiding this comment.
@urso After your input I decided to rewrite / cleanup this part here. I think it is identical to before, but now max 1 Seek instead of 2 are needed and it seems easier to read. Can you have a close look if it looks sane for you?
|
@ruflin needs rebase |
ruflin
force-pushed
the
filebeat-todo-cleanup
branch
2 times, most recently
from
df9e833
to
37e9b4f
Compare
| return nil | ||
| } | ||
|
|
||
| func (h *Harvester) initFileOffset(file *os.File) error { | ||
| offset, err := file.Seek(0, os.SEEK_CUR) | ||
| func (h *Harvester) getFileOffset(file *os.File) (int64, error) { |
There was a problem hiding this comment.
I think initFileOffset still matches this methods functionality better. getFileOffset sounds like re-usable getter.
See elastic#1913 * Make sure ignore_older is not only applied to new files but also existing files * Improve logging in prospector * Add expvar to prospector and harvester * Cleanup offset handling * Add truncation handling to prospector * Handle all offsets as part of the state * Remove path from harvester and use state.Source instead * Remove obsolete exclude / include options * Remove unnecessary getOffset() * Fix potential file handler leak. Close file handler in case of errors. * Implement cleaner file handler closing * Cleanup initOffset handling * Add build-image task to generate json in metricbeat as it was missing * Only send state if file was properly initialised * Only set offset if seek succeeds
ruflin
force-pushed
the
filebeat-todo-cleanup
branch
from
37e9b4f
to
c428d42
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See elastic#1913