No close_write event in watcher - need to move a large file but only if it's closed

[tomek-cz]

I have a scenario, where I want to move a large file (4GB) to an S3 bucket once it's written to a watched directory on the local disk. The process of writing a file to the local directory is time-consuming.

I am waiting for the Create event and then I run 'move to s3 bucket' process, which will, unfortunately, only upload a portion of the file (as the rest of the file is still being written to the disk).

I have tested using Create and Write event, but those two trigger before a file is completed.

Look like we'd need close_write event to be implemented into the watcher, so it works similar to :
inotifywait -e close_write /path/to/file
https://unix.stackexchange.com/questions/176387/move-file-but-only-if-its-closed

[gaby]

I'm running into the same issue!

[tomek-cz]

I've worked this around by:

• watch for CREATE event
• check the mtime of the file in a loop, compare with now()
  (mtime changes whenever the file is written to)
• if the difference is bigger than 30 seconds then it means the file is not being written to anymore so I can initiate the move.

Works well so far, can move large files with no issues.

[gaby]

Worst case, that file won't get process until 30 seconds after you finish writing? I wonder if there's a quicker approach to this.

[Akumzy]

I solved mine like this

func isFileOpen(filePath string) (state bool) {
	state = false
	prevSize := int64(0)
	count := 0
	for {
		stat, err := os.Stat(filePath)
		if err != nil {
			break
		}
		if prevSize == stat.Size() {
			state = false
		} else {
			prevSize = stat.Size()
			state = true
		}
		if count == 2 {
			break
		}
		count++
		time.Sleep(time.Millisecond * 200)
	}
	return state
}

func changeHandler(f fileInfo, op watcher.Op) {
	//time.Sleep(time.Second * 2)
	if (op == watcher.Create || op == watcher.Write) && !f.IsDir && isFileOpen(f.Path) {
		return
	}
	handleFile(f, op)
}