Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync with --cache-file fails when file is removed (KeyError in FileLists.fetch_local_list()) #377

Closed
tail opened this issue Aug 26, 2014 · 0 comments

Comments

@tail
Copy link

tail commented Aug 26, 2014

This sounds similar to #194 and http://sourceforge.net/p/s3tools/mailman/message/32305453/. I looked into the issue a little bit, and I think this has to do with the fact that I'm trying to sync a directory that is periodically removing files, and there are enough large files in the directory that the initial MD5 calculations takes a couple hours. The fix in the aforementioned issues has been to wrap the calls with a try/except on KeyError. I could submit a patch doing the same here, but I'm not sure that doing so is a proper fix.

This was tested against HEAD as of this writing (revision 76eb785). Here is the exception:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   [email protected]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: ./s3cmd --cache-file=/path/to/cache-file -v sync /path/to/sync/ s3://path/to/backup/
Problem: KeyError: 'dev'
S3cmd:   1.5.0-rc1
python:   2.7.3 (default, Apr 10 2013, 06:20:15) 
[GCC 4.6.3]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "./s3cmd", line 2521, in <module>
    rc = main()
  File "./s3cmd", line 2439, in main
    rc = cmd_func(args)
  File "./s3cmd", line 1470, in cmd_sync
    return cmd_sync_local2remote(args)
  File "./s3cmd", line 1440, in cmd_sync_local2remote
    local_list, single_file_local, src_exclude_list = fetch_local_list(args[:-1], is_src = True, recursive = True)
  File "/home/user/s3cmd/S3/FileLists.py", line 321, in fetch_local_list
    _maintain_cache(cache, local_list)
  File "/home/user/s3cmd/S3/FileLists.py", line 277, in _maintain_cache
    cache.unmark_for_purge(local_list[i]['dev'], local_list[i]['inode'], local_list[i]['mtime'], local_list[i]['size'])
KeyError: 'dev'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   [email protected]
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants