Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory management / avoiding temporary files #14

Closed
schnaader opened this issue Jan 10, 2016 · 3 comments
Closed

Memory management / avoiding temporary files #14

schnaader opened this issue Jan 10, 2016 · 3 comments
Assignees
@schnaader
Labels
enhancement much work
Milestone
Precomp v0.4.6

Comments

@schnaader
Copy link
Owner

At the moment, everything is done using temporary files, so even very small streams are processed on disk instead of memory which slows down Precomp a lot. On the other hand, it allows handling of very large streams that are several GB in size.

Proposal: There should be a parameter to set a max. memory size, everything up to that size will be processed in memory and only if more memory is needed, streams are processed on disk using temporary files.
@schnaader schnaader added this to the Precomp v0.4.5 milestone Jan 10, 2016
@schnaader schnaader self-assigned this Jan 10, 2016
@kassane
Copy link

kassane commented Jan 19, 2016

Hello, how are you?
I tested your project being developed and realized a large accumulation of memory leak.
After some tests with files with an average of 2GB each, there was a bug due to lack of precompilation processing, this means failed continued to point to abort without error or exception.

RAM:6GB

The parameters used were:(-intense -v -cn)

@schnaader schnaader modified the milestones: Precomp v0.5, Precomp v0.4.5 Feb 4, 2016
schnaader added a commit that referenced this issue Apr 18, 2016
@schnaader
Recompress MP3 files up to 64 MB in memory
a75eb06 
- Added helper functions to copy data, file<->memory
- Related: Issue #14
schnaader added a commit that referenced this issue Apr 19, 2016
@schnaader
MP3 recompression ("-r") in memory for files up to 64 MB
fefc6a4 
- Helper functions renamed to fast_copy overrides
- Helper functions added to precomp.h
- Related: Issue #14
schnaader added a commit that referenced this issue Apr 21, 2016
@schnaader
Compress JPG files up to 64 MB in memory
a00e0ff 
- Related: Issue #14
schnaader added a commit that referenced this issue Apr 21, 2016
@schnaader
JPG recompression ("-r") in memory for files up to 64 MB
5edc074 
- Related: Issue #14
@schnaader
Copy link
Owner Author

2 temporary files removed for partial matches by commit faa477

schnaader added a commit that referenced this issue May 30, 2016
@schnaader
Use 2 temporary files less for partial bZip2 matches
0d0b754 
- See issue #14
schnaader added a commit that referenced this issue Jun 2, 2016
@schnaader
One temporary file less
f9afdbd 
- Merged compare_file_mem and compare_files_penalty
- There was a temporary file between the two functions before which is
not needed anymore, comparisons are done in memory on 512 byte chunks
- Result on silesia.zip: 15% faster (30,7 s => 26,2 s)
- See issue #14
@schnaader
Copy link
Owner Author

Version 0.4.6 release uses temporary files only in these cases:

  • streams are bigger than 64 MB
  • multi PNG stitching
  • bZip2
  • GIF
  • recursion

I probably won't target any of them except recursion, so I close this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@schnaader schnaader

Labels
enhancement much work
Projects
None yet
Milestone
Precomp v0.4.6
Development

No branches or pull requests
2 participants
@kassane @schnaader