Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading files larger than 2GB on Windows #8678

Closed
tknopp opened this issue Oct 14, 2014 · 6 comments
Closed

Loading files larger than 2GB on Windows #8678

tknopp opened this issue Oct 14, 2014 · 6 comments

Comments

@tknopp
Copy link
Contributor

tknopp commented Oct 14, 2014

Sorry if this is a dump issue:

I tried to read a binary file which is larger than 2GB on Windows with the 64-bit Julia Version (3.1 from the Website). And I get the error:

ERROR: read: end of file
  in read! at iostream.jl:196

On my Macbook it works as expected. Thoughts? @tkelman @vtjnash @Keno

@vtjnash
Copy link
Sponsor Member

vtjnash commented Oct 14, 2014

probably a bug somewhere in support/ios.c. all off_t types should be 64-bit. not much has been done to see that it actually handles 64-bit files on windows however

@tknopp
Copy link
Contributor Author

tknopp commented Oct 14, 2014

Thanks Jameson, so then it really seems to be a bug.

@tknopp
Copy link
Contributor Author

tknopp commented Oct 14, 2014

There seems to be an option -D_FILE_OFFSET_BITS=64 that allows to change the size of off_t to 64 bit. I currently do not have a Windows development environement. Would be great if someone could test putting the Option to the Makefile

@tkelman
Copy link
Contributor

tkelman commented Oct 14, 2014

-D_FILE_OFFSET_BITS=64 is already set in JCFLAGS in Make.inc. Either that flag's not getting sent everywhere it should be, or there's some other problem.

@vtjnash
Copy link
Sponsor Member

vtjnash commented Oct 15, 2014

off_t is still 32-bit with that flag, it only affects stat, AFAICT

The read function (like __APPLE__) only accepts int's: http://msdn.microsoft.com/en-us/library/wyssk1bs.aspx

@tkelman
Copy link
Contributor

tkelman commented Oct 15, 2014

Ah, I think I mostly get what's going on in _os_read now. Should we change this line

#ifdef __APPLE__

to #if defined(__APPLE__) || defined(_OS_WINDOWS_) then?

tkelman added a commit that referenced this issue Oct 15, 2014
fix #8678, reading large files on windows
tkelman added a commit that referenced this issue Oct 22, 2014
read() on windows takes a uint, so apply a similar io limit per call
as done on mac (directly reusing the same code would give a compiler
warning, since long is 32 bits on windows)

(cherry picked from commit 46e1598)
ref #8689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants