New resource delivery system for Emscripten port #352
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Akaricchi
force-pushed
the
emscripten-fetchfs
branch
from
4ce2d1d to
2de39f0
Compare
Akaricchi
force-pushed
the
emscripten-fetchfs
branch
from
2de39f0 to
6ce6a40
Compare
This is currently unused. It's going to be the backbone for a new on-the-fly resource fetching system for the Emscripten port. In this mode, the virtual resource directory structure is embedded into the executable. Files are referenced by their "content IDs", which are currently their sha256 hashes. The "resindex" VFS backend allows one to define a custom file open function, which connects the content IDs to actual data. A example implementation is provided in resindex_layered, which simply opens the content ID as a file under a specified VFS path.
Resource files are no longer packaged and preloaded all in advance. Instead, they are downloaded while the game is running. The implementation is less than ideal, but it works. Resources are requested on demand and cached into IDBFS. Unfortunately, since the resource system was not designed with web-style asynchrony in mind and we aren't ready for threads either, we have no way to do any useful work while a required file is downloading. To somewhat offset that, we also pre-fetch files that were not requested yet. The pre-fetching is limited to 4 files at a time, which seems to be a reasonable compromise between throughput and cold-cache load latency on slow connections. Also unfortunately, it's dumb as rocks: we have no way of knowing which files we will actually be needing soon, so the pre-fetching is done in whatever order the files were indexed. Possibly the easiest way to improve this system would be to bundle (and compress) all of the tiny text files together to alleviate the overhead of hundreds of HTTP requests.
Akaricchi
force-pushed
the
emscripten-fetchfs
branch
from
6ce6a40 to
e2c0282
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resource files are no longer packaged and preloaded all in advance. Instead, they are downloaded while the game is running.
The implementation is less than ideal, but it works. Resources are requested on demand and cached into IDBFS. Unfortunately, since the resource system was not designed with web-style asynchrony in mind and we aren't ready for threads either, we have no way to do any useful work while a required file is downloading. To somewhat offset that, we also pre-fetch files that were not requested yet. The pre-fetching is limited to 4 files at a time, which seems to be a reasonable compromise between throughput and cold-cache load latency on slow connections. Also unfortunately, it's dumb as rocks: we have no way of knowing which files we will actually be needing soon, so the pre-fetching is done in whatever order the files were indexed.
Possibly the easiest way to improve this system would be to bundle (and compress) all of the tiny text files together to alleviate the overhead of hundreds of HTTP requests.