-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4551a3d
commit 1a0c7f1
Showing
4 changed files
with
85 additions
and
61 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,24 @@ | ||
# rsync-gateway | ||
|
||
`rsync-gateway` serves the rsync repository on S3 over HTTP, using the metadata stored in redis. | ||
`rsync-gateway` serves the rsync repository on S3 over HTTP, using the metadata stored in the database. | ||
|
||
## Implementation Details | ||
|
||
1. Connect to Redis. | ||
2. Spawn a watcher task to watch for the latest index. | ||
3. For each request, if the path ends with a trailing slash, it's a directory listing request. Otherwise, it's a file | ||
request. | ||
4. For directory listing requests, redirect to `<path>/index.html` on S3. | ||
5. For file requests, check if the file exists in the index. If not, return 404. Otherwise, redirect to the file on S3. | ||
Symlinks are resolved on the gateway side. | ||
1. Connect to Postgres. | ||
2. Spawn a watcher task to watch for the latest revision. | ||
3. For each request, check if there's a cache hit. Return the cached response if there is. | ||
4. Otherwise, try to resolve the path to in the revision. If the path is a directory, render the directory listing. If | ||
the path is a file, pre-sign the file on S3 and redirect to the pre-signed URL. Symlinks are followed. | ||
|
||
## More details on the cache | ||
|
||
There are two layers of cache: L1 and L2. Both of them are in-memory cache implemented using `moka`, a concurrent LRU | ||
cache. | ||
|
||
L1 cache is raw resolved entries, while L2 cache is compressed entries. The L2 cache is used to reduce memory | ||
usage, since the raw resolved entries can be quite large when there are many files in a directory. | ||
|
||
The size of the L1 cache is 32MB, and the size of the L2 cache is 128MB. There's a TTL for pre-signed URLs in both | ||
caches, which is half the pre-signed URL expiration time. | ||
|
||
It's a NINE cache. The eviction of L1 and L2 caches are independent and asynchronous. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters