Support using remote HDFS as storage in Raptor#13535
Conversation
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/FileSystemConfig.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsStorageService.java
Outdated
Show resolved
Hide resolved
...-raptor/src/main/java/com/facebook/presto/raptor/filesystem/RaptorLocalFileSystemConfig.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Let's keep this config. Use URI dataDirectory as the only source of truth to determine the FS:
- In
FileSystemModule, UsedataDirectory.getScheme()to decide what FS module to install. Local file should be configured as "file:///raptor" (https://en.wikipedia.org/wiki/File_URI_scheme). - Get the value of
dataDirectoryand convert it intoPath baseLocation. Use it everywhere. Ideally we should avoid usingFileas much as possible.
There was a problem hiding this comment.
I am thinking using URI scheme as "FileSystemModule" selector is not very feasible. For example, under "HdfsStorageModule" mode, we can support URI schemes for both "hdfs" and "local". It's a many to many mapping so I suggest we still use a separate config to indicate what file system we are going to use.
...src/main/java/com/facebook/presto/raptor/filesystem/RaptorRemoteFileSystemConfiguration.java
Outdated
Show resolved
Hide resolved
...src/main/java/com/facebook/presto/raptor/filesystem/RaptorRemoteFileSystemConfiguration.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsOrcDataEnvironment.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/FileSystemModule.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/StorageModule.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/FileStorageService.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsOrcDataEnvironment.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsStorageService.java
Outdated
Show resolved
Hide resolved
highker
left a comment
There was a problem hiding this comment.
Could you hide fileSystem.rename(stagingFile, storageFile); in OrcStorageManager inside StorageService with a new interface? (e.g., promoteFromStagingToStorage). We can hide the rename logic all in StorageService.
19dd15e to
6b96e02
Compare
highker
left a comment
There was a problem hiding this comment.
LGTM; the test failure looks related
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/FileSystemContext.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/StorageManagerConfig.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsStorageService.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/filesystem/HdfsStorageService.java
Outdated
Show resolved
Hide resolved
6b96e02 to
4ef7606
Compare
4ef7606 to
fea8d21
Compare
TBD