Skip to content

HDDS-4147. Add OFS to FileSystem META-INF #1352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
smengcl merged 6 commits into from
Aug 28, 2020
Merged

HDDS-4147. Add OFS to FileSystem META-INF #1352

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2572e01
[Experimental] Remove all explicit fs.o3fs.impl and fs.ofs.impl config.
smengcl Jun 18, 2020
4593e8a
Set fs.defaultFS to OFS in docker-config.
smengcl Jun 18, 2020
9670c6e
trigger new CI check
adoroszlai Aug 26, 2020
06eda61
Revert "Set fs.defaultFS to OFS in docker-config."
smengcl Aug 26, 2020
0e5ed80
Revert docker-config fs.defaultFS changes.
smengcl Aug 26, 2020
e2086f9
Recheck
smengcl Aug 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=o3fs://bucket.volume.id1
OZONE-SITE.XML_ozone.om.service.ids=id1
OZONE-SITE.XML_ozone.om.nodes.id1=om1,om2,om3
Expand Down
1 change: 0 additions & 1 deletion hadoop-ozone/dist/src/main/compose/ozone-om-ha/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=o3fs://bucket1.volume1.omservice
OZONE-SITE.XML_ozone.om.service.ids=omservice
OZONE-SITE.XML_ozone.om.nodes.omservice=om1,om2,om3
Expand Down
2 changes: 0 additions & 2 deletions hadoop-ozone/dist/src/main/compose/ozone/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.scm.container.size=1GB
Expand Down
1 change: 0 additions & 1 deletion hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=o3fs://bucket.volume.id1
OZONE-SITE.XML_ozone.om.service.ids=id1
OZONE-SITE.XML_ozone.om.internal.service.id=id1
Expand Down
2 changes: 0 additions & 2 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
OZONE-SITE.XML_ozone.om.volume.listall.allowed=false

OZONE-SITE.XML_ozone.om.address=om
Expand Down
1 change: 1 addition & 0 deletions ...est/mini-chaos-tests/src/test/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
9 changes: 2 additions & 7 deletions .../integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,7 @@ public static void init() throws Exception {
conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
// Set the number of keys to be processed during batch operate.
conf.setInt(OZONE_FS_ITERATE_BATCH_SIZE, 5);
// Note: FileSystem#loadFileSystems won't load OFS class due to META-INF
// hence this workaround.
conf.set("fs.ofs.impl", "org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
// fs.ofs.impl would be loaded from META-INF, no need to manually set it
fs = FileSystem.get(conf);
trash = new Trash(conf);
ofs = (RootedOzoneFileSystem) fs;
Expand All @@ -163,10 +161,7 @@ public static void teardown() {
@Test
public void testOzoneFsServiceLoader() throws IOException {
OzoneConfiguration confTestLoader = new OzoneConfiguration();
// Note: FileSystem#loadFileSystems won't load OFS class due to META-INF
// hence this workaround.
confTestLoader.set("fs.ofs.impl",
"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
// fs.ofs.impl should be loaded from META-INF, no need to explicitly set it
Assert.assertEquals(FileSystem.getFileSystemClass(
OzoneConsts.OZONE_OFS_URI_SCHEME, confTestLoader),
RootedOzoneFileSystem.class);
Expand Down
7 changes: 1 addition & 6 deletions ...on-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/RootedOzoneContract.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,7 @@ public FileSystem getTestFileSystem() throws IOException {
OzoneConsts.OZONE_OFS_URI_SCHEME,
cluster.getOzoneManager().getRpcPort());
getConf().set("fs.defaultFS", uri);

// Note: FileSystem#loadFileSystems doesn't load OFS class because
// META-INF points to org.apache.hadoop.fs.ozone.OzoneFileSystem
getConf().set("fs.ofs.impl",
"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");

// fs.ofs.impl should be loaded from META-INF, no need to explicitly set it
copyClusterConfigs(OMConfigKeys.OZONE_OM_ADDRESS_KEY);
copyClusterConfigs(ScmConfigKeys.OZONE_SCM_CLIENT_ADDRESS_KEY);
return FileSystem.get(getConf());
Expand Down
3 changes: 1 addition & 2 deletions ...-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,8 +456,7 @@ private OzoneConfiguration getClientConfForOFS(
String hostPrefix, OzoneConfiguration configuration) {

OzoneConfiguration clientConf = new OzoneConfiguration(configuration);
clientConf.set("fs.ofs.impl",
"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
// fs.ofs.impl should be loaded from META-INF, no need to explicitly set it
clientConf.set(FS_DEFAULT_NAME_KEY, hostPrefix);
clientConf.setInt(FS_TRASH_INTERVAL_KEY, 60);
return clientConf;
Expand Down
1 change: 1 addition & 0 deletions ...ozone/ozonefs-common/src/test/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions ...zone/ozonefs-hadoop2/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions ...zone/ozonefs-hadoop3/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions hadoop-ozone/ozonefs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions hadoop-ozone/tools/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem