Skip to content

HDDS-4147. Add OFS to FileSystem META-INF #1352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
smengcl merged 6 commits into from
Aug 28, 2020
Merged

HDDS-4147. Add OFS to FileSystem META-INF #1352

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2572e01
[Experimental] Remove all explicit fs.o3fs.impl and fs.ofs.impl config.
smengcl Jun 18, 2020
4593e8a
Set fs.defaultFS to OFS in docker-config.
smengcl Jun 18, 2020
9670c6e
trigger new CI check
adoroszlai Aug 26, 2020
06eda61
Revert "Set fs.defaultFS to OFS in docker-config."
smengcl Aug 26, 2020
0e5ed80
Revert docker-config fs.defaultFS changes.
smengcl Aug 26, 2020
e2086f9
Recheck
smengcl Aug 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-csi/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.defaultFS=ofs://om

OZONE-SITE.XML_ozone.csi.owner=hadoop
OZONE-SITE.XML_ozone.csi.socket=/tmp/csi.sock

Expand Down
4 changes: 2 additions & 2 deletions hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=o3fs://bucket.volume.id1
CORE-SITE.XML_fs.defaultFS=ofs://id1

OZONE-SITE.XML_ozone.om.service.ids=id1
OZONE-SITE.XML_ozone.om.nodes.id1=om1,om2,om3
OZONE-SITE.XML_ozone.om.address.id1.om1=om1
Expand Down
4 changes: 2 additions & 2 deletions hadoop-ozone/dist/src/main/compose/ozone-om-ha/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=o3fs://bucket1.volume1.omservice
CORE-SITE.XML_fs.defaultFS=ofs://omservice
Copy link
Member

@elek elek Aug 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this line to use META-INF services based discovery?

Copy link
Contributor

@adoroszlai adoroszlai Aug 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think only MR tests (ozone-mr and ozonesecure-mr) use defaultFS when setting up MR environment:

https://github.com/apache/hadoop-ozone/blob/8102ac799816f417e096873b6f351fd531f7bfbf/hadoop-ozone/dist/src/main/smoketest/createmrenv.robot#L46-L47

Changing defaultFS to ofs:// in these environments can help validate the META-INF changes.

Changing/adding defaultFS in the other ones is not needed IMO.

Copy link
Member

@elek elek Aug 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK these are just helper lines, not really the test of the file system. META-INF/services are already tested as test.sh is modified to do both o3fs/ofs tests.

I am fine to replace all the defaultFS just let's agree first what do we need to test. Do we need different MR test with both kind of defaultFs?

Or is it enough to test o3fs/ofs low level functionality and MR can use just one of them?

(And you see it's a different discussion, that's the reason why I suggested to do it in a separated PR. But can be included if you think it's better.)

Copy link
Contributor

@adoroszlai adoroszlai Aug 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

META-INF/services are already tested as test.sh is modified to do both o3fs/ofs tests.

Got it.

Do we need different MR test with both kind of defaultFs?

I'm doing that in HDDS-3840. I'll pass it via -D command-line option, as different value is needed for each test in the same cluster.

I guess I can retain the cluster-level config for convenience.


OZONE-SITE.XML_ozone.om.service.ids=omservice
OZONE-SITE.XML_ozone.om.nodes.omservice=om1,om2,om3
OZONE-SITE.XML_ozone.om.address.omservice.om1=om1
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-topology/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.defaultFS=ofs://om

OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.scm.container.size=256MB
Expand Down
4 changes: 2 additions & 2 deletions hadoop-ozone/dist/src/main/compose/ozone/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=ofs://om

OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.scm.container.size=1GB
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozoneblockade/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.defaultFS=ofs://om

OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.scm.names=scm
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.defaultFS=ofs://om

OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.scm.container.size=1GB
OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1
Expand Down
2 changes: 1 addition & 1 deletion hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300

CORE-SITE.XML_fs.AbstractFileSystem.o3fs.impl=org.apache.hadoop.fs.ozone.OzFs
CORE-SITE.XML_fs.AbstractFileSystem.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzFs
CORE-SITE.XML_fs.defaultFS=o3fs://bucket1.volume1/
CORE-SITE.XML_fs.defaultFS=ofs://om

MAPRED-SITE.XML_mapreduce.framework.name=yarn
MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
Expand Down
4 changes: 2 additions & 2 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-om-ha/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
CORE-SITE.XML_fs.defaultFS=o3fs://bucket.volume.id1
CORE-SITE.XML_fs.defaultFS=ofs://id1

OZONE-SITE.XML_ozone.om.service.ids=id1
OZONE-SITE.XML_ozone.om.internal.service.id=id1
OZONE-SITE.XML_ozone.om.nodes.id1=om1,om2,om3
Expand Down
6 changes: 2 additions & 4 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.

CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
CORE-SITE.XML_fs.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem
OZONE-SITE.XML_ozone.om.volume.listall.allowed=false
CORE-SITE.XML_fs.defaultFS=ofs://om

OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.om.volume.listall.allowed=false
OZONE-SITE.XML_ozone.scm.container.size=1GB
OZONE-SITE.XML_ozone.scm.pipeline.owner.container.count=1
OZONE-SITE.XML_ozone.scm.names=scm
Expand Down Expand Up @@ -89,7 +88,6 @@ CORE-SITE.XML_hadoop.http.authentication.type=kerberos
CORE-SITE.XML_hadoop.http.authentication.kerberos.principal=HTTP/[email protected]
CORE-SITE.XML_hadoop.http.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab


CORE-SITE.XML_hadoop.security.authorization=true
HADOOP-POLICY.XML_ozone.om.security.client.protocol.acl=*
HADOOP-POLICY.XML_hdds.security.client.datanode.container.protocol.acl=*
Expand Down
1 change: 1 addition & 0 deletions ...est/mini-chaos-tests/src/test/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
9 changes: 2 additions & 7 deletions .../integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,7 @@ public static void init() throws Exception {
conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
// Set the number of keys to be processed during batch operate.
conf.setInt(OZONE_FS_ITERATE_BATCH_SIZE, 5);
// Note: FileSystem#loadFileSystems won't load OFS class due to META-INF
// hence this workaround.
conf.set("fs.ofs.impl", "org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
// fs.ofs.impl would be loaded from META-INF, no need to manually set it
fs = FileSystem.get(conf);
trash = new Trash(conf);
ofs = (RootedOzoneFileSystem) fs;
Expand All @@ -163,10 +161,7 @@ public static void teardown() {
@Test
public void testOzoneFsServiceLoader() throws IOException {
OzoneConfiguration confTestLoader = new OzoneConfiguration();
// Note: FileSystem#loadFileSystems won't load OFS class due to META-INF
// hence this workaround.
confTestLoader.set("fs.ofs.impl",
"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
// fs.ofs.impl should be loaded from META-INF, no need to explicitly set it
Assert.assertEquals(FileSystem.getFileSystemClass(
OzoneConsts.OZONE_OFS_URI_SCHEME, confTestLoader),
RootedOzoneFileSystem.class);
Expand Down
7 changes: 1 addition & 6 deletions ...on-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/RootedOzoneContract.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,7 @@ public FileSystem getTestFileSystem() throws IOException {
OzoneConsts.OZONE_OFS_URI_SCHEME,
cluster.getOzoneManager().getRpcPort());
getConf().set("fs.defaultFS", uri);

// Note: FileSystem#loadFileSystems doesn't load OFS class because
// META-INF points to org.apache.hadoop.fs.ozone.OzoneFileSystem
getConf().set("fs.ofs.impl",
"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");

// fs.ofs.impl should be loaded from META-INF, no need to explicitly set it
copyClusterConfigs(OMConfigKeys.OZONE_OM_ADDRESS_KEY);
copyClusterConfigs(ScmConfigKeys.OZONE_SCM_CLIENT_ADDRESS_KEY);
return FileSystem.get(getConf());
Expand Down
3 changes: 1 addition & 2 deletions ...-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,8 +456,7 @@ private OzoneConfiguration getClientConfForOFS(
String hostPrefix, OzoneConfiguration configuration) {

OzoneConfiguration clientConf = new OzoneConfiguration(configuration);
clientConf.set("fs.ofs.impl",
"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem");
// fs.ofs.impl should be loaded from META-INF, no need to explicitly set it
clientConf.set(FS_DEFAULT_NAME_KEY, hostPrefix);
clientConf.setInt(FS_TRASH_INTERVAL_KEY, 60);
return clientConf;
Expand Down
1 change: 1 addition & 0 deletions ...ozone/ozonefs-common/src/test/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions ...zone/ozonefs-hadoop2/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions ...zone/ozonefs-hadoop3/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions hadoop-ozone/ozonefs/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
1 change: 1 addition & 0 deletions hadoop-ozone/tools/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
# limitations under the License.

org.apache.hadoop.fs.ozone.OzoneFileSystem
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem