Skip to content

HDDS-10553. Add test case for creating file with EC replication config #6405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 5 commits into from
Mar 24, 2024
Merged

HDDS-10553. Add test case for creating file with EC replication config #6405

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0d223a7
HDDS-10553. Add unit test for creating keys/files with EC replication…
YuanbenWang Mar 19, 2024
4a4a7d5
take advice
YuanbenWang Mar 20, 2024
c9ef5bd
take advice 2
YuanbenWang Mar 20, 2024
d3f131c
Revert "take advice 2"
adoroszlai Mar 21, 2024
6e9ac65
copy input config, close filesystem
YuanbenWang Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions hadoop-hdds/docs/content/feature/ErasureCoding.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,9 @@ the configuration keys `ozone.server.default.replication.type` and `ozone.server
<name>ozone.server.default.replication.type</name>
<value>EC</value>
</property>
```

```XML
<property>
<name>ozone.server.default.replication</name>
<value>RS-6-3-1024k</value>
Expand Down Expand Up @@ -208,6 +210,22 @@ We can pass the EC Replication Config while creating the keys irrespective of bu
ozone sh key put <Ozone Key Object Path> <Local File> --type EC --replication rs-6-3-1024k
```

When using ofs/o3fs, we can pass the EC Replication Config by setting the configuration keys `ozone.replication.type` and `ozone.replication`.

```XML
<property>
<name>ozone.replication.type</name>
<value>EC</value>
</property>
```

```XML
<property>
<name>ozone.replication</name>
<value>rs-3-2-1024k</value>
</property>
```

In the case bucket already has default EC Replication Config, there is no need of passing EC Replication Config while creating key.

### Enable Intel ISA-L
Expand Down
29 changes: 25 additions & 4 deletions ...ntegration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractOzoneFileSystemTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@
import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_FS_ITERATE_BATCH_SIZE;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION_TYPE;
import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_DELIMITER;
import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.KEY_NOT_FOUND;
Expand Down Expand Up @@ -165,6 +167,7 @@ abstract class AbstractOzoneFileSystemTest {
private OzoneManagerProtocol writeClient;
private FileSystem fs;
private OzoneFileSystem o3fs;
private OzoneConfiguration conf;
private OzoneFSStorageStatistics statistics;
private OzoneBucket ozoneBucket;
private String volumeName;
Expand All @@ -174,7 +177,8 @@ abstract class AbstractOzoneFileSystemTest {

@BeforeAll
void init() throws Exception {
OzoneConfiguration conf = new OzoneConfiguration();
conf = new OzoneConfiguration();
conf.set("fs.o3fs.impl.disable.cache", "true");

@adoroszlai adoroszlai Mar 19, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please disable the cache only in the specific test case that needs new FileSystem with changed config, similar to:

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/OzoneFileSystemTests.java

Lines 63 to 65 in 264cbc6

config.setBoolean(
String.format("fs.%s.impl.disable.cache", uri.getScheme()), true);
FileSystem subject = FileSystem.get(uri, config);

conf.setFloat(OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY, TRASH_INTERVAL);
conf.setFloat(FS_TRASH_INTERVAL_KEY, TRASH_INTERVAL);
conf.setFloat(FS_TRASH_CHECKPOINT_INTERVAL_KEY, TRASH_INTERVAL / 2);
Expand Down Expand Up @@ -426,6 +430,23 @@ public void testCreateDoesNotAddParentDirKeys() throws Exception {
assertTrue(fs.getFileStatus(parent).isDirectory(), "Parent directory does not appear to be a directory");
}

@Test
public void testCreateKeyWithECReplicationConfig() throws Exception {
Path root = new Path("/" + volumeName + "/" + bucketName);
Path testKeyPath = new Path(root, "testKey");
conf.set(OZONE_REPLICATION, "rs-3-2-1024k");
conf.set(OZONE_REPLICATION_TYPE, "EC");
FileSystem fileSystem = FileSystem.get(conf);
ContractTestUtils.touch(fileSystem, testKeyPath);

OzoneKeyDetails key = getKey(testKeyPath, false);
assertEquals(HddsProtos.ReplicationType.EC,
key.getReplicationConfig().getReplicationType());
assertEquals("rs-3-2-1024k",
key.getReplicationConfig().getReplication());
fileSystem.close();

@adoroszlai adoroszlai Mar 19, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move the implementation to a new method in OzoneFileSystemTests and call that from both test classes, to avoid code duplication.

}

@Test
public void testDeleteCreatesFakeParentDir() throws Exception {
Path grandparent = new Path("/testDeleteCreatesFakeParentDir");
Expand Down Expand Up @@ -1451,10 +1472,10 @@ public void testCreateKeyShouldUseRefreshedBucketReplicationConfig()
bucket.getVolumeName());

// Set the fs.defaultFS and start the filesystem
Configuration conf = new OzoneConfiguration(cluster.getConf());
conf.set(FS_DEFAULT_NAME_KEY, rootPath);
Configuration configuration = new OzoneConfiguration(cluster.getConf());
configuration.set(FS_DEFAULT_NAME_KEY, rootPath);
// Set the number of keys to be processed during batch operate.
OzoneFileSystem o3FS = (OzoneFileSystem) FileSystem.get(conf);
OzoneFileSystem o3FS = (OzoneFileSystem) FileSystem.get(configuration);

//Let's reset the clock to control the time.
((BasicOzoneClientAdapterImpl) (o3FS.getAdapter())).setClock(testClock);
Expand Down
20 changes: 20 additions & 0 deletions ...tion-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractRootedOzoneFileSystemTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,8 @@
import static org.apache.hadoop.hdds.client.ECReplicationConfig.EcCodec.RS;
import static org.apache.hadoop.ozone.OzoneAcl.AclScope.ACCESS;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_FS_ITERATE_BATCH_SIZE;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION_TYPE;
import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_DELIMITER;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ADDRESS_KEY;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ENABLE_OFS_SHARED_TMP_DIR;
Expand Down Expand Up @@ -232,6 +234,7 @@ public Path getBucketPath() {
@BeforeAll
void initClusterAndEnv() throws IOException, InterruptedException, TimeoutException {
conf = new OzoneConfiguration();
conf.set("fs.ofs.impl.disable.cache", "true");
conf.setFloat(OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY, TRASH_INTERVAL);
conf.setFloat(FS_TRASH_INTERVAL_KEY, TRASH_INTERVAL);
conf.setFloat(FS_TRASH_CHECKPOINT_INTERVAL_KEY, TRASH_INTERVAL / 2);
Expand Down Expand Up @@ -325,6 +328,23 @@ void testCreateDoesNotAddParentDirKeys() throws Exception {
fs.delete(grandparent, true);
}

@Test
public void testCreateKeyWithECReplicationConfig() throws Exception {
String testKeyName = "testKey";
Path testKeyPath = new Path(bucketPath, testKeyName);
conf.set(OZONE_REPLICATION, "rs-3-2-1024k");
conf.set(OZONE_REPLICATION_TYPE, "EC");
FileSystem fileSystem = FileSystem.get(conf);
ContractTestUtils.touch(fileSystem, testKeyPath);

OzoneKeyDetails key = getKey(testKeyPath, false);
assertEquals(HddsProtos.ReplicationType.EC,
key.getReplicationConfig().getReplicationType());
assertEquals("rs-3-2-1024k",
key.getReplicationConfig().getReplication());
Comment thread
adoroszlai marked this conversation as resolved.
fileSystem.close();
}

@Test
void testListStatusWithIntermediateDirWithECEnabled()
throws Exception {
Expand Down