HDDS-4503. Provide info on block size via FileSystem #1619
Conversation
| @Override | ||
| public long getDefaultBlockSize() { | ||
| return (long) getConf().getStorageSize( | ||
| OZONE_SCM_BLOCK_SIZE, OZONE_SCM_BLOCK_SIZE_DEFAULT, StorageUnit.BYTES); |
There was a problem hiding this comment.
Is this block size used by client while writing? In create and createNonRecursive it tends to ignore the specified block size.
So, if it isn't correct, Isn't it same as not implementing it? FileSystem will anyway return some value, that can be tweaked as well by fs.local.block.size
There was a problem hiding this comment.
In create and createNonRecursive it tends to ignore the specified block size.
Ozone also ignores it and uses uniform block size on server side (configurable via this OZONE_SCM_BLOCK_SIZE property).
can be tweaked as well by
fs.local.block.size
You are right, we could achieve the same result by forcing this config to the same value as SCM block size. I think it's simpler and cleaner to override the method than to keep the config in sync.
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
Show resolved
Hide resolved
3 participants
What changes were proposed in this pull request?
Provide information about Ozone's configured block size via Hadoop-compatible file system implementations by overriding
getDefaultBlockSize()fromFileSystem.Ozone does not make new files visible until they are closed. This change allows Impala to query the actual configured block size and use this info during file write, instead of relying on the default block size, which can be overridden in config.
https://issues.apache.org/jira/browse/HDDS-4503
How was this patch tested?
Added unit test.