Disable file system caching when size is set to zero#16500
Disable file system caching when size is set to zero#16500alexjo2144 wants to merge 1 commit intotrinodb:masterfrom
Conversation
There was a problem hiding this comment.
is new HdfsConfigurationProvider(new HdfsConfig()) no-op?
what does it do?
There was a problem hiding this comment.
It is a no-op. I included it in the tests here to show that it is a no-op.
alexjo2144
force-pushed
the
hdfs/allow-disabling-caching
branch
from
efe3d44 to
264ff2e
Compare
lib/trino-hdfs/src/main/java/io/trino/hdfs/HdfsConfigurationProvider.java
Outdated
Show resolved
Hide resolved
lib/trino-hdfs/src/main/java/io/trino/hdfs/HdfsConfigurationProvider.java
Outdated
Show resolved
Hide resolved
lib/trino-hdfs/src/main/java/io/trino/hdfs/HdfsConfigurationProvider.java
Outdated
Show resolved
Hide resolved
If the `hive.fs.cache.max-size` property is set to zero the first time a FileSystem is loaded an exception is thrown "FileSystem max cache size has been reached"
alexjo2144
force-pushed
the
hdfs/allow-disabling-caching
branch
from
264ff2e to
42ce3c8
Compare
|
Addressed @ebyhr 's comments, thanks for the reviews |
There was a problem hiding this comment.
This breaks the assumption that all file system interaction goes through our FileSystemCache interface. I'd have to carefully consider the implications of this. Certainly, it's not tested, nor is it something we would want an end user to set.
Since we don't need this for real usage, don't even need it for testing (given the direction of the Iceberg PR), and are looking to minimize the scope of Hadoop usage, let's not merge this.
|
We could enforce |
4 participants
Description
If the
hive.fs.cache.max-sizeproperty is set to zero the first time a FileSystem is loaded an exception is thrown"FileSystem max cache size has been reached"
Extracted from: #16455
I don't expect this to be useful for production but I found it useful for testing the fix in the PR above. The FileSystem cache prevented the tests from hitting the improperly configured FileSystem consistently.
Release notes
(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: