Improve docs for support hadoop compatible file system when use HDFS …#24918
Conversation
mosabua
left a comment
mosabua
left a comment
There was a problem hiding this comment.
We should clarify if this is about Alluxio or some other partially Hadoop compatible system. And if we make it more explicit as Alluxio as the concrete example we should link to where to get the JAR files from and what they are called maybe
There was a problem hiding this comment.
| - Skip directory scheme validation to support hadoop compatible file system. | |
| - Skip directory scheme validation to support Hadoop-compatible file system. |
arguably "partially Hadoop-compatible" right?
There was a problem hiding this comment.
| You can enable `exchange.hdfs.skip-directory-scheme-validation` to support hadoop compatible file system. Please do the following steps: | |
| You can enable `exchange.hdfs.skip-directory-scheme-validation` to support other Hadoop- compatible file systems: | |
There was a problem hiding this comment.
| 1. Configure AbstractFileSystem implementation in `core-site.xml`. | |
| 1. Configure the `AbstractFileSystem` implementation in `core-site.xml`. |
There was a problem hiding this comment.
| 2. Put the relevant client jars into the directory `${Trino_HOME}/plugin/exchange-hdfs` on all Trino servers. | |
| 2. Add the relevant client JAR files into the directory `${Trino_HOME}/plugin/exchange-hdfs` on all Trino cluster nodes. |
There was a problem hiding this comment.
| as the spooling storage destination. | |
| as the spooling storage location. |
There was a problem hiding this comment.
Why is that not done via the Alluxio file system support instead of HDFS? We have to probably explain that
There was a problem hiding this comment.
Thank you for feedback. Since the HDFS client supports accessing other Hadoop-compatible file system, I believe adding this config can provide user with more options. By modifying the core-site.xml and adding the relevant client JARs, user can freely choose Hadoop-compatible file system. Different file system will require different configurations and client JARs. Here, Alluxio is only an example.
There was a problem hiding this comment.
Fair enough .. we should call that out in the written docs
There was a problem hiding this comment.
I've made the suggested changes. Please let me know if there's anything else.
hqbhoho
force-pushed
the
feature/improve_docs_for_exchange_hdfs
branch
from
27bb725 to
46c04bd
Compare
2 participants
Description
Follow up to #24627
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: