Issue on Hive orc vectorisation

[tprelle]

Hi,
I try to test #2613 but when i test it on my cluster with hive 3.1.2.
I have a no such method error on the constructor of OrcSplit because of the shaded of OrcTail.
java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.io.orc.OrcSplit.<init>(Lorg/apache/hadoop/fs/Path;Ljava/lang/Object;JJ[Ljava/lang/String;Lorg/apache/iceberg/shaded/org/apache/orc/impl/OrcTail;ZZLjava/util/List;JJLorg/apache/hadoop/fs/Path;)V at org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.reader(HiveVectorizedReader.java:102)

I was thinking using reflection but i do not know if someone hive a better way to do it ?

[omarsmak]

I also spent sometime on this issue trying to find a good solution for it, as @tprelle mentioned, the issue is that compiled class HiveVectorizedReader refers to the shaded org.apache.iceberg.shaded.org.apache.orc.OrcTail instead of org.apache.orc.OrcTail which triggers the above error. Some ways I thought about going around this issue?

• From Gradle, after relocating org.apache.orc to org.apache.iceberg.shaded.org.apache.orc.OrcTail, modify the class HiveVectorizedReader to let it point back to org.apache.orc.OrcTail instead of org.apache.iceberg.shaded.org.apache.orc.OrcTail.
• Copy the OrcSplit class from Hive source code and in gradle, relocate it to the shaded folder in order to avoid any conflicts with Hive libraries. This should work but I am not convinced to copy things as it may give maintenance overhead.
• Shade the whole org.apache.hadoop.hive.ql.io.orc under org.apache.iceberg.shaded.org.apache.hadoop.hive.ql.io.orc.

Any other ideas @szlta as as I can see you were the original author of this PR: #2613 ?

[tprelle]

Hi @omarsmak
I try to shade the entire org.apache.hadoop.hive.ql.io.orc i do not achieve do it because hive-exec the native jar it's a uber jar so it's hard.
I achieve to make it work before my vacation with the copy of OrcSplit as hive iceberg it's currently or already moved into hive 4 it's just temporary so it can be ok.
I can send the PR with this change if it's the good solution for you.

[omarsmak]

Hey @tprelle, I am not a committer for this project and thus me myself is seeking for solution for this as well :). What you propose actually is exactly what I tried to do (and it works) to just copy OrcSplit since as you said, it is a temporary solution until hive 4 is out. What I would recommend is, to send a PR with the proposed solution to see what the maintainers think of that.

[omarsmak]

@tprelle I have submitted a PR here: #3155 . Please feel free to comment and share your ideas!