Skip to content

Conversation

@avijayanhwx
Copy link
Contributor

@avijayanhwx avijayanhwx commented Jun 16, 2021

What changes were proposed in this pull request?

On a secure cluster with GRPC TLS enabled, Hive/Spark/HDFS commands that read data blocks from Ozone do not work.

This happens due to the shading of the ozone filesystem client JAR that changes the value of a String constant representing the io.netty package in code to org.apache.hadoop.ozone.shaded.io.netty in the class io.netty.util.internal.NativeLibraryLoader. The code in that class seems to be written in such a way that it can elude "shading", but in vain.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5352

How was this patch tested?

Manually tested.
Integration/Acceptance test pending.

@jojochuang
Copy link
Contributor

does it mean the the io.netty packages are shaded into the uber jar, but they are not relocated to classpath org.apache.hadoop.ozone.io.netty? If so it could cause problems if downstream application don't use the same netty version.

@avijayanhwx
Copy link
Contributor Author

does it mean the the io.netty packages are shaded into the uber jar, but they are not relocated to classpath org.apache.hadoop.ozone.io.netty? If so it could cause problems if downstream application don't use the same netty version.

Netty is already shaded by Apache Ratis (which is the source of the dependency here). Moreover, we are only skipping the shading where the match phrase is io!netty! (Note the '!'). This is to done skip the shading of this piece of code that is meant to not be shaded.

@jojochuang
Copy link
Contributor

ah interesting hack. good to know thanks.

@arp7 arp7 merged commit fa9616d into apache:master Jun 22, 2021
errose28 added a commit to errose28/ozone that referenced this pull request Jul 7, 2021
* master: (28 commits)
  HDDS-5332. Add a new column family and a service provider in Recon DB for Namespace Summaries (apache#2366)
  HDDS-5405. Refactor pom files for HadoopRpc and Grpc/Ratis compilation properties. (apache#2386)
  HDDS-5406. add proto version to all the proto files. (apache#2385)
  HDDS-5398. Avoid object creation in ReplicationManger debug log statements (apache#2379)
  HDDS-5396. Fix negligence issue conditional expressions in MockCRLStore.java (apache#2380)
  HDDS-5395. Avoid unnecessary numKeyOps.incr() call in OMMetrics (apache#2374)
  HDDS-5389. Include ozoneserviceid in fs.defaultFS when configuring o3fs (apache#2370)
  HDDS-5383. Eliminate expensive string creation in debug log messages (apache#2372)
  HDDS-5380. Get more accurate space info for DedicatedDiskSpaceUsage. (apache#2365)
  HDDS-5341. Container report processing is single threaded (apache#2338)
  HDDS-5387. ProfileServlet to move the default output location to an ozone specific directory (apache#2368)
  HDDS-5289. Update container's deleteTransactionId on creation of the transaction in SCM. (apache#2361)
  HDDS-5369. Cleanup unused configuration related to SCM HA (apache#2359)
  HDDS-5381. SCM terminated with exit status 1: null. (apache#2362)
  HDDS-5353. Avoid unnecessary executeBatch call in insertAudits (apache#2342)
  HDDS-5350 :  Add allocate block support in MockOmTransport (apache#2341). Contributed by Uma Maheswara Rao G.
  HDDS-4926. Support start/stop for container balancer via command line (apache#2278)
  HDDS-5269. Datandoe with low ratis log volume space should not be considered for new pipeline allocation. (apache#2344)
  HDDS-5367. Update modification time when updating quota/storageType/versioning (apache#2355)
  HDDS-5352. java.lang.ClassNotFoundException: org/eclipse/jetty/alpn/ALPN (apache#2347)
  ...
bharatviswa504 pushed a commit to bharatviswa504/hadoop-ozone that referenced this pull request Jul 25, 2021
…LPN (apache#2347)

(cherry picked from commit fa9616d)
Change-Id: I37caae2aff898686d47e900ba52b82b7ed4e7d7c
(cherry picked from commit a8a67bf1dc24063bf7f8429c15ac7dceb997a279)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants