HDDS-11967. [Docs]DistCP Integration in Kerberized environment.

[jojochuang]

What changes were proposed in this pull request?

HDDS-11967. [Docs]DistCP Integration in Kerberized environment.

Please describe your PR in detail:

• Generated-by: Google Geimini Pro 2.5 (Preview) with the following prompt:

Read Ozone Distcp user doc https://ozone.apache.org/docs/edge/integration/distcp.html
Create an updated user doc with the following instructions
Be succinct, update and integrate with the existing text rather than overwrite the existing text
Generate Markdown source output.
Delegation Token Issues

If a command fails due to being unable to retrieve a delegation token from the destination cluster (indicated by “OzoneToken” in the error output), add the following configuration to core-site.xml or ozone-site.xml:

<property>
  <name>ozone.security.enabled</name>
  <value>true</value>
</property>

Cross-Realm Kerberos

Affected Versions:

For Ozone 1.x, issuing commands across clusters in different Kerberos realms may produce the following error:

# hdfs dfs -ls ofs://ozone1707264383/

24/02/07 18:47:36 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.io.IOException: DestHost:destPort ccycloud-1.weichiu-dst.root.comops.site:9862, LocalHost:localPort ccycloud-1.weichiu-src.local/10.140.99.144:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: om/[email protected], expecting: OM/ccycloud-1.weichiu-dst.local@REALM, while invoking $Proxy10.submitRequest over nodeId=om26,nodeAddress=ccycloud-1.weichiu-dst.local:9862 after 3 failover attempts. Trying to failover immediately.

Cause:

This occurs because the ozone.om.kerberos.principal property is not defined correctly. This issue is fixed in Ozone 2.0.

Workaround:

To resolve this, add the following property to ozone-site.xml:

<property>

  <name>ozone.om.kerberos.principal.pattern</name>
  <value>*</value>
</property>

Fix:

This bug is addressed by HDDS-10328 in Ozone 2.0.

Bidirectional Cross-Realm Trust Environment

In environments with bidirectional cross-realm trust, the command may fail with a token renewal error such as:

24/02/08 00:35:00 ERROR tools.DistCp: Exception encountered

java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1707350431298_0001 to YARN: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: 10.140.99.144:8020, Ident: (token for systest: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1707352474394, maxDate=1707957274394, sequenceNumber=44, masterKeyId=14)

Solution:

Add the following parameter to prevent the DistCp job from attempting to renew the remote Ozone delegation token:

-Dmapreduce.job.hdfs-servers.token-renewal.exclude=ozone1707264383

Example:

If running the command on the destination cluster, use the following syntax:

hadoop distcp \

  -Dmapreduce.job.hdfs-servers.token-renewal.exclude=ccycloud-1.weichiu-src.root.comops.site \

  -Ddfs.checksum.combine.mode=COMPOSITE_CRC                 \

  -Dozone.client.checksum.type=CRC32C                       \

  hdfs://ccycloud-1.weichiu-src.root.comops.site:8020/tmp/  \

  ofs://ozone1707264383/tmp/dest

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11967

How was this patch tested?

User doc only

[Copilot]

Pull Request Overview

HDDS-11967. [Docs] DistCP Integration in Kerberized environment
This PR updates the user documentation for Hadoop DistCp to better address security issues in kerberized environments by adding troubleshooting sections for delegation token failures, cross-realm Kerberos issues, and token renewal errors.

• Added a "Troubleshooting Common Issues" section with subsections on Delegation Token Issues, Cross-Realm Kerberos, and Token Renewal Failures.
• Introduced XML and shell command examples to illustrate configuration changes and workarounds.
• Updated error examples and provided workaround instructions referencing HDDS-10328.

[Copilot]

Consider using fenced code blocks (e.g., triple backticks with 'xml') for XML configuration snippets to improve readability and consistency in the documentation.

[Copilot]

Ensure that the example command is formatted in a fenced code block with shell syntax to enhance clarity for users.

[jojochuang]

@SaketaChalamchala pls review

[jojochuang]

Thanks @adoroszlai !