[SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2.7.3

[wangyum]

What changes were proposed in this pull request?

How to reproduce permission issue:

# build spark
./dev/make-distribution.sh --name SPARK-25330 --tgz  -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn

tar -zxf spark-2.4.0-SNAPSHOT-bin-SPARK-25330.tar && cd spark-2.4.0-SNAPSHOT-bin-SPARK-25330
export HADOOP_PROXY_USER=user_a
bin/spark-sql

export HADOOP_PROXY_USER=user_b
bin/spark-sql

Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=user_b, access=EXECUTE, inode="/tmp/hive-$%7Buser.name%7D/user_b/668748f2-f6c5-4325-a797-fd0a7ee7f4d4":user_b:hadoop:drwx------
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)

The issue occurred in this commit: apache/hadoop@feb886f. This pr revert Hadoop 2.7 to 2.7.3 to avoid this issue.

How was this patch tested?

unit tests and manual tests.

[wangyum]

cc @srowen @steveloughran

[srowen]

Hm is this really a Hadoop thing? Looks like a local env issue? Not sure what may have changed or be different about the temp dir here.

Users are going to use this with Hadoop 2.7.7+; I wonder if we face the issue regardless.

Yes over to @steveloughran for any thoughts on what this may be or if it is a known issue.

[wangyum]

Yes. This is a Hadoop thing. I try to build Hadoop 2.7.7 with Configuration.getRestrictParserDefault(Object resource) = true and false.
It succeeded when Configuration.getRestrictParserDefault(Object resource)=false, but failed when Configuration.getRestrictParserDefault(Object resource)=true.

[SparkQA]

Test build #95664 has finished for PR 22327 at commit f89448b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

The 2.7.x branches updates are generally all security plus some fixes for JVM/JDK regressions.

without looking at the details, you can assume that the regression will be related to one of these and for that reason, wouldn't recommend rolling back. Better to find the problem and come up with a fix or workaround.

Created HADOOP-15722 to cover this.

[srowen]

If it's an unintentional regression in Hadoop, and is impacting users, I'd agree with down-grading the version here (at least to the version that doesn't exhibit the issue).

If it were intentional behavior, we'd really need to work around it.

@steveloughran seems like you think it's a regression insofar as behavior changes like this aren't expected in maintenance releases?

[steveloughran]

The goal for the 2.7.x line should be "nothing breaks", which is precisely why it's only getting critical patches. Reverting might make the problem go away, but you can assume that everyone running HDFS clusters with something based on the 2.7.x line will get this patch in before long: it's best to identify what's up and address it

[wangyum]

How about revert it to branch-2.3 as we are going to release 2.3.2?
We have time to fix it before releasing 2.4.0.

[srowen]

I think that is reasonable personally

[srowen]

From further discussion on the JIRA, looks like there is not an obvious immediate resolution, and we are about to release 2.3.2 and 2.4, and have a nontrivial problem vs a theoretical set of bug fixes in 2.7.7. I'd say roll back now. If we do figure something out soon we can update again

[srowen]

Merged to master/2.4/2.3