Hadoop 2.9.0-palantir.2

[sjrand]

Just for testing right now. Once that's done, will cut 2.9.0-palantir.1 and update the PR.

[sjrand]

This looks good. I built a Spark dist based on this branch and did some reads and writes against:

• HDFS
• s3a
• FFS backed by s3a

All operations succeeded. Will make 2.9.0-palantir.1 now.

[sjrand]

Not sure what's up with the test failure. The value of localHttpJar.head does start with file: when I look at in in the debugger. Can't actually get the tests running/passing locally for silly reasons, so not sure if the test would pass locally or not.

EDIT: Turns out I was using Hadoop 2.6.5 locally when I wrote this.

[sjrand]

Okay, fine, the test failure is legit. Somehow the behavior of FileSystem.getFileSystemClass changed b/w 2.8.2 and 2.9.0 s.t. this line returns true in the former and false in the latter: https://github.com/palantir/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L407.

[sjrand]

And to make things even harder to figure out, the culprit isn't even a code change

srand06-mac:hadoop srand$ git branch
* branch-2.8.2
  branch-2.9.0
srand06-mac:hadoop srand$ grep 'org\.apache' ./hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.LocalFileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystem
org.apache.hadoop.fs.ftp.FTPFileSystem
org.apache.hadoop.fs.HarFileSystem
srand06-mac:hadoop srand$ git co branch-2.9.0
Switched to branch 'branch-2.9.0'
Your branch is up-to-date with 'origin/branch-2.9.0'.
srand06-mac:hadoop srand$ grep 'org\.apache' ./hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.LocalFileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystem
org.apache.hadoop.fs.ftp.FTPFileSystem
org.apache.hadoop.fs.HarFileSystem
org.apache.hadoop.fs.http.HttpFileSystem
org.apache.hadoop.fs.http.HttpsFileSystem

That line doesn't throw anymore just because org.apache.hadoop.fs.http.HttpFileSystem got added to the provider configuration file.