Merge from apache trunk

.gitignore

@@ -1,3 +1,4 @@
+.DS_Store
 *.iml
 *.ipr
 *.iws

BUILDING.txt

@@ -10,9 +10,44 @@ Requirements:
 * ProtocolBuffer 2.5.0
 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
 * Zlib devel (if compiling native code)
-* openssl devel ( if compiling native hadoop-pipes )
+* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
+* Jansson C XML parsing library ( if compiling libwebhdfs )
+* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
 * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
 
+----------------------------------------------------------------------------------
+Installing required packages for clean install of Ubuntu 14.04 LTS Desktop:
+
+* Oracle JDK 1.7 (preferred)
+  $ sudo apt-get purge openjdk*
+  $ sudo apt-get install software-properties-common
+  $ sudo add-apt-repository ppa:webupd8team/java
+  $ sudo apt-get update
+  $ sudo apt-get install oracle-java7-installer
+* Maven
+  $ sudo apt-get -y install maven
+* Native libraries
+  $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
+* ProtocolBuffer 2.5.0
+  $ wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
+  $ tar -zxvf protobuf-2.5.0.tar.gz
+  $ cd protobuf-2.5.0.tar.gz
+  $ ./configure
+  $ make
+  $ sudo make install
+  $ sudo ldconfig
+
+Optional packages:
+
+* Snappy compression
+  $ sudo apt-get install snappy libsnappy-dev
+* Bzip2
+  $ sudo apt-get install bzip2 libbz2-dev
+* Jansson (C Library for JSON)
+  $ sudo apt-get install libjansson-dev
+* Linux FUSE
+  $ sudo apt-get install fuse libfuse-dev
+
 ----------------------------------------------------------------------------------
 Maven main modules:
 
@@ -90,7 +125,7 @@ Maven build goals:
   * Use -Drequire.openssl to fail the build if libcrypto.so is not found.
     If this option is not specified and the openssl library is missing,
     we silently build a version of libhadoop.so that cannot make use of
-    openssl. This option is recommended if you plan on making use of openssl 
+    openssl. This option is recommended if you plan on making use of openssl
     and want to get more repeatable builds.
   * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto
     header files and library files. You do not need this option if you have
@@ -128,8 +163,8 @@ Protocol Buffer compiler
 The version of Protocol Buffer compiler, protoc, must match the version of the
 protobuf JAR.
 
-If you have multiple versions of protoc in your system, you can set in your 
-build shell the HADOOP_PROTOC_PATH environment variable to point to the one you 
+If you have multiple versions of protoc in your system, you can set in your
+build shell the HADOOP_PROTOC_PATH environment variable to point to the one you
 want to use for the Hadoop build. If you don't define this environment variable,
 protoc is looked up in the PATH.
 ----------------------------------------------------------------------------------
@@ -209,7 +244,8 @@ Requirements:
 * Findbugs 1.3.9 (if running findbugs)
 * ProtocolBuffer 2.5.0
 * CMake 2.6 or newer
-* Windows SDK or Visual Studio 2010 Professional
+* Windows SDK 7.1 or Visual Studio 2010 Professional
+* Windows SDK 8.1 (if building CPU rate control for the container executor)
 * zlib headers (if building native code bindings for zlib)
 * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
 * Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
@@ -220,11 +256,15 @@ can be downloaded from http://git-scm.com/download/win.
 
 If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
 Do not use Visual Studio Express.  It does not support compiling for 64-bit,
-which is problematic if running a 64-bit system.  The Windows SDK is free to
+which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to
 download here:
 
 http://www.microsoft.com/en-us/download/details.aspx?id=8279
 
+The Windows SDK 8.1 is available to download at:
+
+http://msdn.microsoft.com/en-us/windows/bg162891.aspx
+
 Cygwin is neither required nor supported.
 
 ----------------------------------------------------------------------------------
@@ -253,8 +293,8 @@ Several tests require that the user must have the Create Symbolic Links
 privilege.
 
 All Maven goals are the same as described above with the exception that
-native code is built by enabling the 'native-win' Maven profile. -Pnative-win 
-is enabled by default when building on Windows since the native components 
+native code is built by enabling the 'native-win' Maven profile. -Pnative-win
+is enabled by default when building on Windows since the native components
 are required (not optional) on Windows.
 
 If native code bindings for zlib are required, then the zlib headers must be
@@ -273,4 +313,3 @@ http://www.zlib.net/
 Building distributions:
 
  * Build distribution with native code    : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]
-

dev-support/shelldocs.py

@@ -0,0 +1,250 @@
+#!/usr/bin/python
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+
+import re
+import sys
+import string
+from optparse import OptionParser
+
+def docstrip(key,string):
+  string=re.sub("^## @%s " % key ,"",string)
+  string=string.lstrip()
+  string=string.rstrip()
+  return string
+
+def toc(list):
+  tocout=[]
+  header=()
+  for i in list:
+    if header != i.getinter():
+      header=i.getinter()
+      line="  * %s\n" % (i.headerbuild())
+      tocout.append(line)
+    line="    * [%s](#%s)\n" % (i.getname().replace("_","\_"),i.getname())
+    tocout.append(line)
+  return tocout
+
+class ShellFunction:
+  def __init__(self):
+    self.reset()
+
+  def __cmp__(self,other):
+    if (self.audience == other.audience):
+      if (self.stability == other.stability):
+        if (self.replaceb == other.replaceb):
+          return(cmp(self.name,other.name))
+        else:
+          if (self.replaceb == "Yes"):
+            return -1
+          else:
+            return 1
+      else:
+          if (self.stability == "Stable"):
+            return -1
+          else:
+            return 1
+    else:
+      if (self.audience == "Public"):
+        return -1
+      else:
+        return 1
+
+  def reset(self):
+    self.name=None
+    self.audience=None
+    self.stability=None
+    self.replaceb=None
+    self.returnt=None
+    self.desc=None
+    self.params=None
+
+  def setname(self,text):
+    definition=text.split();
+    self.name=definition[1]
+
+  def getname(self):
+    if (self.name is None):
+      return "None"
+    else:
+      return self.name
+
+  def setaudience(self,text):
+    self.audience=docstrip("audience",text)
+    self.audience=self.audience.capitalize()
+
+  def getaudience(self):
+    if (self.audience is None):
+      return "None"
+    else:
+      return self.audience
+
+  def setstability(self,text):
+    self.stability=docstrip("stability",text)
+    self.stability=self.stability.capitalize()
+
+  def getstability(self):
+    if (self.stability is None):
+      return "None"
+    else:
+      return self.stability
+
+  def setreplace(self,text):
+    self.replaceb=docstrip("replaceable",text)
+    self.replaceb=self.replaceb.capitalize()
+
+  def getreplace(self):
+    if (self.replaceb is None):
+      return "None"
+    else:
+      return self.replaceb
+
+  def getinter(self):
+    return( (self.getaudience(), self.getstability(), self.getreplace()))
+
+  def addreturn(self,text):
+    if (self.returnt is None):
+      self.returnt = []
+    self.returnt.append(docstrip("return",text))
+
+  def getreturn(self):
+    if (self.returnt is None):
+      return "Nothing"
+    else:
+      return "\n\n".join(self.returnt)
+
+  def adddesc(self,text):
+    if (self.desc is None):
+      self.desc = []
+    self.desc.append(docstrip("description",text))
+
+  def getdesc(self):
+    if (self.desc is None):
+      return "None"
+    else:
+      return " ".join(self.desc)
+
+  def addparam(self,text):
+    if (self.params is None):
+      self.params = []
+    self.params.append(docstrip("param",text))
+
+  def getparams(self):
+    if (self.params is None):
+      return ""
+    else:
+      return " ".join(self.params)
+
+  def getusage(self):
+    line="%s %s" % (self.name, self.getparams())
+    return line
+
+  def headerbuild(self):
+    if self.getreplace() == "Yes":
+      replacetext="Replaceable"
+    else:
+      replacetext="Not Replaceable"
+    line="%s/%s/%s" % (self.getaudience(), self.getstability(), replacetext)
+    return(line)
+
+  def getdocpage(self):
+    line="### `%s`\n\n"\
+         "* Synopsis\n\n"\
+         "```\n%s\n"\
+         "```\n\n" \
+         "* Description\n\n" \
+         "%s\n\n" \
+         "* Returns\n\n" \
+         "%s\n\n" \
+         "| Classification | Level |\n" \
+         "| :--- | :--- |\n" \
+         "| Audience | %s |\n" \
+         "| Stability | %s |\n" \
+         "| Replaceable | %s |\n\n" \
+         % (self.getname(),
+            self.getusage(),
+            self.getdesc(),
+            self.getreturn(),
+            self.getaudience(),
+            self.getstability(),
+            self.getreplace())
+    return line
+
+  def __str__(self):
+    line="{%s %s %s %s}" \
+      % (self.getname(),
+         self.getaudience(),
+         self.getstability(),
+         self.getreplace())
+    return line
+
+def main():
+  parser=OptionParser(usage="usage: %prog --skipprnorep --output OUTFILE --input INFILE [--input INFILE ...]")
+  parser.add_option("-o","--output", dest="outfile",
+     action="store", type="string",
+     help="file to create", metavar="OUTFILE")
+  parser.add_option("-i","--input", dest="infile",
+     action="append", type="string",
+     help="file to read", metavar="INFILE")
+  parser.add_option("--skipprnorep", dest="skipprnorep",
+     action="store_true", help="Skip Private & Not Replaceable")
+
+  (options, args)=parser.parse_args()
+
+  allfuncs=[]
+  for filename in options.infile:
+    with open(filename,"r") as shellcode:
+      funcdef=ShellFunction()
+      for line in shellcode:
+        if line.startswith('## @description'):
+          funcdef.adddesc(line)
+        elif line.startswith('## @audience'):
+          funcdef.setaudience(line)
+        elif line.startswith('## @stability'):
+          funcdef.setstability(line)
+        elif line.startswith('## @replaceable'):
+          funcdef.setreplace(line)
+        elif line.startswith('## @param'):
+          funcdef.addparam(line)
+        elif line.startswith('## @return'):
+          funcdef.addreturn(line)
+        elif line.startswith('function'):
+          funcdef.setname(line)
+          if options.skipprnorep:
+            if funcdef.getaudience() == "Private" and \
+               funcdef.getreplace() == "No":
+               pass
+            else:
+              allfuncs.append(funcdef)
+          funcdef=ShellFunction()
+
+  allfuncs=sorted(allfuncs)
+
+  outfile=open(options.outfile, "w")
+  for line in toc(allfuncs):
+    outfile.write(line)
+
+  outfile.write("\n------\n\n")
+
+  header=[]
+  for funcs in allfuncs:
+    if header != funcs.getinter():
+      header=funcs.getinter()
+      line="## %s\n" % (funcs.headerbuild())
+      outfile.write(line)
+    outfile.write(funcs.getdocpage())
+  outfile.close()
+
+if __name__ == "__main__":
+  main()
+

dev-support/test-patch.sh

@@ -292,6 +292,10 @@ prebuildWithoutPatch () {
     cd -
   fi
   echo "Compiling $(pwd)"
+  if [[ -d "$(pwd)"/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs ]]; then
+    echo "Changing permission $(pwd)/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs to avoid broken builds "
+    chmod +x -R "$(pwd)/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs"
+  fi
   echo "$MVN clean test -DskipTests -D${PROJECT_NAME}PatchProcess -Ptest-patch > $PATCH_DIR/trunkJavacWarnings.txt 2>&1"
   $MVN clean test -DskipTests -D${PROJECT_NAME}PatchProcess -Ptest-patch > $PATCH_DIR/trunkJavacWarnings.txt 2>&1
   if [[ $? != 0 ]] ; then

hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml

@@ -34,5 +34,15 @@
     <Method name="getCurrentSecret" />
     <Bug pattern="EI_EXPOSE_REP" />
   </Match>
+  <Match>
+    <Class name="org.apache.hadoop.security.authentication.util.FileSignerSecretProvider" />
+    <Method name="getAllSecrets" />
+    <Bug pattern="EI_EXPOSE_REP" />
+  </Match>
+  <Match>
+    <Class name="org.apache.hadoop.security.authentication.util.FileSignerSecretProvider" />
+    <Method name="getCurrentSecret" />
+    <Bug pattern="EI_EXPOSE_REP" />
+  </Match>
 
 </FindBugsFilter>