diff --git a/.gitignore b/.gitignore index 15c040cc300c9..a49ad4b48c2e3 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +.DS_Store *.iml *.ipr *.iws diff --git a/BUILDING.txt b/BUILDING.txt index 6e38ad374a174..02b86107e3175 100644 --- a/BUILDING.txt +++ b/BUILDING.txt @@ -10,9 +10,44 @@ Requirements: * ProtocolBuffer 2.5.0 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac * Zlib devel (if compiling native code) -* openssl devel ( if compiling native hadoop-pipes ) +* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance ) +* Jansson C XML parsing library ( if compiling libwebhdfs ) +* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs ) * Internet connection for first build (to fetch all Maven and Hadoop dependencies) +---------------------------------------------------------------------------------- +Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: + +* Oracle JDK 1.7 (preferred) + $ sudo apt-get purge openjdk* + $ sudo apt-get install software-properties-common + $ sudo add-apt-repository ppa:webupd8team/java + $ sudo apt-get update + $ sudo apt-get install oracle-java7-installer +* Maven + $ sudo apt-get -y install maven +* Native libraries + $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev +* ProtocolBuffer 2.5.0 + $ wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz + $ tar -zxvf protobuf-2.5.0.tar.gz + $ cd protobuf-2.5.0.tar.gz + $ ./configure + $ make + $ sudo make install + $ sudo ldconfig + +Optional packages: + +* Snappy compression + $ sudo apt-get install snappy libsnappy-dev +* Bzip2 + $ sudo apt-get install bzip2 libbz2-dev +* Jansson (C Library for JSON) + $ sudo apt-get install libjansson-dev +* Linux FUSE + $ sudo apt-get install fuse libfuse-dev + ---------------------------------------------------------------------------------- Maven main modules: @@ -90,7 +125,7 @@ Maven build goals: * Use -Drequire.openssl to fail the build if libcrypto.so is not found. If this option is not specified and the openssl library is missing, we silently build a version of libhadoop.so that cannot make use of - openssl. This option is recommended if you plan on making use of openssl + openssl. This option is recommended if you plan on making use of openssl and want to get more repeatable builds. * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto header files and library files. You do not need this option if you have @@ -128,8 +163,8 @@ Protocol Buffer compiler The version of Protocol Buffer compiler, protoc, must match the version of the protobuf JAR. -If you have multiple versions of protoc in your system, you can set in your -build shell the HADOOP_PROTOC_PATH environment variable to point to the one you +If you have multiple versions of protoc in your system, you can set in your +build shell the HADOOP_PROTOC_PATH environment variable to point to the one you want to use for the Hadoop build. If you don't define this environment variable, protoc is looked up in the PATH. ---------------------------------------------------------------------------------- @@ -209,7 +244,8 @@ Requirements: * Findbugs 1.3.9 (if running findbugs) * ProtocolBuffer 2.5.0 * CMake 2.6 or newer -* Windows SDK or Visual Studio 2010 Professional +* Windows SDK 7.1 or Visual Studio 2010 Professional +* Windows SDK 8.1 (if building CPU rate control for the container executor) * zlib headers (if building native code bindings for zlib) * Internet connection for first build (to fetch all Maven and Hadoop dependencies) * Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These @@ -220,11 +256,15 @@ can be downloaded from http://git-scm.com/download/win. If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012). Do not use Visual Studio Express. It does not support compiling for 64-bit, -which is problematic if running a 64-bit system. The Windows SDK is free to +which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 +The Windows SDK 8.1 is available to download at: + +http://msdn.microsoft.com/en-us/windows/bg162891.aspx + Cygwin is neither required nor supported. ---------------------------------------------------------------------------------- @@ -253,8 +293,8 @@ Several tests require that the user must have the Create Symbolic Links privilege. All Maven goals are the same as described above with the exception that -native code is built by enabling the 'native-win' Maven profile. -Pnative-win -is enabled by default when building on Windows since the native components +native code is built by enabling the 'native-win' Maven profile. -Pnative-win +is enabled by default when building on Windows since the native components are required (not optional) on Windows. If native code bindings for zlib are required, then the zlib headers must be @@ -273,4 +313,3 @@ http://www.zlib.net/ Building distributions: * Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar] - diff --git a/dev-support/shelldocs.py b/dev-support/shelldocs.py new file mode 100755 index 0000000000000..25474508c07e5 --- /dev/null +++ b/dev-support/shelldocs.py @@ -0,0 +1,250 @@ +#!/usr/bin/python +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import re +import sys +import string +from optparse import OptionParser + +def docstrip(key,string): + string=re.sub("^## @%s " % key ,"",string) + string=string.lstrip() + string=string.rstrip() + return string + +def toc(list): + tocout=[] + header=() + for i in list: + if header != i.getinter(): + header=i.getinter() + line=" * %s\n" % (i.headerbuild()) + tocout.append(line) + line=" * [%s](#%s)\n" % (i.getname().replace("_","\_"),i.getname()) + tocout.append(line) + return tocout + +class ShellFunction: + def __init__(self): + self.reset() + + def __cmp__(self,other): + if (self.audience == other.audience): + if (self.stability == other.stability): + if (self.replaceb == other.replaceb): + return(cmp(self.name,other.name)) + else: + if (self.replaceb == "Yes"): + return -1 + else: + return 1 + else: + if (self.stability == "Stable"): + return -1 + else: + return 1 + else: + if (self.audience == "Public"): + return -1 + else: + return 1 + + def reset(self): + self.name=None + self.audience=None + self.stability=None + self.replaceb=None + self.returnt=None + self.desc=None + self.params=None + + def setname(self,text): + definition=text.split(); + self.name=definition[1] + + def getname(self): + if (self.name is None): + return "None" + else: + return self.name + + def setaudience(self,text): + self.audience=docstrip("audience",text) + self.audience=self.audience.capitalize() + + def getaudience(self): + if (self.audience is None): + return "None" + else: + return self.audience + + def setstability(self,text): + self.stability=docstrip("stability",text) + self.stability=self.stability.capitalize() + + def getstability(self): + if (self.stability is None): + return "None" + else: + return self.stability + + def setreplace(self,text): + self.replaceb=docstrip("replaceable",text) + self.replaceb=self.replaceb.capitalize() + + def getreplace(self): + if (self.replaceb is None): + return "None" + else: + return self.replaceb + + def getinter(self): + return( (self.getaudience(), self.getstability(), self.getreplace())) + + def addreturn(self,text): + if (self.returnt is None): + self.returnt = [] + self.returnt.append(docstrip("return",text)) + + def getreturn(self): + if (self.returnt is None): + return "Nothing" + else: + return "\n\n".join(self.returnt) + + def adddesc(self,text): + if (self.desc is None): + self.desc = [] + self.desc.append(docstrip("description",text)) + + def getdesc(self): + if (self.desc is None): + return "None" + else: + return " ".join(self.desc) + + def addparam(self,text): + if (self.params is None): + self.params = [] + self.params.append(docstrip("param",text)) + + def getparams(self): + if (self.params is None): + return "" + else: + return " ".join(self.params) + + def getusage(self): + line="%s %s" % (self.name, self.getparams()) + return line + + def headerbuild(self): + if self.getreplace() == "Yes": + replacetext="Replaceable" + else: + replacetext="Not Replaceable" + line="%s/%s/%s" % (self.getaudience(), self.getstability(), replacetext) + return(line) + + def getdocpage(self): + line="### `%s`\n\n"\ + "* Synopsis\n\n"\ + "```\n%s\n"\ + "```\n\n" \ + "* Description\n\n" \ + "%s\n\n" \ + "* Returns\n\n" \ + "%s\n\n" \ + "| Classification | Level |\n" \ + "| :--- | :--- |\n" \ + "| Audience | %s |\n" \ + "| Stability | %s |\n" \ + "| Replaceable | %s |\n\n" \ + % (self.getname(), + self.getusage(), + self.getdesc(), + self.getreturn(), + self.getaudience(), + self.getstability(), + self.getreplace()) + return line + + def __str__(self): + line="{%s %s %s %s}" \ + % (self.getname(), + self.getaudience(), + self.getstability(), + self.getreplace()) + return line + +def main(): + parser=OptionParser(usage="usage: %prog --skipprnorep --output OUTFILE --input INFILE [--input INFILE ...]") + parser.add_option("-o","--output", dest="outfile", + action="store", type="string", + help="file to create", metavar="OUTFILE") + parser.add_option("-i","--input", dest="infile", + action="append", type="string", + help="file to read", metavar="INFILE") + parser.add_option("--skipprnorep", dest="skipprnorep", + action="store_true", help="Skip Private & Not Replaceable") + + (options, args)=parser.parse_args() + + allfuncs=[] + for filename in options.infile: + with open(filename,"r") as shellcode: + funcdef=ShellFunction() + for line in shellcode: + if line.startswith('## @description'): + funcdef.adddesc(line) + elif line.startswith('## @audience'): + funcdef.setaudience(line) + elif line.startswith('## @stability'): + funcdef.setstability(line) + elif line.startswith('## @replaceable'): + funcdef.setreplace(line) + elif line.startswith('## @param'): + funcdef.addparam(line) + elif line.startswith('## @return'): + funcdef.addreturn(line) + elif line.startswith('function'): + funcdef.setname(line) + if options.skipprnorep: + if funcdef.getaudience() == "Private" and \ + funcdef.getreplace() == "No": + pass + else: + allfuncs.append(funcdef) + funcdef=ShellFunction() + + allfuncs=sorted(allfuncs) + + outfile=open(options.outfile, "w") + for line in toc(allfuncs): + outfile.write(line) + + outfile.write("\n------\n\n") + + header=[] + for funcs in allfuncs: + if header != funcs.getinter(): + header=funcs.getinter() + line="## %s\n" % (funcs.headerbuild()) + outfile.write(line) + outfile.write(funcs.getdocpage()) + outfile.close() + +if __name__ == "__main__": + main() + diff --git a/dev-support/test-patch.sh b/dev-support/test-patch.sh index b0fbb80a8b03c..574a4fd7e8abb 100755 --- a/dev-support/test-patch.sh +++ b/dev-support/test-patch.sh @@ -292,6 +292,10 @@ prebuildWithoutPatch () { cd - fi echo "Compiling $(pwd)" + if [[ -d "$(pwd)"/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs ]]; then + echo "Changing permission $(pwd)/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs to avoid broken builds " + chmod +x -R "$(pwd)/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs" + fi echo "$MVN clean test -DskipTests -D${PROJECT_NAME}PatchProcess -Ptest-patch > $PATCH_DIR/trunkJavacWarnings.txt 2>&1" $MVN clean test -DskipTests -D${PROJECT_NAME}PatchProcess -Ptest-patch > $PATCH_DIR/trunkJavacWarnings.txt 2>&1 if [[ $? != 0 ]] ; then diff --git a/hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml b/hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml index 1ecf37a577605..ddda63c689818 100644 --- a/hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml +++ b/hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml @@ -34,5 +34,15 @@ + + + + + + + + + + diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java index e891ed2623dd5..684e91c2bc941 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java @@ -18,18 +18,14 @@ import org.apache.hadoop.security.authentication.client.AuthenticatedURL; import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.security.authentication.client.KerberosAuthenticator; -import org.apache.hadoop.security.authentication.util.Signer; -import org.apache.hadoop.security.authentication.util.SignerException; -import org.apache.hadoop.security.authentication.util.RandomSignerSecretProvider; -import org.apache.hadoop.security.authentication.util.SignerSecretProvider; -import org.apache.hadoop.security.authentication.util.StringSignerSecretProvider; -import org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider; +import org.apache.hadoop.security.authentication.util.*; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import javax.servlet.Filter; import javax.servlet.FilterChain; import javax.servlet.FilterConfig; +import javax.servlet.ServletContext; import javax.servlet.ServletException; import javax.servlet.ServletRequest; import javax.servlet.ServletResponse; @@ -147,6 +143,8 @@ public class AuthenticationFilter implements Filter { */ public static final String SIGNATURE_SECRET = "signature.secret"; + public static final String SIGNATURE_SECRET_FILE = SIGNATURE_SECRET + ".file"; + /** * Constant for the configuration property that indicates the validity of the generated token. */ @@ -186,8 +184,6 @@ public class AuthenticationFilter implements Filter { private Signer signer; private SignerSecretProvider secretProvider; private AuthenticationHandler authHandler; - private boolean randomSecret; - private boolean customSecretProvider; private long validity; private String cookieDomain; private String cookiePath; @@ -229,7 +225,6 @@ public void init(FilterConfig filterConfig) throws ServletException { initializeAuthHandler(authHandlerClassName, filterConfig); - cookieDomain = config.getProperty(COOKIE_DOMAIN, null); cookiePath = config.getProperty(COOKIE_PATH, null); } @@ -240,11 +235,8 @@ protected void initializeAuthHandler(String authHandlerClassName, FilterConfig f Class klass = Thread.currentThread().getContextClassLoader().loadClass(authHandlerClassName); authHandler = (AuthenticationHandler) klass.newInstance(); authHandler.init(config); - } catch (ClassNotFoundException ex) { - throw new ServletException(ex); - } catch (InstantiationException ex) { - throw new ServletException(ex); - } catch (IllegalAccessException ex) { + } catch (ClassNotFoundException | InstantiationException | + IllegalAccessException ex) { throw new ServletException(ex); } } @@ -254,60 +246,59 @@ protected void initializeSecretProvider(FilterConfig filterConfig) secretProvider = (SignerSecretProvider) filterConfig.getServletContext(). getAttribute(SIGNER_SECRET_PROVIDER_ATTRIBUTE); if (secretProvider == null) { - Class providerClass - = getProviderClass(config); + // As tomcat cannot specify the provider object in the configuration. + // It'll go into this path try { - secretProvider = providerClass.newInstance(); - } catch (InstantiationException ex) { - throw new ServletException(ex); - } catch (IllegalAccessException ex) { - throw new ServletException(ex); - } - try { - secretProvider.init(config, filterConfig.getServletContext(), validity); + secretProvider = constructSecretProvider( + filterConfig.getServletContext(), + config, false); } catch (Exception ex) { throw new ServletException(ex); } - } else { - customSecretProvider = true; } signer = new Signer(secretProvider); } - @SuppressWarnings("unchecked") - private Class getProviderClass(Properties config) - throws ServletException { - String providerClassName; - String signerSecretProviderName - = config.getProperty(SIGNER_SECRET_PROVIDER, null); - // fallback to old behavior - if (signerSecretProviderName == null) { - String signatureSecret = config.getProperty(SIGNATURE_SECRET, null); - if (signatureSecret != null) { - providerClassName = StringSignerSecretProvider.class.getName(); - } else { - providerClassName = RandomSignerSecretProvider.class.getName(); - randomSecret = true; + public static SignerSecretProvider constructSecretProvider( + ServletContext ctx, Properties config, + boolean disallowFallbackToRandomSecretProvider) throws Exception { + String name = config.getProperty(SIGNER_SECRET_PROVIDER, "file"); + long validity = Long.parseLong(config.getProperty(AUTH_TOKEN_VALIDITY, + "36000")) * 1000; + + if (!disallowFallbackToRandomSecretProvider + && "file".equals(name) + && config.getProperty(SIGNATURE_SECRET_FILE) == null) { + name = "random"; + } + + SignerSecretProvider provider; + if ("file".equals(name)) { + provider = new FileSignerSecretProvider(); + try { + provider.init(config, ctx, validity); + } catch (Exception e) { + if (!disallowFallbackToRandomSecretProvider) { + LOG.info("Unable to initialize FileSignerSecretProvider, " + + "falling back to use random secrets."); + provider = new RandomSignerSecretProvider(); + provider.init(config, ctx, validity); + } else { + throw e; + } } + } else if ("random".equals(name)) { + provider = new RandomSignerSecretProvider(); + provider.init(config, ctx, validity); + } else if ("zookeeper".equals(name)) { + provider = new ZKSignerSecretProvider(); + provider.init(config, ctx, validity); } else { - if ("random".equals(signerSecretProviderName)) { - providerClassName = RandomSignerSecretProvider.class.getName(); - randomSecret = true; - } else if ("string".equals(signerSecretProviderName)) { - providerClassName = StringSignerSecretProvider.class.getName(); - } else if ("zookeeper".equals(signerSecretProviderName)) { - providerClassName = ZKSignerSecretProvider.class.getName(); - } else { - providerClassName = signerSecretProviderName; - customSecretProvider = true; - } - } - try { - return (Class) Thread.currentThread(). - getContextClassLoader().loadClass(providerClassName); - } catch (ClassNotFoundException ex) { - throw new ServletException(ex); + provider = (SignerSecretProvider) Thread.currentThread(). + getContextClassLoader().loadClass(name).newInstance(); + provider.init(config, ctx, validity); } + return provider; } /** @@ -336,7 +327,7 @@ protected AuthenticationHandler getAuthenticationHandler() { * @return if a random secret is being used. */ protected boolean isRandomSecret() { - return randomSecret; + return secretProvider.getClass() == RandomSignerSecretProvider.class; } /** @@ -345,7 +336,10 @@ protected boolean isRandomSecret() { * @return if a custom implementation of a SignerSecretProvider is being used. */ protected boolean isCustomSignerSecretProvider() { - return customSecretProvider; + Class clazz = secretProvider.getClass(); + return clazz != FileSignerSecretProvider.class && clazz != + RandomSignerSecretProvider.class && clazz != ZKSignerSecretProvider + .class; } /** @@ -386,9 +380,6 @@ public void destroy() { authHandler.destroy(); authHandler = null; } - if (secretProvider != null) { - secretProvider.destroy(); - } } /** diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java new file mode 100644 index 0000000000000..e8aa160a20877 --- /dev/null +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java @@ -0,0 +1,84 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.util; + +import com.google.common.base.Charsets; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.security.authentication.server.AuthenticationFilter; +import org.apache.hadoop.security.authentication.util.SignerSecretProvider; + +import javax.servlet.ServletContext; +import java.io.*; +import java.nio.charset.Charset; +import java.util.Properties; + +/** + * A SignerSecretProvider that simply loads a secret from a specified file. + */ +@InterfaceStability.Unstable +@InterfaceAudience.Private +public class FileSignerSecretProvider extends SignerSecretProvider { + + private byte[] secret; + private byte[][] secrets; + + public FileSignerSecretProvider() {} + + @Override + public void init(Properties config, ServletContext servletContext, + long tokenValidity) throws Exception { + + String signatureSecretFile = config.getProperty( + AuthenticationFilter.SIGNATURE_SECRET_FILE, null); + + Reader reader = null; + if (signatureSecretFile != null) { + try { + StringBuilder sb = new StringBuilder(); + reader = new InputStreamReader( + new FileInputStream(signatureSecretFile), Charsets.UTF_8); + int c = reader.read(); + while (c > -1) { + sb.append((char) c); + c = reader.read(); + } + secret = sb.toString().getBytes(Charset.forName("UTF-8")); + } catch (IOException ex) { + throw new RuntimeException("Could not read signature secret file: " + + signatureSecretFile); + } finally { + if (reader != null) { + try { + reader.close(); + } catch (IOException e) { + // nothing to do + } + } + } + } + + secrets = new byte[][]{secret}; + } + + @Override + public byte[] getCurrentSecret() { + return secret; + } + + @Override + public byte[][] getAllSecrets() { + return secrets; + } +} diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java index c01c182db1515..63b812d5e3541 100644 --- a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java @@ -13,9 +13,14 @@ */ package org.apache.hadoop.security.authentication.server; +import java.io.File; +import java.io.FileWriter; import java.io.IOException; +import java.io.Writer; import java.net.HttpCookie; +import java.util.ArrayList; import java.util.Arrays; +import java.util.Enumeration; import java.util.HashMap; import java.util.List; import java.util.Properties; @@ -35,7 +40,7 @@ import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.security.authentication.util.Signer; import org.apache.hadoop.security.authentication.util.SignerSecretProvider; -import org.apache.hadoop.security.authentication.util.StringSignerSecretProvider; +import org.apache.hadoop.security.authentication.util.StringSignerSecretProviderCreator; import org.junit.Assert; import org.junit.Test; import org.mockito.Mockito; @@ -148,22 +153,21 @@ public AuthenticationToken authenticate(HttpServletRequest request, HttpServletR } @Test - public void testInit() throws Exception { - + public void testFallbackToRandomSecretProvider() throws Exception { // minimal configuration & simple auth handler (Pseudo) AuthenticationFilter filter = new AuthenticationFilter(); try { FilterConfig config = Mockito.mock(FilterConfig.class); Mockito.when(config.getInitParameter(AuthenticationFilter.AUTH_TYPE)).thenReturn("simple"); - Mockito.when(config.getInitParameter(AuthenticationFilter.AUTH_TOKEN_VALIDITY)).thenReturn( + Mockito.when(config.getInitParameter( + AuthenticationFilter.AUTH_TOKEN_VALIDITY)).thenReturn( (new Long(TOKEN_VALIDITY_SEC)).toString()); Mockito.when(config.getInitParameterNames()).thenReturn( - new Vector(Arrays.asList(AuthenticationFilter.AUTH_TYPE, - AuthenticationFilter.AUTH_TOKEN_VALIDITY)).elements()); + new Vector<>(Arrays.asList(AuthenticationFilter.AUTH_TYPE, + AuthenticationFilter.AUTH_TOKEN_VALIDITY)).elements()); ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); + Mockito.when(context.getAttribute(AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) + .thenReturn(null); Mockito.when(config.getServletContext()).thenReturn(context); filter.init(config); Assert.assertEquals(PseudoAuthenticationHandler.class, filter.getAuthenticationHandler().getClass()); @@ -175,37 +179,17 @@ public void testInit() throws Exception { } finally { filter.destroy(); } - - // string secret - filter = new AuthenticationFilter(); - try { - FilterConfig config = Mockito.mock(FilterConfig.class); - Mockito.when(config.getInitParameter(AuthenticationFilter.AUTH_TYPE)).thenReturn("simple"); - Mockito.when(config.getInitParameter(AuthenticationFilter.SIGNATURE_SECRET)).thenReturn("secret"); - Mockito.when(config.getInitParameterNames()).thenReturn( - new Vector(Arrays.asList(AuthenticationFilter.AUTH_TYPE, - AuthenticationFilter.SIGNATURE_SECRET)).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); - filter.init(config); - Assert.assertFalse(filter.isRandomSecret()); - Assert.assertFalse(filter.isCustomSignerSecretProvider()); - } finally { - filter.destroy(); - } - - // custom secret - filter = new AuthenticationFilter(); + } + @Test + public void testInit() throws Exception { + // custom secret as inline + AuthenticationFilter filter = new AuthenticationFilter(); try { FilterConfig config = Mockito.mock(FilterConfig.class); Mockito.when(config.getInitParameter(AuthenticationFilter.AUTH_TYPE)).thenReturn("simple"); - Mockito.when(config.getInitParameter(AuthenticationFilter.SIGNATURE_SECRET)).thenReturn("secret"); Mockito.when(config.getInitParameterNames()).thenReturn( - new Vector(Arrays.asList(AuthenticationFilter.AUTH_TYPE, - AuthenticationFilter.SIGNATURE_SECRET)).elements()); + new Vector<>(Arrays.asList(AuthenticationFilter.AUTH_TYPE)) + .elements()); ServletContext context = Mockito.mock(ServletContext.class); Mockito.when(context.getAttribute( AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)).thenReturn( @@ -231,6 +215,39 @@ public byte[][] getAllSecrets() { filter.destroy(); } + // custom secret by file + File testDir = new File(System.getProperty("test.build.data", + "target/test-dir")); + testDir.mkdirs(); + String secretValue = "hadoop"; + File secretFile = new File(testDir, "http-secret.txt"); + Writer writer = new FileWriter(secretFile); + writer.write(secretValue); + writer.close(); + + filter = new AuthenticationFilter(); + try { + FilterConfig config = Mockito.mock(FilterConfig.class); + Mockito.when(config.getInitParameter( + AuthenticationFilter.AUTH_TYPE)).thenReturn("simple"); + Mockito.when(config.getInitParameter( + AuthenticationFilter.SIGNATURE_SECRET_FILE)) + .thenReturn(secretFile.getAbsolutePath()); + Mockito.when(config.getInitParameterNames()).thenReturn( + new Vector(Arrays.asList(AuthenticationFilter.AUTH_TYPE, + AuthenticationFilter.SIGNATURE_SECRET_FILE)).elements()); + ServletContext context = Mockito.mock(ServletContext.class); + Mockito.when(context.getAttribute( + AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) + .thenReturn(null); + Mockito.when(config.getServletContext()).thenReturn(context); + filter.init(config); + Assert.assertFalse(filter.isRandomSecret()); + Assert.assertFalse(filter.isCustomSignerSecretProvider()); + } finally { + filter.destroy(); + } + // custom cookie domain and cookie path filter = new AuthenticationFilter(); try { @@ -242,11 +259,7 @@ public byte[][] getAllSecrets() { new Vector(Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.COOKIE_DOMAIN, AuthenticationFilter.COOKIE_PATH)).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); Assert.assertEquals(".foo.com", filter.getCookieDomain()); Assert.assertEquals("/bar", filter.getCookiePath()); @@ -267,11 +280,7 @@ public byte[][] getAllSecrets() { new Vector( Arrays.asList(AuthenticationFilter.AUTH_TYPE, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); Assert.assertTrue(DummyAuthenticationHandler.init); } finally { @@ -309,11 +318,7 @@ public void testInitCaseSensitivity() throws Exception { Mockito.when(config.getInitParameterNames()).thenReturn( new Vector(Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.AUTH_TOKEN_VALIDITY)).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); Assert.assertEquals(PseudoAuthenticationHandler.class, @@ -336,11 +341,7 @@ public void testGetRequestURL() throws Exception { new Vector( Arrays.asList(AuthenticationFilter.AUTH_TYPE, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -356,6 +357,7 @@ public void testGetRequestURL() throws Exception { @Test public void testGetToken() throws Exception { AuthenticationFilter filter = new AuthenticationFilter(); + try { FilterConfig config = Mockito.mock(FilterConfig.class); Mockito.when(config.getInitParameter("management.operation.return")). @@ -368,21 +370,13 @@ public void testGetToken() throws Exception { Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.SIGNATURE_SECRET, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + SignerSecretProvider secretProvider = + getMockedServletContextWithStringSigner(config); filter.init(config); AuthenticationToken token = new AuthenticationToken("u", "p", DummyAuthenticationHandler.TYPE); token.setExpires(System.currentTimeMillis() + TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); - Properties secretProviderProps = new Properties(); - secretProviderProps.setProperty( - AuthenticationFilter.SIGNATURE_SECRET, "secret"); - secretProvider.init(secretProviderProps, null, TOKEN_VALIDITY_SEC); + Signer signer = new Signer(secretProvider); String tokenSigned = signer.sign(token.toString()); @@ -412,18 +406,14 @@ public void testGetTokenExpired() throws Exception { Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.SIGNATURE_SECRET, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); AuthenticationToken token = new AuthenticationToken("u", "p", DummyAuthenticationHandler.TYPE); token.setExpires(System.currentTimeMillis() - TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, "secret"); @@ -464,17 +454,13 @@ public void testGetTokenInvalidType() throws Exception { Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.SIGNATURE_SECRET, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); AuthenticationToken token = new AuthenticationToken("u", "p", "invalidtype"); token.setExpires(System.currentTimeMillis() + TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, "secret"); @@ -500,6 +486,23 @@ public void testGetTokenInvalidType() throws Exception { } } + private static SignerSecretProvider getMockedServletContextWithStringSigner( + FilterConfig config) throws Exception { + Properties secretProviderProps = new Properties(); + secretProviderProps.setProperty(AuthenticationFilter.SIGNATURE_SECRET, + "secret"); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); + secretProvider.init(secretProviderProps, null, TOKEN_VALIDITY_SEC); + + ServletContext context = Mockito.mock(ServletContext.class); + Mockito.when(context.getAttribute( + AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) + .thenReturn(secretProvider); + Mockito.when(config.getServletContext()).thenReturn(context); + return secretProvider; + } + @Test public void testDoFilterNotAuthenticated() throws Exception { AuthenticationFilter filter = new AuthenticationFilter(); @@ -513,11 +516,7 @@ public void testDoFilterNotAuthenticated() throws Exception { new Vector( Arrays.asList(AuthenticationFilter.AUTH_TYPE, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -567,11 +566,7 @@ private void _testDoFilterAuthentication(boolean withDomainPath, AuthenticationFilter.AUTH_TOKEN_VALIDITY, AuthenticationFilter.SIGNATURE_SECRET, "management.operation" + ".return", "expired.token")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); if (withDomainPath) { Mockito.when(config.getInitParameter(AuthenticationFilter @@ -625,8 +620,8 @@ public Object answer(InvocationOnMock invocation) throws Throwable { Mockito.verify(chain).doFilter(Mockito.any(ServletRequest.class), Mockito.any(ServletResponse.class)); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, "secret"); @@ -698,11 +693,7 @@ public void testDoFilterAuthenticated() throws Exception { new Vector( Arrays.asList(AuthenticationFilter.AUTH_TYPE, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -710,8 +701,8 @@ public void testDoFilterAuthenticated() throws Exception { AuthenticationToken token = new AuthenticationToken("u", "p", "t"); token.setExpires(System.currentTimeMillis() + TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, "secret"); @@ -759,11 +750,7 @@ public void testDoFilterAuthenticationFailure() throws Exception { new Vector( Arrays.asList(AuthenticationFilter.AUTH_TYPE, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -827,11 +814,7 @@ public void testDoFilterAuthenticatedExpired() throws Exception { Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.SIGNATURE_SECRET, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -839,8 +822,8 @@ public void testDoFilterAuthenticatedExpired() throws Exception { AuthenticationToken token = new AuthenticationToken("u", "p", DummyAuthenticationHandler.TYPE); token.setExpires(System.currentTimeMillis() - TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, secret); @@ -906,11 +889,7 @@ public void testDoFilterAuthenticatedInvalidType() throws Exception { Arrays.asList(AuthenticationFilter.AUTH_TYPE, AuthenticationFilter.SIGNATURE_SECRET, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -918,8 +897,8 @@ public void testDoFilterAuthenticatedInvalidType() throws Exception { AuthenticationToken token = new AuthenticationToken("u", "p", "invalidtype"); token.setExpires(System.currentTimeMillis() + TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, secret); @@ -953,11 +932,7 @@ public void testManagementOperation() throws Exception { new Vector( Arrays.asList(AuthenticationFilter.AUTH_TYPE, "management.operation.return")).elements()); - ServletContext context = Mockito.mock(ServletContext.class); - Mockito.when(context.getAttribute( - AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE)) - .thenReturn(null); - Mockito.when(config.getServletContext()).thenReturn(context); + getMockedServletContextWithStringSigner(config); filter.init(config); HttpServletRequest request = Mockito.mock(HttpServletRequest.class); @@ -977,8 +952,8 @@ public void testManagementOperation() throws Exception { AuthenticationToken token = new AuthenticationToken("u", "p", "t"); token.setExpires(System.currentTimeMillis() + TOKEN_VALIDITY_SEC); - StringSignerSecretProvider secretProvider - = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty( AuthenticationFilter.SIGNATURE_SECRET, "secret"); diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java similarity index 92% rename from hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java rename to hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java index 57ddd372fe4b1..7e5b10e641889 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProvider.java @@ -16,6 +16,8 @@ import java.nio.charset.Charset; import java.util.Properties; import javax.servlet.ServletContext; + +import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; @@ -24,8 +26,8 @@ * A SignerSecretProvider that simply creates a secret based on a given String. */ @InterfaceStability.Unstable -@InterfaceAudience.Private -public class StringSignerSecretProvider extends SignerSecretProvider { +@VisibleForTesting +class StringSignerSecretProvider extends SignerSecretProvider { private byte[] secret; private byte[][] secrets; diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProviderCreator.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProviderCreator.java new file mode 100644 index 0000000000000..e567e7bfbafb3 --- /dev/null +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/StringSignerSecretProviderCreator.java @@ -0,0 +1,33 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.util; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.classification.InterfaceStability; + +/** + * Helper class for creating StringSignerSecretProviders in unit tests + */ +@InterfaceStability.Unstable +@VisibleForTesting +public class StringSignerSecretProviderCreator { + /** + * @return a new StringSignerSecretProvider + * @throws Exception + */ + public static StringSignerSecretProvider newStringSignerSecretProvider() + throws Exception { + return new StringSignerSecretProvider(); + } +} diff --git a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java new file mode 100644 index 0000000000000..1856410fd2943 --- /dev/null +++ b/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java @@ -0,0 +1,51 @@ +/** + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. See accompanying LICENSE file. + */ +package org.apache.hadoop.security.authentication.util; + +import org.apache.hadoop.security.authentication.server.AuthenticationFilter; +import org.junit.Assert; +import org.junit.Test; + +import java.io.File; +import java.io.FileWriter; +import java.io.Writer; +import java.util.Properties; + +public class TestFileSignerSecretProvider { + + @Test + public void testGetSecrets() throws Exception { + File testDir = new File(System.getProperty("test.build.data", + "target/test-dir")); + testDir.mkdirs(); + String secretValue = "hadoop"; + File secretFile = new File(testDir, "http-secret.txt"); + Writer writer = new FileWriter(secretFile); + writer.write(secretValue); + writer.close(); + + FileSignerSecretProvider secretProvider + = new FileSignerSecretProvider(); + Properties secretProviderProps = new Properties(); + secretProviderProps.setProperty( + AuthenticationFilter.SIGNATURE_SECRET_FILE, + secretFile.getAbsolutePath()); + secretProvider.init(secretProviderProps, null, -1); + Assert.assertArrayEquals(secretValue.getBytes(), + secretProvider.getCurrentSecret()); + byte[][] allSecrets = secretProvider.getAllSecrets(); + Assert.assertEquals(1, allSecrets.length); + Assert.assertArrayEquals(secretValue.getBytes(), allSecrets[0]); + } +} diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt index 11785f24311b6..13d74fd39afc8 100644 --- a/hadoop-common-project/hadoop-common/CHANGES.txt +++ b/hadoop-common-project/hadoop-common/CHANGES.txt @@ -13,9 +13,14 @@ Trunk (Unreleased) HADOOP-10950. rework heap management vars (John Smith via aw) + HADOOP-6857. FsShell should report raw disk usage including replication + factor. (Byron Wong via shv) + HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. (aajisaka) + HADOOP-11553. Formalize the shell API (aw) + NEW FEATURES HADOOP-6590. Add a username check for hadoop sub-commands (John Smith via aw) @@ -65,9 +70,6 @@ Trunk (Unreleased) HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS filesystems (harsh) - HADOOP-8059. Add javadoc to InterfaceAudience and InterfaceStability. - (Brandon Li via suresh) - HADOOP-8434. Add tests for Configuration setter methods. (Madhukara Phatak via suresh) @@ -181,6 +183,15 @@ Trunk (Unreleased) HADOOP-11593. Convert site documentation from apt to markdown (stragglers) (Masatake Iwasaki via aw) + HADOOP-11673. Skip using JUnit Assume in TestCodec. (Brahma Reddy Battula + via cdouglas) + + HADOOP-10115. Exclude duplicate jars in hadoop package under different + component's lib (Vinayakumar B via aw) + + HADOOP-11524. hadoop_do_classpath_subcommand throws a shellcheck warning. + (cnauroth) + BUG FIXES HADOOP-11473. test-patch says "-1 overall" even when all checks are +1 @@ -409,7 +420,14 @@ Trunk (Unreleased) HADOOP-10774. Update KerberosTestUtils for hadoop-auth tests when using IBM Java (sangamesh via aw) - HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) + HADOOP-11653. shellprofiles should require .sh extension + (Brahma Reddy Battula via aw) + + HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change + (Vinayakumar B via aw) + + HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab + via aw) OPTIMIZATIONS @@ -417,6 +435,58 @@ Trunk (Unreleased) HADOOP-8589. ViewFs tests fail when tests and home dirs are nested (sanjay Radia) +Release 2.8.0 - UNRELEASED + + INCOMPATIBLE CHANGES + + NEW FEATURES + + HADOOP-11226. Add a configuration to set ipc.Client's traffic class with + IPTOS_LOWDELAY|IPTOS_RELIABILITY. (Gopal V via ozawa) + + HADOOP-9477. Add posixGroups support for LDAP groups mapping service. + (Dapeng Sun via Yongjun Zhang) + + IMPROVEMENTS + + HADOOP-11719. [Fsshell] Remove bin/hadoop reference from + GenericOptionsParser default help text. + (Brahma Reddy Battula via harsh) + + HADOOP-11692. Improve authentication failure WARN message to avoid user + confusion. (Yongjun Zhang) + + HADOOP-11659. o.a.h.fs.FileSystem.Cache#remove should use a single hash map + lookup. (Brahma Reddy Battula via aajisaka) + + HADOOP-11709. Time.NANOSECONDS_PER_MILLISECOND - use class-level final + constant instead of method variable (Ajith S via ozawa) + + HADOOP-11447. Add a more meaningful toString method to SampleStat and + MutableStat. (kasha) + + HADOOP-11737. mockito's version in hadoop-nfs’ pom.xml shouldn't be + specified. (Kengo Seki via ozawa) + + HADOOP-11741. Add LOG.isDebugEnabled() guard for some LOG.debug(). + (Walter Su via ozawa) + + HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 + architecture (Edward Nevill via Colin P. McCabe) + + OPTIMIZATIONS + + BUG FIXES + + HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of + jclass to GetStaticObjectField. (Hui Zheng via cnauroth) + + HADOOP-11724. DistCp throws NPE when the target directory is root. + (Lei Eddy Xu via Yongjun Zhang) + + HADOOP-11760. Fix typo of javadoc in DistCp. (Brahma Reddy Battula via + ozawa). + Release 2.7.0 - UNRELEASED INCOMPATIBLE CHANGES @@ -465,9 +535,6 @@ Release 2.7.0 - UNRELEASED HADOOP-10748. HttpServer2 should not load JspServlet. (wheat9) - HADOOP-6857. FsShell should report raw disk usage including replication - factor. (Byron Wong via shv) - HADOOP-10847. Remove the usage of sun.security.x509.* in testing code. (Pascal Oliva via wheat9) @@ -647,6 +714,23 @@ Release 2.7.0 - UNRELEASED HADOOP-11658. Externalize io.compression.codecs property. (Kai Zheng via aajisaka) + HADOOP-11648. Set DomainSocketWatcher thread name explicitly. + (Liang Xie via ozawa) + + HADOOP-11642. Upgrade azure sdk version from 0.6.0 to 2.0.0. + (Shashank Khandelwal and Ivan Mitic via cnauroth) + + HADOOP-11714. Add more trace log4j messages to SpanReceiverHost (cmccabe) + + HADOOP-8059. Add javadoc to InterfaceAudience and InterfaceStability. + (Brandon Li via suresh) + + HADOOP-9329. document native build dependencies in BUILDING.txt (Vijay Bhat + via Colin P. McCabe) + + HADOOP-10670. Allow AuthenticationFilters to load secret from signature + secret files. (Kai Zheng via wheat9) + OPTIMIZATIONS HADOOP-11323. WritableComparator#compare keeps reference to byte array. @@ -667,6 +751,8 @@ Release 2.7.0 - UNRELEASED HADOOP-11620. Add support for load balancing across a group of KMS for HA. (Arun Suresh via wang) + HADOOP-11183. Memory-based S3AOutputstream. (Thomas Demoor via stevel) + BUG FIXES HADOOP-11512. Use getTrimmedStrings when reading serialization keys @@ -1036,6 +1122,68 @@ Release 2.7.0 - UNRELEASED HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it to wrapped FS. (gera) + HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream + should be non static. (Sean Busbey via yliu) + + HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) + + HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) + + HADOOP-11686. MiniKDC cannot change ORG_NAME or ORG_DOMAIN. + (Duo Zhang via wheat9) + + HADOOP-11618. DelegateToFileSystem erroneously uses default FS's port in + constructor. (Brahma Reddy Battula via gera) + + HADOOP-11693. Azure Storage FileSystem rename operations are throttled too + aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth) + + HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt + synchronization. (Sean Busbey via yliu) + + HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre + Matsumoto via ozawa) + + HADOOP-11638. OpensslSecureRandom.c pthreads_thread_id should support FreeBSD + and Solaris in addition to Linux. (Kiran Kumar M R via cnauroth) + + HADOOP-11720. [JDK8] Fix javadoc errors caused by incorrect or illegal + tags in hadoop-tools. (Akira AJISAKA via ozawa) + + HADOOP-11722. Some Instances of Services using + ZKDelegationTokenSecretManager go down when old token cannot be deleted. + (Arun Suresh via atm) + + HADOOP-10703. HttpServer2 creates multiple authentication filters. + (Benoy Antony via wheat9) + + HADOOP-11729. Fix link to cgroups doc in site.xml. (Masatake Iwasaki via + ozawa) + + HADOOP-11609. Correct credential commands info in + CommandsManual.html#credential. (Varun Saxena via ozawa) + + HADOOP-11014. Potential resource leak in JavaKeyStoreProvider due to + unclosed stream. (ozawa) + + HADOOP-11738. Fix a link of Protocol Buffers 2.5 for download in BUILDING.txt. + (ozawa) + + HADOOP-11748. The secrets of auth cookies should not be specified in + configuration in clear text. (Li Lu and Haohui Mai via wheat9) + + HADOOP-11691. X86 build of libwinutils is broken. + (Kiran Kumar M R via cnauroth) + + HADOOP-11639. Clean up Windows native code compilation warnings related to + Windows Secure Container Executor. (Remus Rusanu via cnauroth) + + HADOOP-11761. Fix findbugs warnings in org.apache.hadoop.security + .authentication. (Li Lu via wheat9) + + HADOOP-11754. RM fails to start in non-secure mode due to authentication + filter failure. (wheat9) + Release 2.6.1 - UNRELEASED INCOMPATIBLE CHANGES diff --git a/hadoop-common-project/hadoop-common/pom.xml b/hadoop-common-project/hadoop-common/pom.xml index 3ae78f4300042..706f5b5640755 100644 --- a/hadoop-common-project/hadoop-common/pom.xml +++ b/hadoop-common-project/hadoop-common/pom.xml @@ -471,10 +471,10 @@ run - + - + @@ -509,6 +509,53 @@ + + org.codehaus.mojo + exec-maven-plugin + + + shelldocs + pre-site + + exec + + + python + src/site/markdown + + ${basedir}/../../dev-support/shelldocs.py + --skipprnorep + --output + ${basedir}/src/site/markdown/UnixShellAPI.md + --input + ${basedir}/src/main/bin/hadoop-functions.sh + + + + + + + maven-clean-plugin + + + + src/site/markdown + + UnixShellAPI.md + + false + + + src/site/resources + + configuration.xsl + core-default.xml + + false + + + + @@ -550,7 +597,7 @@ unix native build only supported on Mac or Unix - + true @@ -670,7 +717,7 @@ windows native-win build only supported on Windows - + true @@ -786,7 +833,7 @@ mac unix - + true diff --git a/hadoop-common-project/hadoop-common/src/CMakeLists.txt b/hadoop-common-project/hadoop-common/src/CMakeLists.txt index 942b19c752672..7d68fd7b0b7fe 100644 --- a/hadoop-common-project/hadoop-common/src/CMakeLists.txt +++ b/hadoop-common-project/hadoop-common/src/CMakeLists.txt @@ -163,6 +163,14 @@ else (SNAPPY_LIBRARY AND SNAPPY_INCLUDE_DIR) ENDIF(REQUIRE_SNAPPY) endif (SNAPPY_LIBRARY AND SNAPPY_INCLUDE_DIR) +IF (CMAKE_SYSTEM_PROCESSOR MATCHES "^i.86$" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64") + set(BULK_CRC_ARCH_SOURCE_FIlE "${D}/util/bulk_crc32_x86.c") +ELSEIF (CMAKE_SYSTEM_PROCESSOR STREQUAL "aarch64") + set(BULK_CRC_ARCH_SOURCE_FIlE "${D}/util/bulk_crc32_aarch64.c") +ELSE() + MESSAGE("No HW CRC acceleration for ${CMAKE_SYSTEM_PROCESSOR}, falling back to SW") +ENDIF() + # Find the no-suffix version of libcrypto. # See HADOOP-11216 for details. SET(STORED_CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES}) @@ -228,6 +236,7 @@ CONFIGURE_FILE(${CMAKE_SOURCE_DIR}/config.h.cmake ${CMAKE_BINARY_DIR}/config.h) add_executable(test_bulk_crc32 ${D}/util/bulk_crc32.c + ${BULK_CRC_ARCH_SOURCE_FIlE} ${T}/util/test_bulk_crc32.c ) @@ -256,6 +265,7 @@ add_dual_library(hadoop ${D}/util/NativeCodeLoader.c ${D}/util/NativeCrc32.c ${D}/util/bulk_crc32.c + ${BULK_CRC_ARCH_SOURCE_FIlE} ) if (NEED_LINK_DL) set(LIB_DL dl) diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop b/hadoop-common-project/hadoop-common/src/main/bin/hadoop index 6003927047d18..64c3c13acf18d 100755 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop @@ -125,7 +125,7 @@ case ${COMMAND} in CLASS=org.apache.hadoop.util.NativeLibraryChecker ;; classpath) - hadoop_do_classpath_subcommand "$@" + hadoop_do_classpath_subcommand CLASS "$@" ;; credential) CLASS=org.apache.hadoop.security.alias.CredentialShell diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh index 9e4e6b00fa7f9..2619ab798be10 100755 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh @@ -65,8 +65,13 @@ hadoop_error "WARNING: Attempting to execute replacement \"hdfs --slaves --daemo # we're going to turn this into # hdfs --slaves --daemon (start|stop) (rest of options) # -argv=(${HADOOP_USER_PARAMS[@]/start}) -argv=(${argv[@]/stop}) -argv=(${argv[@]/status}) +for (( i = 0; i < ${#HADOOP_USER_PARAMS[@]}; i++ )) +do + if [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^start$ ]] || + [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^stop$ ]] || + [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^status$ ]]; then + unset HADOOP_USER_PARAMS[$i] + fi +done -${hdfsscript} --slaves --daemon "${daemonmode}" "${argv[@]}" +${hdfsscript} --slaves --daemon "${daemonmode}" "${HADOOP_USER_PARAMS[@]}" diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh index bccbe259d64f8..85f82000596df 100644 --- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh +++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh @@ -14,13 +14,21 @@ # See the License for the specific language governing permissions and # limitations under the License. +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string function hadoop_error { - # NOTE: This function is not user replaceable. - echo "$*" 1>&2 } +## @description Print a message to stderr if --debug is turned on +## @audience public +## @stability stable +## @replaceable no +## @param string function hadoop_debug { if [[ -n "${HADOOP_SHELL_SCRIPT_DEBUG}" ]]; then @@ -28,10 +36,14 @@ function hadoop_debug fi } +## @description Replace `oldvar` with `newvar` if `oldvar` exists. +## @audience public +## @stability stable +## @replaceable yes +## @param oldvar +## @param newvar function hadoop_deprecate_envvar { - # - # Deprecate $1 with $2 local oldvar=$1 local newvar=$2 local oldval=${!oldvar} @@ -50,10 +62,12 @@ function hadoop_deprecate_envvar fi } +## @description Bootstraps the Hadoop shell environment +## @audience private +## @stability evolving +## @replaceable no function hadoop_bootstrap { - # NOTE: This function is not user replaceable. - # the root of the Hadoop installation # See HADOOP-6255 for the expected directory structure layout @@ -94,14 +108,14 @@ function hadoop_bootstrap hadoop_debug "Initial HADOOP_OPTS=${HADOOP_OPTS}" } +## @description Locate Hadoop's configuration directory +## @audience private +## @stability evolving +## @replaceable no function hadoop_find_confdir { - # NOTE: This function is not user replaceable. - local conf_dir - # Look for the basic hadoop configuration area. - # - # + # An attempt at compatibility with some Hadoop 1.x # installs. if [[ -e "${HADOOP_PREFIX}/conf/hadoop-env.sh" ]]; then @@ -114,6 +128,11 @@ function hadoop_find_confdir hadoop_debug "HADOOP_CONF_DIR=${HADOOP_CONF_DIR}" } +## @description Validate ${HADOOP_CONF_DIR} +## @audience public +## @stability stable +## @replaceable yes +## @return will exit on failure conditions function hadoop_verify_confdir { # Check only log4j.properties by default. @@ -123,10 +142,12 @@ function hadoop_verify_confdir fi } +## @description Import the hadoop-env.sh settings +## @audience private +## @stability evolving +## @replaceable no function hadoop_exec_hadoopenv { - # NOTE: This function is not user replaceable. - if [[ -z "${HADOOP_ENV_PROCESSED}" ]]; then if [[ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]]; then export HADOOP_ENV_PROCESSED=true @@ -135,26 +156,35 @@ function hadoop_exec_hadoopenv fi } +## @description Import the replaced functions +## @audience private +## @stability evolving +## @replaceable no function hadoop_exec_userfuncs { - # NOTE: This function is not user replaceable. - if [[ -e "${HADOOP_CONF_DIR}/hadoop-user-functions.sh" ]]; then . "${HADOOP_CONF_DIR}/hadoop-user-functions.sh" fi } +## @description Read the user's settings. This provides for users to +## @description override and/or append hadoop-env.sh. It is not meant +## @description as a complete system override. +## @audience private +## @stability evolving +## @replaceable yes function hadoop_exec_hadooprc { - # Read the user's settings. This provides for users to override - # and/or append hadoop-env.sh. It is not meant as a complete system override. - if [[ -f "${HOME}/.hadooprc" ]]; then hadoop_debug "Applying the user's .hadooprc" . "${HOME}/.hadooprc" fi } +## @description Import shellprofile.d content +## @audience private +## @stability evolving +## @replaceable yes function hadoop_import_shellprofiles { local i @@ -162,13 +192,13 @@ function hadoop_import_shellprofiles local files2 if [[ -d "${HADOOP_LIBEXEC_DIR}/shellprofile.d" ]]; then - files1=(${HADOOP_LIBEXEC_DIR}/shellprofile.d/*) + files1=(${HADOOP_LIBEXEC_DIR}/shellprofile.d/*.sh) else hadoop_error "WARNING: ${HADOOP_LIBEXEC_DIR}/shellprofile.d doesn't exist. Functionality may not work." fi if [[ -d "${HADOOP_CONF_DIR}/shellprofile.d" ]]; then - files2=(${HADOOP_CONF_DIR}/shellprofile.d/*) + files2=(${HADOOP_CONF_DIR}/shellprofile.d/*.sh) fi for i in "${files1[@]}" "${files2[@]}" @@ -180,6 +210,10 @@ function hadoop_import_shellprofiles done } +## @description Initialize the registered shell profiles +## @audience private +## @stability evolving +## @replaceable yes function hadoop_shellprofiles_init { local i @@ -194,6 +228,10 @@ function hadoop_shellprofiles_init done } +## @description Apply the shell profile classpath additions +## @audience private +## @stability evolving +## @replaceable yes function hadoop_shellprofiles_classpath { local i @@ -208,6 +246,10 @@ function hadoop_shellprofiles_classpath done } +## @description Apply the shell profile native library additions +## @audience private +## @stability evolving +## @replaceable yes function hadoop_shellprofiles_nativelib { local i @@ -222,6 +264,10 @@ function hadoop_shellprofiles_nativelib done } +## @description Apply the shell profile final configuration +## @audience private +## @stability evolving +## @replaceable yes function hadoop_shellprofiles_finalize { local i @@ -236,6 +282,11 @@ function hadoop_shellprofiles_finalize done } +## @description Initialize the Hadoop shell environment, now that +## @description user settings have been imported +## @audience private +## @stability evolving +## @replaceable no function hadoop_basic_init { # Some of these are also set in hadoop-env.sh. @@ -290,10 +341,15 @@ function hadoop_basic_init HADOOP_SSH_PARALLEL=${HADOOP_SSH_PARALLEL:-10} } -function hadoop_populate_slaves_file() +## @description Set the slave support information to the contents +## @description of `filename` +## @audience public +## @stability stable +## @replaceable no +## @param filename +## @return will exit if file does not exist +function hadoop_populate_slaves_file { - # NOTE: This function is not user replaceable. - local slavesfile=$1 shift if [[ -f "${slavesfile}" ]]; then @@ -308,10 +364,17 @@ function hadoop_populate_slaves_file() fi } +## @description Rotates the given `file` until `number` of +## @description files exist. +## @audience public +## @stability stable +## @replaceable no +## @param filename +## @param [number] +## @return $? will contain last mv's return value function hadoop_rotate_log { # - # log rotation (mainly used for .out files) # Users are likely to replace this one for something # that gzips or uses dates or who knows what. # @@ -334,6 +397,13 @@ function hadoop_rotate_log fi } +## @description Via ssh, log into `hostname` and run `command` +## @audience private +## @stability evolving +## @replaceable yes +## @param hostname +## @param command +## @param [...] function hadoop_actual_ssh { # we are passing this function to xargs @@ -345,6 +415,13 @@ function hadoop_actual_ssh ssh ${HADOOP_SSH_OPTS} ${slave} $"${@// /\\ }" 2>&1 | sed "s/^/$slave: /" } +## @description Connect to ${HADOOP_SLAVES} or ${HADOOP_SLAVE_NAMES} +## @description and execute command. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param [...] function hadoop_connect_to_hosts { # shellcheck disable=SC2124 @@ -405,6 +482,11 @@ function hadoop_connect_to_hosts fi } +## @description Utility routine to handle --slaves mode +## @audience private +## @stability evolving +## @replaceable yes +## @param commandarray function hadoop_common_slave_mode_execute { # @@ -415,10 +497,30 @@ function hadoop_common_slave_mode_execute # if --slaves is still on the command line, remove it # to prevent loops - argv=(${argv[@]/--slaves}) + # Also remove --hostnames and --hosts along with arg values + local argsSize=${#argv[@]}; + for (( i = 0; i < $argsSize; i++ )) + do + if [[ "${argv[$i]}" =~ ^--slaves$ ]]; then + unset argv[$i] + elif [[ "${argv[$i]}" =~ ^--hostnames$ ]] || + [[ "${argv[$i]}" =~ ^--hosts$ ]]; then + unset argv[$i]; + let i++; + unset argv[$i]; + fi + done hadoop_connect_to_hosts -- "${argv[@]}" } +## @description Verify that a shell command was passed a valid +## @description class name +## @audience public +## @stability stable +## @replaceable yes +## @param classname +## @return 0 = success +## @return 1 = failure w/user message function hadoop_validate_classname { local class=$1 @@ -433,6 +535,14 @@ function hadoop_validate_classname return 0 } +## @description Append the `appendstring` if `checkstring` is not +## @description present in the given `envvar` +## @audience public +## @stability stable +## @replaceable yes +## @param envvar +## @param checkstring +## @param appendstring function hadoop_add_param { # @@ -454,21 +564,30 @@ function hadoop_add_param fi } +## @description Register the given `shellprofile` to the Hadoop +## @description shell subsystem +## @audience public +## @stability stable +## @replaceable yes +## @param shellprofile function hadoop_add_profile { # shellcheck disable=SC2086 hadoop_add_param HADOOP_SHELL_PROFILES $1 $1 } +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the classpath. Optionally provide +## @description a hint as to where in the classpath it should go. +## @audience public +## @stability stable +## @replaceable yes +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) function hadoop_add_classpath { - # two params: - # $1 = directory, file, wildcard, whatever to add - # $2 = before or after, which determines where in the - # classpath this object should go. default is after - # return 0 = success (added or duplicate) - # return 1 = failure (doesn't exist, whatever) - # However, with classpath (& JLP), we can do dedupe # along with some sanity checking (e.g., missing directories) # since we have a better idea of what is legal @@ -505,15 +624,23 @@ function hadoop_add_classpath return 0 } +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the colonpath. Optionally provide +## @description a hint as to where in the colonpath it should go. +## @description Prior to adding, objects are checked for duplication +## @description and check for existence. Many other functions use +## @description this function as their base implementation +## @description including `hadoop_add_javalibpath` and `hadoop_add_ldlibpath`. +## @audience public +## @stability stable +## @replaceable yes +## @param envvar +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) function hadoop_add_colonpath { - # two params: - # $1 = directory, file, wildcard, whatever to add - # $2 = before or after, which determines where in the - # classpath this object should go - # return 0 = success - # return 1 = failure (duplicate) - # this is CLASSPATH, JLP, etc but with dedupe but no # other checking if [[ -d "${2}" ]] && [[ ":${!1}:" != *":$2:"* ]]; then @@ -536,12 +663,34 @@ function hadoop_add_colonpath return 1 } +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the Java JNI path. Optionally +## @description provide a hint as to where in the Java JNI path +## @description it should go. +## @audience public +## @stability stable +## @replaceable yes +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) function hadoop_add_javalibpath { # specialized function for a common use case hadoop_add_colonpath JAVA_LIBRARY_PATH "$1" "$2" } +## @description Add a file system object (directory, file, +## @description wildcard, ...) to the LD_LIBRARY_PATH. Optionally +## @description provide a hint as to where in the LD_LIBRARY_PATH +## @description it should go. +## @audience public +## @stability stable +## @replaceable yes +## @param object +## @param [before|after] +## @return 0 = success (added or duplicate) +## @return 1 = failure (doesn't exist or some other reason) function hadoop_add_ldlibpath { # specialized function for a common use case @@ -551,6 +700,11 @@ function hadoop_add_ldlibpath export LD_LIBRARY_PATH } +## @description Add the common/core Hadoop components to the +## @description environment +## @audience private +## @stability evolving +## @replaceable yes function hadoop_add_common_to_classpath { # @@ -570,6 +724,11 @@ function hadoop_add_common_to_classpath hadoop_add_classpath "${HADOOP_COMMON_HOME}/${HADOOP_COMMON_DIR}"'/*' } +## @description Add the user's custom classpath settings to the +## @description environment +## @audience private +## @stability evolving +## @replaceable yes function hadoop_add_to_classpath_userpath { # Add the user-specified HADOOP_CLASSPATH to the @@ -607,13 +766,15 @@ function hadoop_add_to_classpath_userpath fi } +## @description Routine to configure any OS-specific settings. +## @audience public +## @stability stable +## @replaceable yes +## @return may exit on failure conditions function hadoop_os_tricks { local bindv6only - # Some OSes have special needs. Here's some out of the box examples for OS X, - # Linux and Windows on Cygwin. - # Vendors, replace this with your special sauce. HADOOP_IS_CYGWIN=false case ${HADOOP_OS_TYPE} in Darwin) @@ -652,6 +813,11 @@ function hadoop_os_tricks esac } +## @description Configure/verify ${JAVA_HOME} +## @audience public +## @stability stable +## @replaceable yes +## @return may exit on failure conditions function hadoop_java_setup { # Bail if we did not detect it @@ -673,6 +839,10 @@ function hadoop_java_setup fi } +## @description Finish Java JNI paths prior to execution +## @audience private +## @stability evolving +## @replaceable yes function hadoop_finalize_libpaths { if [[ -n "${JAVA_LIBRARY_PATH}" ]]; then @@ -683,6 +853,10 @@ function hadoop_finalize_libpaths fi } +## @description Finish Java heap parameters prior to execution +## @audience private +## @stability evolving +## @replaceable yes function hadoop_finalize_hadoop_heap { if [[ -n "${HADOOP_HEAPSIZE_MAX}" ]]; then @@ -708,9 +882,15 @@ function hadoop_finalize_hadoop_heap fi } -# Accepts a variable name. If running on Cygwin, sets the variable value to the -# equivalent translated Windows path by running the cygpath utility. If the -# second argument is true, then the variable is treated as a path list. +## @description Converts the contents of the variable name +## @description `varnameref` into the equivalent Windows path. +## @description If the second parameter is true, then `varnameref` +## @description is treated as though it was a path list. +## @audience public +## @stability stable +## @replaceable yes +## @param varnameref +## @param [true] function hadoop_translate_cygwin_path { if [[ "${HADOOP_IS_CYGWIN}" = "true" ]]; then @@ -724,9 +904,11 @@ function hadoop_translate_cygwin_path fi } -# -# fill in any last minute options that might not have been defined yet -# +## @description Finish configuring Hadoop specific system properties +## @description prior to executing Java +## @audience private +## @stability evolving +## @replaceable yes function hadoop_finalize_hadoop_opts { hadoop_translate_cygwin_path HADOOP_LOG_DIR @@ -742,6 +924,10 @@ function hadoop_finalize_hadoop_opts hadoop_add_param HADOOP_OPTS hadoop.security.logger "-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER}" } +## @description Finish Java classpath prior to execution +## @audience private +## @stability evolving +## @replaceable yes function hadoop_finalize_classpath { hadoop_add_classpath "${HADOOP_CONF_DIR}" before @@ -752,6 +938,10 @@ function hadoop_finalize_classpath hadoop_translate_cygwin_path CLASSPATH true } +## @description Finish Catalina configuration prior to execution +## @audience private +## @stability evolving +## @replaceable yes function hadoop_finalize_catalina_opts { @@ -771,9 +961,14 @@ function hadoop_finalize_catalina_opts hadoop_add_param CATALINA_OPTS "${prefix}.ssl.keystore.file" "-D${prefix}.ssl.keystore.file=${HADOOP_CATALINA_SSL_KEYSTORE_FILE}" } +## @description Finish all the remaining environment settings prior +## @description to executing Java. This is a wrapper that calls +## @description the other `finalize` routines. +## @audience private +## @stability evolving +## @replaceable yes function hadoop_finalize { - hadoop_shellprofiles_finalize hadoop_finalize_classpath @@ -789,10 +984,15 @@ function hadoop_finalize hadoop_translate_cygwin_path HADOOP_MAPRED_HOME } +## @description Print usage information and exit with the passed +## @description `exitcode` +## @audience public +## @stability stable +## @replaceable no +## @param exitcode +## @return This function will always exit. function hadoop_exit_with_usage { - # NOTE: This function is not user replaceable. - local exitcode=$1 if [[ -z $exitcode ]]; then exitcode=1 @@ -807,6 +1007,12 @@ function hadoop_exit_with_usage exit $exitcode } +## @description Verify that prerequisites have been met prior to +## @description excuting a privileged program. +## @audience private +## @stability evolving +## @replaceable yes +## @return This routine may exit. function hadoop_verify_secure_prereq { # if you are on an OS like Illumos that has functional roles @@ -822,6 +1028,9 @@ function hadoop_verify_secure_prereq fi } +## @audience private +## @stability evolving +## @replaceable yes function hadoop_setup_secure_service { # need a more complicated setup? replace me! @@ -830,6 +1039,9 @@ function hadoop_setup_secure_service HADOOP_LOG_DIR=${HADOOP_SECURE_LOG_DIR} } +## @audience private +## @stability evolving +## @replaceable yes function hadoop_verify_piddir { if [[ -z "${HADOOP_PID_DIR}" ]]; then @@ -852,6 +1064,9 @@ function hadoop_verify_piddir rm "${HADOOP_PID_DIR}/$$" >/dev/null 2>&1 } +## @audience private +## @stability evolving +## @replaceable yes function hadoop_verify_logdir { if [[ -z "${HADOOP_LOG_DIR}" ]]; then @@ -874,7 +1089,14 @@ function hadoop_verify_logdir rm "${HADOOP_LOG_DIR}/$$" >/dev/null 2>&1 } -function hadoop_status_daemon() +## @description Determine the status of the daemon referenced +## @description by `pidfile` +## @audience public +## @stability stable +## @replaceable yes +## @param pidfile +## @return (mostly) LSB 4.1.0 compatible status +function hadoop_status_daemon { # # LSB 4.1.0 compatible status command (1) @@ -907,6 +1129,14 @@ function hadoop_status_daemon() return 3 } +## @description Execute the Java `class`, passing along any `options`. +## @description Additionally, set the Java property -Dproc_`command`. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param class +## @param [options] function hadoop_java_exec { # run a java command. this is used for @@ -924,6 +1154,14 @@ function hadoop_java_exec exec "${JAVA}" "-Dproc_${command}" ${HADOOP_OPTS} "${class}" "$@" } +## @description Start a non-privileged daemon in the foreground. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param pidfile +## @param [options] function hadoop_start_daemon { # this is our non-privileged daemon starter @@ -949,10 +1187,17 @@ function hadoop_start_daemon exec "${JAVA}" "-Dproc_${command}" ${HADOOP_OPTS} "${class}" "$@" } +## @description Start a non-privileged daemon in the background. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param pidfile +## @param outfile +## @param [options] function hadoop_start_daemon_wrapper { - # this is our non-privileged daemon start - # that fires up a daemon in the *background* local daemonname=$1 local class=$2 local pidfile=$3 @@ -1007,6 +1252,17 @@ function hadoop_start_daemon_wrapper return 0 } +## @description Start a privileged daemon in the foreground. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param daemonerrfile +## @param wrapperpidfile +## @param [options] function hadoop_start_secure_daemon { # this is used to launch a secure daemon in the *foreground* @@ -1063,6 +1319,18 @@ function hadoop_start_secure_daemon "${class}" "$@" } +## @description Start a privileged daemon in the background. +## @audience private +## @stability evolving +## @replaceable yes +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param wrapperpidfile +## @param warpperoutfile +## @param daemonerrfile +## @param [options] function hadoop_start_secure_daemon_wrapper { # this wraps hadoop_start_secure_daemon to take care @@ -1143,6 +1411,13 @@ function hadoop_start_secure_daemon_wrapper return 0 } +## @description Stop the non-privileged `command` daemon with that +## @description that is running at `pidfile`. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param pidfile function hadoop_stop_daemon { local cmd=$1 @@ -1168,6 +1443,15 @@ function hadoop_stop_daemon fi } +## @description Stop the privileged `command` daemon with that +## @description that is running at `daemonpidfile` and launched with +## @description the wrapper at `wrapperpidfile`. +## @audience public +## @stability stable +## @replaceable yes +## @param command +## @param daemonpidfile +## @param wrapperpidfile function hadoop_stop_secure_daemon { local command=$1 @@ -1182,6 +1466,16 @@ function hadoop_stop_secure_daemon return ${ret} } +## @description Manage a non-privileged daemon. +## @audience private +## @stability evolving +## @replaceable yes +## @param [start|stop|status|default] +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param [options] function hadoop_daemon_handler { local daemonmode=$1 @@ -1226,6 +1520,19 @@ function hadoop_daemon_handler esac } +## @description Manage a privileged daemon. +## @audience private +## @stability evolving +## @replaceable yes +## @param [start|stop|status|default] +## @param command +## @param class +## @param daemonpidfile +## @param daemonoutfile +## @param wrapperpidfile +## @param wrapperoutfile +## @param wrappererrfile +## @param [options] function hadoop_secure_daemon_handler { local daemonmode=$1 @@ -1278,6 +1585,13 @@ function hadoop_secure_daemon_handler esac } +## @description Verify that ${USER} is allowed to execute the +## @description given subcommand. +## @audience public +## @stability stable +## @replaceable yes +## @param subcommand +## @return will exit on failure conditions function hadoop_verify_user { local command=$1 @@ -1291,10 +1605,17 @@ function hadoop_verify_user fi } +## @description Perform the 'hadoop classpath', etc subcommand with the given +## @description parameters +## @audience private +## @stability evolving +## @replaceable yes +## @param [parameters] +## @return will print & exit with no params function hadoop_do_classpath_subcommand { - if [[ "$#" -gt 0 ]]; then - CLASS=org.apache.hadoop.util.Classpath + if [[ "$#" -gt 1 ]]; then + eval "$1"=org.apache.hadoop.util.Classpath else hadoop_finalize echo "${CLASSPATH}" diff --git a/hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example b/hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example.sh similarity index 100% rename from hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example rename to hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example.sh diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java index 753f51530d01b..8a312ffda810b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java @@ -2479,7 +2479,9 @@ public Iterator> iterator() { private Document parse(DocumentBuilder builder, URL url) throws IOException, SAXException { if (!quietmode) { - LOG.debug("parsing URL " + url); + if (LOG.isDebugEnabled()) { + LOG.debug("parsing URL " + url); + } } if (url == null) { return null; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java index c5ac2ae242d80..493e23de74d20 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java @@ -24,6 +24,7 @@ import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configurable; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.util.PerformanceAdvisory; import org.apache.hadoop.util.ReflectionUtils; import org.slf4j.Logger; @@ -105,7 +106,14 @@ private static List> getCodecClasses( List> result = Lists.newArrayList(); String configName = HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_KEY_PREFIX + cipherSuite.getConfigSuffix(); - String codecString = conf.get(configName); + String codecString; + if (configName.equals(CommonConfigurationKeysPublic + .HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY)) { + codecString = conf.get(configName, CommonConfigurationKeysPublic + .HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_DEFAULT); + } else { + codecString = conf.get(configName); + } if (codecString == null) { PerformanceAdvisory.LOG.debug( "No crypto codec classes with cipher suite configured."); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java index f3e5b90afa99d..2e87f916eb6eb 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java @@ -60,7 +60,7 @@ public class CryptoInputStream extends FilterInputStream implements Seekable, PositionedReadable, ByteBufferReadable, HasFileDescriptor, CanSetDropBehind, CanSetReadahead, HasEnhancedByteBufferAccess, ReadableByteChannel { - private static final byte[] oneByteBuf = new byte[1]; + private final byte[] oneByteBuf = new byte[1]; private final CryptoCodec codec; private final Decryptor decryptor; private final int bufferSize; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java index 876ffd6376615..bc09b8c179f43 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java @@ -40,12 +40,15 @@ * padding = pos%(algorithm blocksize); *

* The underlying stream offset is maintained as state. + * + * Note that while some of this class' methods are synchronized, this is just to + * match the threadsafety behavior of DFSOutputStream. See HADOOP-11710. */ @InterfaceAudience.Private @InterfaceStability.Evolving public class CryptoOutputStream extends FilterOutputStream implements Syncable, CanSetDropBehind { - private static final byte[] oneByteBuf = new byte[1]; + private final byte[] oneByteBuf = new byte[1]; private final CryptoCodec codec; private final Encryptor encryptor; private final int bufferSize; @@ -126,7 +129,7 @@ public OutputStream getWrappedStream() { * @throws IOException */ @Override - public void write(byte[] b, int off, int len) throws IOException { + public synchronized void write(byte[] b, int off, int len) throws IOException { checkStream(); if (b == null) { throw new NullPointerException(); @@ -213,14 +216,16 @@ private byte[] getTmpBuf() { } @Override - public void close() throws IOException { + public synchronized void close() throws IOException { if (closed) { return; } - - super.close(); - freeBuffers(); - closed = true; + try { + super.close(); + freeBuffers(); + } finally { + closed = true; + } } /** @@ -228,7 +233,7 @@ public void close() throws IOException { * underlying stream, then do the flush. */ @Override - public void flush() throws IOException { + public synchronized void flush() throws IOException { checkStream(); encrypt(); super.flush(); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java index c0d510d51f992..c6d60a33b1155 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java @@ -22,6 +22,7 @@ import org.apache.commons.io.IOUtils; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; @@ -214,9 +215,11 @@ private FsPermission tryLoadFromPath(Path path, Path backupPath) renameOrFail(path, new Path(path.toString() + "_CORRUPTED_" + System.currentTimeMillis())); renameOrFail(backupPath, path); - LOG.debug(String.format( - "KeyStore loaded successfully from '%s' since '%s'" - + "was corrupted !!", backupPath, path)); + if (LOG.isDebugEnabled()) { + LOG.debug(String.format( + "KeyStore loaded successfully from '%s' since '%s'" + + "was corrupted !!", backupPath, path)); + } } else { throw ioe; } @@ -265,8 +268,10 @@ private FsPermission loadAndReturnPerm(Path pathToLoad, Path pathToDelete) try { perm = loadFromPath(pathToLoad, password); renameOrFail(pathToLoad, path); - LOG.debug(String.format("KeyStore loaded successfully from '%s'!!", - pathToLoad)); + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("KeyStore loaded successfully from '%s'!!", + pathToLoad)); + } if (fs.exists(pathToDelete)) { fs.delete(pathToDelete, true); } @@ -299,9 +304,11 @@ private boolean isBadorWrongPassword(IOException ioe) { private FsPermission loadFromPath(Path p, char[] password) throws IOException, NoSuchAlgorithmException, CertificateException { - FileStatus s = fs.getFileStatus(p); - keyStore.load(fs.open(p), password); - return s.getPermission(); + try (FSDataInputStream in = fs.open(p)) { + FileStatus s = fs.getFileStatus(p); + keyStore.load(in, password); + return s.getPermission(); + } } private Path constructNewPath(Path path) { @@ -595,9 +602,8 @@ private void cleanupNewAndOld(Path newPath, Path oldPath) throws IOException { } protected void writeToNew(Path newPath) throws IOException { - FSDataOutputStream out = - FileSystem.create(fs, newPath, permissions); - try { + try (FSDataOutputStream out = + FileSystem.create(fs, newPath, permissions);) { keyStore.store(out, password); } catch (KeyStoreException e) { throw new IOException("Can't store keystore " + this, e); @@ -608,7 +614,6 @@ protected void writeToNew(Path newPath) throws IOException { throw new IOException( "Certificate exception storing keystore " + this, e); } - out.close(); } protected boolean backupToOld(Path oldPath) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java index 00c8d78bf77d9..87c2abab2acaf 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java @@ -19,6 +19,9 @@ package org.apache.hadoop.fs; import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.crypto.CipherSuite; +import org.apache.hadoop.crypto.JceAesCtrCryptoCodec; +import org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec; /** * This class contains constants for configuration keys used @@ -206,8 +209,12 @@ public class CommonConfigurationKeysPublic { /** See core-default.xml */ public static final String IPC_CLIENT_TCPNODELAY_KEY = "ipc.client.tcpnodelay"; - /** Defalt value for IPC_CLIENT_TCPNODELAY_KEY */ + /** Default value for IPC_CLIENT_TCPNODELAY_KEY */ public static final boolean IPC_CLIENT_TCPNODELAY_DEFAULT = true; + /** Enable low-latency connections from the client */ + public static final String IPC_CLIENT_LOW_LATENCY = "ipc.client.low-latency"; + /** Default value of IPC_CLIENT_LOW_LATENCY */ + public static final boolean IPC_CLIENT_LOW_LATENCY_DEFAULT = false; /** See core-default.xml */ public static final String IPC_SERVER_LISTEN_QUEUE_SIZE_KEY = "ipc.server.listen.queue.size"; @@ -295,6 +302,14 @@ public class CommonConfigurationKeysPublic { "hadoop.security.saslproperties.resolver.class"; public static final String HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_KEY_PREFIX = "hadoop.security.crypto.codec.classes"; + public static final String + HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY = + HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_KEY_PREFIX + + CipherSuite.AES_CTR_NOPADDING.getConfigSuffix(); + public static final String + HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_DEFAULT = + OpensslAesCtrCryptoCodec.class.getName() + "," + + JceAesCtrCryptoCodec.class.getName(); /** See core-default.xml */ public static final String HADOOP_SECURITY_CRYPTO_CIPHER_SUITE_KEY = "hadoop.security.crypto.cipher.suite"; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java index 6276dda2addbd..66137d02c16c6 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java @@ -21,6 +21,7 @@ import java.io.DataOutput; import java.io.IOException; +import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.io.Writable; @@ -36,17 +37,106 @@ public class ContentSummary implements Writable{ private long quota; private long spaceConsumed; private long spaceQuota; - + private long typeConsumed[]; + private long typeQuota[]; + + public static class Builder{ + public Builder() { + this.quota = -1; + this.spaceQuota = -1; + + typeConsumed = new long[StorageType.values().length]; + typeQuota = new long[StorageType.values().length]; + for (int i = 0; i < typeQuota.length; i++) { + typeQuota[i] = -1; + } + } + + public Builder length(long length) { + this.length = length; + return this; + } + + public Builder fileCount(long fileCount) { + this.fileCount = fileCount; + return this; + } + + public Builder directoryCount(long directoryCount) { + this.directoryCount = directoryCount; + return this; + } + + public Builder quota(long quota){ + this.quota = quota; + return this; + } + + public Builder spaceConsumed(long spaceConsumed) { + this.spaceConsumed = spaceConsumed; + return this; + } + + public Builder spaceQuota(long spaceQuota) { + this.spaceQuota = spaceQuota; + return this; + } + + public Builder typeConsumed(long typeConsumed[]) { + for (int i = 0; i < typeConsumed.length; i++) { + this.typeConsumed[i] = typeConsumed[i]; + } + return this; + } + + public Builder typeQuota(StorageType type, long quota) { + this.typeQuota[type.ordinal()] = quota; + return this; + } + + public Builder typeConsumed(StorageType type, long consumed) { + this.typeConsumed[type.ordinal()] = consumed; + return this; + } + + public Builder typeQuota(long typeQuota[]) { + for (int i = 0; i < typeQuota.length; i++) { + this.typeQuota[i] = typeQuota[i]; + } + return this; + } + + public ContentSummary build() { + return new ContentSummary(length, fileCount, directoryCount, quota, + spaceConsumed, spaceQuota, typeConsumed, typeQuota); + } + + private long length; + private long fileCount; + private long directoryCount; + private long quota; + private long spaceConsumed; + private long spaceQuota; + private long typeConsumed[]; + private long typeQuota[]; + } - /** Constructor */ + /** Constructor deprecated by ContentSummary.Builder*/ + @Deprecated public ContentSummary() {} - /** Constructor */ + /** Constructor, deprecated by ContentSummary.Builder + * This constructor implicitly set spaceConsumed the same as length. + * spaceConsumed and length must be set explicitly with + * ContentSummary.Builder + * */ + @Deprecated public ContentSummary(long length, long fileCount, long directoryCount) { this(length, fileCount, directoryCount, -1L, length, -1L); } - /** Constructor */ + /** Constructor, deprecated by ContentSummary.Builder */ + @Deprecated public ContentSummary( long length, long fileCount, long directoryCount, long quota, long spaceConsumed, long spaceQuota) { @@ -58,6 +148,21 @@ public ContentSummary( this.spaceQuota = spaceQuota; } + /** Constructor for ContentSummary.Builder*/ + private ContentSummary( + long length, long fileCount, long directoryCount, long quota, + long spaceConsumed, long spaceQuota, long typeConsumed[], + long typeQuota[]) { + this.length = length; + this.fileCount = fileCount; + this.directoryCount = directoryCount; + this.quota = quota; + this.spaceConsumed = spaceConsumed; + this.spaceQuota = spaceQuota; + this.typeConsumed = typeConsumed; + this.typeQuota = typeQuota; + } + /** @return the length */ public long getLength() {return length;} @@ -70,12 +175,48 @@ public ContentSummary( /** Return the directory quota */ public long getQuota() {return quota;} - /** Retuns (disk) space consumed */ + /** Retuns storage space consumed */ public long getSpaceConsumed() {return spaceConsumed;} - /** Returns (disk) space quota */ + /** Returns storage space quota */ public long getSpaceQuota() {return spaceQuota;} - + + /** Returns storage type quota */ + public long getTypeQuota(StorageType type) { + return (typeQuota != null) ? typeQuota[type.ordinal()] : -1; + } + + /** Returns storage type consumed*/ + public long getTypeConsumed(StorageType type) { + return (typeConsumed != null) ? typeConsumed[type.ordinal()] : 0; + } + + /** Returns true if any storage type quota has been set*/ + public boolean isTypeQuotaSet() { + if (typeQuota == null) { + return false; + } + for (StorageType t : StorageType.getTypesSupportingQuota()) { + if (typeQuota[t.ordinal()] > 0) { + return true; + } + } + return false; + } + + /** Returns true if any storage type consumption information is available*/ + public boolean isTypeConsumedAvailable() { + if (typeConsumed == null) { + return false; + } + for (StorageType t : StorageType.getTypesSupportingQuota()) { + if (typeConsumed[t.ordinal()] > 0) { + return true; + } + } + return false; + } + @Override @InterfaceAudience.Private public void write(DataOutput out) throws IOException { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java index 09707c6c0792b..6b7f387b0fb62 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java @@ -29,7 +29,6 @@ import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Options.ChecksumOpt; -import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.security.token.Token; import org.apache.hadoop.util.Progressable; @@ -47,7 +46,7 @@ protected DelegateToFileSystem(URI theUri, FileSystem theFsImpl, Configuration conf, String supportedScheme, boolean authorityRequired) throws IOException, URISyntaxException { super(theUri, supportedScheme, authorityRequired, - FileSystem.getDefaultUri(conf).getPort()); + theFsImpl.getDefaultPort()); fsImpl = theFsImpl; fsImpl.initialize(theUri, conf); fsImpl.statistics = getStatistics(); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java index 27131449e7631..aad8be9e2cf79 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java @@ -1644,20 +1644,27 @@ public ContentSummary getContentSummary(Path f) UnsupportedFileSystemException, IOException { FileStatus status = FileContext.this.getFileStatus(f); if (status.isFile()) { - return new ContentSummary(status.getLen(), 1, 0); + long length = status.getLen(); + return new ContentSummary.Builder().length(length). + fileCount(1).directoryCount(0).spaceConsumed(length). + build(); } long[] summary = {0, 0, 1}; - RemoteIterator statusIterator = + RemoteIterator statusIterator = FileContext.this.listStatus(f); while(statusIterator.hasNext()) { FileStatus s = statusIterator.next(); + long length = s.getLen(); ContentSummary c = s.isDirectory() ? getContentSummary(s.getPath()) : - new ContentSummary(s.getLen(), 1, 0); + new ContentSummary.Builder().length(length).fileCount(1). + directoryCount(0).spaceConsumed(length).build(); summary[0] += c.getLength(); summary[1] += c.getFileCount(); summary[2] += c.getDirectoryCount(); } - return new ContentSummary(summary[0], summary[1], summary[2]); + return new ContentSummary.Builder().length(summary[0]). + fileCount(summary[1]).directoryCount(summary[2]). + spaceConsumed(summary[0]).build(); } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java index 42434f1945413..305fef2d60333 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java @@ -1467,18 +1467,24 @@ public ContentSummary getContentSummary(Path f) throws IOException { FileStatus status = getFileStatus(f); if (status.isFile()) { // f is a file - return new ContentSummary(status.getLen(), 1, 0); + long length = status.getLen(); + return new ContentSummary.Builder().length(length). + fileCount(1).directoryCount(0).spaceConsumed(length).build(); } // f is a directory long[] summary = {0, 0, 1}; for(FileStatus s : listStatus(f)) { + long length = s.getLen(); ContentSummary c = s.isDirectory() ? getContentSummary(s.getPath()) : - new ContentSummary(s.getLen(), 1, 0); + new ContentSummary.Builder().length(length). + fileCount(1).directoryCount(0).spaceConsumed(length).build(); summary[0] += c.getLength(); summary[1] += c.getFileCount(); summary[2] += c.getDirectoryCount(); } - return new ContentSummary(summary[0], summary[1], summary[2]); + return new ContentSummary.Builder().length(summary[0]). + fileCount(summary[1]).directoryCount(summary[2]). + spaceConsumed(summary[0]).build(); } final private static PathFilter DEFAULT_FILTER = new PathFilter() { @@ -2700,10 +2706,12 @@ private FileSystem getInternal(URI uri, Configuration conf, Key key) throws IOEx } synchronized void remove(Key key, FileSystem fs) { - if (map.containsKey(key) && fs == map.get(key)) { - map.remove(key); + FileSystem cachedFs = map.remove(key); + if (fs == cachedFs) { toAutoClose.remove(key); - } + } else if (cachedFs != null) { + map.put(key, cachedFs); + } } synchronized void closeAll() throws IOException { @@ -2730,7 +2738,8 @@ synchronized void closeAll(boolean onlyAutomatic) throws IOException { } //remove from cache - remove(key, fs); + map.remove(key); + toAutoClose.remove(key); if (fs != null) { try { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java index 947baa93e1e8e..e520a165c821f 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java @@ -256,7 +256,9 @@ public synchronized void joinElection(byte[] data) appData = new byte[data.length]; System.arraycopy(data, 0, appData, 0, data.length); - LOG.debug("Attempting active election for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Attempting active election for " + this); + } joinElectionInternal(); } @@ -406,9 +408,11 @@ public synchronized byte[] getActiveData() throws ActiveNotFoundException, public synchronized void processResult(int rc, String path, Object ctx, String name) { if (isStaleClient(ctx)) return; - LOG.debug("CreateNode result: " + rc + " for path: " + path - + " connectionState: " + zkConnectionState + - " for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("CreateNode result: " + rc + " for path: " + path + + " connectionState: " + zkConnectionState + + " for " + this); + } Code code = Code.get(rc); if (isSuccess(code)) { @@ -467,10 +471,11 @@ public synchronized void processResult(int rc, String path, Object ctx, assert wantToBeInElection : "Got a StatNode result after quitting election"; - - LOG.debug("StatNode result: " + rc + " for path: " + path - + " connectionState: " + zkConnectionState + " for " + this); - + + if (LOG.isDebugEnabled()) { + LOG.debug("StatNode result: " + rc + " for path: " + path + + " connectionState: " + zkConnectionState + " for " + this); + } Code code = Code.get(rc); if (isSuccess(code)) { @@ -535,10 +540,12 @@ private void reJoinElectionAfterFailureToBecomeActive() { synchronized void processWatchEvent(ZooKeeper zk, WatchedEvent event) { Event.EventType eventType = event.getType(); if (isStaleClient(zk)) return; - LOG.debug("Watcher event type: " + eventType + " with state:" - + event.getState() + " for path:" + event.getPath() - + " connectionState: " + zkConnectionState - + " for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Watcher event type: " + eventType + " with state:" + + event.getState() + " for path:" + event.getPath() + + " connectionState: " + zkConnectionState + + " for " + this); + } if (eventType == Event.EventType.None) { // the connection state has changed @@ -597,7 +604,9 @@ synchronized void processWatchEvent(ZooKeeper zk, WatchedEvent event) { monitorActiveStatus(); break; default: - LOG.debug("Unexpected node event: " + eventType + " for path: " + path); + if (LOG.isDebugEnabled()) { + LOG.debug("Unexpected node event: " + eventType + " for path: " + path); + } monitorActiveStatus(); } @@ -646,7 +655,9 @@ private void fatalError(String errorMessage) { private void monitorActiveStatus() { assert wantToBeInElection; - LOG.debug("Monitoring active leader for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Monitoring active leader for " + this); + } statRetryCount = 0; monitorLockNodeAsync(); } @@ -737,7 +748,9 @@ private boolean reEstablishSession() { int connectionRetryCount = 0; boolean success = false; while(!success && connectionRetryCount < maxRetryNum) { - LOG.debug("Establishing zookeeper connection for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Establishing zookeeper connection for " + this); + } try { createConnection(); success = true; @@ -765,7 +778,9 @@ private void createConnection() throws IOException, KeeperException { watcher = null; } zkClient = getNewZooKeeper(); - LOG.debug("Created new connection for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Created new connection for " + this); + } } @InterfaceAudience.Private @@ -773,7 +788,9 @@ public synchronized void terminateConnection() { if (zkClient == null) { return; } - LOG.debug("Terminating ZK connection for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Terminating ZK connection for " + this); + } ZooKeeper tempZk = zkClient; zkClient = null; watcher = null; @@ -800,8 +817,10 @@ private boolean becomeActive() { try { Stat oldBreadcrumbStat = fenceOldActive(); writeBreadCrumbNode(oldBreadcrumbStat); - - LOG.debug("Becoming active for " + this); + + if (LOG.isDebugEnabled()) { + LOG.debug("Becoming active for " + this); + } appClient.becomeActive(); state = State.ACTIVE; return true; @@ -906,7 +925,9 @@ public byte[] run() throws KeeperException, InterruptedException { private void becomeStandby() { if (state != State.STANDBY) { - LOG.debug("Becoming standby for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Becoming standby for " + this); + } state = State.STANDBY; appClient.becomeStandby(); } @@ -914,7 +935,9 @@ private void becomeStandby() { private void enterNeutralMode() { if (state != State.NEUTRAL) { - LOG.debug("Entering neutral mode for " + this); + if (LOG.isDebugEnabled()) { + LOG.debug("Entering neutral mode for " + this); + } state = State.NEUTRAL; appClient.enterNeutralMode(); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java index 80831e976aa57..0f1c22287e2dc 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java @@ -31,6 +31,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Properties; import javax.servlet.Filter; import javax.servlet.FilterChain; @@ -53,6 +54,11 @@ import org.apache.hadoop.conf.ConfServlet; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; +import org.apache.hadoop.security.AuthenticationFilterInitializer; +import org.apache.hadoop.security.authentication.util.FileSignerSecretProvider; +import org.apache.hadoop.security.authentication.util.RandomSignerSecretProvider; +import org.apache.hadoop.security.authentication.util.SignerSecretProvider; +import org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider; import org.apache.hadoop.security.ssl.SslSocketConnectorSecure; import org.apache.hadoop.jmx.JMXJsonServlet; import org.apache.hadoop.log.LogLevel; @@ -91,6 +97,8 @@ import com.google.common.collect.Lists; import com.sun.jersey.spi.container.servlet.ServletContainer; +import static org.apache.hadoop.security.authentication.server + .AuthenticationFilter.*; /** * Create a Jetty embedded server to answer http requests. The primary goal is * to serve up status information for the server. There are three contexts: @@ -160,6 +168,8 @@ public static class Builder { private boolean findPort; private String hostName; + private boolean disallowFallbackToRandomSignerSecretProvider; + private String authFilterConfigurationPrefix = "hadoop.http.authentication."; public Builder setName(String name){ this.name = name; @@ -254,6 +264,16 @@ public Builder setKeytabConfKey(String keytabConfKey) { return this; } + public Builder disallowFallbackToRandomSingerSecretProvider(boolean value) { + this.disallowFallbackToRandomSignerSecretProvider = value; + return this; + } + + public Builder authFilterConfigurationPrefix(String value) { + this.authFilterConfigurationPrefix = value; + return this; + } + public HttpServer2 build() throws IOException { Preconditions.checkNotNull(name, "name is not set"); Preconditions.checkState(!endpoints.isEmpty(), "No endpoints specified"); @@ -314,6 +334,18 @@ private HttpServer2(final Builder b) throws IOException { this.webServer = new Server(); this.adminsAcl = b.adminsAcl; this.webAppContext = createWebAppContext(b.name, b.conf, adminsAcl, appDir); + try { + SignerSecretProvider secretProvider = + constructSecretProvider(b, webAppContext.getServletContext()); + this.webAppContext.getServletContext().setAttribute + (AuthenticationFilter.SIGNER_SECRET_PROVIDER_ATTRIBUTE, + secretProvider); + } catch(IOException e) { + throw e; + } catch (Exception e) { + throw new IOException(e); + } + this.findPort = b.findPort; initializeWebServer(b.name, b.hostName, b.conf, b.pathSpecs); } @@ -405,9 +437,28 @@ private static WebAppContext createWebAppContext(String name, return ctx; } + private static SignerSecretProvider constructSecretProvider(final Builder b, + ServletContext ctx) + throws Exception { + final Configuration conf = b.conf; + Properties config = getFilterProperties(conf, + b.authFilterConfigurationPrefix); + return AuthenticationFilter.constructSecretProvider( + ctx, config, b.disallowFallbackToRandomSignerSecretProvider); + } + + private static Properties getFilterProperties(Configuration conf, String + prefix) { + Properties prop = new Properties(); + Map filterConfig = AuthenticationFilterInitializer + .getFilterConfigMap(conf, prefix); + prop.putAll(filterConfig); + return prop; + } + private static void addNoCacheFilter(WebAppContext ctxt) { defineFilter(ctxt, NO_CACHE_FILTER, NoCacheFilter.class.getName(), - Collections. emptyMap(), new String[] { "/*" }); + Collections. emptyMap(), new String[] { "/*" }); } @InterfaceAudience.Private @@ -594,15 +645,18 @@ public void addInternalServlet(String name, String pathSpec, public void addFilter(String name, String classname, Map parameters) { + FilterHolder filterHolder = getFilterHolder(name, classname, parameters); final String[] USER_FACING_URLS = { "*.html", "*.jsp" }; - defineFilter(webAppContext, name, classname, parameters, USER_FACING_URLS); + FilterMapping fmap = getFilterMapping(name, USER_FACING_URLS); + defineFilter(webAppContext, filterHolder, fmap); LOG.info( "Added filter " + name + " (class=" + classname + ") to context " + webAppContext.getDisplayName()); final String[] ALL_URLS = { "/*" }; + fmap = getFilterMapping(name, ALL_URLS); for (Map.Entry e : defaultContexts.entrySet()) { if (e.getValue()) { Context ctx = e.getKey(); - defineFilter(ctx, name, classname, parameters, ALL_URLS); + defineFilter(ctx, filterHolder, fmap); LOG.info("Added filter " + name + " (class=" + classname + ") to context " + ctx.getDisplayName()); } @@ -614,9 +668,11 @@ public void addFilter(String name, String classname, public void addGlobalFilter(String name, String classname, Map parameters) { final String[] ALL_URLS = { "/*" }; - defineFilter(webAppContext, name, classname, parameters, ALL_URLS); + FilterHolder filterHolder = getFilterHolder(name, classname, parameters); + FilterMapping fmap = getFilterMapping(name, ALL_URLS); + defineFilter(webAppContext, filterHolder, fmap); for (Context ctx : defaultContexts.keySet()) { - defineFilter(ctx, name, classname, parameters, ALL_URLS); + defineFilter(ctx, filterHolder, fmap); } LOG.info("Added global filter '" + name + "' (class=" + classname + ")"); } @@ -626,17 +682,35 @@ public void addGlobalFilter(String name, String classname, */ public static void defineFilter(Context ctx, String name, String classname, Map parameters, String[] urls) { + FilterHolder filterHolder = getFilterHolder(name, classname, parameters); + FilterMapping fmap = getFilterMapping(name, urls); + defineFilter(ctx, filterHolder, fmap); + } - FilterHolder holder = new FilterHolder(); - holder.setName(name); - holder.setClassName(classname); - holder.setInitParameters(parameters); + /** + * Define a filter for a context and set up default url mappings. + */ + private static void defineFilter(Context ctx, FilterHolder holder, + FilterMapping fmap) { + ServletHandler handler = ctx.getServletHandler(); + handler.addFilter(holder, fmap); + } + + private static FilterMapping getFilterMapping(String name, String[] urls) { FilterMapping fmap = new FilterMapping(); fmap.setPathSpecs(urls); fmap.setDispatches(Handler.ALL); fmap.setFilterName(name); - ServletHandler handler = ctx.getServletHandler(); - handler.addFilter(holder, fmap); + return fmap; + } + + private static FilterHolder getFilterHolder(String name, String classname, + Map parameters) { + FilterHolder holder = new FilterHolder(); + holder.setName(name); + holder.setClassName(classname); + holder.setInitParameters(parameters); + return holder; } /** diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java index 0f333bb36cef9..a973dc93340f3 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.java @@ -44,9 +44,6 @@ public class Bzip2Compressor implements Compressor { private static final Log LOG = LogFactory.getLog(Bzip2Compressor.class); - // HACK - Use this as a global lock in the JNI layer. - private static Class clazz = Bzip2Compressor.class; - private long stream; private int blockSize; private int workFactor; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java index 672090209db3f..3135165e8794c 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.java @@ -38,9 +38,6 @@ public class Bzip2Decompressor implements Decompressor { private static final Log LOG = LogFactory.getLog(Bzip2Decompressor.class); - // HACK - Use this as a global lock in the JNI layer. - private static Class clazz = Bzip2Decompressor.class; - private long stream; private boolean conserveMemory; private int directBufferSize; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java index b5db99f92dc23..ccfae8b3c36b9 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Compressor.java @@ -37,10 +37,6 @@ public class Lz4Compressor implements Compressor { LogFactory.getLog(Lz4Compressor.class.getName()); private static final int DEFAULT_DIRECT_BUFFER_SIZE = 64 * 1024; - // HACK - Use this as a global lock in the JNI layer - @SuppressWarnings({"unchecked", "unused"}) - private static Class clazz = Lz4Compressor.class; - private int directBufferSize; private Buffer compressedDirectBuf = null; private int uncompressedDirectBufLen; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java index 22a3118f5f9e5..685956cc1bf2d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.java @@ -36,10 +36,6 @@ public class Lz4Decompressor implements Decompressor { LogFactory.getLog(Lz4Compressor.class.getName()); private static final int DEFAULT_DIRECT_BUFFER_SIZE = 64 * 1024; - // HACK - Use this as a global lock in the JNI layer - @SuppressWarnings({"unchecked", "unused"}) - private static Class clazz = Lz4Decompressor.class; - private int directBufferSize; private Buffer compressedDirectBuf = null; private int compressedDirectBufLen; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java index ab45f250585b7..814718d99ef02 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java @@ -37,10 +37,6 @@ public class SnappyCompressor implements Compressor { LogFactory.getLog(SnappyCompressor.class.getName()); private static final int DEFAULT_DIRECT_BUFFER_SIZE = 64 * 1024; - // HACK - Use this as a global lock in the JNI layer - @SuppressWarnings({"unchecked", "unused"}) - private static Class clazz = SnappyCompressor.class; - private int directBufferSize; private Buffer compressedDirectBuf = null; private int uncompressedDirectBufLen; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java index b5f5acf86649e..dbffba811b14e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java @@ -37,10 +37,6 @@ public class SnappyDecompressor implements Decompressor { LogFactory.getLog(SnappyCompressor.class.getName()); private static final int DEFAULT_DIRECT_BUFFER_SIZE = 64 * 1024; - // HACK - Use this as a global lock in the JNI layer - @SuppressWarnings({"unchecked", "unused"}) - private static Class clazz = SnappyDecompressor.class; - private int directBufferSize; private Buffer compressedDirectBuf = null; private int compressedDirectBufLen; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java index 6799403b16039..b9550449f7fec 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java @@ -41,9 +41,6 @@ public class ZlibCompressor implements Compressor { private static final int DEFAULT_DIRECT_BUFFER_SIZE = 64*1024; - // HACK - Use this as a global lock in the JNI layer - private static Class clazz = ZlibCompressor.class; - private long stream; private CompressionLevel level; private CompressionStrategy strategy; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java index 89c879a03c875..d728fad817b74 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java @@ -34,10 +34,7 @@ */ public class ZlibDecompressor implements Decompressor { private static final int DEFAULT_DIRECT_BUFFER_SIZE = 64*1024; - - // HACK - Use this as a global lock in the JNI layer - private static Class clazz = ZlibDecompressor.class; - + private long stream; private CompressionHeader header; private int directBufferSize; diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java index 32558bcb8772d..97b715bcfa82b 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java @@ -384,7 +384,8 @@ private class Connection extends Thread { private final RetryPolicy connectionRetryPolicy; private final int maxRetriesOnSasl; private int maxRetriesOnSocketTimeouts; - private boolean tcpNoDelay; // if T then disable Nagle's Algorithm + private final boolean tcpNoDelay; // if T then disable Nagle's Algorithm + private final boolean tcpLowLatency; // if T then use low-delay QoS private boolean doPing; //do we need to send ping message private int pingInterval; // how often sends ping to the server in msecs private ByteArrayOutputStream pingRequest; // ping message @@ -413,6 +414,7 @@ public Connection(ConnectionId remoteId, int serviceClass) throws IOException { this.maxRetriesOnSasl = remoteId.getMaxRetriesOnSasl(); this.maxRetriesOnSocketTimeouts = remoteId.getMaxRetriesOnSocketTimeouts(); this.tcpNoDelay = remoteId.getTcpNoDelay(); + this.tcpLowLatency = remoteId.getTcpLowLatency(); this.doPing = remoteId.getDoPing(); if (doPing) { // construct a RPC header with the callId as the ping callId @@ -585,6 +587,20 @@ private synchronized void setupConnection() throws IOException { this.socket.setTcpNoDelay(tcpNoDelay); this.socket.setKeepAlive(true); + if (tcpLowLatency) { + /* + * This allows intermediate switches to shape IPC traffic + * differently from Shuffle/HDFS DataStreamer traffic. + * + * IPTOS_RELIABILITY (0x04) | IPTOS_LOWDELAY (0x10) + * + * Prefer to optimize connect() speed & response latency over net + * throughput. + */ + this.socket.setTrafficClass(0x04 | 0x10); + this.socket.setPerformancePreferences(1, 2, 0); + } + /* * Bind the socket to the host specified in the principal name of the * client, to ensure Server matching address of the client connection @@ -1549,6 +1565,7 @@ public static class ConnectionId { // the max. no. of retries for socket connections on time out exceptions private final int maxRetriesOnSocketTimeouts; private final boolean tcpNoDelay; // if T then disable Nagle's Algorithm + private final boolean tcpLowLatency; // if T then use low-delay QoS private final boolean doPing; //do we need to send ping message private final int pingInterval; // how often sends ping to the server in msecs private String saslQop; // here for testing @@ -1575,6 +1592,10 @@ public static class ConnectionId { this.tcpNoDelay = conf.getBoolean( CommonConfigurationKeysPublic.IPC_CLIENT_TCPNODELAY_KEY, CommonConfigurationKeysPublic.IPC_CLIENT_TCPNODELAY_DEFAULT); + this.tcpLowLatency = conf.getBoolean( + CommonConfigurationKeysPublic.IPC_CLIENT_LOW_LATENCY, + CommonConfigurationKeysPublic.IPC_CLIENT_LOW_LATENCY_DEFAULT + ); this.doPing = conf.getBoolean( CommonConfigurationKeys.IPC_CLIENT_PING_KEY, CommonConfigurationKeys.IPC_CLIENT_PING_DEFAULT); @@ -1610,11 +1631,17 @@ public int getMaxRetriesOnSasl() { public int getMaxRetriesOnSocketTimeouts() { return maxRetriesOnSocketTimeouts; } - + + /** disable nagle's algorithm */ boolean getTcpNoDelay() { return tcpNoDelay; } - + + /** use low-latency QoS bits over TCP */ + boolean getTcpLowLatency() { + return tcpLowLatency; + } + boolean getDoPing() { return doPing; } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java index 8ada0fff9801f..47661758bc3a5 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java @@ -876,9 +876,12 @@ void registerProtocolAndImpl(RpcKind rpcKind, Class protocolClass, getProtocolImplMap(rpcKind).put(new ProtoNameVer(protocolName, version), new ProtoClassProtoImpl(protocolClass, protocolImpl)); - LOG.debug("RpcKind = " + rpcKind + " Protocol Name = " + protocolName + " version=" + version + - " ProtocolImpl=" + protocolImpl.getClass().getName() + - " protocolClass=" + protocolClass.getName()); + if (LOG.isDebugEnabled()) { + LOG.debug("RpcKind = " + rpcKind + " Protocol Name = " + protocolName + + " version=" + version + + " ProtocolImpl=" + protocolImpl.getClass().getName() + + " protocolClass=" + protocolClass.getName()); + } } static class VerProtocolImpl { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java index 893e0eb8903bc..9aa362ef0228d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java @@ -234,9 +234,11 @@ public static void registerProtocolEngine(RPC.RpcKind rpcKind, throw new IllegalArgumentException("ReRegistration of rpcKind: " + rpcKind); } - LOG.debug("rpcKind=" + rpcKind + - ", rpcRequestWrapperClass=" + rpcRequestWrapperClass + - ", rpcInvoker=" + rpcInvoker); + if (LOG.isDebugEnabled()) { + LOG.debug("rpcKind=" + rpcKind + + ", rpcRequestWrapperClass=" + rpcRequestWrapperClass + + ", rpcInvoker=" + rpcInvoker); + } } public Class getRpcRequestWrapper( @@ -1324,10 +1326,15 @@ private void saslProcess(RpcSaslProto saslMessage) saslResponse = processSaslMessage(saslMessage); } catch (IOException e) { rpcMetrics.incrAuthenticationFailures(); + if (LOG.isDebugEnabled()) { + LOG.debug(StringUtils.stringifyException(e)); + } // attempting user could be null + IOException tce = (IOException) getTrueCause(e); AUDITLOG.warn(AUTH_FAILED_FOR + this.toString() + ":" - + attemptingUser + " (" + e.getLocalizedMessage() + ")"); - throw (IOException) getTrueCause(e); + + attemptingUser + " (" + e.getLocalizedMessage() + + ") with true cause: (" + tce.getLocalizedMessage() + ")"); + throw tce; } if (saslServer != null && saslServer.isComplete()) { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java index f35ad1816f2b5..9ad2d39ea93b6 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/GangliaContext31.java @@ -82,8 +82,10 @@ protected void emitMetric(String name, String type, String value) return; } - LOG.debug("Emitting metric " + name + ", type " + type + ", value " + - value + " from hostname" + hostName); + if (LOG.isDebugEnabled()) { + LOG.debug("Emitting metric " + name + ", type " + type + ", value " + + value + " from hostname" + hostName); + } String units = getUnits(name); int slope = getSlope(name); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java index ba377570efb4e..d794e8eeed384 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java @@ -151,4 +151,8 @@ public void resetMinMax() { minMax.reset(); } + @Override + public String toString() { + return lastStat().toString(); + } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleStat.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleStat.java index 589062a691cd2..cd9aaa49e100c 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleStat.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleStat.java @@ -137,6 +137,19 @@ public double max() { return minmax.max(); } + @Override + public String toString() { + try { + return "Samples = " + numSamples() + + " Min = " + min() + + " Mean = " + mean() + + " Std Dev = " + stddev() + + " Max = " + max(); + } catch (Throwable t) { + return super.toString(); + } + } + /** * Helper to keep running min/max */ diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java index 8c617dc8c8209..03b52e0f8c45e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/unix/DomainSocketWatcher.java @@ -238,7 +238,8 @@ private static class FdSet { */ private boolean kicked = false; - public DomainSocketWatcher(int interruptCheckPeriodMs) throws IOException { + public DomainSocketWatcher(int interruptCheckPeriodMs, String src) + throws IOException { if (loadingFailureReason != null) { throw new UnsupportedOperationException(loadingFailureReason); } @@ -246,8 +247,9 @@ public DomainSocketWatcher(int interruptCheckPeriodMs) throws IOException { this.interruptCheckPeriodMs = interruptCheckPeriodMs; notificationSockets = DomainSocket.socketpair(); watcherThread.setDaemon(true); - watcherThread.setUncaughtExceptionHandler( - new Thread.UncaughtExceptionHandler() { + watcherThread.setName(src + " DomainSocketWatcher"); + watcherThread + .setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() { @Override public void uncaughtException(Thread thread, Throwable t) { LOG.error(thread + " terminating on unexpected exception", t); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java index 43d1b66d44f6d..ca221f5b3dcd2 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java @@ -17,7 +17,6 @@ */ package org.apache.hadoop.security; -import com.google.common.base.Charsets; import org.apache.hadoop.http.HttpServer2; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; import org.apache.hadoop.conf.Configuration; @@ -25,11 +24,7 @@ import org.apache.hadoop.http.FilterInitializer; import org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler; -import java.io.FileInputStream; -import java.io.FileReader; import java.io.IOException; -import java.io.InputStreamReader; -import java.io.Reader; import java.util.HashMap; import java.util.Map; @@ -50,8 +45,6 @@ public class AuthenticationFilterInitializer extends FilterInitializer { static final String PREFIX = "hadoop.http.authentication."; - static final String SIGNATURE_SECRET_FILE = AuthenticationFilter.SIGNATURE_SECRET + ".file"; - /** * Initializes hadoop-auth AuthenticationFilter. *

@@ -63,6 +56,15 @@ public class AuthenticationFilterInitializer extends FilterInitializer { */ @Override public void initFilter(FilterContainer container, Configuration conf) { + Map filterConfig = getFilterConfigMap(conf, PREFIX); + + container.addFilter("authentication", + AuthenticationFilter.class.getName(), + filterConfig); + } + + public static Map getFilterConfigMap(Configuration conf, + String prefix) { Map filterConfig = new HashMap(); //setting the cookie path to root '/' so it is used for all resources. @@ -70,32 +72,13 @@ public void initFilter(FilterContainer container, Configuration conf) { for (Map.Entry entry : conf) { String name = entry.getKey(); - if (name.startsWith(PREFIX)) { + if (name.startsWith(prefix)) { String value = conf.get(name); - name = name.substring(PREFIX.length()); + name = name.substring(prefix.length()); filterConfig.put(name, value); } } - String signatureSecretFile = filterConfig.get(SIGNATURE_SECRET_FILE); - if (signatureSecretFile == null) { - throw new RuntimeException("Undefined property: " + SIGNATURE_SECRET_FILE); - } - - StringBuilder secret = new StringBuilder(); - try (Reader reader = new InputStreamReader( - new FileInputStream(signatureSecretFile), Charsets.UTF_8)) { - int c = reader.read(); - while (c > -1) { - secret.append((char)c); - c = reader.read(); - } - reader.close(); - filterConfig.put(AuthenticationFilter.SIGNATURE_SECRET, secret.toString()); - } catch (IOException ex) { - throw new RuntimeException("Could not read HTTP signature secret file: " + signatureSecretFile); - } - //Resolve _HOST into bind address String bindAddress = conf.get(HttpServer2.BIND_ADDRESS); String principal = filterConfig.get(KerberosAuthenticationHandler.PRINCIPAL); @@ -108,10 +91,7 @@ public void initFilter(FilterContainer container, Configuration conf) { } filterConfig.put(KerberosAuthenticationHandler.PRINCIPAL, principal); } - - container.addFilter("authentication", - AuthenticationFilter.class.getName(), - filterConfig); + return filterConfig; } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java index d463ac7a2be3f..df91b701bcd15 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/LdapGroupsMapping.java @@ -149,6 +149,14 @@ public class LdapGroupsMapping public static final String GROUP_NAME_ATTR_KEY = LDAP_CONFIG_PREFIX + ".search.attr.group.name"; public static final String GROUP_NAME_ATTR_DEFAULT = "cn"; + /* + * Posix attributes + */ + public static final String POSIX_UIDNUMBER = "uidNumber"; + public static final String POSIX_GIDNUMBER = "gidNumber"; + public static final String POSIX_GROUP = "posixGroup"; + public static final String POSIX_ACCOUNT = "posixAccount"; + /* * LDAP {@link SearchControls} attribute to set the time limit * for an invoked directory search. Prevents infinite wait cases. @@ -178,6 +186,7 @@ public class LdapGroupsMapping private String userSearchFilter; private String groupMemberAttr; private String groupNameAttr; + private boolean isPosix; public static int RECONNECT_RETRY_COUNT = 3; @@ -242,15 +251,40 @@ List doGetGroups(String user) throws NamingException { SearchResult result = results.nextElement(); String userDn = result.getNameInNamespace(); - NamingEnumeration groupResults = - ctx.search(baseDN, - "(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))", - new Object[]{userDn}, - SEARCH_CONTROLS); - while (groupResults.hasMoreElements()) { - SearchResult groupResult = groupResults.nextElement(); - Attribute groupName = groupResult.getAttributes().get(groupNameAttr); - groups.add(groupName.get().toString()); + NamingEnumeration groupResults = null; + + if (isPosix) { + String gidNumber = null; + String uidNumber = null; + Attribute gidAttribute = result.getAttributes().get(POSIX_GIDNUMBER); + Attribute uidAttribute = result.getAttributes().get(POSIX_UIDNUMBER); + if (gidAttribute != null) { + gidNumber = gidAttribute.get().toString(); + } + if (uidAttribute != null) { + uidNumber = uidAttribute.get().toString(); + } + if (uidNumber != null && gidNumber != null) { + groupResults = + ctx.search(baseDN, + "(&"+ groupSearchFilter + "(|(" + POSIX_GIDNUMBER + "={0})" + + "(" + groupMemberAttr + "={1})))", + new Object[] { gidNumber, uidNumber }, + SEARCH_CONTROLS); + } + } else { + groupResults = + ctx.search(baseDN, + "(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))", + new Object[]{userDn}, + SEARCH_CONTROLS); + } + if (groupResults != null) { + while (groupResults.hasMoreElements()) { + SearchResult groupResult = groupResults.nextElement(); + Attribute groupName = groupResult.getAttributes().get(groupNameAttr); + groups.add(groupName.get().toString()); + } } } @@ -334,6 +368,8 @@ public synchronized void setConf(Configuration conf) { conf.get(GROUP_SEARCH_FILTER_KEY, GROUP_SEARCH_FILTER_DEFAULT); userSearchFilter = conf.get(USER_SEARCH_FILTER_KEY, USER_SEARCH_FILTER_DEFAULT); + isPosix = groupSearchFilter.contains(POSIX_GROUP) && userSearchFilter + .contains(POSIX_ACCOUNT); groupMemberAttr = conf.get(GROUP_MEMBERSHIP_ATTR_KEY, GROUP_MEMBERSHIP_ATTR_DEFAULT); groupNameAttr = diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java index e8a721f65b97b..265ed164e080a 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialShell.java @@ -81,7 +81,7 @@ public int run(String[] args) throws Exception { * 

    * % hadoop credential create alias [-provider providerPath]
    * % hadoop credential list [-provider providerPath]
-   * % hadoop credential delete alias [-provider providerPath] [-i]
+   * % hadoop credential delete alias [-provider providerPath] [-f]
    *
* @param args * @return 0 if the argument(s) were recognized, 1 otherwise diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java index 05958a058a3c1..5e5cebbc6291f 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java @@ -22,6 +22,7 @@ import org.apache.commons.io.IOUtils; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; @@ -98,11 +99,8 @@ private JavaKeyStoreProvider(URI uri, Configuration conf) throws IOException { ClassLoader cl = Thread.currentThread().getContextClassLoader(); URL pwdFile = cl.getResource(pwFile); if (pwdFile != null) { - InputStream is = pwdFile.openStream(); - try { + try (InputStream is = pwdFile.openStream()) { password = IOUtils.toString(is).trim().toCharArray(); - } finally { - is.close(); } } } @@ -110,6 +108,7 @@ private JavaKeyStoreProvider(URI uri, Configuration conf) throws IOException { if (password == null) { password = KEYSTORE_PASSWORD_DEFAULT.toCharArray(); } + try { keyStore = KeyStore.getInstance(SCHEME_NAME); if (fs.exists(path)) { @@ -118,7 +117,9 @@ private JavaKeyStoreProvider(URI uri, Configuration conf) throws IOException { FileStatus s = fs.getFileStatus(path); permissions = s.getPermission(); - keyStore.load(fs.open(path), password); + try (FSDataInputStream in = fs.open(path)) { + keyStore.load(in, password); + } } else { permissions = new FsPermission("700"); // required to create an empty keystore. *sigh* @@ -257,8 +258,7 @@ public void flush() throws IOException { return; } // write out the keystore - FSDataOutputStream out = FileSystem.create(fs, path, permissions); - try { + try (FSDataOutputStream out = FileSystem.create(fs, path, permissions)) { keyStore.store(out, password); } catch (KeyStoreException e) { throw new IOException("Can't store keystore " + this, e); @@ -268,7 +268,6 @@ public void flush() throws IOException { throw new IOException("Certificate exception storing keystore " + this, e); } - out.close(); changed = false; } finally { diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java index 609c71f5f5620..41634a8f44589 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/ssl/FileBasedKeyStoresFactory.java @@ -164,7 +164,9 @@ public void init(SSLFactory.Mode mode) // configuration property for key password. keystoreKeyPassword = getPassword( conf, keyPasswordProperty, keystorePassword); - LOG.debug(mode.toString() + " KeyStore: " + keystoreLocation); + if (LOG.isDebugEnabled()) { + LOG.debug(mode.toString() + " KeyStore: " + keystoreLocation); + } InputStream is = new FileInputStream(keystoreLocation); try { @@ -172,7 +174,9 @@ public void init(SSLFactory.Mode mode) } finally { is.close(); } - LOG.debug(mode.toString() + " Loaded KeyStore: " + keystoreLocation); + if (LOG.isDebugEnabled()) { + LOG.debug(mode.toString() + " Loaded KeyStore: " + keystoreLocation); + } } else { keystore.load(null, null); } @@ -204,18 +208,24 @@ public void init(SSLFactory.Mode mode) resolvePropertyName(mode, SSL_TRUSTSTORE_RELOAD_INTERVAL_TPL_KEY), DEFAULT_SSL_TRUSTSTORE_RELOAD_INTERVAL); - LOG.debug(mode.toString() + " TrustStore: " + truststoreLocation); + if (LOG.isDebugEnabled()) { + LOG.debug(mode.toString() + " TrustStore: " + truststoreLocation); + } trustManager = new ReloadingX509TrustManager(truststoreType, truststoreLocation, truststorePassword, truststoreReloadInterval); trustManager.init(); - LOG.debug(mode.toString() + " Loaded TrustStore: " + truststoreLocation); + if (LOG.isDebugEnabled()) { + LOG.debug(mode.toString() + " Loaded TrustStore: " + truststoreLocation); + } trustManagers = new TrustManager[]{trustManager}; } else { - LOG.debug("The property '" + locationProperty + "' has not been set, " + - "no TrustStore will be loaded"); + if (LOG.isDebugEnabled()) { + LOG.debug("The property '" + locationProperty + "' has not been set, " + + "no TrustStore will be loaded"); + } trustManagers = null; } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java index ec522dcff89bf..73c3ab82390da 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java @@ -55,6 +55,7 @@ import org.apache.hadoop.security.token.delegation.web.DelegationTokenManager; import org.apache.zookeeper.CreateMode; import org.apache.zookeeper.KeeperException; +import org.apache.zookeeper.KeeperException.NoNodeException; import org.apache.zookeeper.ZooDefs.Perms; import org.apache.zookeeper.client.ZooKeeperSaslClient; import org.apache.zookeeper.data.ACL; @@ -709,7 +710,15 @@ protected void removeStoredMasterKey(DelegationKey key) { try { if (zkClient.checkExists().forPath(nodeRemovePath) != null) { while(zkClient.checkExists().forPath(nodeRemovePath) != null){ - zkClient.delete().guaranteed().forPath(nodeRemovePath); + try { + zkClient.delete().guaranteed().forPath(nodeRemovePath); + } catch (NoNodeException nne) { + // It is possible that the node might be deleted between the + // check and the actual delete.. which might lead to an + // exception that can bring down the daemon running this + // SecretManager + LOG.debug("Node already deleted by peer " + nodeRemovePath); + } } } else { LOG.debug("Attempted to delete a non-existing znode " + nodeRemovePath); @@ -761,7 +770,15 @@ protected void removeStoredToken(TokenIdent ident) try { if (zkClient.checkExists().forPath(nodeRemovePath) != null) { while(zkClient.checkExists().forPath(nodeRemovePath) != null){ - zkClient.delete().guaranteed().forPath(nodeRemovePath); + try { + zkClient.delete().guaranteed().forPath(nodeRemovePath); + } catch (NoNodeException nne) { + // It is possible that the node might be deleted between the + // check and the actual delete.. which might lead to an + // exception that can bring down the daemon running this + // SecretManager + LOG.debug("Node already deleted by peer " + nodeRemovePath); + } } } else { LOG.debug("Attempted to remove a non-existing znode " + nodeRemovePath); diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java index 01ba76d849a15..f2de0a0ff9186 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java @@ -134,20 +134,27 @@ public synchronized void loadSpanReceivers(Configuration conf) { String[] receiverNames = config.getTrimmedStrings(SPAN_RECEIVERS_CONF_KEY); if (receiverNames == null || receiverNames.length == 0) { + if (LOG.isTraceEnabled()) { + LOG.trace("No span receiver names found in " + + SPAN_RECEIVERS_CONF_KEY + "."); + } return; } // It's convenient to have each daemon log to a random trace file when // testing. if (config.get(LOCAL_FILE_SPAN_RECEIVER_PATH) == null) { - config.set(LOCAL_FILE_SPAN_RECEIVER_PATH, - getUniqueLocalTraceFileName()); + String uniqueFile = getUniqueLocalTraceFileName(); + config.set(LOCAL_FILE_SPAN_RECEIVER_PATH, uniqueFile); + if (LOG.isTraceEnabled()) { + LOG.trace("Set " + LOCAL_FILE_SPAN_RECEIVER_PATH + " to " + uniqueFile); + } } for (String className : receiverNames) { try { SpanReceiver rcvr = loadInstance(className, EMPTY); Trace.addReceiver(rcvr); receivers.put(highestId++, rcvr); - LOG.info("SpanReceiver " + className + " was loaded successfully."); + LOG.info("Loaded SpanReceiver " + className + " successfully."); } catch (IOException e) { LOG.error("Failed to load SpanReceiver", e); } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java index 0a46a7a81caf9..925aad680aab6 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java @@ -513,7 +513,7 @@ public static void printGenericCommandUsage(PrintStream out) { "specify comma separated archives to be unarchived" + " on the compute machines.\n"); out.println("The general command line syntax is"); - out.println("bin/hadoop command [genericOptions] [commandOptions]\n"); + out.println("command [genericOptions] [commandOptions]\n"); } } diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Time.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Time.java index 347cdc5afa9d7..b98892332ed9c 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Time.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Time.java @@ -27,6 +27,11 @@ @InterfaceStability.Unstable public final class Time { + /** + * number of nano seconds in 1 millisecond + */ + private static final long NANOSECONDS_PER_MILLISECOND = 1000000; + /** * Current system time. Do not use this to calculate a duration or interval * to sleep, because it will be broken by settimeofday. Instead, use @@ -45,8 +50,6 @@ public static long now() { * @return a monotonic clock that counts in milliseconds. */ public static long monotonicNow() { - final long NANOSECONDS_PER_MILLISECOND = 1000000; - return System.nanoTime() / NANOSECONDS_PER_MILLISECOND; } } diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c index 6c31d10599c89..8f0c06d17a172 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c @@ -29,6 +29,10 @@ #include #endif +#if defined(__FreeBSD__) +#include +#endif + #ifdef WINDOWS #include #endif @@ -274,7 +278,19 @@ static void pthreads_locking_callback(int mode, int type, char *file, int line) static unsigned long pthreads_thread_id(void) { - return (unsigned long)syscall(SYS_gettid); + unsigned long thread_id = 0; +#if defined(__linux__) + thread_id = (unsigned long)syscall(SYS_gettid); +#elif defined(__FreeBSD__) + thread_id = (unsigned long)pthread_getthreadid_np(); +#elif defined(__sun) + thread_id = (unsigned long)pthread_self(); +#elif defined(__APPLE__) + (void)pthread_threadid_np(pthread_self(), &thread_id); +#else +#error "Platform not supported" +#endif + return thread_id; } #endif /* UNIX */ diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c index ef81bea2ce009..a92123efcd98b 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Compressor.c @@ -25,7 +25,6 @@ #include "org_apache_hadoop_io_compress_bzip2.h" #include "org_apache_hadoop_io_compress_bzip2_Bzip2Compressor.h" -static jfieldID Bzip2Compressor_clazz; static jfieldID Bzip2Compressor_stream; static jfieldID Bzip2Compressor_uncompressedDirectBuf; static jfieldID Bzip2Compressor_uncompressedDirectBufOff; @@ -74,8 +73,6 @@ Java_org_apache_hadoop_io_compress_bzip2_Bzip2Compressor_initIDs( "BZ2_bzCompressEnd"); // Initialize the requisite fieldIds. - Bzip2Compressor_clazz = (*env)->GetStaticFieldID(env, class, "clazz", - "Ljava/lang/Class;"); Bzip2Compressor_stream = (*env)->GetFieldID(env, class, "stream", "J"); Bzip2Compressor_finish = (*env)->GetFieldID(env, class, "finish", "Z"); Bzip2Compressor_finished = (*env)->GetFieldID(env, class, "finished", "Z"); @@ -155,9 +152,7 @@ Java_org_apache_hadoop_io_compress_bzip2_Bzip2Compressor_deflateBytesDirect( return (jint)0; } - jobject clazz = (*env)->GetStaticObjectField(env, this, - Bzip2Compressor_clazz); - jobject uncompressed_direct_buf = (*env)->GetObjectField(env, this, + jobject uncompressed_direct_buf = (*env)->GetObjectField(env, this, Bzip2Compressor_uncompressedDirectBuf); jint uncompressed_direct_buf_off = (*env)->GetIntField(env, this, Bzip2Compressor_uncompressedDirectBufOff); @@ -173,12 +168,10 @@ Java_org_apache_hadoop_io_compress_bzip2_Bzip2Compressor_deflateBytesDirect( Bzip2Compressor_finish); // Get the input and output direct buffers. - LOCK_CLASS(env, clazz, "Bzip2Compressor"); char* uncompressed_bytes = (*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); char* compressed_bytes = (*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Bzip2Compressor"); if (!uncompressed_bytes || !compressed_bytes) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c index ad9bcb72c6c91..3d9fc084f7849 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/bzip2/Bzip2Decompressor.c @@ -25,7 +25,6 @@ #include "org_apache_hadoop_io_compress_bzip2.h" #include "org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor.h" -static jfieldID Bzip2Decompressor_clazz; static jfieldID Bzip2Decompressor_stream; static jfieldID Bzip2Decompressor_compressedDirectBuf; static jfieldID Bzip2Decompressor_compressedDirectBufOff; @@ -73,8 +72,6 @@ Java_org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor_initIDs( "BZ2_bzDecompressEnd"); // Initialize the requisite fieldIds. - Bzip2Decompressor_clazz = (*env)->GetStaticFieldID(env, class, "clazz", - "Ljava/lang/Class;"); Bzip2Decompressor_stream = (*env)->GetFieldID(env, class, "stream", "J"); Bzip2Decompressor_finished = (*env)->GetFieldID(env, class, "finished", "Z"); @@ -144,8 +141,6 @@ Java_org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor_inflateBytesDirect( return (jint)0; } - jobject clazz = (*env)->GetStaticObjectField(env, this, - Bzip2Decompressor_clazz); jarray compressed_direct_buf = (jarray)(*env)->GetObjectField(env, this, Bzip2Decompressor_compressedDirectBuf); jint compressed_direct_buf_off = (*env)->GetIntField(env, this, @@ -159,12 +154,10 @@ Java_org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor_inflateBytesDirect( Bzip2Decompressor_directBufferSize); // Get the input and output direct buffers. - LOCK_CLASS(env, clazz, "Bzip2Decompressor"); char* compressed_bytes = (*env)->GetDirectBufferAddress(env, compressed_direct_buf); char* uncompressed_bytes = (*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Bzip2Decompressor"); if (!compressed_bytes || !uncompressed_bytes) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c index 9f14312cde394..f742384efc743 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Compressor.c @@ -27,7 +27,6 @@ #include "lz4hc.h" -static jfieldID Lz4Compressor_clazz; static jfieldID Lz4Compressor_uncompressedDirectBuf; static jfieldID Lz4Compressor_uncompressedDirectBufLen; static jfieldID Lz4Compressor_compressedDirectBuf; @@ -37,8 +36,6 @@ static jfieldID Lz4Compressor_directBufferSize; JNIEXPORT void JNICALL Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs (JNIEnv *env, jclass clazz){ - Lz4Compressor_clazz = (*env)->GetStaticFieldID(env, clazz, "clazz", - "Ljava/lang/Class;"); Lz4Compressor_uncompressedDirectBuf = (*env)->GetFieldID(env, clazz, "uncompressedDirectBuf", "Ljava/nio/Buffer;"); @@ -57,25 +54,20 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_comp char *compressed_bytes; // Get members of Lz4Compressor - jobject clazz = (*env)->GetStaticObjectField(env, thisj, Lz4Compressor_clazz); jobject uncompressed_direct_buf = (*env)->GetObjectField(env, thisj, Lz4Compressor_uncompressedDirectBuf); jint uncompressed_direct_buf_len = (*env)->GetIntField(env, thisj, Lz4Compressor_uncompressedDirectBufLen); jobject compressed_direct_buf = (*env)->GetObjectField(env, thisj, Lz4Compressor_compressedDirectBuf); jint compressed_direct_buf_len = (*env)->GetIntField(env, thisj, Lz4Compressor_directBufferSize); // Get the input direct buffer - LOCK_CLASS(env, clazz, "Lz4Compressor"); uncompressed_bytes = (const char*)(*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Lz4Compressor"); if (uncompressed_bytes == 0) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "Lz4Compressor"); compressed_bytes = (char *)(*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Lz4Compressor"); if (compressed_bytes == 0) { return (jint)0; @@ -104,25 +96,20 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_comp char* compressed_bytes = NULL; // Get members of Lz4Compressor - jobject clazz = (*env)->GetStaticObjectField(env, thisj, Lz4Compressor_clazz); jobject uncompressed_direct_buf = (*env)->GetObjectField(env, thisj, Lz4Compressor_uncompressedDirectBuf); jint uncompressed_direct_buf_len = (*env)->GetIntField(env, thisj, Lz4Compressor_uncompressedDirectBufLen); jobject compressed_direct_buf = (*env)->GetObjectField(env, thisj, Lz4Compressor_compressedDirectBuf); jint compressed_direct_buf_len = (*env)->GetIntField(env, thisj, Lz4Compressor_directBufferSize); // Get the input direct buffer - LOCK_CLASS(env, clazz, "Lz4Compressor"); uncompressed_bytes = (const char*)(*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Lz4Compressor"); if (uncompressed_bytes == 0) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "Lz4Compressor"); compressed_bytes = (char *)(*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Lz4Compressor"); if (compressed_bytes == 0) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c index 2b8c91c348b32..cdeaa315d1e59 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/lz4/Lz4Decompressor.c @@ -25,7 +25,6 @@ #include "lz4.h" -static jfieldID Lz4Decompressor_clazz; static jfieldID Lz4Decompressor_compressedDirectBuf; static jfieldID Lz4Decompressor_compressedDirectBufLen; static jfieldID Lz4Decompressor_uncompressedDirectBuf; @@ -34,8 +33,6 @@ static jfieldID Lz4Decompressor_directBufferSize; JNIEXPORT void JNICALL Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs (JNIEnv *env, jclass clazz){ - Lz4Decompressor_clazz = (*env)->GetStaticFieldID(env, clazz, "clazz", - "Ljava/lang/Class;"); Lz4Decompressor_compressedDirectBuf = (*env)->GetFieldID(env,clazz, "compressedDirectBuf", "Ljava/nio/Buffer;"); @@ -54,25 +51,20 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_de char *uncompressed_bytes; // Get members of Lz4Decompressor - jobject clazz = (*env)->GetStaticObjectField(env,thisj, Lz4Decompressor_clazz); jobject compressed_direct_buf = (*env)->GetObjectField(env,thisj, Lz4Decompressor_compressedDirectBuf); jint compressed_direct_buf_len = (*env)->GetIntField(env,thisj, Lz4Decompressor_compressedDirectBufLen); jobject uncompressed_direct_buf = (*env)->GetObjectField(env,thisj, Lz4Decompressor_uncompressedDirectBuf); size_t uncompressed_direct_buf_len = (*env)->GetIntField(env, thisj, Lz4Decompressor_directBufferSize); // Get the input direct buffer - LOCK_CLASS(env, clazz, "Lz4Decompressor"); compressed_bytes = (const char*)(*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Lz4Decompressor"); if (compressed_bytes == 0) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "Lz4Decompressor"); uncompressed_bytes = (char *)(*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "Lz4Decompressor"); if (uncompressed_bytes == 0) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c index fe827f02dea75..9a09f078d8260 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyCompressor.c @@ -38,7 +38,6 @@ #define JINT_MAX 0x7fffffff -static jfieldID SnappyCompressor_clazz; static jfieldID SnappyCompressor_uncompressedDirectBuf; static jfieldID SnappyCompressor_uncompressedDirectBufLen; static jfieldID SnappyCompressor_compressedDirectBuf; @@ -84,8 +83,6 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyCompresso LOAD_DYNAMIC_SYMBOL(__dlsym_snappy_compress, dlsym_snappy_compress, env, libsnappy, "snappy_compress"); #endif - SnappyCompressor_clazz = (*env)->GetStaticFieldID(env, clazz, "clazz", - "Ljava/lang/Class;"); SnappyCompressor_uncompressedDirectBuf = (*env)->GetFieldID(env, clazz, "uncompressedDirectBuf", "Ljava/nio/Buffer;"); @@ -104,7 +101,6 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyCompresso char* compressed_bytes; snappy_status ret; // Get members of SnappyCompressor - jobject clazz = (*env)->GetStaticObjectField(env, thisj, SnappyCompressor_clazz); jobject uncompressed_direct_buf = (*env)->GetObjectField(env, thisj, SnappyCompressor_uncompressedDirectBuf); jint uncompressed_direct_buf_len = (*env)->GetIntField(env, thisj, SnappyCompressor_uncompressedDirectBufLen); jobject compressed_direct_buf = (*env)->GetObjectField(env, thisj, SnappyCompressor_compressedDirectBuf); @@ -112,18 +108,14 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyCompresso size_t buf_len; // Get the input direct buffer - LOCK_CLASS(env, clazz, "SnappyCompressor"); uncompressed_bytes = (const char*)(*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "SnappyCompressor"); if (uncompressed_bytes == 0) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "SnappyCompressor"); compressed_bytes = (char *)(*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "SnappyCompressor"); if (compressed_bytes == 0) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c index d1fd13c9e547f..69ec1017526fd 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.c @@ -31,7 +31,6 @@ #include "org_apache_hadoop_io_compress_snappy_SnappyDecompressor.h" -static jfieldID SnappyDecompressor_clazz; static jfieldID SnappyDecompressor_compressedDirectBuf; static jfieldID SnappyDecompressor_compressedDirectBufLen; static jfieldID SnappyDecompressor_uncompressedDirectBuf; @@ -79,8 +78,6 @@ JNIEXPORT void JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyDecompres LOAD_DYNAMIC_SYMBOL(__dlsym_snappy_uncompress, dlsym_snappy_uncompress, env, libsnappy, "snappy_uncompress"); #endif - SnappyDecompressor_clazz = (*env)->GetStaticFieldID(env, clazz, "clazz", - "Ljava/lang/Class;"); SnappyDecompressor_compressedDirectBuf = (*env)->GetFieldID(env,clazz, "compressedDirectBuf", "Ljava/nio/Buffer;"); @@ -99,25 +96,20 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_snappy_SnappyDecompres char* uncompressed_bytes = NULL; snappy_status ret; // Get members of SnappyDecompressor - jobject clazz = (*env)->GetStaticObjectField(env,thisj, SnappyDecompressor_clazz); jobject compressed_direct_buf = (*env)->GetObjectField(env,thisj, SnappyDecompressor_compressedDirectBuf); jint compressed_direct_buf_len = (*env)->GetIntField(env,thisj, SnappyDecompressor_compressedDirectBufLen); jobject uncompressed_direct_buf = (*env)->GetObjectField(env,thisj, SnappyDecompressor_uncompressedDirectBuf); size_t uncompressed_direct_buf_len = (*env)->GetIntField(env, thisj, SnappyDecompressor_directBufferSize); // Get the input direct buffer - LOCK_CLASS(env, clazz, "SnappyDecompressor"); compressed_bytes = (const char*)(*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "SnappyDecompressor"); if (compressed_bytes == 0) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "SnappyDecompressor"); uncompressed_bytes = (char *)(*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "SnappyDecompressor"); if (uncompressed_bytes == 0) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c index f7c0cb9db999e..51f7bed9bd70d 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c @@ -28,7 +28,6 @@ #include "org_apache_hadoop_io_compress_zlib.h" #include "org_apache_hadoop_io_compress_zlib_ZlibCompressor.h" -static jfieldID ZlibCompressor_clazz; static jfieldID ZlibCompressor_stream; static jfieldID ZlibCompressor_uncompressedDirectBuf; static jfieldID ZlibCompressor_uncompressedDirectBufOff; @@ -141,8 +140,6 @@ Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs( #endif // Initialize the requisite fieldIds - ZlibCompressor_clazz = (*env)->GetStaticFieldID(env, class, "clazz", - "Ljava/lang/Class;"); ZlibCompressor_stream = (*env)->GetFieldID(env, class, "stream", "J"); ZlibCompressor_finish = (*env)->GetFieldID(env, class, "finish", "Z"); ZlibCompressor_finished = (*env)->GetFieldID(env, class, "finished", "Z"); @@ -239,7 +236,6 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect( JNIEnv *env, jobject this ) { - jobject clazz = NULL; jobject uncompressed_direct_buf = NULL; jint uncompressed_direct_buf_off = 0; jint uncompressed_direct_buf_len = 0; @@ -260,9 +256,6 @@ Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect( return (jint)0; } - // Get members of ZlibCompressor - clazz = (*env)->GetStaticObjectField(env, this, - ZlibCompressor_clazz); uncompressed_direct_buf = (*env)->GetObjectField(env, this, ZlibCompressor_uncompressedDirectBuf); uncompressed_direct_buf_off = (*env)->GetIntField(env, this, @@ -278,20 +271,16 @@ Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_deflateBytesDirect( finish = (*env)->GetBooleanField(env, this, ZlibCompressor_finish); // Get the input direct buffer - LOCK_CLASS(env, clazz, "ZlibCompressor"); uncompressed_bytes = (*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "ZlibCompressor"); if (uncompressed_bytes == 0) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "ZlibCompressor"); compressed_bytes = (*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "ZlibCompressor"); if (compressed_bytes == 0) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c index 8b78f41e1a10f..b9f23b181b0cd 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.c @@ -28,7 +28,6 @@ #include "org_apache_hadoop_io_compress_zlib.h" #include "org_apache_hadoop_io_compress_zlib_ZlibDecompressor.h" -static jfieldID ZlibDecompressor_clazz; static jfieldID ZlibDecompressor_stream; static jfieldID ZlibDecompressor_compressedDirectBuf; static jfieldID ZlibDecompressor_compressedDirectBufOff; @@ -104,8 +103,6 @@ JNIEnv *env, jclass class // Initialize the requisite fieldIds - ZlibDecompressor_clazz = (*env)->GetStaticFieldID(env, class, "clazz", - "Ljava/lang/Class;"); ZlibDecompressor_stream = (*env)->GetFieldID(env, class, "stream", "J"); ZlibDecompressor_needDict = (*env)->GetFieldID(env, class, "needDict", "Z"); ZlibDecompressor_finished = (*env)->GetFieldID(env, class, "finished", "Z"); @@ -197,7 +194,6 @@ JNIEXPORT jint JNICALL Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_inflateBytesDirect( JNIEnv *env, jobject this ) { - jobject clazz = NULL; jarray compressed_direct_buf = NULL; jint compressed_direct_buf_off = 0; jint compressed_direct_buf_len = 0; @@ -218,8 +214,6 @@ Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_inflateBytesDirect( } // Get members of ZlibDecompressor - clazz = (*env)->GetStaticObjectField(env, this, - ZlibDecompressor_clazz); compressed_direct_buf = (jarray)(*env)->GetObjectField(env, this, ZlibDecompressor_compressedDirectBuf); compressed_direct_buf_off = (*env)->GetIntField(env, this, @@ -233,20 +227,16 @@ Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_inflateBytesDirect( ZlibDecompressor_directBufferSize); // Get the input direct buffer - LOCK_CLASS(env, clazz, "ZlibDecompressor"); compressed_bytes = (*env)->GetDirectBufferAddress(env, compressed_direct_buf); - UNLOCK_CLASS(env, clazz, "ZlibDecompressor"); if (!compressed_bytes) { return (jint)0; } // Get the output direct buffer - LOCK_CLASS(env, clazz, "ZlibDecompressor"); uncompressed_bytes = (*env)->GetDirectBufferAddress(env, uncompressed_direct_buf); - UNLOCK_CLASS(env, clazz, "ZlibDecompressor"); if (!uncompressed_bytes) { return (jint)0; diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c index c7efb8db95ce1..b3bb69959b233 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c @@ -38,22 +38,23 @@ #include "bulk_crc32.h" #include "gcc_optimizations.h" -#if (!defined(__FreeBSD__) && !defined(WINDOWS)) -#define USE_PIPELINED -#endif - #define CRC_INITIAL_VAL 0xffffffff -typedef uint32_t (*crc_update_func_t)(uint32_t, const uint8_t *, size_t); static uint32_t crc_val(uint32_t crc); -static uint32_t crc32_zlib_sb8(uint32_t crc, const uint8_t *buf, size_t length); -static uint32_t crc32c_sb8(uint32_t crc, const uint8_t *buf, size_t length); -#ifdef USE_PIPELINED -static void pipelined_crc32c(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf, size_t block_size, int num_blocks); -#endif -static int cached_cpu_supports_crc32; // initialized by constructor below -static uint32_t crc32c_hardware(uint32_t crc, const uint8_t* data, size_t length); +typedef void (*crc_pipelined_func_t)(uint32_t *, uint32_t *, uint32_t *, const uint8_t *, size_t, int); + +// The software versions of pipelined crc +static void pipelined_crc32c_sb8(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, + const uint8_t *p_buf, size_t block_size, int num_blocks); +static void pipelined_crc32_zlib_sb8(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, + const uint8_t *p_buf, size_t block_size, int num_blocks); + +// Satically initialise the function pointers to the software versions +// If a HW implementation is available they will subsequently be initialised in the dynamic +// initialisers to point to the HW routines. +crc_pipelined_func_t pipelined_crc32c_func = pipelined_crc32c_sb8; +crc_pipelined_func_t pipelined_crc32_zlib_func = pipelined_crc32_zlib_sb8; static inline int store_or_verify(uint32_t *sums, uint32_t crc, int is_verify) { @@ -72,94 +73,66 @@ int bulk_crc(const uint8_t *data, size_t data_len, int is_verify = error_info != NULL; -#ifdef USE_PIPELINED uint32_t crc1, crc2, crc3; int n_blocks = data_len / bytes_per_checksum; int remainder = data_len % bytes_per_checksum; - int do_pipelined = 0; -#endif uint32_t crc; - crc_update_func_t crc_update_func; + crc_pipelined_func_t crc_pipelined_func; switch (checksum_type) { case CRC32_ZLIB_POLYNOMIAL: - crc_update_func = crc32_zlib_sb8; + crc_pipelined_func = pipelined_crc32_zlib_func; break; case CRC32C_POLYNOMIAL: - if (likely(cached_cpu_supports_crc32)) { - crc_update_func = crc32c_hardware; -#ifdef USE_PIPELINED - do_pipelined = 1; -#endif - } else { - crc_update_func = crc32c_sb8; - } + crc_pipelined_func = pipelined_crc32c_func; break; default: return is_verify ? INVALID_CHECKSUM_TYPE : -EINVAL; } -#ifdef USE_PIPELINED - if (do_pipelined) { - /* Process three blocks at a time */ - while (likely(n_blocks >= 3)) { - crc1 = crc2 = crc3 = CRC_INITIAL_VAL; - pipelined_crc32c(&crc1, &crc2, &crc3, data, bytes_per_checksum, 3); + /* Process three blocks at a time */ + while (likely(n_blocks >= 3)) { + crc1 = crc2 = crc3 = CRC_INITIAL_VAL; + crc_pipelined_func(&crc1, &crc2, &crc3, data, bytes_per_checksum, 3); - if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc1))), is_verify))) - goto return_crc_error; - sums++; - data += bytes_per_checksum; - if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc2))), is_verify))) - goto return_crc_error; - sums++; - data += bytes_per_checksum; - if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc3))), is_verify))) - goto return_crc_error; - sums++; - data += bytes_per_checksum; - n_blocks -= 3; - } + if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc1))), is_verify))) + goto return_crc_error; + sums++; + data += bytes_per_checksum; + if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc2))), is_verify))) + goto return_crc_error; + sums++; + data += bytes_per_checksum; + if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc3))), is_verify))) + goto return_crc_error; + sums++; + data += bytes_per_checksum; + n_blocks -= 3; + } - /* One or two blocks */ - if (n_blocks) { - crc1 = crc2 = crc3 = CRC_INITIAL_VAL; - pipelined_crc32c(&crc1, &crc2, &crc3, data, bytes_per_checksum, n_blocks); + /* One or two blocks */ + if (n_blocks) { + crc1 = crc2 = crc3 = CRC_INITIAL_VAL; + crc_pipelined_func(&crc1, &crc2, &crc3, data, bytes_per_checksum, n_blocks); - if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc1))), is_verify))) + if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc1))), is_verify))) + goto return_crc_error; + data += bytes_per_checksum; + sums++; + if (n_blocks == 2) { + if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc2))), is_verify))) goto return_crc_error; - data += bytes_per_checksum; sums++; - if (n_blocks == 2) { - if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc2))), is_verify))) - goto return_crc_error; - sums++; - data += bytes_per_checksum; - } - } - - /* For something smaller than a block */ - if (remainder) { - crc1 = crc2 = crc3 = CRC_INITIAL_VAL; - pipelined_crc32c(&crc1, &crc2, &crc3, data, remainder, 1); - - if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc1))), is_verify))) - goto return_crc_error; + data += bytes_per_checksum; } - return is_verify ? CHECKSUMS_VALID : 0; } -#endif - while (likely(data_len > 0)) { - int len = likely(data_len >= bytes_per_checksum) ? bytes_per_checksum : data_len; - crc = CRC_INITIAL_VAL; - crc = crc_update_func(crc, data, len); - crc = ntohl(crc_val(crc)); - if (unlikely(!store_or_verify(sums, crc, is_verify))) { + /* For something smaller than a block */ + if (remainder) { + crc1 = crc2 = crc3 = CRC_INITIAL_VAL; + crc_pipelined_func(&crc1, &crc2, &crc3, data, remainder, 1); + + if (unlikely(!store_or_verify(sums, (crc = ntohl(crc_val(crc1))), is_verify))) goto return_crc_error; - } - data += len; - data_len -= len; - sums++; } return is_verify ? CHECKSUMS_VALID : 0; @@ -175,7 +148,7 @@ int bulk_crc(const uint8_t *data, size_t data_len, /** * Extract the final result of a CRC */ -uint32_t crc_val(uint32_t crc) { +static uint32_t crc_val(uint32_t crc) { return ~crc; } @@ -214,6 +187,16 @@ static uint32_t crc32c_sb8(uint32_t crc, const uint8_t *buf, size_t length) { return crc; } +static void pipelined_crc32c_sb8(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, + const uint8_t *p_buf, size_t block_size, int num_blocks) { + assert(num_blocks >= 1 && num_blocks <=3 && "invalid num_blocks"); + *crc1 = crc32c_sb8(*crc1, p_buf, block_size); + if (num_blocks >= 2) + *crc2 = crc32c_sb8(*crc2, p_buf+block_size, block_size); + if (num_blocks >= 3) + *crc3 = crc32c_sb8(*crc3, p_buf+2*block_size, block_size); +} + /** * Update a CRC using the "zlib" polynomial -- what Hadoop calls CHECKSUM_CRC32 * using slicing-by-8 @@ -250,416 +233,12 @@ static uint32_t crc32_zlib_sb8( return crc; } -/////////////////////////////////////////////////////////////////////////// -// Begin code for SSE4.2 specific hardware support of CRC32C -/////////////////////////////////////////////////////////////////////////// - -#if (defined(__amd64__) || defined(__i386)) && defined(__GNUC__) && !defined(__FreeBSD__) -# define SSE42_FEATURE_BIT (1 << 20) -# define CPUID_FEATURES 1 -/** - * Call the cpuid instruction to determine CPU feature flags. - */ -static uint32_t cpuid(uint32_t eax_in) { - uint32_t eax, ebx, ecx, edx; -# if defined(__PIC__) && !defined(__LP64__) -// 32-bit PIC code uses the ebx register for the base offset -- -// have to save and restore it on the stack - asm("pushl %%ebx\n\t" - "cpuid\n\t" - "movl %%ebx, %[ebx]\n\t" - "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" (eax_in) - : "cc"); -# else - asm("cpuid" : "=a" (eax), "=b"(ebx), "=c"(ecx), "=d"(edx) : "a"(eax_in) - : "cc"); -# endif - - return ecx; -} - -/** - * On library load, initiailize the cached value above for - * whether the cpu supports SSE4.2's crc32 instruction. - */ -void __attribute__ ((constructor)) init_cpu_support_flag(void) { - uint32_t ecx = cpuid(CPUID_FEATURES); - cached_cpu_supports_crc32 = ecx & SSE42_FEATURE_BIT; -} - - -// -// Definitions of the SSE4.2 crc32 operations. Using these instead of -// the GCC __builtin_* intrinsics allows this code to compile without -// -msse4.2, since we do dynamic CPU detection at runtime. -// - -# ifdef __LP64__ -inline uint64_t _mm_crc32_u64(uint64_t crc, uint64_t value) { - asm("crc32q %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); - return crc; -} -# endif - -inline uint32_t _mm_crc32_u32(uint32_t crc, uint32_t value) { - asm("crc32l %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); - return crc; -} - -inline uint32_t _mm_crc32_u16(uint32_t crc, uint16_t value) { - asm("crc32w %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); - return crc; -} - -inline uint32_t _mm_crc32_u8(uint32_t crc, uint8_t value) { - asm("crc32b %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); - return crc; +static void pipelined_crc32_zlib_sb8(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, + const uint8_t *p_buf, size_t block_size, int num_blocks) { + assert(num_blocks >= 1 && num_blocks <=3 && "invalid num_blocks"); + *crc1 = crc32_zlib_sb8(*crc1, p_buf, block_size); + if (num_blocks >= 2) + *crc2 = crc32_zlib_sb8(*crc2, p_buf+block_size, block_size); + if (num_blocks >= 3) + *crc3 = crc32_zlib_sb8(*crc3, p_buf+2*block_size, block_size); } - - -# ifdef __LP64__ -/** - * Hardware-accelerated CRC32C calculation using the 64-bit instructions. - */ -static uint32_t crc32c_hardware(uint32_t crc, const uint8_t* p_buf, size_t length) { - // start directly at p_buf, even if it's an unaligned address. According - // to the original author of this code, doing a small run of single bytes - // to word-align the 64-bit instructions doesn't seem to help, but - // we haven't reconfirmed those benchmarks ourselves. - uint64_t crc64bit = crc; - size_t i; - for (i = 0; i < length / sizeof(uint64_t); i++) { - crc64bit = _mm_crc32_u64(crc64bit, *(uint64_t*) p_buf); - p_buf += sizeof(uint64_t); - } - - // This ugly switch is slightly faster for short strings than the straightforward loop - uint32_t crc32bit = (uint32_t) crc64bit; - length &= sizeof(uint64_t) - 1; - switch (length) { - case 7: - crc32bit = _mm_crc32_u8(crc32bit, *p_buf++); - case 6: - crc32bit = _mm_crc32_u16(crc32bit, *(uint16_t*) p_buf); - p_buf += 2; - // case 5 is below: 4 + 1 - case 4: - crc32bit = _mm_crc32_u32(crc32bit, *(uint32_t*) p_buf); - break; - case 3: - crc32bit = _mm_crc32_u8(crc32bit, *p_buf++); - case 2: - crc32bit = _mm_crc32_u16(crc32bit, *(uint16_t*) p_buf); - break; - case 5: - crc32bit = _mm_crc32_u32(crc32bit, *(uint32_t*) p_buf); - p_buf += 4; - case 1: - crc32bit = _mm_crc32_u8(crc32bit, *p_buf); - break; - case 0: - break; - default: - // This should never happen; enable in debug code - assert(0 && "ended up with 8 or more bytes at tail of calculation"); - } - - return crc32bit; -} - -#ifdef USE_PIPELINED -/** - * Pipelined version of hardware-accelerated CRC32C calculation using - * the 64 bit crc32q instruction. - * One crc32c instruction takes three cycles, but two more with no data - * dependency can be in the pipeline to achieve something close to single - * instruction/cycle. Here we feed three blocks in RR. - * - * crc1, crc2, crc3 : Store initial checksum for each block before - * calling. When it returns, updated checksums are stored. - * p_buf : The base address of the data buffer. The buffer should be - * at least as big as block_size * num_blocks. - * block_size : The size of each block in bytes. - * num_blocks : The number of blocks to work on. Min = 1, Max = 3 - */ -static void pipelined_crc32c(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf, size_t block_size, int num_blocks) { - uint64_t c1 = *crc1; - uint64_t c2 = *crc2; - uint64_t c3 = *crc3; - uint64_t *data = (uint64_t*)p_buf; - int counter = block_size / sizeof(uint64_t); - int remainder = block_size % sizeof(uint64_t); - uint8_t *bdata; - - /* We do switch here because the loop has to be tight in order - * to fill the pipeline. Any other statement inside the loop - * or inbetween crc32 instruction can slow things down. Calling - * individual crc32 instructions three times from C also causes - * gcc to insert other instructions inbetween. - * - * Do not rearrange the following code unless you have verified - * the generated machine code is as efficient as before. - */ - switch (num_blocks) { - case 3: - /* Do three blocks */ - while (likely(counter)) { - __asm__ __volatile__( - "crc32q (%7), %0;\n\t" - "crc32q (%7,%6,1), %1;\n\t" - "crc32q (%7,%6,2), %2;\n\t" - : "=r"(c1), "=r"(c2), "=r"(c3) - : "0"(c1), "1"(c2), "2"(c3), "r"(block_size), "r"(data) - ); - data++; - counter--; - } - - /* Take care of the remainder. They are only up to seven bytes, - * so performing byte-level crc32 won't take much time. - */ - bdata = (uint8_t*)data; - while (likely(remainder)) { - __asm__ __volatile__( - "crc32b (%7), %0;\n\t" - "crc32b (%7,%6,1), %1;\n\t" - "crc32b (%7,%6,2), %2;\n\t" - : "=r"(c1), "=r"(c2), "=r"(c3) - : "0"(c1), "1"(c2), "2"(c3), "r"(block_size), "r"(bdata) - ); - bdata++; - remainder--; - } - break; - case 2: - /* Do two blocks */ - while (likely(counter)) { - __asm__ __volatile__( - "crc32q (%5), %0;\n\t" - "crc32q (%5,%4,1), %1;\n\t" - : "=r"(c1), "=r"(c2) - : "0"(c1), "1"(c2), "r"(block_size), "r"(data) - ); - data++; - counter--; - } - - bdata = (uint8_t*)data; - while (likely(remainder)) { - __asm__ __volatile__( - "crc32b (%5), %0;\n\t" - "crc32b (%5,%4,1), %1;\n\t" - : "=r"(c1), "=r"(c2) - : "0"(c1), "1"(c2), "r"(block_size), "r"(bdata) - ); - bdata++; - remainder--; - } - break; - case 1: - /* single block */ - while (likely(counter)) { - __asm__ __volatile__( - "crc32q (%2), %0;\n\t" - : "=r"(c1) - : "0"(c1), "r"(data) - ); - data++; - counter--; - } - bdata = (uint8_t*)data; - while (likely(remainder)) { - __asm__ __volatile__( - "crc32b (%2), %0;\n\t" - : "=r"(c1) - : "0"(c1), "r"(bdata) - ); - bdata++; - remainder--; - } - break; - case 0: - return; - default: - assert(0 && "BUG: Invalid number of checksum blocks"); - } - - *crc1 = c1; - *crc2 = c2; - *crc3 = c3; - return; -} -#endif /* USE_PIPELINED */ - -# else // 32-bit - -/** - * Hardware-accelerated CRC32C calculation using the 32-bit instructions. - */ -static uint32_t crc32c_hardware(uint32_t crc, const uint8_t* p_buf, size_t length) { - // start directly at p_buf, even if it's an unaligned address. According - // to the original author of this code, doing a small run of single bytes - // to word-align the 64-bit instructions doesn't seem to help, but - // we haven't reconfirmed those benchmarks ourselves. - size_t i; - for (i = 0; i < length / sizeof(uint32_t); i++) { - crc = _mm_crc32_u32(crc, *(uint32_t*) p_buf); - p_buf += sizeof(uint32_t); - } - - // This ugly switch is slightly faster for short strings than the straightforward loop - length &= sizeof(uint32_t) - 1; - switch (length) { - case 3: - crc = _mm_crc32_u8(crc, *p_buf++); - case 2: - crc = _mm_crc32_u16(crc, *(uint16_t*) p_buf); - break; - case 1: - crc = _mm_crc32_u8(crc, *p_buf); - break; - case 0: - break; - default: - // This should never happen; enable in debug code - assert(0 && "ended up with 4 or more bytes at tail of calculation"); - } - - return crc; -} - -#ifdef USE_PIPELINED -/** - * Pipelined version of hardware-accelerated CRC32C calculation using - * the 32 bit crc32l instruction. - * One crc32c instruction takes three cycles, but two more with no data - * dependency can be in the pipeline to achieve something close to single - * instruction/cycle. Here we feed three blocks in RR. - * - * crc1, crc2, crc3 : Store initial checksum for each block before - * calling. When it returns, updated checksums are stored. - * data : The base address of the data buffer. The buffer should be - * at least as big as block_size * num_blocks. - * block_size : The size of each block in bytes. - * num_blocks : The number of blocks to work on. Min = 1, Max = 3 - */ -static void pipelined_crc32c(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf, size_t block_size, int num_blocks) { - uint32_t c1 = *crc1; - uint32_t c2 = *crc2; - uint32_t c3 = *crc3; - int counter = block_size / sizeof(uint32_t); - int remainder = block_size % sizeof(uint32_t); - uint32_t *data = (uint32_t*)p_buf; - uint8_t *bdata; - - /* We do switch here because the loop has to be tight in order - * to fill the pipeline. Any other statement inside the loop - * or inbetween crc32 instruction can slow things down. Calling - * individual crc32 instructions three times from C also causes - * gcc to insert other instructions inbetween. - * - * Do not rearrange the following code unless you have verified - * the generated machine code is as efficient as before. - */ - switch (num_blocks) { - case 3: - /* Do three blocks */ - while (likely(counter)) { - __asm__ __volatile__( - "crc32l (%7), %0;\n\t" - "crc32l (%7,%6,1), %1;\n\t" - "crc32l (%7,%6,2), %2;\n\t" - : "=r"(c1), "=r"(c2), "=r"(c3) - : "r"(c1), "r"(c2), "r"(c3), "r"(block_size), "r"(data) - ); - data++; - counter--; - } - /* Take care of the remainder. They are only up to three bytes, - * so performing byte-level crc32 won't take much time. - */ - bdata = (uint8_t*)data; - while (likely(remainder)) { - __asm__ __volatile__( - "crc32b (%7), %0;\n\t" - "crc32b (%7,%6,1), %1;\n\t" - "crc32b (%7,%6,2), %2;\n\t" - : "=r"(c1), "=r"(c2), "=r"(c3) - : "r"(c1), "r"(c2), "r"(c3), "r"(block_size), "r"(bdata) - ); - bdata++; - remainder--; - } - break; - case 2: - /* Do two blocks */ - while (likely(counter)) { - __asm__ __volatile__( - "crc32l (%5), %0;\n\t" - "crc32l (%5,%4,1), %1;\n\t" - : "=r"(c1), "=r"(c2) - : "r"(c1), "r"(c2), "r"(block_size), "r"(data) - ); - data++; - counter--; - } - - bdata = (uint8_t*)data; - while (likely(remainder)) { - __asm__ __volatile__( - "crc32b (%5), %0;\n\t" - "crc32b (%5,%4,1), %1;\n\t" - : "=r"(c1), "=r"(c2) - : "r"(c1), "r"(c2), "r"(block_size), "r"(bdata) - ); - bdata++; - remainder--; - } - break; - case 1: - /* single block */ - while (likely(counter)) { - __asm__ __volatile__( - "crc32l (%2), %0;\n\t" - : "=r"(c1) - : "r"(c1), "r"(data) - ); - data++; - counter--; - } - bdata = (uint8_t*)data; - while (likely(remainder)) { - __asm__ __volatile__( - "crc32b (%2), %0;\n\t" - : "=r"(c1) - : "r"(c1), "r"(bdata) - ); - bdata++; - remainder--; - } - break; - case 0: - return; - default: - assert(0 && "BUG: Invalid number of checksum blocks"); - } - - *crc1 = c1; - *crc2 = c2; - *crc3 = c3; - return; -} - -#endif /* USE_PIPELINED */ - -# endif // 64-bit vs 32-bit - -#else // end x86 architecture - -static uint32_t crc32c_hardware(uint32_t crc, const uint8_t* data, size_t length) { - // never called! - assert(0 && "hardware crc called on an unsupported platform"); - return 0; -} - -#endif diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c new file mode 100644 index 0000000000000..ab4690bcdfd3f --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c @@ -0,0 +1,362 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include // for size_t + +#include "bulk_crc32.h" +#include "gcc_optimizations.h" + +/** + * Hardware-accelerated CRC32 calculation using the 64-bit instructions. + * 2 variants:- + * pipelined_crc32c uses the Castagnoli polynomial 0x1EDC6F41 + * pipelined_crc32_zlib uses the Zlib polynomial 0x04C11DB7 + */ + +// gcc doesn't know how to vectorize a 128 bit load, so use the following to tell it +#define LDP(x,y,p) asm("ldp %x[a], %x[b], [%x[c]], #16" : [a]"=r"(x),[b]"=r"(y),[c]"+r"(p)) + +#define CRC32CX(crc,value) asm("crc32cx %w[c], %w[c], %x[v]" : [c]"+r"(*&crc) : [v]"r"(+value)) +#define CRC32CW(crc,value) asm("crc32cw %w[c], %w[c], %w[v]" : [c]"+r"(*&crc) : [v]"r"(+value)) +#define CRC32CH(crc,value) asm("crc32ch %w[c], %w[c], %w[v]" : [c]"+r"(*&crc) : [v]"r"(+value)) +#define CRC32CB(crc,value) asm("crc32cb %w[c], %w[c], %w[v]" : [c]"+r"(*&crc) : [v]"r"(+value)) + +#define CRC32ZX(crc,value) asm("crc32x %w[c], %w[c], %x[v]" : [c]"+r"(crc) : [v]"r"(value)) +#define CRC32ZW(crc,value) asm("crc32w %w[c], %w[c], %w[v]" : [c]"+r"(crc) : [v]"r"(value)) +#define CRC32ZH(crc,value) asm("crc32h %w[c], %w[c], %w[v]" : [c]"+r"(crc) : [v]"r"(value)) +#define CRC32ZB(crc,value) asm("crc32b %w[c], %w[c], %w[v]" : [c]"+r"(crc) : [v]"r"(value)) + +/** + * Pipelined version of hardware-accelerated CRC32 calculation using + * the 64 bit crc32 instructions. + * One crc32 instruction takes three cycles, but two more with no data + * dependency can be in the pipeline to achieve something close to single + * instruction/cycle. Here we feed three blocks in RR. + * + * 2 variants:- + * pipelined_crc32c uses the Castagnoli polynomial 0x1EDC6F41 + * pipelined_crc32_zlib uses the Zlib polynomial 0x04C11DB7 + * + * crc1, crc2, crc3 : Store initial checksum for each block before + * calling. When it returns, updated checksums are stored. + * p_buf : The base address of the data buffer. The buffer should be + * at least as big as block_size * num_blocks. + * block_size : The size of each block in bytes. + * num_blocks : The number of blocks to work on. Min = 1, Max = 3 + */ +static void pipelined_crc32c(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf1, size_t block_size, int num_blocks) { + uint64_t c1 = *crc1; + uint64_t c2 = *crc2; + uint64_t c3 = *crc3; + const uint8_t *p_buf2 = p_buf1 + block_size; + const uint8_t *p_buf3 = p_buf1 + block_size * 2; + uint64_t x1, y1, x2, y2, x3, y3; + long len = block_size; + + /* We do switch here because the loop has to be tight in order + * to fill the pipeline. Any other statement inside the loop + * or inbetween crc32 instruction can slow things down. + * + * Do verify that this code generates the expected assembler + * by disassembling test_bulk_crc32 + */ + + asm(".cpu generic+crc"); // Allow crc instructions in asm + switch (num_blocks) { + case 3: + /* Do three blocks */ + while ((len -= 2*sizeof(uint64_t)) >= 0) { + LDP(x1,y1,p_buf1); + LDP(x2,y2,p_buf2); + LDP(x3,y3,p_buf3); + CRC32CX(c1, x1); + CRC32CX(c2, x2); + CRC32CX(c3, x3); + CRC32CX(c1, y1); + CRC32CX(c2, y2); + CRC32CX(c3, y3); + } + + if (unlikely(len & sizeof(uint64_t))) { + x1 = *(uint64_t*)p_buf1; p_buf1 += sizeof(uint64_t); + x2 = *(uint64_t*)p_buf2; p_buf2 += sizeof(uint64_t); + x3 = *(uint64_t*)p_buf3; p_buf3 += sizeof(uint64_t); + CRC32CX(c1, x1); + CRC32CX(c2, x2); + CRC32CX(c3, x3); + } + if (unlikely(len & sizeof(uint32_t))) { + x1 = *(uint32_t*)p_buf1; p_buf1 += sizeof(uint32_t); + x2 = *(uint32_t*)p_buf2; p_buf2 += sizeof(uint32_t); + x3 = *(uint32_t*)p_buf3; p_buf3 += sizeof(uint32_t); + CRC32CW(c1, x1); + CRC32CW(c2, x2); + CRC32CW(c3, x3); + } + if (unlikely(len & sizeof(uint16_t))) { + x1 = *(uint16_t*)p_buf1; p_buf1 += sizeof(uint16_t); + x2 = *(uint16_t*)p_buf2; p_buf2 += sizeof(uint16_t); + x3 = *(uint16_t*)p_buf3; p_buf3 += sizeof(uint16_t); + CRC32CH(c1, x1); + CRC32CH(c2, x2); + CRC32CH(c3, x3); + } + if (unlikely(len & sizeof(uint8_t))) { + x1 = *p_buf1; + x2 = *p_buf2; + x3 = *p_buf3; + CRC32CB(c1, x1); + CRC32CB(c2, x2); + CRC32CB(c3, x3); + } + break; + case 2: + /* Do two blocks */ + while ((len -= 2*sizeof(uint64_t)) >= 0) { + LDP(x1,y1,p_buf1); + LDP(x2,y2,p_buf2); + CRC32CX(c1, x1); + CRC32CX(c2, x2); + CRC32CX(c1, y1); + CRC32CX(c2, y2); + } + + if (unlikely(len & sizeof(uint64_t))) { + x1 = *(uint64_t*)p_buf1; p_buf1 += sizeof(uint64_t); + x2 = *(uint64_t*)p_buf2; p_buf2 += sizeof(uint64_t); + CRC32CX(c1, x1); + CRC32CX(c2, x2); + } + if (unlikely(len & sizeof(uint32_t))) { + x1 = *(uint32_t*)p_buf1; p_buf1 += sizeof(uint32_t); + x2 = *(uint32_t*)p_buf2; p_buf2 += sizeof(uint32_t); + CRC32CW(c1, x1); + CRC32CW(c2, x2); + } + if (unlikely(len & sizeof(uint16_t))) { + x1 = *(uint16_t*)p_buf1; p_buf1 += sizeof(uint16_t); + x2 = *(uint16_t*)p_buf2; p_buf2 += sizeof(uint16_t); + CRC32CH(c1, x1); + CRC32CH(c2, x2); + } + if (unlikely(len & sizeof(uint8_t))) { + x1 = *p_buf1; + x2 = *p_buf2; + CRC32CB(c1, x1); + CRC32CB(c2, x2); + } + break; + case 1: + /* single block */ + while ((len -= 2*sizeof(uint64_t)) >= 0) { + LDP(x1,y1,p_buf1); + CRC32CX(c1, x1); + CRC32CX(c1, y1); + } + + if (unlikely(len & sizeof(uint64_t))) { + x1 = *(uint64_t*)p_buf1; p_buf1 += sizeof(uint64_t); + CRC32CX(c1, x1); + } + if (unlikely(len & sizeof(uint32_t))) { + x1 = *(uint32_t*)p_buf1; p_buf1 += sizeof(uint32_t); + CRC32CW(c1, x1); + } + if (unlikely(len & sizeof(uint16_t))) { + x1 = *(uint16_t*)p_buf1; p_buf1 += sizeof(uint16_t); + CRC32CH(c1, x1); + } + if (unlikely(len & sizeof(uint8_t))) { + x1 = *p_buf1; + CRC32CB(c1, x1); + } + break; + case 0: + return; + default: + assert(0 && "BUG: Invalid number of checksum blocks"); + } + + *crc1 = c1; + *crc2 = c2; + *crc3 = c3; + return; +} + +static void pipelined_crc32_zlib(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf1, size_t block_size, int num_blocks) { + uint64_t c1 = *crc1; + uint64_t c2 = *crc2; + uint64_t c3 = *crc3; + const uint8_t *p_buf2 = p_buf1 + block_size; + const uint8_t *p_buf3 = p_buf1 + block_size * 2; + uint64_t x1, y1, x2, y2, x3, y3; + long len = block_size; + + /* We do switch here because the loop has to be tight in order + * to fill the pipeline. Any other statement inside the loop + * or inbetween crc32 instruction can slow things down. + * + * Do verify that this code generates the expected assembler + * by disassembling test_bulk_crc32 + */ + + asm(".cpu generic+crc"); // Allow crc instructions in asm + switch (num_blocks) { + case 3: + /* Do three blocks */ + while ((len -= 2*sizeof(uint64_t)) >= 0) { + LDP(x1,y1,p_buf1); + LDP(x2,y2,p_buf2); + LDP(x3,y3,p_buf3); + CRC32ZX(c1, x1); + CRC32ZX(c2, x2); + CRC32ZX(c3, x3); + CRC32ZX(c1, y1); + CRC32ZX(c2, y2); + CRC32ZX(c3, y3); + } + + if (unlikely(len & sizeof(uint64_t))) { + x1 = *(uint64_t*)p_buf1; p_buf1 += sizeof(uint64_t); + x2 = *(uint64_t*)p_buf2; p_buf2 += sizeof(uint64_t); + x3 = *(uint64_t*)p_buf3; p_buf3 += sizeof(uint64_t); + CRC32ZX(c1, x1); + CRC32ZX(c2, x2); + CRC32ZX(c3, x3); + } + if (unlikely(len & sizeof(uint32_t))) { + x1 = *(uint32_t*)p_buf1; p_buf1 += sizeof(uint32_t); + x2 = *(uint32_t*)p_buf2; p_buf2 += sizeof(uint32_t); + x3 = *(uint32_t*)p_buf3; p_buf3 += sizeof(uint32_t); + CRC32ZW(c1, x1); + CRC32ZW(c2, x2); + CRC32ZW(c3, x3); + } + if (unlikely(len & sizeof(uint16_t))) { + x1 = *(uint16_t*)p_buf1; p_buf1 += sizeof(uint16_t); + x2 = *(uint16_t*)p_buf2; p_buf2 += sizeof(uint16_t); + x3 = *(uint16_t*)p_buf3; p_buf3 += sizeof(uint16_t); + CRC32ZH(c1, x1); + CRC32ZH(c2, x2); + CRC32ZH(c3, x3); + } + if (unlikely(len & sizeof(uint8_t))) { + x1 = *p_buf1; + x2 = *p_buf2; + x3 = *p_buf3; + CRC32ZB(c1, x1); + CRC32ZB(c2, x2); + CRC32ZB(c3, x3); + } + break; + case 2: + /* Do two blocks */ + while ((len -= 2*sizeof(uint64_t)) >= 0) { + LDP(x1,y1,p_buf1); + LDP(x2,y2,p_buf2); + CRC32ZX(c1, x1); + CRC32ZX(c2, x2); + CRC32ZX(c1, y1); + CRC32ZX(c2, y2); + } + + if (unlikely(len & sizeof(uint64_t))) { + x1 = *(uint64_t*)p_buf1; p_buf1 += sizeof(uint64_t); + x2 = *(uint64_t*)p_buf2; p_buf2 += sizeof(uint64_t); + CRC32ZX(c1, x1); + CRC32ZX(c2, x2); + } + if (unlikely(len & sizeof(uint32_t))) { + x1 = *(uint32_t*)p_buf1; p_buf1 += sizeof(uint32_t); + x2 = *(uint32_t*)p_buf2; p_buf2 += sizeof(uint32_t); + CRC32ZW(c1, x1); + CRC32ZW(c2, x2); + } + if (unlikely(len & sizeof(uint16_t))) { + x1 = *(uint16_t*)p_buf1; p_buf1 += sizeof(uint16_t); + x2 = *(uint16_t*)p_buf2; p_buf2 += sizeof(uint16_t); + CRC32ZH(c1, x1); + CRC32ZH(c2, x2); + } + if (unlikely(len & sizeof(uint8_t))) { + x1 = *p_buf1; + x2 = *p_buf2; + CRC32ZB(c1, x1); + CRC32ZB(c2, x2); + } + break; + case 1: + /* single block */ + while ((len -= 2*sizeof(uint64_t)) >= 0) { + LDP(x1,y1,p_buf1); + CRC32ZX(c1, x1); + CRC32ZX(c1, y1); + } + + if (unlikely(len & sizeof(uint64_t))) { + x1 = *(uint64_t*)p_buf1; p_buf1 += sizeof(uint64_t); + CRC32ZX(c1, x1); + } + if (unlikely(len & sizeof(uint32_t))) { + x1 = *(uint32_t*)p_buf1; p_buf1 += sizeof(uint32_t); + CRC32ZW(c1, x1); + } + if (unlikely(len & sizeof(uint16_t))) { + x1 = *(uint16_t*)p_buf1; p_buf1 += sizeof(uint16_t); + CRC32ZH(c1, x1); + } + if (unlikely(len & sizeof(uint8_t))) { + x1 = *p_buf1; + CRC32ZB(c1, x1); + } + break; + case 0: + return; + default: + assert(0 && "BUG: Invalid number of checksum blocks"); + } + + *crc1 = c1; + *crc2 = c2; + *crc3 = c3; + return; +} + +typedef void (*crc_pipelined_func_t)(uint32_t *, uint32_t *, uint32_t *, const uint8_t *, size_t, int); +extern crc_pipelined_func_t pipelined_crc32c_func; +extern crc_pipelined_func_t pipelined_crc32_zlib_func; + +#include +#include + +#ifndef HWCAP_CRC32 +#define HWCAP_CRC32 (1 << 7) +#endif + +/** + * On library load, determine what sort of crc we are going to do + * and set crc function pointers appropriately. + */ +void __attribute__ ((constructor)) init_cpu_support_flag(void) { + unsigned long auxv = getauxval(AT_HWCAP); + if (auxv & HWCAP_CRC32) { + pipelined_crc32c_func = pipelined_crc32c; + pipelined_crc32_zlib_func = pipelined_crc32_zlib; + } +} diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c new file mode 100644 index 0000000000000..290b8a619955d --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c @@ -0,0 +1,345 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * Portions of this file are from http://www.evanjones.ca/crc32c.html under + * the BSD license: + * Copyright 2008,2009,2010 Massachusetts Institute of Technology. + * All rights reserved. Use of this source code is governed by a + * BSD-style license that can be found in the LICENSE file. + */ + +#include +#include // for size_t + +#include "bulk_crc32.h" +#include "gcc_optimizations.h" +#include "gcc_optimizations.h" + +/////////////////////////////////////////////////////////////////////////// +// Begin code for SSE4.2 specific hardware support of CRC32C +/////////////////////////////////////////////////////////////////////////// + +# define SSE42_FEATURE_BIT (1 << 20) +# define CPUID_FEATURES 1 +/** + * Call the cpuid instruction to determine CPU feature flags. + */ +static uint32_t cpuid(uint32_t eax_in) { + uint32_t eax, ebx, ecx, edx; +# if defined(__PIC__) && !defined(__LP64__) +// 32-bit PIC code uses the ebx register for the base offset -- +// have to save and restore it on the stack + asm("pushl %%ebx\n\t" + "cpuid\n\t" + "movl %%ebx, %[ebx]\n\t" + "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" (eax_in) + : "cc"); +# else + asm("cpuid" : "=a" (eax), "=b"(ebx), "=c"(ecx), "=d"(edx) : "a"(eax_in) + : "cc"); +# endif + + return ecx; +} + +// +// Definitions of the SSE4.2 crc32 operations. Using these instead of +// the GCC __builtin_* intrinsics allows this code to compile without +// -msse4.2, since we do dynamic CPU detection at runtime. +// + +# ifdef __LP64__ +inline uint64_t _mm_crc32_u64(uint64_t crc, uint64_t value) { + asm("crc32q %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); + return crc; +} +# endif + +inline uint32_t _mm_crc32_u32(uint32_t crc, uint32_t value) { + asm("crc32l %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); + return crc; +} + +inline uint32_t _mm_crc32_u16(uint32_t crc, uint16_t value) { + asm("crc32w %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); + return crc; +} + +inline uint32_t _mm_crc32_u8(uint32_t crc, uint8_t value) { + asm("crc32b %[value], %[crc]\n" : [crc] "+r" (crc) : [value] "rm" (value)); + return crc; +} + +# ifdef __LP64__ +/** + * Pipelined version of hardware-accelerated CRC32C calculation using + * the 64 bit crc32q instruction. + * One crc32c instruction takes three cycles, but two more with no data + * dependency can be in the pipeline to achieve something close to single + * instruction/cycle. Here we feed three blocks in RR. + * + * crc1, crc2, crc3 : Store initial checksum for each block before + * calling. When it returns, updated checksums are stored. + * p_buf : The base address of the data buffer. The buffer should be + * at least as big as block_size * num_blocks. + * block_size : The size of each block in bytes. + * num_blocks : The number of blocks to work on. Min = 1, Max = 3 + */ +static void pipelined_crc32c(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf, size_t block_size, int num_blocks) { + uint64_t c1 = *crc1; + uint64_t c2 = *crc2; + uint64_t c3 = *crc3; + uint64_t *data = (uint64_t*)p_buf; + int counter = block_size / sizeof(uint64_t); + int remainder = block_size % sizeof(uint64_t); + uint8_t *bdata; + + /* We do switch here because the loop has to be tight in order + * to fill the pipeline. Any other statement inside the loop + * or inbetween crc32 instruction can slow things down. Calling + * individual crc32 instructions three times from C also causes + * gcc to insert other instructions inbetween. + * + * Do not rearrange the following code unless you have verified + * the generated machine code is as efficient as before. + */ + switch (num_blocks) { + case 3: + /* Do three blocks */ + while (likely(counter)) { + __asm__ __volatile__( + "crc32q (%7), %0;\n\t" + "crc32q (%7,%6,1), %1;\n\t" + "crc32q (%7,%6,2), %2;\n\t" + : "=r"(c1), "=r"(c2), "=r"(c3) + : "0"(c1), "1"(c2), "2"(c3), "r"(block_size), "r"(data) + ); + data++; + counter--; + } + + /* Take care of the remainder. They are only up to seven bytes, + * so performing byte-level crc32 won't take much time. + */ + bdata = (uint8_t*)data; + while (likely(remainder)) { + __asm__ __volatile__( + "crc32b (%7), %0;\n\t" + "crc32b (%7,%6,1), %1;\n\t" + "crc32b (%7,%6,2), %2;\n\t" + : "=r"(c1), "=r"(c2), "=r"(c3) + : "0"(c1), "1"(c2), "2"(c3), "r"(block_size), "r"(bdata) + ); + bdata++; + remainder--; + } + break; + case 2: + /* Do two blocks */ + while (likely(counter)) { + __asm__ __volatile__( + "crc32q (%5), %0;\n\t" + "crc32q (%5,%4,1), %1;\n\t" + : "=r"(c1), "=r"(c2) + : "0"(c1), "1"(c2), "r"(block_size), "r"(data) + ); + data++; + counter--; + } + + bdata = (uint8_t*)data; + while (likely(remainder)) { + __asm__ __volatile__( + "crc32b (%5), %0;\n\t" + "crc32b (%5,%4,1), %1;\n\t" + : "=r"(c1), "=r"(c2) + : "0"(c1), "1"(c2), "r"(block_size), "r"(bdata) + ); + bdata++; + remainder--; + } + break; + case 1: + /* single block */ + while (likely(counter)) { + __asm__ __volatile__( + "crc32q (%2), %0;\n\t" + : "=r"(c1) + : "0"(c1), "r"(data) + ); + data++; + counter--; + } + bdata = (uint8_t*)data; + while (likely(remainder)) { + __asm__ __volatile__( + "crc32b (%2), %0;\n\t" + : "=r"(c1) + : "0"(c1), "r"(bdata) + ); + bdata++; + remainder--; + } + break; + case 0: + return; + default: + assert(0 && "BUG: Invalid number of checksum blocks"); + } + + *crc1 = c1; + *crc2 = c2; + *crc3 = c3; + return; +} + +# else // 32-bit + +/** + * Pipelined version of hardware-accelerated CRC32C calculation using + * the 32 bit crc32l instruction. + * One crc32c instruction takes three cycles, but two more with no data + * dependency can be in the pipeline to achieve something close to single + * instruction/cycle. Here we feed three blocks in RR. + * + * crc1, crc2, crc3 : Store initial checksum for each block before + * calling. When it returns, updated checksums are stored. + * data : The base address of the data buffer. The buffer should be + * at least as big as block_size * num_blocks. + * block_size : The size of each block in bytes. + * num_blocks : The number of blocks to work on. Min = 1, Max = 3 + */ +static void pipelined_crc32c(uint32_t *crc1, uint32_t *crc2, uint32_t *crc3, const uint8_t *p_buf, size_t block_size, int num_blocks) { + uint32_t c1 = *crc1; + uint32_t c2 = *crc2; + uint32_t c3 = *crc3; + int counter = block_size / sizeof(uint32_t); + int remainder = block_size % sizeof(uint32_t); + uint32_t *data = (uint32_t*)p_buf; + uint8_t *bdata; + + /* We do switch here because the loop has to be tight in order + * to fill the pipeline. Any other statement inside the loop + * or inbetween crc32 instruction can slow things down. Calling + * individual crc32 instructions three times from C also causes + * gcc to insert other instructions inbetween. + * + * Do not rearrange the following code unless you have verified + * the generated machine code is as efficient as before. + */ + switch (num_blocks) { + case 3: + /* Do three blocks */ + while (likely(counter)) { + __asm__ __volatile__( + "crc32l (%7), %0;\n\t" + "crc32l (%7,%6,1), %1;\n\t" + "crc32l (%7,%6,2), %2;\n\t" + : "=r"(c1), "=r"(c2), "=r"(c3) + : "r"(c1), "r"(c2), "r"(c3), "r"(block_size), "r"(data) + ); + data++; + counter--; + } + /* Take care of the remainder. They are only up to three bytes, + * so performing byte-level crc32 won't take much time. + */ + bdata = (uint8_t*)data; + while (likely(remainder)) { + __asm__ __volatile__( + "crc32b (%7), %0;\n\t" + "crc32b (%7,%6,1), %1;\n\t" + "crc32b (%7,%6,2), %2;\n\t" + : "=r"(c1), "=r"(c2), "=r"(c3) + : "r"(c1), "r"(c2), "r"(c3), "r"(block_size), "r"(bdata) + ); + bdata++; + remainder--; + } + break; + case 2: + /* Do two blocks */ + while (likely(counter)) { + __asm__ __volatile__( + "crc32l (%5), %0;\n\t" + "crc32l (%5,%4,1), %1;\n\t" + : "=r"(c1), "=r"(c2) + : "r"(c1), "r"(c2), "r"(block_size), "r"(data) + ); + data++; + counter--; + } + + bdata = (uint8_t*)data; + while (likely(remainder)) { + __asm__ __volatile__( + "crc32b (%5), %0;\n\t" + "crc32b (%5,%4,1), %1;\n\t" + : "=r"(c1), "=r"(c2) + : "r"(c1), "r"(c2), "r"(block_size), "r"(bdata) + ); + bdata++; + remainder--; + } + break; + case 1: + /* single block */ + while (likely(counter)) { + __asm__ __volatile__( + "crc32l (%2), %0;\n\t" + : "=r"(c1) + : "r"(c1), "r"(data) + ); + data++; + counter--; + } + bdata = (uint8_t*)data; + while (likely(remainder)) { + __asm__ __volatile__( + "crc32b (%2), %0;\n\t" + : "=r"(c1) + : "r"(c1), "r"(bdata) + ); + bdata++; + remainder--; + } + break; + case 0: + return; + default: + assert(0 && "BUG: Invalid number of checksum blocks"); + } + + *crc1 = c1; + *crc2 = c2; + *crc3 = c3; + return; +} + +# endif // 64-bit vs 32-bit + +/** + * On library load, initiailize the cached function pointer + * if cpu supports SSE4.2's crc32 instruction. + */ +typedef void (*crc_pipelined_func_t)(uint32_t *, uint32_t *, uint32_t *, const uint8_t *, size_t, int); +extern crc_pipelined_func_t pipelined_crc32c_func; + +void __attribute__ ((constructor)) init_cpu_support_flag(void) { + uint32_t ecx = cpuid(CPUID_FEATURES); + if (ecx & SSE42_FEATURE_BIT) pipelined_crc32c_func = pipelined_crc32c; +} diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/yarn/server/nodemanager/windows_secure_container_executor.c b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/yarn/server/nodemanager/windows_secure_container_executor.c index 7e6506571ebb5..b37359dad51d5 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/yarn/server/nodemanager/windows_secure_container_executor.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/yarn/server/nodemanager/windows_secure_container_executor.c @@ -409,7 +409,7 @@ Java_org_apache_hadoop_yarn_server_nodemanager_WindowsSecureContainerExecutor_00 done: if (path) (*env)->ReleaseStringChars(env, jpath, path); - return hFile; + return (jlong) hFile; #endif } diff --git a/hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c b/hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c index 5a8c9f2d85577..ef3dbecceca08 100644 --- a/hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c +++ b/hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c @@ -23,6 +23,7 @@ #include #include #include +#include #define EXPECT_ZERO(x) \ do { \ @@ -57,6 +58,36 @@ static int testBulkVerifyCrc(int dataLen, int crcType, int bytesPerChecksum) return 0; } +static int timeBulkCrc(int dataLen, int crcType, int bytesPerChecksum, int iterations) +{ + int i; + uint8_t *data; + uint32_t *sums; + crc32_error_t errorData; + clock_t start, fini; + + data = malloc(dataLen); + for (i = 0; i < dataLen; i++) { + data[i] = (i % 16) + 1; + } + sums = calloc(sizeof(uint32_t), + (dataLen + bytesPerChecksum - 1) / bytesPerChecksum); + + start = clock(); + for (i = 0; i < iterations; i++) { + EXPECT_ZERO(bulk_crc(data, dataLen, sums, crcType, + bytesPerChecksum, NULL)); + EXPECT_ZERO(bulk_crc(data, dataLen, sums, crcType, + bytesPerChecksum, &errorData)); + } + fini = clock(); + printf("CRC %d bytes @ %d bytes per checksum X %d iterations = %g\n", + dataLen, bytesPerChecksum, iterations, (double)(fini-start)/CLOCKS_PER_SEC); + free(data); + free(sums); + return 0; +} + int main(int argc, char **argv) { /* Test running bulk_calculate_crc with some different algorithms and @@ -74,6 +105,9 @@ int main(int argc, char **argv) EXPECT_ZERO(testBulkVerifyCrc(17, CRC32C_POLYNOMIAL, 4)); EXPECT_ZERO(testBulkVerifyCrc(17, CRC32_ZLIB_POLYNOMIAL, 4)); + EXPECT_ZERO(timeBulkCrc(16 * 1024, CRC32C_POLYNOMIAL, 512, 1000000)); + EXPECT_ZERO(timeBulkCrc(16 * 1024, CRC32_ZLIB_POLYNOMIAL, 512, 1000000)); + fprintf(stderr, "%s: SUCCESS.\n", argv[0]); return EXIT_SUCCESS; } diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml index 80dd15b21879f..46eae0a85c65d 100644 --- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml +++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml @@ -580,6 +580,12 @@ for ldap providers in the same way as above does. (ie client side mount table:). + + fs.AbstractFileSystem.ftp.impl + org.apache.hadoop.fs.ftp.FtpFs + The FileSystem for Ftp: uris. + + fs.ftp.host 0.0.0.0 @@ -763,13 +769,13 @@ for ldap providers in the same way as above does. fs.s3a.connection.establish.timeout 5000 - Socket connection setup timeout in seconds. + Socket connection setup timeout in milliseconds. fs.s3a.connection.timeout 50000 - Socket connection timeout in seconds. + Socket connection timeout in milliseconds. @@ -845,6 +851,22 @@ for ldap providers in the same way as above does. uploads to. + + fs.s3a.fast.upload + false + Upload directly from memory instead of buffering to + disk first. Memory usage and parallelism can be controlled as up to + fs.s3a.multipart.size memory is consumed for each (part)upload actively + uploading (fs.s3a.threads.max) or queueing (fs.s3a.max.total.tasks) + + + + fs.s3a.fast.buffer.size + 1048576 + Size of initial memory buffer in bytes allocated for an + upload. No effect if fs.s3a.fast.upload is false. + + fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem @@ -961,6 +983,20 @@ for ldap providers in the same way as above does. + + ipc.client.tcpnodelay + true + Use TCP_NODELAY flag to bypass Nagle's algorithm transmission delays. + + + + + ipc.client.low-latency + false + Use low-latency QoS markers for IPC connections. + + + ipc.server.listen.queue.size 128 diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/client.c b/hadoop-common-project/hadoop-common/src/main/winutils/client.c index 047bfb5440fd1..e3a2c37240de9 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/client.c +++ b/hadoop-common-project/hadoop-common/src/main/winutils/client.c @@ -28,8 +28,6 @@ static ACCESS_MASK CLIENT_MASK = 1; VOID ReportClientError(LPWSTR lpszLocation, DWORD dwError) { LPWSTR debugMsg = NULL; int len; - WCHAR hexError[32]; - HRESULT hr; if (IsDebuggerPresent()) { len = FormatMessageW( @@ -49,7 +47,6 @@ DWORD PrepareRpcBindingHandle( DWORD dwError = EXIT_FAILURE; RPC_STATUS status; LPWSTR lpszStringBinding = NULL; - ULONG ulCode; RPC_SECURITY_QOS_V3 qos; SID_IDENTIFIER_AUTHORITY authNT = SECURITY_NT_AUTHORITY; BOOL rpcBindingInit = FALSE; @@ -104,7 +101,7 @@ DWORD PrepareRpcBindingHandle( RPC_C_AUTHN_WINNT, // AuthnSvc NULL, // AuthnIdentity (self) RPC_C_AUTHZ_NONE, // AuthzSvc - &qos); + (RPC_SECURITY_QOS*) &qos); if (RPC_S_OK != status) { ReportClientError(L"RpcBindingSetAuthInfoEx", status); dwError = status; @@ -375,7 +372,7 @@ DWORD RpcCall_WinutilsCreateFile( RpcEndExcept; if (ERROR_SUCCESS == dwError) { - *hFile = response->hFile; + *hFile = (HANDLE) response->hFile; } done: @@ -479,11 +476,11 @@ DWORD RpcCall_TaskCreateAsUser( RpcEndExcept; if (ERROR_SUCCESS == dwError) { - *phProcess = response->hProcess; - *phThread = response->hThread; - *phStdIn = response->hStdIn; - *phStdOut = response->hStdOut; - *phStdErr = response->hStdErr; + *phProcess = (HANDLE) response->hProcess; + *phThread = (HANDLE) response->hThread; + *phStdIn = (HANDLE) response->hStdIn; + *phStdOut = (HANDLE) response->hStdOut; + *phStdErr = (HANDLE) response->hStdErr; } done: diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/config.cpp b/hadoop-common-project/hadoop-common/src/main/winutils/config.cpp index 1e07b7fe34274..74be68932cb74 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/config.cpp +++ b/hadoop-common-project/hadoop-common/src/main/winutils/config.cpp @@ -18,7 +18,7 @@ #include "winutils.h" #include #include -#import "msxml6.dll" +#import "msxml6.dll" exclude("ISequentialStream", "_FILETIME") #define ERROR_CHECK_HRESULT_DONE(hr, message) \ if (FAILED(hr)) { \ diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h b/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h index f72802c6d04c2..6c33b5ae79e6f 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h +++ b/hadoop-common-project/hadoop-common/src/main/winutils/include/winutils.h @@ -248,8 +248,8 @@ DWORD BuildServiceSecurityDescriptor( __out PSECURITY_DESCRIPTOR* pSD); DWORD AddNodeManagerAndUserACEsToObject( - __in HANDLE hObject, - __in LPWSTR user, + __in HANDLE hProcess, + __in LPCWSTR user, __in ACCESS_MASK accessMask); @@ -283,15 +283,29 @@ DWORD RpcCall_WinutilsCreateFile( __out HANDLE* hFile); DWORD RpcCall_WinutilsMoveFile( - __in LPCWSTR sourcePath, - __in LPCWSTR destinationPath, - __in BOOL replaceExisting); + __in int operation, + __in LPCWSTR sourcePath, + __in LPCWSTR destinationPath, + __in BOOL replaceExisting); + DWORD RpcCall_WinutilsDeletePath( __in LPCWSTR path, __in BOOL isDir, __out BOOL* pDeleted); +DWORD RpcCall_WinutilsChown( + __in LPCWSTR filePath, + __in_opt LPCWSTR ownerName, + __in_opt LPCWSTR groupName); + +DWORD RpcCall_WinutilsMkDir( + __in LPCWSTR filePath); + +DWORD RpcCall_WinutilsChmod( + __in LPCWSTR filePath, + __in int mode); + #ifdef __cplusplus } #endif diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c b/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c index 98fe3ab81e7fe..676f1b2dad7dd 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c +++ b/hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.c @@ -2596,7 +2596,7 @@ LPCWSTR GetSystemTimeString() { QueryPerformanceFrequency(&frequency); qpc = (double) counter.QuadPart / (double) frequency.QuadPart; - subSec = ((qpc - (long)qpc) * 1000000); + subSec = (int)((qpc - (long)qpc) * 1000000); hr = StringCbPrintf(buffer, sizeof(buffer), L"%02d:%02d:%02d.%06d", (int)systime.wHour, (int)systime.wMinute, (int)systime.wSecond, (int)subSec); @@ -2619,7 +2619,7 @@ LPCWSTR GetSystemTimeString() { // Native debugger: windbg, ntsd, cdb, visual studio // VOID LogDebugMessage(LPCWSTR format, ...) { - LPWSTR buffer[8192]; + wchar_t buffer[8192]; va_list args; HRESULT hr; @@ -2657,8 +2657,8 @@ DWORD SplitStringIgnoreSpaceW( size_t tokenCount = 0; size_t crtSource; size_t crtToken = 0; - WCHAR* lpwszTokenStart = NULL; - WCHAR* lpwszTokenEnd = NULL; + const WCHAR* lpwszTokenStart = NULL; + const WCHAR* lpwszTokenEnd = NULL; WCHAR* lpwszBuffer = NULL; size_t tokenLength = 0; size_t cchBufferLength = 0; @@ -2849,7 +2849,7 @@ DWORD BuildServiceSecurityDescriptor( } } - pTokenGroup = (PTOKEN_USER) LocalAlloc(LPTR, dwBufferSize); + pTokenGroup = (PTOKEN_PRIMARY_GROUP) LocalAlloc(LPTR, dwBufferSize); if (NULL == pTokenGroup) { dwError = GetLastError(); LogDebugMessage(L"LocalAlloc:pTokenGroup: %d\n", dwError); @@ -2870,11 +2870,11 @@ DWORD BuildServiceSecurityDescriptor( owner.TrusteeForm = TRUSTEE_IS_SID; owner.TrusteeType = TRUSTEE_IS_UNKNOWN; - owner.ptstrName = (LPCWSTR) pOwner; + owner.ptstrName = (LPWSTR) pOwner; group.TrusteeForm = TRUSTEE_IS_SID; group.TrusteeType = TRUSTEE_IS_UNKNOWN; - group.ptstrName = (LPCWSTR) pTokenGroup->PrimaryGroup; + group.ptstrName = (LPWSTR) pTokenGroup->PrimaryGroup; eas = (EXPLICIT_ACCESS*) LocalAlloc(LPTR, sizeof(EXPLICIT_ACCESS) * (grantSidCount + denySidCount)); if (NULL == eas) { @@ -2890,7 +2890,7 @@ DWORD BuildServiceSecurityDescriptor( eas[crt].grfInheritance = NO_INHERITANCE; eas[crt].Trustee.TrusteeForm = TRUSTEE_IS_SID; eas[crt].Trustee.TrusteeType = TRUSTEE_IS_UNKNOWN; - eas[crt].Trustee.ptstrName = (LPCWSTR) pGrantSids[crt]; + eas[crt].Trustee.ptstrName = (LPWSTR) pGrantSids[crt]; eas[crt].Trustee.pMultipleTrustee = NULL; eas[crt].Trustee.MultipleTrusteeOperation = NO_MULTIPLE_TRUSTEE; } @@ -2902,7 +2902,7 @@ DWORD BuildServiceSecurityDescriptor( eas[crt].grfInheritance = NO_INHERITANCE; eas[crt].Trustee.TrusteeForm = TRUSTEE_IS_SID; eas[crt].Trustee.TrusteeType = TRUSTEE_IS_UNKNOWN; - eas[crt].Trustee.ptstrName = (LPCWSTR) pDenySids[crt - grantSidCount]; + eas[crt].Trustee.ptstrName = (LPWSTR) pDenySids[crt - grantSidCount]; eas[crt].Trustee.pMultipleTrustee = NULL; eas[crt].Trustee.MultipleTrusteeOperation = NO_MULTIPLE_TRUSTEE; } diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/service.c b/hadoop-common-project/hadoop-common/src/main/winutils/service.c index ba35003771104..fca5dbc6be6a0 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/service.c +++ b/hadoop-common-project/hadoop-common/src/main/winutils/service.c @@ -206,7 +206,7 @@ DWORD ValidateConfigurationFile() { BOOL daclPresent = FALSE; BOOL daclDefaulted = FALSE; PACL pDacl = NULL; - unsigned int crt = 0, crtSid = 0; + DWORD crt = 0; WELL_KNOWN_SID_TYPE allowedSidTypes[] = { WinLocalSystemSid, WinBuiltinAdministratorsSid}; @@ -214,7 +214,6 @@ DWORD ValidateConfigurationFile() { DWORD cbSid = SECURITY_MAX_SID_SIZE; PSID* allowedSids = NULL; int cAllowedSids = 0; - BOOL isSidDefaulted; PSID sidOwner = NULL; PSID sidGroup = NULL; @@ -324,7 +323,7 @@ DWORD InitJobName() { int crt = 0; // Services can be restarted - if (gJobName) LocalFree(gJobName); + if (gJobName) LocalFree((HLOCAL)gJobName); gJobName = NULL; dwError = GetConfigValue( @@ -382,7 +381,7 @@ DWORD InitLocalDirs() { } done: - if (value) LocalFree(value); + if (value) LocalFree((HLOCAL)value); return dwError; } @@ -437,7 +436,7 @@ DWORD ValidateLocalPath(LPCWSTR lpszPath) { gLocalDirs[crt], gCchLocalDir[crt], NULL, // lpVersionInformation NULL, // lpReserved - NULL); // lParam + (LPARAM) NULL); // lParam if (0 == compareResult) { dwError = GetLastError(); @@ -500,7 +499,7 @@ DWORD RunService(__in int argc, __in_ecount(argc) wchar_t *argv[]) // Description: // Service main entry point. // -VOID WINAPI SvcMain() { +VOID WINAPI SvcMain(DWORD dwArg, LPTSTR* lpszArgv) { DWORD dwError = ERROR_SUCCESS; gSvcStatusHandle = RegisterServiceCtrlHandler( @@ -693,15 +692,15 @@ RPC_STATUS CALLBACK RpcAuthorizeCallback ( // DWORD AuthInit() { DWORD dwError = ERROR_SUCCESS; - int count = 0; - int crt = 0; + size_t count = 0; + size_t crt = 0; size_t len = 0; LPCWSTR value = NULL; WCHAR** tokens = NULL; LPWSTR lpszSD = NULL; ULONG cchSD = 0; DWORD dwBufferSize = 0; - int allowedCount = 0; + size_t allowedCount = 0; PSID* allowedSids = NULL; @@ -737,7 +736,7 @@ DWORD AuthInit() { done: if (lpszSD) LocalFree(lpszSD); - if (value) LocalFree(value); + if (value) LocalFree((HLOCAL)value); if (tokens) LocalFree(tokens); return dwError; } @@ -1167,11 +1166,12 @@ error_status_t WinutilsCreateProcessAsUser( // Note that there are no more API calls, only assignments. A failure could occur only if // foced (process kill) or hardware error (faulty memory, processort bit flip etc). - (*response)->hProcess = hDuplicateProcess; - (*response)->hThread = hDuplicateThread; - (*response)->hStdIn = hDuplicateStdIn; - (*response)->hStdOut = hDuplicateStdOut; - (*response)->hStdErr = hDuplicateStdErr; + // as MIDL has no 'HANDLE' type, the (LONG_PTR) is used instead + (*response)->hProcess = (LONG_PTR)hDuplicateProcess; + (*response)->hThread = (LONG_PTR)hDuplicateThread; + (*response)->hStdIn = (LONG_PTR)hDuplicateStdIn; + (*response)->hStdOut = (LONG_PTR)hDuplicateStdOut; + (*response)->hStdErr = (LONG_PTR)hDuplicateStdErr; fMustCleanupProcess = FALSE; @@ -1276,7 +1276,8 @@ error_status_t WinutilsCreateFile( goto done; } - (*response)->hFile = hDuplicateFile; + // As MIDL has no 'HANDLE' type, (LONG_PTR) is used instead + (*response)->hFile = (LONG_PTR)hDuplicateFile; hDuplicateFile = INVALID_HANDLE_VALUE; done: @@ -1302,7 +1303,6 @@ error_status_t WinutilsKillTask( /* [in] */ handle_t IDL_handle, /* [in] */ KILLTASK_REQUEST *request) { DWORD dwError = ERROR_SUCCESS; - HRESULT hr; WCHAR bufferName[MAX_PATH]; dwError = GetSecureJobObjectName(request->taskName, MAX_PATH, bufferName); diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/systeminfo.c b/hadoop-common-project/hadoop-common/src/main/winutils/systeminfo.c index 7fce4241b956e..48f03ed3b6318 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/systeminfo.c +++ b/hadoop-common-project/hadoop-common/src/main/winutils/systeminfo.c @@ -19,6 +19,9 @@ #include #include +#ifdef PSAPI_VERSION +#undef PSAPI_VERSION +#endif #define PSAPI_VERSION 1 #pragma comment(lib, "psapi.lib") #pragma comment(lib, "Powrprof.lib") diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/task.c b/hadoop-common-project/hadoop-common/src/main/winutils/task.c index 21b1893953373..057fd8aa6064b 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/task.c +++ b/hadoop-common-project/hadoop-common/src/main/winutils/task.c @@ -22,6 +22,9 @@ #include #include +#ifdef PSAPI_VERSION +#undef PSAPI_VERSION +#endif #define PSAPI_VERSION 1 #pragma comment(lib, "psapi.lib") @@ -49,6 +52,31 @@ typedef enum TaskCommandOptionType TaskProcessList } TaskCommandOption; + //---------------------------------------------------------------------------- +// Function: GetLimit +// +// Description: +// Get the resource limit value in long type given the command line argument. +// +// Returns: +// TRUE: If successfully get the value +// FALSE: otherwise +static BOOL GetLimit(__in const wchar_t *str, __out long *value) +{ + wchar_t *end = NULL; + if (str == NULL || value == NULL) return FALSE; + *value = wcstol(str, &end, 10); + if (end == NULL || *end != '\0') + { + *value = -1; + return FALSE; + } + else + { + return TRUE; + } +} + //---------------------------------------------------------------------------- // Function: ParseCommandLine // @@ -61,7 +89,9 @@ typedef enum TaskCommandOptionType // FALSE: otherwise static BOOL ParseCommandLine(__in int argc, __in_ecount(argc) wchar_t *argv[], - __out TaskCommandOption *command) + __out TaskCommandOption *command, + __out_opt long *memory, + __out_opt long *vcore) { *command = TaskInvalid; @@ -88,9 +118,44 @@ static BOOL ParseCommandLine(__in int argc, } } - if (argc == 4) { + if (argc >= 4 && argc <= 8) { if (wcscmp(argv[1], L"create") == 0) { + int i; + for (i = 2; i < argc - 3; i++) + { + if (wcscmp(argv[i], L"-c") == 0) + { + if (vcore != NULL && !GetLimit(argv[i + 1], vcore)) + { + return FALSE; + } + else + { + i++; + continue; + } + } + else if (wcscmp(argv[i], L"-m") == 0) + { + if (memory != NULL && !GetLimit(argv[i + 1], memory)) + { + return FALSE; + } + else + { + i++; + continue; + } + } + else + { + break; + } + } + if (argc - i != 2) + return FALSE; + *command = TaskCreate; return TRUE; } @@ -169,7 +234,7 @@ DWORD BuildImpersonateSecurityDescriptor(__out PSECURITY_DESCRIPTOR* ppSD) { LocalFree(tokens); tokens = NULL; - LocalFree(value); + LocalFree((HLOCAL)value); value = NULL; dwError = GetConfigValue(wsceConfigRelativePath, NM_WSCE_IMPERSONATE_DENIED, &len, &value); @@ -236,18 +301,18 @@ DWORD BuildImpersonateSecurityDescriptor(__out PSECURITY_DESCRIPTOR* ppSD) { // DWORD AddNodeManagerAndUserACEsToObject( __in HANDLE hObject, - __in LPWSTR user, + __in LPCWSTR user, __in ACCESS_MASK accessMask) { DWORD dwError = ERROR_SUCCESS; - int countTokens = 0; + size_t countTokens = 0; size_t len = 0; LPCWSTR value = NULL; WCHAR** tokens = NULL; - int crt = 0; + DWORD crt = 0; PACL pDacl = NULL; PSECURITY_DESCRIPTOR psdProcess = NULL; - LPSTR lpszOldDacl = NULL, lpszNewDacl = NULL; + LPWSTR lpszOldDacl = NULL, lpszNewDacl = NULL; ULONG daclLen = 0; PACL pNewDacl = NULL; ACL_SIZE_INFORMATION si; @@ -319,8 +384,8 @@ DWORD AddNodeManagerAndUserACEsToObject( // ACCESS_ALLOWED_ACE struct contains the first DWORD of the SID // dwNewAclSize = si.AclBytesInUse + - (countTokens + 1 + sizeof(forcesSidTypes)/sizeof(forcesSidTypes[0])) * - (sizeof(ACCESS_ALLOWED_ACE) + SECURITY_MAX_SID_SIZE - sizeof(DWORD)); + (DWORD)(countTokens + 1 + sizeof(forcesSidTypes)/sizeof(forcesSidTypes[0])) * + (sizeof(ACCESS_ALLOWED_ACE) + SECURITY_MAX_SID_SIZE - sizeof(DWORD)); pNewDacl = (PSID) LocalAlloc(LPTR, dwNewAclSize); if (!pNewDacl) { @@ -449,7 +514,7 @@ DWORD AddNodeManagerAndUserACEsToObject( goto done; } - LogDebugMessage(L"Old DACL: %s\nNew DACL: %s\n", lpszOldDacl, lpszNewDacl); + LogDebugMessage(L"Old DACL: %ls\nNew DACL: %ls\n", lpszOldDacl, lpszNewDacl); } done: @@ -572,8 +637,8 @@ DWORD ValidateImpersonateAccessCheck(__in HANDLE logonHandle) { // Returns: // ERROR_SUCCESS: On success // GetLastError: otherwise -DWORD CreateTaskImpl(__in_opt HANDLE logonHandle, __in PCWSTR jobObjName,__in PCWSTR cmdLine, - __in LPCWSTR userName) +DWORD CreateTaskImpl(__in_opt HANDLE logonHandle, __in PCWSTR jobObjName,__in PWSTR cmdLine, + __in LPCWSTR userName, __in long memory, __in long cpuRate) { DWORD dwErrorCode = ERROR_SUCCESS; DWORD exitCode = EXIT_FAILURE; @@ -616,6 +681,12 @@ DWORD CreateTaskImpl(__in_opt HANDLE logonHandle, __in PCWSTR jobObjName,__in PC return dwErrorCode; } jeli.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; + if (memory > 0) + { + jeli.BasicLimitInformation.LimitFlags |= JOB_OBJECT_LIMIT_JOB_MEMORY; + jeli.ProcessMemoryLimit = ((SIZE_T) memory) * 1024 * 1024; + jeli.JobMemoryLimit = ((SIZE_T) memory) * 1024 * 1024; + } if(SetInformationJobObject(jobObject, JobObjectExtendedLimitInformation, &jeli, @@ -626,6 +697,24 @@ DWORD CreateTaskImpl(__in_opt HANDLE logonHandle, __in PCWSTR jobObjName,__in PC CloseHandle(jobObject); return dwErrorCode; } +#ifdef NTDDI_WIN8 + if (cpuRate > 0) + { + JOBOBJECT_CPU_RATE_CONTROL_INFORMATION jcrci = { 0 }; + SYSTEM_INFO sysinfo; + GetSystemInfo(&sysinfo); + jcrci.ControlFlags = JOB_OBJECT_CPU_RATE_CONTROL_ENABLE | + JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP; + jcrci.CpuRate = min(10000, cpuRate); + if(SetInformationJobObject(jobObject, JobObjectCpuRateControlInformation, + &jcrci, sizeof(jcrci)) == 0) + { + dwErrorCode = GetLastError(); + CloseHandle(jobObject); + return dwErrorCode; + } + } +#endif if (logonHandle != NULL) { dwErrorCode = AddNodeManagerAndUserACEsToObject(jobObject, userName, JOB_OBJECT_ALL_ACCESS); @@ -809,10 +898,10 @@ DWORD CreateTaskImpl(__in_opt HANDLE logonHandle, __in PCWSTR jobObjName,__in PC // Returns: // ERROR_SUCCESS: On success // GetLastError: otherwise -DWORD CreateTask(__in PCWSTR jobObjName,__in PWSTR cmdLine) +DWORD CreateTask(__in PCWSTR jobObjName,__in PWSTR cmdLine, __in long memory, __in long cpuRate) { // call with null logon in order to create tasks utilizing the current logon - return CreateTaskImpl( NULL, jobObjName, cmdLine, NULL); + return CreateTaskImpl( NULL, jobObjName, cmdLine, NULL, memory, cpuRate); } //---------------------------------------------------------------------------- @@ -826,7 +915,7 @@ DWORD CreateTask(__in PCWSTR jobObjName,__in PWSTR cmdLine) // ERROR_SUCCESS: On success // GetLastError: otherwise DWORD CreateTaskAsUser(__in PCWSTR jobObjName, - __in PCWSTR user, __in PCWSTR pidFilePath, __in PCWSTR cmdLine) + __in PCWSTR user, __in PCWSTR pidFilePath, __in PWSTR cmdLine) { DWORD err = ERROR_SUCCESS; DWORD exitCode = EXIT_FAILURE; @@ -837,6 +926,7 @@ DWORD CreateTaskAsUser(__in PCWSTR jobObjName, FILE* pidFile = NULL; DWORD retLen = 0; HANDLE logonHandle = NULL; + errno_t pidErrNo = 0; err = EnableImpersonatePrivileges(); if( err != ERROR_SUCCESS ) { @@ -875,8 +965,8 @@ DWORD CreateTaskAsUser(__in PCWSTR jobObjName, profileIsLoaded = TRUE; // Create the PID file - - if (!(pidFile = _wfopen(pidFilePath, "w"))) { + pidErrNo = _wfopen_s(&pidFile, pidFilePath, L"w"); + if (pidErrNo) { err = GetLastError(); ReportErrorCode(L"_wfopen:pidFilePath", err); goto done; @@ -893,7 +983,7 @@ DWORD CreateTaskAsUser(__in PCWSTR jobObjName, goto done; } - err = CreateTaskImpl(logonHandle, jobObjName, cmdLine, user); + err = CreateTaskImpl(logonHandle, jobObjName, cmdLine, user, -1, -1); done: if( profileIsLoaded ) { @@ -1095,6 +1185,8 @@ int Task(__in int argc, __in_ecount(argc) wchar_t *argv[]) { DWORD dwErrorCode = ERROR_SUCCESS; TaskCommandOption command = TaskInvalid; + long memory = -1; + long cpuRate = -1; wchar_t* cmdLine = NULL; wchar_t buffer[16*1024] = L""; // 32K max command line size_t charCountBufferLeft = sizeof(buffer)/sizeof(wchar_t); @@ -1111,7 +1203,7 @@ int Task(__in int argc, __in_ecount(argc) wchar_t *argv[]) ARGC_COMMAND_ARGS }; - if (!ParseCommandLine(argc, argv, &command)) { + if (!ParseCommandLine(argc, argv, &command, &memory, &cpuRate)) { dwErrorCode = ERROR_INVALID_COMMAND_LINE; fwprintf(stderr, L"Incorrect command line arguments.\n\n"); @@ -1123,7 +1215,7 @@ int Task(__in int argc, __in_ecount(argc) wchar_t *argv[]) { // Create the task jobobject // - dwErrorCode = CreateTask(argv[2], argv[3]); + dwErrorCode = CreateTask(argv[argc-2], argv[argc-1], memory, cpuRate); if (dwErrorCode != ERROR_SUCCESS) { ReportErrorCode(L"CreateTask", dwErrorCode); @@ -1238,18 +1330,30 @@ void TaskUsage() // jobobject's are being used. // ProcessTree.isSetsidSupported() fwprintf(stdout, L"\ - Usage: task create [TASKNAME] [COMMAND_LINE] |\n\ - task createAsUser [TASKNAME] [USERNAME] [PIDFILE] [COMMAND_LINE] |\n\ - task isAlive [TASKNAME] |\n\ - task kill [TASKNAME]\n\ - task processList [TASKNAME]\n\ - Creates a new task jobobject with taskname\n\ - Creates a new task jobobject with taskname as the user provided\n\ - Checks if task jobobject is alive\n\ - Kills task jobobject\n\ - Prints to stdout a list of processes in the task\n\ - along with their resource usage. One process per line\n\ - and comma separated info per process\n\ - ProcessId,VirtualMemoryCommitted(bytes),\n\ - WorkingSetSize(bytes),CpuTime(Millisec,Kernel+User)\n"); +Usage: task create [OPTOINS] [TASKNAME] [COMMAND_LINE]\n\ + Creates a new task job object with taskname and options to set CPU\n\ + and memory limits on the job object\n\ +\n\ + OPTIONS: -c [cup rate] set the cpu rate limit on the job object.\n\ + -m [memory] set the memory limit on the job object.\n\ + The cpu limit is an integral value of percentage * 100. The memory\n\ + limit is an integral number of memory in MB. \n\ + The limit will not be set if 0 or negative value is passed in as\n\ + parameter(s).\n\ +\n\ + task createAsUser [TASKNAME] [USERNAME] [PIDFILE] [COMMAND_LINE]\n\ + Creates a new task jobobject with taskname as the user provided\n\ +\n\ + task isAlive [TASKNAME]\n\ + Checks if task job object is alive\n\ +\n\ + task kill [TASKNAME]\n\ + Kills task job object\n\ +\n\ + task processList [TASKNAME]\n\ + Prints to stdout a list of processes in the task\n\ + along with their resource usage. One process per line\n\ + and comma separated info per process\n\ + ProcessId,VirtualMemoryCommitted(bytes),\n\ + WorkingSetSize(bytes),CpuTime(Millisec,Kernel+User)\n"); } diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props b/hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props new file mode 100644 index 0000000000000..29ae2b7bc5442 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/main/winutils/win8sdk.props @@ -0,0 +1,47 @@ + + + + + + + $(MSBuildProgramFiles32)\Windows Kits\8.1\ + + + + $(Windows8SdkDir)Include\um;$(Windows8SdkDir)Include\shared; + $(Windows8SdkDir)bin\x86; + $(Windows8SdkDir)bin\x64; + $(Windows8SdkDir)lib\winv6.3\um\x86; + $(Windows8SdkDir)lib\winv6.3\um\x64; + + + + $(ExecutablePath);$(Windows8SDK_ExecutablePath_x86); + $(Windows8SDK_IncludePath);$(IncludePath); + $(LibraryPath);$(Windows8SDK_LibraryPath_x86); + $(ExcludePath);$(Windows8SDK_IncludePath);$(Windows8SDK_LibraryPath_x86); + + + + $(ExecutablePath);$(Windows8SDK_ExecutablePath_x64); + $(Windows8SDK_IncludePath);$(IncludePath); + $(LibraryPath);$(Windows8SDK_LibraryPath_x64); + $(ExcludePath);$(Windows8SDK_IncludePath);$(Windows8SDK_LibraryPath_x64); + + + diff --git a/hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj b/hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj index 9ecba0a87a7d6..76a74147e1f3b 100644 --- a/hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj +++ b/hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj @@ -67,6 +67,9 @@ + + + diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md b/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md index 33986ae43426d..207160e93629f 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md @@ -128,8 +128,8 @@ Usage: `hadoop credential [options]` | COMMAND\_OPTION | Description | |:---- |:---- | -| create *alias* [-v *value*][-provider *provider-path*] | Prompts the user for a credential to be stored as the given alias when a value is not provided via `-v`. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. | -| delete *alias* [-i][-provider *provider-path*] | Deletes the credential with the provided alias and optionally warns the user when `--interactive` is used. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. | +| create *alias* [-provider *provider-path*] | Prompts the user for a credential to be stored as the given alias. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. | +| delete *alias* [-provider *provider-path*] [-f] | Deletes the credential with the provided alias. The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. The command asks for confirmation unless `-f` is specified | | list [-provider *provider-path*] | Lists all of the credential aliases The *hadoop.security.credential.provider.path* within the core-site.xml file will be used unless a `-provider` is indicated. | Command to manage credentials, passwords and secrets within credential providers. diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md index 0e0fc091cf6ba..4a10a008a2ac8 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md @@ -303,6 +303,8 @@ Each metrics record contains tags such as SessionId and Hostname as additional i | `HeartbeatsAvgTime` | Average heartbeat time in milliseconds | | `BlockReportsNumOps` | Total number of block report operations | | `BlockReportsAvgTime` | Average time of block report operations in milliseconds | +| `IncrementalBlockReportsNumOps` | Total number of incremental block report operations | +| `IncrementalBlockReportsAvgTime` | Average time of incremental block report operations in milliseconds | | `CacheReportsNumOps` | Total number of cache report operations | | `CacheReportsAvgTime` | Average time of cache report operations in milliseconds | | `PacketAckRoundTripTimeNanosNumOps` | Total number of ack round trip | diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md b/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md new file mode 100644 index 0000000000000..006fca5c49166 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md @@ -0,0 +1,98 @@ + + +# Unix Shell Guide + +Much of Hadoop's functionality is controlled via [the shell](CommandsManual.html). There are several ways to modify the default behavior of how these commands execute. + +## Important End-User Environment Variables + +Hadoop has many environment variables that control various aspects of the software. (See `hadoop-env.sh` and related files.) Some of these environment variables are dedicated to helping end users manage their runtime. + +### `HADOOP_CLIENT_OPTS` + +This environment variable is used for almost all end-user operations. It can be used to set any Java options as well as any Hadoop options via a system property definition. For example: + +```bash +HADOOP_CLIENT_OPTS="-Xmx1g -Dhadoop.socks.server=localhost:4000" hadoop fs -ls /tmp +``` + +will increase the memory and send this command via a SOCKS proxy server. + +### `HADOOP_USER_CLASSPATH` + +The Hadoop scripts have the capability to inject more content into the classpath of the running command by setting this environment variable. It should be a colon delimited list of directories, files, or wildcard locations. + +```bash +HADOOP_USER_CLASSPATH=${HOME}/lib/myjars/*.jar hadoop classpath +``` + +A user can provides hints to the location of the paths via the `HADOOP_USER_CLASSPATH_FIRST` variable. Setting this to any value will tell the system to try and push these paths near the front. + +### Auto-setting of Variables + +If a user has a common set of settings, they can be put into the `${HOME}/.hadooprc` file. This file is always read to initialize and override any variables that the user may want to customize. It uses bash syntax, similar to the `.bashrc` file: + +For example: + +```bash +# +# my custom Hadoop settings! +# + +HADOOP_USER_CLASSPATH=${HOME}/hadoopjars/* +HADOOP_USER_CLASSPATH_FIRST=yes +HADOOP_CLIENT_OPTS="-Xmx1g" +``` + +The `.hadooprc` file can also be used to extend functionality and teach Hadoop new tricks. For example, to run hadoop commands accessing the server referenced in the environment variable `${HADOOP_SERVER}`, the following in the `.hadooprc` will do just that: + +```bash + +if [[ -n ${HADOOP_SERVER} ]]; then + HADOOP_CONF_DIR=/etc/hadoop.${HADOOP_SERVER} +fi +``` + +## Administrator Environment + +There are many environment variables that impact how the system operates. By far, the most important are the series of `_OPTS` variables that control how daemons work. These variables should contain all of the relevant settings for those daemons. + +More, detailed information is contained in `hadoop-env.sh` and the other env.sh files. + +Advanced administrators may wish to supplement or do some platform-specific fixes to the existing scripts. In some systems, this means copying the errant script or creating a custom build with these changes. Hadoop provides the capabilities to do function overrides so that the existing code base may be changed in place without all of that work. Replacing functions is covered later under the Shell API documentation. + +## Developer and Advanced Administrator Environment + +### Shell Profiles + +Apache Hadoop allows for third parties to easily add new features through a variety of pluggable interfaces. This includes a shell code subsystem that makes it easy to inject the necessary content into the base installation. + +Core to this functionality is the concept of a shell profile. Shell profiles are shell snippets that can do things such as add jars to the classpath, configure Java system properties and more. + +Shell profiles may be installed in either `${HADOOP_CONF_DIR}/shellprofile.d` or `${HADOOP_PREFIX}/libexec/shellprofile.d`. Shell profiles in the `libexec` directory are part of the base installation and cannot be overriden by the user. Shell profiles in the configuration directory may be ignored if the end user changes the configuration directory at runtime. + +An example of a shell profile is in the libexec directory. + +## Shell API + +Hadoop's shell code has a [function library](./UnixShellAPI.html) that is open for administrators and developers to use to assist in their configuration and advanced feature management. These APIs follow the standard [Hadoop Interface Classification](./InterfaceClassification.html), with one addition: Replaceable. + +The shell code allows for core functions to be overridden. However, not all functions can be or are safe to be replaced. If a function is not safe to replace, it will have an attribute of Replaceable: No. If a function is safe to replace, it will have the attribute of Replaceable: Yes. + +In order to replace a function, create a file called `hadoop-user-functions.sh` in the `${HADOOP_CONF_DIR}` directory. Simply define the new, replacement function in this file and the system will pick it up automatically. There may be as many replacement functions as needed in this file. Examples of function replacement are in the `hadoop-user-functions.sh.examples` file. + + +Functions that are marked Public and Stable are safe to use in shell profiles as-is. Other functions may change in a minor release. + diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java index 765a364faa6d5..0a65085e167af 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java @@ -47,15 +47,9 @@ public class TestCryptoStreamsForLocalFS extends CryptoStreamsTestBase { @BeforeClass public static void init() throws Exception { - Configuration conf = new Configuration(); - conf = new Configuration(false); + Configuration conf = new Configuration(false); conf.set("fs.file.impl", LocalFileSystem.class.getName()); fileSys = FileSystem.getLocal(conf); - conf.set( - CommonConfigurationKeysPublic.HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_KEY_PREFIX - + CipherSuite.AES_CTR_NOPADDING.getConfigSuffix(), - OpensslAesCtrCryptoCodec.class.getName() + "," - + JceAesCtrCryptoCodec.class.getName()); codec = CryptoCodec.getInstance(conf); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java new file mode 100644 index 0000000000000..76c39d694b04c --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithJceAesCtrCryptoCodec.java @@ -0,0 +1,38 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.crypto; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeysPublic; +import org.junit.Assert; +import org.junit.BeforeClass; + +public class TestCryptoStreamsWithJceAesCtrCryptoCodec extends + TestCryptoStreams { + + @BeforeClass + public static void init() throws Exception { + Configuration conf = new Configuration(); + conf.set( + CommonConfigurationKeysPublic.HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY, + JceAesCtrCryptoCodec.class.getName()); + codec = CryptoCodec.getInstance(conf); + Assert.assertEquals(JceAesCtrCryptoCodec.class.getCanonicalName(), + codec.getClass().getCanonicalName()); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java index f64e8dcca698d..4f90a0c27fb0e 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java @@ -18,6 +18,8 @@ package org.apache.hadoop.crypto; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeysPublic; +import org.junit.Assert; import org.junit.BeforeClass; public class TestCryptoStreamsWithOpensslAesCtrCryptoCodec @@ -26,6 +28,11 @@ public class TestCryptoStreamsWithOpensslAesCtrCryptoCodec @BeforeClass public static void init() throws Exception { Configuration conf = new Configuration(); + conf.set( + CommonConfigurationKeysPublic.HADOOP_SECURITY_CRYPTO_CODEC_CLASSES_AES_CTR_NOPADDING_KEY, + OpensslAesCtrCryptoCodec.class.getName()); codec = CryptoCodec.getInstance(conf); + Assert.assertEquals(OpensslAesCtrCryptoCodec.class.getCanonicalName(), + codec.getClass().getCanonicalName()); } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestContentSummary.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestContentSummary.java index 5db0de33943b8..7cc7ae4094974 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestContentSummary.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestContentSummary.java @@ -32,13 +32,13 @@ public class TestContentSummary { // check the empty constructor correctly initialises the object @Test public void testConstructorEmpty() { - ContentSummary contentSummary = new ContentSummary(); + ContentSummary contentSummary = new ContentSummary.Builder().build(); assertEquals("getLength", 0, contentSummary.getLength()); assertEquals("getFileCount", 0, contentSummary.getFileCount()); assertEquals("getDirectoryCount", 0, contentSummary.getDirectoryCount()); - assertEquals("getQuota", 0, contentSummary.getQuota()); + assertEquals("getQuota", -1, contentSummary.getQuota()); assertEquals("getSpaceConsumed", 0, contentSummary.getSpaceConsumed()); - assertEquals("getSpaceQuota", 0, contentSummary.getSpaceQuota()); + assertEquals("getSpaceQuota", -1, contentSummary.getSpaceQuota()); } // check the full constructor with quota information @@ -51,8 +51,9 @@ public void testConstructorWithQuota() { long spaceConsumed = 55555; long spaceQuota = 66666; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); assertEquals("getLength", length, contentSummary.getLength()); assertEquals("getFileCount", fileCount, contentSummary.getFileCount()); assertEquals("getDirectoryCount", directoryCount, @@ -70,8 +71,9 @@ public void testConstructorNoQuota() { long fileCount = 22222; long directoryCount = 33333; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount). + spaceConsumed(length).build(); assertEquals("getLength", length, contentSummary.getLength()); assertEquals("getFileCount", fileCount, contentSummary.getFileCount()); assertEquals("getDirectoryCount", directoryCount, @@ -91,8 +93,9 @@ public void testWrite() throws IOException { long spaceConsumed = 55555; long spaceQuota = 66666; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); DataOutput out = mock(DataOutput.class); InOrder inOrder = inOrder(out); @@ -116,7 +119,7 @@ public void testReadFields() throws IOException { long spaceConsumed = 55555; long spaceQuota = 66666; - ContentSummary contentSummary = new ContentSummary(); + ContentSummary contentSummary = new ContentSummary.Builder().build(); DataInput in = mock(DataInput.class); when(in.readLong()).thenReturn(length).thenReturn(fileCount) @@ -159,8 +162,9 @@ public void testToStringWithQuota() { long spaceConsumed = 55555; long spaceQuota = 66665; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); String expected = " 44444 -11111 66665 11110" + " 33333 22222 11111 "; assertEquals(expected, contentSummary.toString(true)); @@ -173,8 +177,8 @@ public void testToStringNoQuota() { long fileCount = 22222; long directoryCount = 33333; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).build(); String expected = " none inf none" + " inf 33333 22222 11111 "; assertEquals(expected, contentSummary.toString(true)); @@ -190,8 +194,9 @@ public void testToStringNoShowQuota() { long spaceConsumed = 55555; long spaceQuota = 66665; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); String expected = " 33333 22222 11111 "; assertEquals(expected, contentSummary.toString(false)); } @@ -206,8 +211,9 @@ public void testToString() { long spaceConsumed = 55555; long spaceQuota = 66665; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); String expected = " 44444 -11111 66665" + " 11110 33333 22222 11111 "; assertEquals(expected, contentSummary.toString()); @@ -223,8 +229,9 @@ public void testToStringHumanWithQuota() { long spaceConsumed = 1073741825; long spaceQuota = 1; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); String expected = " 212.0 M 1023 1 " + " -1 G 32.6 K 211.9 M 8.0 E "; assertEquals(expected, contentSummary.toString(true, true)); @@ -240,8 +247,9 @@ public void testToStringHumanNoShowQuota() { long spaceConsumed = 55555; long spaceQuota = Long.MAX_VALUE; - ContentSummary contentSummary = new ContentSummary(length, fileCount, - directoryCount, quota, spaceConsumed, spaceQuota); + ContentSummary contentSummary = new ContentSummary.Builder().length(length). + fileCount(fileCount).directoryCount(directoryCount).quota(quota). + spaceConsumed(spaceConsumed).spaceQuota(spaceQuota).build(); String expected = " 32.6 K 211.9 M 8.0 E "; assertEquals(expected, contentSummary.toString(false, true)); } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegateToFileSystem.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegateToFileSystem.java new file mode 100644 index 0000000000000..5de32861db68d --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegateToFileSystem.java @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.fs; + +import java.net.URI; + +import org.apache.commons.net.ftp.FTP; +import org.apache.hadoop.conf.Configuration; +import org.junit.Assert; +import org.junit.Test; + +public class TestDelegateToFileSystem { + + private static final String FTP_DUMMYHOST = "ftp://dummyhost"; + private static final URI FTP_URI_NO_PORT = URI.create(FTP_DUMMYHOST); + private static final URI FTP_URI_WITH_PORT = URI.create(FTP_DUMMYHOST + ":" + + FTP.DEFAULT_PORT); + + private void testDefaultUriInternal(String defaultUri) + throws UnsupportedFileSystemException { + final Configuration conf = new Configuration(); + FileSystem.setDefaultUri(conf, defaultUri); + final AbstractFileSystem ftpFs = + AbstractFileSystem.get(FTP_URI_NO_PORT, conf); + Assert.assertEquals(FTP_URI_WITH_PORT, ftpFs.getUri()); + } + + @Test + public void testDefaultURIwithOutPort() throws Exception { + testDefaultUriInternal("hdfs://dummyhost"); + } + + @Test + public void testDefaultURIwithPort() throws Exception { + testDefaultUriInternal("hdfs://dummyhost:8020"); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java index 1f2f2d4804632..d5f097d38655b 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java @@ -315,6 +315,8 @@ public void getDescription() { // mock content system static class MockContentSummary extends ContentSummary { + @SuppressWarnings("deprecation") + // suppress warning on the usage of deprecated ContentSummary constructor public MockContentSummary() { } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java index 7d0727abd15c3..5f031337b75f0 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/ClientBaseWithFixes.java @@ -90,6 +90,14 @@ protected static class CountdownWatcher implements Watcher { // XXX this doesn't need to be volatile! (Should probably be final) volatile CountDownLatch clientConnected; volatile boolean connected; + protected ZooKeeper client; + + public void initializeWatchedClient(ZooKeeper zk) { + if (client != null) { + throw new RuntimeException("Watched Client was already set"); + } + client = zk; + } public CountdownWatcher() { reset(); @@ -191,8 +199,7 @@ protected TestableZooKeeper createClient(CountdownWatcher watcher, zk.close(); } } - - + watcher.initializeWatchedClient(zk); return zk; } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java index 98b3934ea4800..7246bf50bb1ea 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java @@ -74,6 +74,7 @@ import org.junit.Assume; import org.junit.Test; import static org.junit.Assert.*; +import static org.junit.Assume.*; public class TestCodec { @@ -364,10 +365,7 @@ private static Path writeSplitTestFile(FileSystem fs, Random rand, public void testCodecPoolGzipReuse() throws Exception { Configuration conf = new Configuration(); conf.setBoolean(CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_KEY, true); - if (!ZlibFactory.isNativeZlibLoaded(conf)) { - LOG.warn("testCodecPoolGzipReuse skipped: native libs not loaded"); - return; - } + assumeTrue(ZlibFactory.isNativeZlibLoaded(conf)); GzipCodec gzc = ReflectionUtils.newInstance(GzipCodec.class, conf); DefaultCodec dfc = ReflectionUtils.newInstance(DefaultCodec.class, conf); Compressor c1 = CodecPool.getCompressor(gzc); @@ -723,10 +721,7 @@ public void testBuiltInGzipConcat() throws IOException { public void testNativeGzipConcat() throws IOException { Configuration conf = new Configuration(); conf.setBoolean(CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_KEY, true); - if (!ZlibFactory.isNativeZlibLoaded(conf)) { - LOG.warn("skipped: native libs not loaded"); - return; - } + assumeTrue(ZlibFactory.isNativeZlibLoaded(conf)); GzipConcatTest(conf, GzipCodec.GzipZlibDecompressor.class); } @@ -840,10 +835,7 @@ public void testGzipCodecWrite(boolean useNative) throws IOException { Configuration conf = new Configuration(); conf.setBoolean(CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_KEY, useNative); if (useNative) { - if (!ZlibFactory.isNativeZlibLoaded(conf)) { - LOG.warn("testGzipCodecWrite skipped: native libs not loaded"); - return; - } + assumeTrue(ZlibFactory.isNativeZlibLoaded(conf)); } else { assertFalse("ZlibFactory is using native libs against request", ZlibFactory.isNativeZlibLoaded(conf)); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java new file mode 100644 index 0000000000000..c585a463e46b1 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/bzip2/TestBzip2CompressorDecompressor.java @@ -0,0 +1,104 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.io.compress.bzip2; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.DataInputBuffer; +import org.apache.hadoop.io.DataOutputBuffer; +import org.apache.hadoop.io.compress.*; +import org.apache.hadoop.io.compress.bzip2.Bzip2Compressor; +import org.apache.hadoop.io.compress.bzip2.Bzip2Decompressor; +import org.apache.hadoop.test.MultithreadedTestUtil; +import org.junit.Before; +import org.junit.Test; + +import java.io.*; +import java.util.Random; + +import static org.junit.Assert.*; +import static org.junit.Assume.*; +import static org.junit.Assume.assumeTrue; + +public class TestBzip2CompressorDecompressor { + + private static final Random rnd = new Random(12345l); + + @Before + public void before() { + assumeTrue(Bzip2Factory.isNativeBzip2Loaded(new Configuration())); + } + + // test compress/decompress process + @Test + public void testCompressDecompress() { + byte[] rawData = null; + int rawDataSize = 0; + rawDataSize = 1024 * 64; + rawData = generate(rawDataSize); + try { + Bzip2Compressor compressor = new Bzip2Compressor(); + Bzip2Decompressor decompressor = new Bzip2Decompressor(); + assertFalse("testBzip2CompressDecompress finished error", + compressor.finished()); + compressor.setInput(rawData, 0, rawData.length); + assertTrue("testBzip2CompressDecompress getBytesRead before error", + compressor.getBytesRead() == 0); + compressor.finish(); + + byte[] compressedResult = new byte[rawDataSize]; + int cSize = compressor.compress(compressedResult, 0, rawDataSize); + assertTrue("testBzip2CompressDecompress getBytesRead after error", + compressor.getBytesRead() == rawDataSize); + assertTrue( + "testBzip2CompressDecompress compressed size no less than original size", + cSize < rawDataSize); + decompressor.setInput(compressedResult, 0, cSize); + byte[] decompressedBytes = new byte[rawDataSize]; + decompressor.decompress(decompressedBytes, 0, decompressedBytes.length); + assertArrayEquals("testBzip2CompressDecompress arrays not equals ", + rawData, decompressedBytes); + compressor.reset(); + decompressor.reset(); + } catch (IOException ex) { + fail("testBzip2CompressDecompress ex !!!" + ex); + } + } + + public static byte[] generate(int size) { + byte[] array = new byte[size]; + for (int i = 0; i < size; i++) + array[i] = (byte)rnd.nextInt(16); + return array; + } + + @Test + public void testBzip2CompressDecompressInMultiThreads() throws Exception { + MultithreadedTestUtil.TestContext ctx = new MultithreadedTestUtil.TestContext(); + for(int i=0;i<10;i++) { + ctx.addThread( new MultithreadedTestUtil.TestingThread(ctx) { + @Override + public void doWork() throws Exception { + testCompressDecompress(); + } + }); + } + ctx.startThreads(); + + ctx.waitFor(60000); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java index e8555b23887cc..6f3b076097aee 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/lz4/TestLz4CompressorDecompressor.java @@ -36,6 +36,7 @@ import org.apache.hadoop.io.compress.Lz4Codec; import org.apache.hadoop.io.compress.lz4.Lz4Compressor; import org.apache.hadoop.io.compress.lz4.Lz4Decompressor; +import org.apache.hadoop.test.MultithreadedTestUtil; import org.junit.Before; import org.junit.Test; import static org.junit.Assume.*; @@ -313,4 +314,20 @@ public static byte[] generate(int size) { array[i] = (byte)rnd.nextInt(16); return array; } + + @Test + public void testLz4CompressDecompressInMultiThreads() throws Exception { + MultithreadedTestUtil.TestContext ctx = new MultithreadedTestUtil.TestContext(); + for(int i=0;i<10;i++) { + ctx.addThread( new MultithreadedTestUtil.TestingThread(ctx) { + @Override + public void doWork() throws Exception { + testCompressDecompress(); + } + }); + } + ctx.startThreads(); + + ctx.waitFor(60000); + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java index 77fbcc099ac99..cc986c7e0aea4 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/snappy/TestSnappyCompressorDecompressor.java @@ -40,6 +40,7 @@ import org.apache.hadoop.io.compress.CompressionOutputStream; import org.apache.hadoop.io.compress.SnappyCodec; import org.apache.hadoop.io.compress.snappy.SnappyDecompressor.SnappyDirectDecompressor; +import org.apache.hadoop.test.MultithreadedTestUtil; import org.junit.Assert; import org.junit.Before; import org.junit.Test; @@ -391,4 +392,20 @@ public static byte[] get(int size) { return array; } } + + @Test + public void testSnappyCompressDecompressInMultiThreads() throws Exception { + MultithreadedTestUtil.TestContext ctx = new MultithreadedTestUtil.TestContext(); + for(int i=0;i<10;i++) { + ctx.addThread( new MultithreadedTestUtil.TestingThread(ctx) { + @Override + public void doWork() throws Exception { + testSnappyCompressDecompress(); + } + }); + } + ctx.startThreads(); + + ctx.waitFor(60000); + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java index db5784c07aa86..e7511251b648f 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zlib/TestZlibCompressorDecompressor.java @@ -38,6 +38,7 @@ import org.apache.hadoop.io.compress.zlib.ZlibCompressor.CompressionLevel; import org.apache.hadoop.io.compress.zlib.ZlibCompressor.CompressionStrategy; import org.apache.hadoop.io.compress.zlib.ZlibDecompressor.ZlibDirectDecompressor; +import org.apache.hadoop.test.MultithreadedTestUtil; import org.apache.hadoop.util.NativeCodeLoader; import org.junit.Before; import org.junit.Test; @@ -419,4 +420,20 @@ public static byte[] generate(int size) { data[i] = (byte)random.nextInt(16); return data; } + + @Test + public void testZlibCompressDecompressInMultiThreads() throws Exception { + MultithreadedTestUtil.TestContext ctx = new MultithreadedTestUtil.TestContext(); + for(int i=0;i<10;i++) { + ctx.addThread( new MultithreadedTestUtil.TestingThread(ctx) { + @Override + public void doWork() throws Exception { + testZlibCompressDecompress(); + } + }); + } + ctx.startThreads(); + + ctx.waitFor(60000); + } } diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java index e85e414135418..4b0e2a80b7b18 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/unix/TestDomainSocketWatcher.java @@ -195,7 +195,7 @@ public void run() { private DomainSocketWatcher newDomainSocketWatcher(int interruptCheckPeriodMs) throws Exception { DomainSocketWatcher watcher = new DomainSocketWatcher( - interruptCheckPeriodMs); + interruptCheckPeriodMs, getClass().getSimpleName()); watcher.watcherThread.setUncaughtExceptionHandler( new Thread.UncaughtExceptionHandler() { @Override diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java index b6aae0eb6377d..c8179e2edc076 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java @@ -37,17 +37,6 @@ public class TestAuthenticationFilter extends TestCase { public void testConfiguration() throws Exception { Configuration conf = new Configuration(); conf.set("hadoop.http.authentication.foo", "bar"); - - File testDir = new File(System.getProperty("test.build.data", - "target/test-dir")); - testDir.mkdirs(); - File secretFile = new File(testDir, "http-secret.txt"); - Writer writer = new FileWriter(new File(testDir, "http-secret.txt")); - writer.write("hadoop"); - writer.close(); - conf.set(AuthenticationFilterInitializer.PREFIX + - AuthenticationFilterInitializer.SIGNATURE_SECRET_FILE, - secretFile.getAbsolutePath()); conf.set(HttpServer2.BIND_ADDRESS, "barhost"); @@ -68,7 +57,6 @@ public Object answer(InvocationOnMock invocationOnMock) assertEquals("simple", conf.get("type")); assertEquals("36000", conf.get("token.validity")); - assertEquals("hadoop", conf.get("signature.secret")); assertNull(conf.get("cookie.domain")); assertEquals("true", conf.get("simple.anonymous.allowed")); assertEquals("HTTP/barhost@LOCALHOST", diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMapping.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMapping.java index a021a8afd4a98..17a14d18894ed 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMapping.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMapping.java @@ -29,13 +29,7 @@ import java.util.List; import javax.naming.CommunicationException; -import javax.naming.NamingEnumeration; import javax.naming.NamingException; -import javax.naming.directory.Attribute; -import javax.naming.directory.Attributes; -import javax.naming.directory.BasicAttribute; -import javax.naming.directory.BasicAttributes; -import javax.naming.directory.DirContext; import javax.naming.directory.SearchControls; import javax.naming.directory.SearchResult; @@ -49,46 +43,12 @@ import org.junit.Test; @SuppressWarnings("unchecked") -public class TestLdapGroupsMapping { - private DirContext mockContext; - - private LdapGroupsMapping mappingSpy = spy(new LdapGroupsMapping()); - private NamingEnumeration mockUserNamingEnum = mock(NamingEnumeration.class); - private NamingEnumeration mockGroupNamingEnum = mock(NamingEnumeration.class); - private String[] testGroups = new String[] {"group1", "group2"}; - +public class TestLdapGroupsMapping extends TestLdapGroupsMappingBase { @Before public void setupMocks() throws NamingException { - mockContext = mock(DirContext.class); - doReturn(mockContext).when(mappingSpy).getDirContext(); - SearchResult mockUserResult = mock(SearchResult.class); - // We only ever call hasMoreElements once for the user NamingEnum, so - // we can just have one return value - when(mockUserNamingEnum.hasMoreElements()).thenReturn(true); when(mockUserNamingEnum.nextElement()).thenReturn(mockUserResult); when(mockUserResult.getNameInNamespace()).thenReturn("CN=some_user,DC=test,DC=com"); - - SearchResult mockGroupResult = mock(SearchResult.class); - // We're going to have to define the loop here. We want two iterations, - // to get both the groups - when(mockGroupNamingEnum.hasMoreElements()).thenReturn(true, true, false); - when(mockGroupNamingEnum.nextElement()).thenReturn(mockGroupResult); - - // Define the attribute for the name of the first group - Attribute group1Attr = new BasicAttribute("cn"); - group1Attr.add(testGroups[0]); - Attributes group1Attrs = new BasicAttributes(); - group1Attrs.put(group1Attr); - - // Define the attribute for the name of the second group - Attribute group2Attr = new BasicAttribute("cn"); - group2Attr.add(testGroups[1]); - Attributes group2Attrs = new BasicAttributes(); - group2Attrs.put(group2Attr); - - // This search result gets reused, so return group1, then group2 - when(mockGroupResult.getAttributes()).thenReturn(group1Attrs, group2Attrs); } @Test diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingBase.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingBase.java new file mode 100644 index 0000000000000..c54ac4c7367d4 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingBase.java @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.security; + +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +import javax.naming.NamingEnumeration; +import javax.naming.NamingException; +import javax.naming.directory.Attribute; +import javax.naming.directory.Attributes; +import javax.naming.directory.BasicAttribute; +import javax.naming.directory.BasicAttributes; +import javax.naming.directory.DirContext; +import javax.naming.directory.SearchResult; + +import org.junit.Before; + +public class TestLdapGroupsMappingBase { + protected DirContext mockContext; + + protected LdapGroupsMapping mappingSpy = spy(new LdapGroupsMapping()); + protected NamingEnumeration mockUserNamingEnum = + mock(NamingEnumeration.class); + protected NamingEnumeration mockGroupNamingEnum = + mock(NamingEnumeration.class); + protected String[] testGroups = new String[] {"group1", "group2"}; + + @Before + public void setupMocksBase() throws NamingException { + mockContext = mock(DirContext.class); + doReturn(mockContext).when(mappingSpy).getDirContext(); + + // We only ever call hasMoreElements once for the user NamingEnum, so + // we can just have one return value + when(mockUserNamingEnum.hasMoreElements()).thenReturn(true); + + SearchResult mockGroupResult = mock(SearchResult.class); + // We're going to have to define the loop here. We want two iterations, + // to get both the groups + when(mockGroupNamingEnum.hasMoreElements()).thenReturn(true, true, false); + when(mockGroupNamingEnum.nextElement()).thenReturn(mockGroupResult); + + // Define the attribute for the name of the first group + Attribute group1Attr = new BasicAttribute("cn"); + group1Attr.add(testGroups[0]); + Attributes group1Attrs = new BasicAttributes(); + group1Attrs.put(group1Attr); + + // Define the attribute for the name of the second group + Attribute group2Attr = new BasicAttribute("cn"); + group2Attr.add(testGroups[1]); + Attributes group2Attrs = new BasicAttributes(); + group2Attrs.put(group2Attr); + + // This search result gets reused, so return group1, then group2 + when(mockGroupResult.getAttributes()).thenReturn(group1Attrs, group2Attrs); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithPosixGroup.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithPosixGroup.java new file mode 100644 index 0000000000000..1d1a354b11fcd --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestLdapGroupsMappingWithPosixGroup.java @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.security; + +import static org.mockito.Matchers.anyString; + +import static org.mockito.Mockito.any; +import static org.mockito.Mockito.contains; +import static org.mockito.Mockito.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.Arrays; +import java.util.List; + +import javax.naming.NamingException; +import javax.naming.directory.Attribute; +import javax.naming.directory.Attributes; +import javax.naming.directory.SearchControls; +import javax.naming.directory.SearchResult; + +import org.apache.hadoop.conf.Configuration; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +@SuppressWarnings("unchecked") +public class TestLdapGroupsMappingWithPosixGroup + extends TestLdapGroupsMappingBase { + + @Before + public void setupMocks() throws NamingException { + SearchResult mockUserResult = mock(SearchResult.class); + when(mockUserNamingEnum.nextElement()).thenReturn(mockUserResult); + + Attribute mockUidAttr = mock(Attribute.class); + Attribute mockGidAttr = mock(Attribute.class); + Attributes mockAttrs = mock(Attributes.class); + + when(mockUidAttr.get()).thenReturn("700"); + when(mockGidAttr.get()).thenReturn("600"); + when(mockAttrs.get(eq("uidNumber"))).thenReturn(mockUidAttr); + when(mockAttrs.get(eq("gidNumber"))).thenReturn(mockGidAttr); + + when(mockUserResult.getAttributes()).thenReturn(mockAttrs); + } + + @Test + public void testGetGroups() throws IOException, NamingException { + // The search functionality of the mock context is reused, so we will + // return the user NamingEnumeration first, and then the group + when(mockContext.search(anyString(), contains("posix"), + any(Object[].class), any(SearchControls.class))) + .thenReturn(mockUserNamingEnum, mockGroupNamingEnum); + + doTestGetGroups(Arrays.asList(testGroups), 2); + } + + private void doTestGetGroups(List expectedGroups, int searchTimes) + throws IOException, NamingException { + Configuration conf = new Configuration(); + // Set this, so we don't throw an exception + conf.set(LdapGroupsMapping.LDAP_URL_KEY, "ldap://test"); + conf.set(LdapGroupsMapping.GROUP_SEARCH_FILTER_KEY, + "(objectClass=posixGroup)(cn={0})"); + conf.set(LdapGroupsMapping.USER_SEARCH_FILTER_KEY, + "(objectClass=posixAccount)"); + conf.set(LdapGroupsMapping.GROUP_MEMBERSHIP_ATTR_KEY, "memberUid"); + conf.set(LdapGroupsMapping.GROUP_NAME_ATTR_KEY, "cn"); + + mappingSpy.setConf(conf); + // Username is arbitrary, since the spy is mocked to respond the same, + // regardless of input + List groups = mappingSpy.getGroups("some_user"); + + Assert.assertEquals(expectedGroups, groups); + + // We should have searched for a user, and then two groups + verify(mockContext, times(searchTimes)).search(anyString(), + anyString(), + any(Object[].class), + any(SearchControls.class)); + } +} diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java index aecdc8f58e684..ecc6c906ab90d 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java @@ -96,6 +96,22 @@ public void testVerifyChunkedSumsFail() throws ChecksumException { checksums, data, fileName, BASE_POSITION); } + @Test + public void testVerifyChunkedSumsSuccessOddSize() throws ChecksumException { + // Test checksum with an odd number of bytes. This is a corner case that + // is often broken in checksum calculation, because there is an loop which + // handles an even multiple or 4 or 8 bytes and then some additional code + // to finish the few odd bytes at the end. This code can often be broken + // but is never tested because we are always calling it with an even value + // such as 512. + bytesPerChecksum--; + allocateDirectByteBuffers(); + fillDataAndValidChecksums(); + NativeCrc32.verifyChunkedSums(bytesPerChecksum, checksumType.id, + checksums, data, fileName, BASE_POSITION); + bytesPerChecksum++; + } + @Test public void testVerifyChunkedSumsByteArraySuccess() throws ChecksumException { allocateArrayByteBuffers(); diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java index 8ac6e40a0452d..987c7068a82c3 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestWinUtils.java @@ -547,4 +547,66 @@ public void testTaskCreate() throws IOException { assertThat(outNumber, containsString(testNumber)); } + + @Test (timeout = 30000) + public void testTaskCreateWithLimits() throws IOException { + // Generate a unique job id + String jobId = String.format("%f", Math.random()); + + // Run a task without any options + String out = Shell.execCommand(Shell.WINUTILS, "task", "create", + "job" + jobId, "cmd /c echo job" + jobId); + assertTrue(out.trim().equals("job" + jobId)); + + // Run a task without any limits + jobId = String.format("%f", Math.random()); + out = Shell.execCommand(Shell.WINUTILS, "task", "create", "-c", "-1", "-m", + "-1", "job" + jobId, "cmd /c echo job" + jobId); + assertTrue(out.trim().equals("job" + jobId)); + + // Run a task with limits (128MB should be enough for a cmd) + jobId = String.format("%f", Math.random()); + out = Shell.execCommand(Shell.WINUTILS, "task", "create", "-c", "10000", "-m", + "128", "job" + jobId, "cmd /c echo job" + jobId); + assertTrue(out.trim().equals("job" + jobId)); + + // Run a task without enough memory + try { + jobId = String.format("%f", Math.random()); + out = Shell.execCommand(Shell.WINUTILS, "task", "create", "-m", "128", "job" + + jobId, "java -Xmx256m -version"); + fail("Failed to get Shell.ExitCodeException with insufficient memory"); + } catch (Shell.ExitCodeException ece) { + assertThat(ece.getExitCode(), is(1)); + } + + // Run tasks with wrong parameters + // + try { + jobId = String.format("%f", Math.random()); + Shell.execCommand(Shell.WINUTILS, "task", "create", "-c", "-1", "-m", + "-1", "foo", "job" + jobId, "cmd /c echo job" + jobId); + fail("Failed to get Shell.ExitCodeException with bad parameters"); + } catch (Shell.ExitCodeException ece) { + assertThat(ece.getExitCode(), is(1639)); + } + + try { + jobId = String.format("%f", Math.random()); + Shell.execCommand(Shell.WINUTILS, "task", "create", "-c", "-m", "-1", + "job" + jobId, "cmd /c echo job" + jobId); + fail("Failed to get Shell.ExitCodeException with bad parameters"); + } catch (Shell.ExitCodeException ece) { + assertThat(ece.getExitCode(), is(1639)); + } + + try { + jobId = String.format("%f", Math.random()); + Shell.execCommand(Shell.WINUTILS, "task", "create", "-c", "foo", + "job" + jobId, "cmd /c echo job" + jobId); + fail("Failed to get Shell.ExitCodeException with bad parameters"); + } catch (Shell.ExitCodeException ece) { + assertThat(ece.getExitCode(), is(1639)); + } + } } diff --git a/hadoop-common-project/hadoop-minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java b/hadoop-common-project/hadoop-minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java index a649bd224e9ca..938836051e0d1 100644 --- a/hadoop-common-project/hadoop-minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java +++ b/hadoop-common-project/hadoop-minikdc/src/main/java/org/apache/hadoop/minikdc/MiniKdc.java @@ -36,6 +36,7 @@ import org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex; import org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition; import org.apache.directory.server.core.partition.ldif.LdifPartition; +import org.apache.directory.server.kerberos.KerberosConfig; import org.apache.directory.server.kerberos.kdc.KdcServer; import org.apache.directory.server.kerberos.shared.crypto.encryption.KerberosKeyFactory; import org.apache.directory.server.kerberos.shared.keytab.Keytab; @@ -418,7 +419,15 @@ private void initKDCServer() throws Exception { IOUtils.closeQuietly(is1); } - kdc = new KdcServer(); + KerberosConfig kerberosConfig = new KerberosConfig(); + kerberosConfig.setMaximumRenewableLifetime(Long.parseLong(conf + .getProperty(MAX_RENEWABLE_LIFETIME))); + kerberosConfig.setMaximumTicketLifetime(Long.parseLong(conf + .getProperty(MAX_TICKET_LIFETIME))); + kerberosConfig.setSearchBaseDn(String.format("dc=%s,dc=%s", orgName, + orgDomain)); + kerberosConfig.setPaEncTimestampRequired(false); + kdc = new KdcServer(kerberosConfig); kdc.setDirectoryService(ds); // transport @@ -431,12 +440,6 @@ private void initKDCServer() throws Exception { throw new IllegalArgumentException("Invalid transport: " + transport); } kdc.setServiceName(conf.getProperty(INSTANCE)); - kdc.getConfig().setMaximumRenewableLifetime( - Long.parseLong(conf.getProperty(MAX_RENEWABLE_LIFETIME))); - kdc.getConfig().setMaximumTicketLifetime( - Long.parseLong(conf.getProperty(MAX_TICKET_LIFETIME))); - - kdc.getConfig().setPaEncTimestampRequired(false); kdc.start(); StringBuilder sb = new StringBuilder(); diff --git a/hadoop-common-project/hadoop-minikdc/src/test/java/org/apache/hadoop/minikdc/TestChangeOrgNameAndDomain.java b/hadoop-common-project/hadoop-minikdc/src/test/java/org/apache/hadoop/minikdc/TestChangeOrgNameAndDomain.java new file mode 100644 index 0000000000000..384313011a315 --- /dev/null +++ b/hadoop-common-project/hadoop-minikdc/src/test/java/org/apache/hadoop/minikdc/TestChangeOrgNameAndDomain.java @@ -0,0 +1,32 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.minikdc; + +import java.util.Properties; + +public class TestChangeOrgNameAndDomain extends TestMiniKdc { + + @Override + public void createMiniKdcConf() { + super.createMiniKdcConf(); + Properties properties = getConf(); + properties.setProperty(MiniKdc.ORG_NAME, "APACHE"); + properties.setProperty(MiniKdc.ORG_DOMAIN, "COM"); + } + +} diff --git a/hadoop-common-project/hadoop-nfs/pom.xml b/hadoop-common-project/hadoop-nfs/pom.xml index 409ed757b1043..e8156d9f84d0c 100644 --- a/hadoop-common-project/hadoop-nfs/pom.xml +++ b/hadoop-common-project/hadoop-nfs/pom.xml @@ -55,7 +55,6 @@ org.mockito mockito-all - 1.8.5 commons-logging diff --git a/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java b/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java index 8b6b46a723717..af965655ed33b 100644 --- a/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java +++ b/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/NfsExports.java @@ -391,7 +391,7 @@ private static Match getMatch(String line) { return new CIDRMatch(privilege, new SubnetUtils(pair[0], pair[1]).getInfo()); } else if (host.contains("*") || host.contains("?") || host.contains("[") - || host.contains("]")) { + || host.contains("]") || host.contains("(") || host.contains(")")) { if (LOG.isDebugEnabled()) { LOG.debug("Using Regex match for '" + host + "' and " + privilege); } diff --git a/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/TestNfsExports.java b/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/TestNfsExports.java index 349e82adbad59..542975d1292b5 100644 --- a/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/TestNfsExports.java +++ b/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/TestNfsExports.java @@ -23,8 +23,8 @@ public class TestNfsExports { - private final String address1 = "192.168.0.1"; - private final String address2 = "10.0.0.1"; + private final String address1 = "192.168.0.12"; + private final String address2 = "10.0.0.12"; private final String hostname1 = "a.b.com"; private final String hostname2 = "a.b.org"; @@ -164,6 +164,24 @@ public void testRegexHostRO() { matcher.getAccessPrivilege(address1, hostname2)); } + @Test + public void testRegexGrouping() { + NfsExports matcher = new NfsExports(CacheSize, ExpirationPeriod, + "192.168.0.(12|34)"); + Assert.assertEquals(AccessPrivilege.READ_ONLY, + matcher.getAccessPrivilege(address1, hostname1)); + // address1 will hit the cache + Assert.assertEquals(AccessPrivilege.READ_ONLY, + matcher.getAccessPrivilege(address1, hostname2)); + + matcher = new NfsExports(CacheSize, ExpirationPeriod, "\\w*.a.b.com"); + Assert.assertEquals(AccessPrivilege.READ_ONLY, + matcher.getAccessPrivilege("1.2.3.4", "web.a.b.com")); + // address "1.2.3.4" will hit the cache + Assert.assertEquals(AccessPrivilege.READ_ONLY, + matcher.getAccessPrivilege("1.2.3.4", "email.a.b.org")); + } + @Test public void testMultiMatchers() throws Exception { long shortExpirationPeriod = 1 * 1000 * 1000 * 1000; // 1s diff --git a/hadoop-dist/pom.xml b/hadoop-dist/pom.xml index 0c82332f42843..f894c016a3b66 100644 --- a/hadoop-dist/pom.xml +++ b/hadoop-dist/pom.xml @@ -107,25 +107,86 @@ fi } - ROOT=`cd ../..;pwd` + findFileInDir(){ + local file="$1"; + local dir="${2:-./share}"; + local count=$(find "$dir" -iname "$file"|wc -l) + echo "$count"; + } + + copyIfNotExists(){ + local src="$1" + local srcName=$(basename "$src") + local dest="$2"; + if [ -f "$src" ]; then + if [[ "$srcName" != *.jar ]] || [ $(findFileInDir "$srcName") -eq "0" ]; then + local destDir=$(dirname "$dest") + mkdir -p "$destDir" + cp "$src" "$dest" + fi + else + for childPath in "$src"/* ; + do + child=$(basename "$childPath"); + if [ "$child" == "doc" ] || [ "$child" == "webapps" ]; then + mkdir -p "$dest"/"$child" + cp -r "$src"/"$child"/* "$dest"/"$child" + continue; + fi + copyIfNotExists "$src"/"$child" "$dest"/"$child" + done + fi + } + + #Copy all contents as is except the lib. + #for libs check for existence in share directory, if not exist then only copy. + copy(){ + local src="$1"; + local dest="$2"; + if [ -d "$src" ]; then + for childPath in "$src"/* ; + do + child=$(basename "$childPath"); + if [ "$child" == "share" ]; then + copyIfNotExists "$src"/"$child" "$dest"/"$child" + else + if [ -d "$src"/"$child" ]; then + mkdir -p "$dest"/"$child" + cp -r "$src"/"$child"/* "$dest"/"$child" + else + cp -r "$src"/"$child" "$dest"/"$child" + fi + fi + done + fi + } + + # Shellcheck SC2086 + ROOT=$(cd "${project.build.directory}"/../..;pwd) echo - echo "Current directory `pwd`" + echo "Current directory $(pwd)" echo run rm -rf hadoop-${project.version} run mkdir hadoop-${project.version} run cd hadoop-${project.version} - run cp $ROOT/LICENSE.txt . - run cp $ROOT/NOTICE.txt . - run cp $ROOT/README.txt . - run cp -r $ROOT/hadoop-common-project/hadoop-common/target/hadoop-common-${project.version}/* . - run cp -r $ROOT/hadoop-common-project/hadoop-nfs/target/hadoop-nfs-${project.version}/* . - run cp -r $ROOT/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-${project.version}/* . - run cp -r $ROOT/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-${project.version}/* . - run cp -r $ROOT/hadoop-common-project/hadoop-kms/target/hadoop-kms-${project.version}/* . - run cp -r $ROOT/hadoop-hdfs-project/hadoop-hdfs-nfs/target/hadoop-hdfs-nfs-${project.version}/* . - run cp -r $ROOT/hadoop-yarn-project/target/hadoop-yarn-project-${project.version}/* . - run cp -r $ROOT/hadoop-mapreduce-project/target/hadoop-mapreduce-${project.version}/* . - run cp -r $ROOT/hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-${project.version}/* . + run cp "$ROOT"/LICENSE.txt . + run cp "$ROOT"/NOTICE.txt . + run cp "$ROOT"/README.txt . + + # Copy hadoop-common first so that it have always have all dependencies. + # Remaining projects will copy only libraries which are not present already in 'share' directory. + run copy "$ROOT"/hadoop-common-project/hadoop-common/target/hadoop-common-${project.version} . + run copy "$ROOT"/hadoop-common-project/hadoop-nfs/target/hadoop-nfs-${project.version} . + run copy "$ROOT"/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-${project.version} . + run copy "$ROOT"/hadoop-hdfs-project/hadoop-hdfs-nfs/target/hadoop-hdfs-nfs-${project.version} . + run copy "$ROOT"/hadoop-yarn-project/target/hadoop-yarn-project-${project.version} . + run copy "$ROOT"/hadoop-mapreduce-project/target/hadoop-mapreduce-${project.version} . + run copy "$ROOT"/hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-${project.version} . + + #copy httpfs and kms as is + run cp -r "$ROOT"/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-${project.version}/* . + run cp -r "$ROOT"/hadoop-common-project/hadoop-kms/target/hadoop-kms-${project.version}/* . + echo echo "Hadoop dist layout available at: ${project.build.directory}/hadoop-${project.version}" echo diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml b/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml index ddc603396693b..520e30fb325c7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml @@ -195,6 +195,12 @@ test test-jar + + org.apache.hadoop + hadoop-auth + test + test-jar + log4j log4j diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java index 20b212e60eeab..e797d1207dc0b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java @@ -1013,13 +1013,13 @@ public ContentSummary getContentSummary(Path f) throws IOException { HttpExceptionUtils.validateResponse(conn, HttpURLConnection.HTTP_OK); JSONObject json = (JSONObject) ((JSONObject) HttpFSUtils.jsonParse(conn)).get(CONTENT_SUMMARY_JSON); - return new ContentSummary((Long) json.get(CONTENT_SUMMARY_LENGTH_JSON), - (Long) json.get(CONTENT_SUMMARY_FILE_COUNT_JSON), - (Long) json.get(CONTENT_SUMMARY_DIRECTORY_COUNT_JSON), - (Long) json.get(CONTENT_SUMMARY_QUOTA_JSON), - (Long) json.get(CONTENT_SUMMARY_SPACE_CONSUMED_JSON), - (Long) json.get(CONTENT_SUMMARY_SPACE_QUOTA_JSON) - ); + return new ContentSummary.Builder(). + length((Long) json.get(CONTENT_SUMMARY_LENGTH_JSON)). + fileCount((Long) json.get(CONTENT_SUMMARY_FILE_COUNT_JSON)). + directoryCount((Long) json.get(CONTENT_SUMMARY_DIRECTORY_COUNT_JSON)). + quota((Long) json.get(CONTENT_SUMMARY_QUOTA_JSON)). + spaceConsumed((Long) json.get(CONTENT_SUMMARY_SPACE_CONSUMED_JSON)). + spaceQuota((Long) json.get(CONTENT_SUMMARY_SPACE_QUOTA_JSON)).build(); } @Override diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java index 763d168d19824..14b7a43654d31 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java @@ -18,6 +18,8 @@ package org.apache.hadoop.fs.http.server; import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.security.authentication.util.SignerSecretProvider; +import org.apache.hadoop.security.authentication.util.StringSignerSecretProviderCreator; import org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator; import org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticationHandler; import org.json.simple.JSONArray; @@ -68,7 +70,6 @@ import com.google.common.collect.Maps; import java.util.Properties; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; -import org.apache.hadoop.security.authentication.util.StringSignerSecretProvider; public class TestHttpFSServer extends HFSTestCase { @@ -687,7 +688,8 @@ public void testDelegationTokenOperations() throws Exception { new AuthenticationToken("u", "p", new KerberosDelegationTokenAuthenticationHandler().getType()); token.setExpires(System.currentTimeMillis() + 100000000); - StringSignerSecretProvider secretProvider = new StringSignerSecretProvider(); + SignerSecretProvider secretProvider = + StringSignerSecretProviderCreator.newStringSignerSecretProvider(); Properties secretProviderProps = new Properties(); secretProviderProps.setProperty(AuthenticationFilter.SIGNATURE_SECRET, "secret"); secretProvider.init(secretProviderProps, null, -1); diff --git a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java index 9e4aaf538f0ad..05cc0b5cf63ea 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java @@ -72,5 +72,18 @@ public class NfsConfigKeys { public static final String NFS_HTTPS_ADDRESS_DEFAULT = "0.0.0.0:" + NFS_HTTPS_PORT_DEFAULT; public static final String NFS_METRICS_PERCENTILES_INTERVALS_KEY = "nfs.metrics.percentiles.intervals"; - public static final String NFS_METRICS_PERCENTILES_INTERVALS_DEFAULT = ""; + + /* + * HDFS super-user is the user with the same identity as NameNode process + * itself and the super-user can do anything in that permissions checks never + * fail for the super-user. If the following property is configured, the + * superuser on NFS client can access any file on HDFS. By default, the super + * user is not configured in the gateway. Note that, even the the superuser is + * configured, "nfs.exports.allowed.hosts" still takes effect. For example, + * the superuser will not have write access to HDFS files through the gateway + * if the NFS client host is not allowed to have write access in + * "nfs.exports.allowed.hosts". + */ + public static final String NFS_SUPERUSER_KEY = "nfs.superuser"; + public static final String NFS_SUPERUSER_DEFAULT = ""; } diff --git a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Metrics.java b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Metrics.java index d36ea732f0f2d..880a8a60637f5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Metrics.java +++ b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Metrics.java @@ -90,9 +90,9 @@ public Nfs3Metrics(String name, String sessionId, int[] intervals, readNanosQuantiles[i] = registry.newQuantiles("readProcessNanos" + interval + "s", "Read process in ns", "ops", "latency", interval); writeNanosQuantiles[i] = registry.newQuantiles("writeProcessNanos" - + interval + "s", " process in ns", "ops", "latency", interval); + + interval + "s", "Write process in ns", "ops", "latency", interval); commitNanosQuantiles[i] = registry.newQuantiles("commitProcessNanos" - + interval + "s", "Read process in ns", "ops", "latency", interval); + + interval + "s", "Commit process in ns", "ops", "latency", interval); } } @@ -101,10 +101,9 @@ public static Nfs3Metrics create(Configuration conf, String gatewayName) { MetricsSystem ms = DefaultMetricsSystem.instance(); JvmMetrics jm = JvmMetrics.create(gatewayName, sessionId, ms); - // Percentile measurement is [,,,] by default - int[] intervals = conf.getInts(conf.get( - NfsConfigKeys.NFS_METRICS_PERCENTILES_INTERVALS_KEY, - NfsConfigKeys.NFS_METRICS_PERCENTILES_INTERVALS_DEFAULT)); + // Percentile measurement is [50th,75th,90th,95th,99th] currently + int[] intervals = conf + .getInts(NfsConfigKeys.NFS_METRICS_PERCENTILES_INTERVALS_KEY); return ms.register(new Nfs3Metrics(gatewayName, sessionId, intervals, jm)); } @@ -217,4 +216,4 @@ public void addCommit(long latencyNanos) { } } -} \ No newline at end of file +} diff --git a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java index 05d067410a4d2..268abbab98675 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java +++ b/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java @@ -34,7 +34,6 @@ import org.apache.hadoop.fs.CreateFlag; import org.apache.hadoop.fs.DirectoryListingStartAfterNotFoundException; import org.apache.hadoop.fs.FSDataInputStream; -import org.apache.hadoop.fs.FileSystem.Statistics; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.FsStatus; import org.apache.hadoop.fs.Options; @@ -166,6 +165,7 @@ public class RpcProgramNfs3 extends RpcProgram implements Nfs3Interface { private JvmPauseMonitor pauseMonitor; private Nfs3HttpServer infoServer = null; static Nfs3Metrics metrics; + private String superuser; public RpcProgramNfs3(NfsConfiguration config, DatagramSocket registrationSocket, boolean allowInsecurePorts) throws IOException { @@ -200,6 +200,9 @@ public RpcProgramNfs3(NfsConfiguration config, DatagramSocket registrationSocket UserGroupInformation.setConfiguration(config); SecurityUtil.login(config, NfsConfigKeys.DFS_NFS_KEYTAB_FILE_KEY, NfsConfigKeys.DFS_NFS_KERBEROS_PRINCIPAL_KEY); + superuser = config.get(NfsConfigKeys.NFS_SUPERUSER_KEY, + NfsConfigKeys.NFS_SUPERUSER_DEFAULT); + LOG.info("Configured HDFS superuser is " + superuser); if (!enableDump) { writeDumpDir = null; @@ -583,13 +586,18 @@ ACCESS3Response access(XDR xdr, SecurityHandler securityHandler, } try { - // HDFS-5804 removed supserUserClient access attrs = writeManager.getFileAttr(dfsClient, handle, iug); if (attrs == null) { LOG.error("Can't get path for fileId: " + handle.getFileId()); return new ACCESS3Response(Nfs3Status.NFS3ERR_STALE); } + if(iug.getUserName(securityHandler.getUid(), "unknown").equals(superuser)) { + int access = Nfs3Constant.ACCESS3_LOOKUP | Nfs3Constant.ACCESS3_DELETE + | Nfs3Constant.ACCESS3_EXECUTE | Nfs3Constant.ACCESS3_EXTEND + | Nfs3Constant.ACCESS3_MODIFY | Nfs3Constant.ACCESS3_READ; + return new ACCESS3Response(Nfs3Status.NFS3_OK, attrs, access); + } int access = Nfs3Utils.getAccessRightsForUserGroup( securityHandler.getUid(), securityHandler.getGid(), securityHandler.getAuxGids(), attrs); diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index fe78097abcff1..4247ea688f0a5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -18,12 +18,12 @@ Trunk (Unreleased) option since it may incorrectly finalize an ongoing rolling upgrade. (Kai Sasaki via szetszwo) + HDFS-7985. WebHDFS should be always enabled. (Li Lu via wheat9) + NEW FEATURES HDFS-3125. Add JournalService to enable Journal Daemon. (suresh) - HDFS-3689. Add support for variable length block. (jing9) - IMPROVEMENTS HDFS-4665. Move TestNetworkTopologyWithNodeGroup to common. @@ -147,6 +147,8 @@ Trunk (Unreleased) HDFS-7460. Rewrite httpfs to use new shell framework (John Smith via aw) + HDFS-6353. Check and make checkpoint before stopping the NameNode. (jing9) + OPTIMIZATIONS BUG FIXES @@ -313,6 +315,92 @@ Trunk (Unreleased) HDFS-4681. TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks fails using IBM java (Ayappan via aw) +Release 2.8.0 - UNRELEASED + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + HDFS-3918. EditLogTailer shouldn't log WARN when other node + is in standby mode (todd via harsh) + + HDFS-4396. Add START_MSG/SHUTDOWN_MSG for ZKFC + (Liang Xie via harsh) + + HDFS-7875. Improve log message when wrong value configured for + dfs.datanode.failed.volumes.tolerated. + (nijel via harsh) + + HDFS-2360. Ugly stacktrace when quota exceeds. (harsh) + + HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for + DFSClient. (Zhihai Xu via Yongjun Zhang) + + HDFS-7829. Code clean up for LocatedBlock. (Takanobu Asanuma via jing9) + + HDFS-7854. Separate class DataStreamer out of DFSOutputStream. (Li Bo via + jing9) + + HDFS-7713. Implement mkdirs in the HDFS Web UI. (Ravi Prakash via wheat9) + + HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes + a lot of time if disks are busy (Rushabh S Shah via kihwal) + + HDFS-7990. IBR delete ack should not be delayed. (daryn via kihwal) + + HDFS-8004. Use KeyProviderCryptoExtension#warmUpEncryptedKeys when creating + an encryption zone. (awang via asuresh) + + HDFS-6263. Remove DRFA.MaxBackupIndex config from log4j.properties. + (Abhiraj Butala via aajisaka) + + HDFS-6408. Remove redundant definitions in log4j.properties. + (Abhiraj Butala via aajisaka) + + HDFS-7890. Improve information on Top users for metrics in + RollingWindowsManager and lower log level (J.Andreina via vinayakumarb) + + HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times. + (Vinayakumar B and Keisuke Ogiwara via Arpit Agarwal) + + HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads. + (Arpit Agarwal) + + HDFS-7671. hdfs user guide should point to the common rack awareness doc. + (Kai Sasaki via aajisaka) + + OPTIMIZATIONS + + BUG FIXES + + HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. + (Gautam Gopalakrishnan via harsh) + + HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown() + (Rakesh R via vinayakumarb) + + HDFS-7867. Update action param from "start" to "prepare" in rolling upgrade + javadoc (J.Andreina via vinayakumarb) + + HDFS-3325. When configuring "dfs.namenode.safemode.threshold-pct" to a value + greater or equal to 1 there is mismatch in the UI report + (J.Andreina via vinayakumarb) + + HDFS-8002. Website refers to /trash directory. (Brahma Reddy Battula via + aajisaka) + + HDFS-7261. storageMap is accessed without synchronization in + DatanodeDescriptor#updateHeartbeatState() (Brahma Reddy Battula via Colin + P. McCabe) + + HDFS-7997. The first non-existing xattr should also throw IOException. + (zhouyingchao via yliu) + + HDFS-6945. BlockManager should remove a block from excessReplicateMap and + decrement ExcessBlocks metric when the block is removed. (aajisaka) + Release 2.7.0 - UNRELEASED INCOMPATIBLE CHANGES @@ -343,11 +431,20 @@ Release 2.7.0 - UNRELEASED HDFS-6133. Add a feature for replica pinning so that a pinned replica will not be moved by Balancer/Mover. (zhaoyunjiong via szetszwo) + HDFS-3689. Add support for variable length block. (jing9) + HDFS-7584. Enable Quota Support for Storage Types (See breakdown of tasks below) HDFS-7656. Expose truncate API for HDFS httpfs. (yliu) + HDFS-6488. Support HDFS superuser in NFS gateway. (brandonli) + + HDFS-7838. Expose truncate API for libhdfs. (yliu) + + HDFS-6826. Plugin interface to enable delegation of HDFS authorization + assertions. (Arun Suresh via jitendra) + IMPROVEMENTS HDFS-7752. Improve description for @@ -704,6 +801,62 @@ Release 2.7.0 - UNRELEASED HDFS-7789. DFSck should resolve the path to support cross-FS symlinks. (gera) + HDFS-7535. Utilize Snapshot diff report for distcp. (jing9) + + HDFS-1522. Combine two BLOCK_FILE_PREFIX constants into one. + (Dongming Liang via shv) + + HDFS-7746. Add a test randomly mixing append, truncate and snapshot + operations. (szetszwo) + + HADOOP-11648. Set DomainSocketWatcher thread name explicitly. + (Liang Xie via ozawa) + + HDFS-7855. Separate class Packet from DFSOutputStream. (Li Bo bia jing9) + + HDFS-7411. Change decommission logic to throttle by blocks rather than + nodes in each interval. (Andrew Wang via cdouglas) + + HDFS-7898. Change TestAppendSnapshotTruncate to fail-fast. + (Tsz Wo Nicholas Sze via jing9) + + HDFS-6806. HDFS Rolling upgrade document should mention the versions + available. (J.Andreina via aajisaka) + + HDFS-7491. Add incremental blockreport latency to DN metrics. + (Ming Ma via cnauroth) + + HDFS-7435. PB encoding of block reports is very inefficient. + (Daryn Sharp via kihwal) + + HDFS-2605. Remove redundant "Release 0.21.1" section from CHANGES.txt. + (Allen Wittenauer via shv) + + HDFS-7940. Add tracing to DFSClient#setQuotaByStorageType (Rakesh R via + Colin P. McCabe) + + HDFS-7054. Make DFSOutputStream tracing more fine-grained (cmccabe) + + HDFS.7849. Update documentation for enabling a new feature in rolling + upgrade ( J.Andreina via vinayakumarb ) + + HDFS-7962. Remove duplicated logs in BlockManager. (yliu) + + HDFS-7917. Use file to replace data dirs in test to simulate a disk failure. + (Lei (Eddy) Xu via cnauroth) + + HDFS-7956. Improve logging for DatanodeRegistration. + (Plamen Jeliazkov via shv) + + HDFS-7976. Update NFS user guide for mount option "sync" to minimize or + avoid reordered writes. (brandonli) + + HDFS-7410. Support CreateFlags with append() to support hsync() for + appending streams (Vinayakumar B via Colin P. McCabe) + + HDFS-7742. Favoring decommissioning node for replication can cause a block + to stay underreplicated for long periods (Nathan Roberts via kihwal) + OPTIMIZATIONS HDFS-7454. Reduce memory footprint for AclEntries in NameNode. @@ -1074,6 +1227,113 @@ Release 2.7.0 - UNRELEASED HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" message. (jing9) + HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula + via Arpit Agarwal) + + HDFS-6565. Use jackson instead jetty json in hdfs-client. + (Akira Ajisaka via wheat9) + + HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file + includes non-snapshotted content. (Charles Lamb via atm) + + HDFS-7869. Inconsistency in the return information while performing rolling + upgrade ( J.Andreina via vinayakumarb ) + + HDFS-7879. hdfs.dll does not export functions of the public libhdfs API. + (Chris Nauroth via wheat9) + + HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal) + + HDFS-7885. Datanode should not trust the generation stamp provided by + client. (Tsz Wo Nicholas Sze via jing9) + + HDFS-7818. OffsetParam should return the default value instead of throwing + NPE when the value is unspecified. (Eric Payne via wheat9) + + HDFS-7830. DataNode does not release the volume lock when adding a volume + fails. (Lei Xu via Colin P. Mccabe) + + HDFS-6833. DirectoryScanner should not register a deleting block with + memory of DataNode. (Shinichi Yamashita via szetszwo) + + HDFS-7926. NameNode implementation of ClientProtocol.truncate(..) is not + idempotent (Tsz Wo Nicholas Sze via brandonli) + + HDFS-7903. Cannot recover block after truncate and delete snapshot. + (Plamen Jeliazkov via shv) + + HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and + fail to tell the DFSClient about it because of a network error (cmccabe) + + HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv) + + HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao + via Arpit Agarwal) + + HDFS-7953. NN Web UI fails to navigate to paths that contain #. + (kanaka kumar avvaru via wheat9) + + HDFS-7948. TestDataNodeHotSwapVolumes#testAddVolumeFailures failed on + Windows. (Xiaoyu Yao via Arpit Agarwal) + + HDFS-7950. Fix TestFsDatasetImpl#testAddVolumes failure on Windows. + (Xiaoyu Yao via Arpit Agarwal) + + HDFS-7951. Fix NPE for + TestFsDatasetImpl#testAddVolumeFailureReleasesInUseLock on Linux + (Xiaoyu Yao via Arpit Agarwal) + + HDFS-7722. DataNode#checkDiskError should also remove Storage when error + is found. (Lei Xu via Colin P. McCabe) + + HDFS-7697. Mark the PB OIV tool as experimental. (Lei Xu via wheat9) + + HDFS-7914. TestJournalNode#testFailToStartWithBadConfig fails when the + default dfs.journalnode.http-address port 8480 is in use. (Xiaoyu Yao via + Arpit Agarwal) + + HDFS-7945. The WebHdfs system on DN does not honor the length parameter. + (wheat9) + + HDFS-7943. Append cannot handle the last block with length greater than + the preferred block size. (jing9) + + HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade + starts (Zhe Zhang via Colin P. McCabe) + + HDFS-7587. Edit log corruption can happen if append fails with a quota + violation. (jing9) + + HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal) + + HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal) + + HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) + + HDFS-7957. Truncate should verify quota before making changes. (jing9) + + HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now() + (Vinayakumar B via kihwal) + + HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli) + + HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp + provided by the client is larger than the one stored in the datanode. + (Brahma Reddy Battula via szetszwo) + + HDFS-7960. The full block report should prune zombie storages even if + they're not empty. (cmccabe and Eddy Xu via wang) + + HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang) + + HDFS-7977. NFS couldn't take percentile intervals (brandonli) + + HDFS-7963. Fix expected tracing spans in TestTracing along with HDFS-7054. + (Masatake Iwasaki via kihwal) + + HDFS-7748. Separate ECN flags from the Status in the DataTransferPipelineAck. + (Anu Engineer and Haohui Mai via wheat9) + BREAKDOWN OF HDFS-7584 SUBTASKS AND RELATED JIRAS HDFS-7720. Quota by Storage Type API, tools and ClientNameNode @@ -1094,6 +1354,12 @@ Release 2.7.0 - UNRELEASED HDFS-7806. Refactor: move StorageType from hadoop-hdfs to hadoop-common. (Xiaoyu Yao via Arpit Agarwal) + HDFS-7824. GetContentSummary API and its namenode implementation for + Storage Type Quota/Usage. (Xiaoyu Yao via Arpit Agarwal) + + HDFS-7700. Document quota support for storage types. (Xiaoyu Yao via + Arpit Agarwal) + Release 2.6.1 - UNRELEASED INCOMPATIBLE CHANGES @@ -10219,8 +10485,6 @@ Release 0.22.0 - 2011-11-29 HDFS-2287. TestParallelRead has a small off-by-one bug. (todd) -Release 0.21.1 - Unreleased - HDFS-1466. TestFcHdfsSymlink relies on /tmp/test not existing. (eli) HDFS-874. TestHDFSFileContextMainOperations fails on weirdly diff --git a/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml b/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml index dedeeced0ccb9..224d2fb5defda 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml @@ -77,7 +77,7 @@ ResponseProccessor is thread that is designed to catch RuntimeException. --> - + diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt b/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt index aceeac1f8e995..563727b44f513 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt @@ -27,7 +27,15 @@ include(../../../hadoop-common-project/hadoop-common/src/JNIFlags.cmake NO_POLIC function(add_dual_library LIBNAME) add_library(${LIBNAME} SHARED ${ARGN}) add_library(${LIBNAME}_static STATIC ${ARGN}) - set_target_properties(${LIBNAME}_static PROPERTIES OUTPUT_NAME ${LIBNAME}) + # Linux builds traditionally ship a libhdfs.a (static linking) and libhdfs.so + # (dynamic linking). On Windows, we cannot use the same base name for both + # static and dynamic, because Windows does not use distinct file extensions + # for a statically linked library vs. a DLL import library. Both use the + # .lib extension. On Windows, we'll build the static library as + # hdfs_static.lib. + if (NOT WIN32) + set_target_properties(${LIBNAME}_static PROPERTIES OUTPUT_NAME ${LIBNAME}) + endif (NOT WIN32) endfunction(add_dual_library) # Link both a static and a dynamic target against some libraries @@ -105,11 +113,14 @@ else (WIN32) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -Wall -O2") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_REENTRANT -D_GNU_SOURCE") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64") + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fvisibility=hidden") set(OS_DIR main/native/libhdfs/os/posix) set(OS_LINK_LIBRARIES pthread) set(OUT_DIR target/usr/local/lib) endif (WIN32) +add_definitions(-DLIBHDFS_DLL_EXPORT) + include_directories( ${GENERATED_JAVAH} ${CMAKE_CURRENT_SOURCE_DIR} @@ -150,7 +161,7 @@ add_executable(test_libhdfs_ops main/native/libhdfs/test/test_libhdfs_ops.c ) target_link_libraries(test_libhdfs_ops - hdfs + hdfs_static ${JAVA_JVM_LIBRARY} ) @@ -158,7 +169,7 @@ add_executable(test_libhdfs_read main/native/libhdfs/test/test_libhdfs_read.c ) target_link_libraries(test_libhdfs_read - hdfs + hdfs_static ${JAVA_JVM_LIBRARY} ) @@ -166,7 +177,7 @@ add_executable(test_libhdfs_write main/native/libhdfs/test/test_libhdfs_write.c ) target_link_libraries(test_libhdfs_write - hdfs + hdfs_static ${JAVA_JVM_LIBRARY} ) @@ -196,7 +207,7 @@ add_executable(test_libhdfs_threaded ${OS_DIR}/thread.c ) target_link_libraries(test_libhdfs_threaded - hdfs + hdfs_static native_mini_dfs ${OS_LINK_LIBRARIES} ) @@ -206,7 +217,7 @@ add_executable(test_libhdfs_zerocopy main/native/libhdfs/test/test_libhdfs_zerocopy.c ) target_link_libraries(test_libhdfs_zerocopy - hdfs + hdfs_static native_mini_dfs ${OS_LINK_LIBRARIES} ) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBootstrapStandbyWithBKJM.java b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBootstrapStandbyWithBKJM.java index ded9e0e99cb20..18dedc83733b0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBootstrapStandbyWithBKJM.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBootstrapStandbyWithBKJM.java @@ -111,7 +111,7 @@ public void testBootstrapStandbyWithActiveNN() throws Exception { cluster.shutdownNameNode(1); deleteEditLogIfExists(confNN1); cluster.getNameNodeRpc(0).setSafeMode(SafeModeAction.SAFEMODE_ENTER, true); - cluster.getNameNodeRpc(0).saveNamespace(); + cluster.getNameNodeRpc(0).saveNamespace(0, 0); cluster.getNameNodeRpc(0).setSafeMode(SafeModeAction.SAFEMODE_LEAVE, true); // check without -skipSharedEditsCheck, Bootstrap should fail for BKJM diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties index 8a6b2174144de..93c22f7149611 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties +++ b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/resources/log4j.properties @@ -53,10 +53,3 @@ log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p - [%t:%C{ # Max log file size of 10MB log4j.appender.ROLLINGFILE.MaxFileSize=10MB -# uncomment the next line to limit number of backup files -#log4j.appender.ROLLINGFILE.MaxBackupIndex=10 - -log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout -log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n - - diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c index deb11ef3040ca..86b4faf2721cd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c +++ b/hadoop-hdfs-project/hadoop-hdfs/src/contrib/libwebhdfs/src/hdfs_web.c @@ -1124,6 +1124,12 @@ hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags, return file; } +int hdfsTruncateFile(hdfsFS fs, const char* path, tOffset newlength) +{ + errno = ENOTSUP; + return -1; +} + tSize hdfsWrite(hdfsFS fs, hdfsFile file, const void* buffer, tSize length) { if (length == 0) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs index 830ca3694195d..f4642617556c7 100755 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs @@ -94,7 +94,7 @@ case ${COMMAND} in CLASS=org.apache.hadoop.hdfs.tools.CacheAdmin ;; classpath) - hadoop_do_classpath_subcommand "$@" + hadoop_do_classpath_subcommand CLASS "$@" ;; crypto) CLASS=org.apache.hadoop.hdfs.tools.CryptoAdmin @@ -281,6 +281,14 @@ fi hadoop_finalize if [[ -n "${supportdaemonization}" ]]; then + if [[ "${COMMAND}" == "namenode" ]] && + [[ "${HADOOP_DAEMON_MODE}" == "stop" ]]; then + hadoop_debug "Do checkpoint if necessary before stopping NameNode" + export CLASSPATH + "${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS} "org.apache.hadoop.hdfs.tools.DFSAdmin" "-safemode" "enter" + "${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS} "org.apache.hadoop.hdfs.tools.DFSAdmin" "-saveNamespace" "-beforeShutdown" + "${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS} "org.apache.hadoop.hdfs.tools.DFSAdmin" "-safemode" "leave" + fi if [[ -n "${secure_service}" ]]; then hadoop_secure_daemon_handler \ "${HADOOP_DAEMON_MODE}" "${COMMAND}" "${CLASS}"\ diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java index ba48c7971580d..1e915b28122d8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java @@ -17,6 +17,8 @@ */ package org.apache.hadoop.hdfs; +import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.ShortCircuitFdResponse.USE_RECEIPT_VERIFICATION; + import java.io.BufferedOutputStream; import java.io.DataInputStream; import java.io.DataOutputStream; @@ -69,12 +71,23 @@ public class BlockReaderFactory implements ShortCircuitReplicaCreator { static final Log LOG = LogFactory.getLog(BlockReaderFactory.class); + public static class FailureInjector { + public void injectRequestFileDescriptorsFailure() throws IOException { + // do nothing + } + } + @VisibleForTesting static ShortCircuitReplicaCreator createShortCircuitReplicaInfoCallback = null; private final DFSClient.Conf conf; + /** + * Injects failures into specific operations during unit tests. + */ + private final FailureInjector failureInjector; + /** * The file name, for logging and debugging purposes. */ @@ -169,6 +182,7 @@ public class BlockReaderFactory implements ShortCircuitReplicaCreator { public BlockReaderFactory(DFSClient.Conf conf) { this.conf = conf; + this.failureInjector = conf.brfFailureInjector; this.remainingCacheTries = conf.nCachedConnRetry; } @@ -518,11 +532,12 @@ private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer, final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(peer.getOutputStream())); SlotId slotId = slot == null ? null : slot.getSlotId(); - new Sender(out).requestShortCircuitFds(block, token, slotId, 1); + new Sender(out).requestShortCircuitFds(block, token, slotId, 1, true); DataInputStream in = new DataInputStream(peer.getInputStream()); BlockOpResponseProto resp = BlockOpResponseProto.parseFrom( PBHelper.vintPrefixed(in)); DomainSocket sock = peer.getDomainSocket(); + failureInjector.injectRequestFileDescriptorsFailure(); switch (resp.getStatus()) { case SUCCESS: byte buf[] = new byte[1]; @@ -532,8 +547,13 @@ private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer, try { ExtendedBlockId key = new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId()); + if (buf[0] == USE_RECEIPT_VERIFICATION.getNumber()) { + LOG.trace("Sending receipt verification byte for slot " + slot); + sock.getOutputStream().write(0); + } replica = new ShortCircuitReplica(key, fis[0], fis[1], cache, Time.monotonicNow(), slot); + return new ShortCircuitReplicaInfo(replica); } catch (IOException e) { // This indicates an error reading from disk, or a format error. Since // it's not a socket communication problem, we return null rather than @@ -545,7 +565,6 @@ private ShortCircuitReplicaInfo requestFileDescriptors(DomainPeer peer, IOUtils.cleanup(DFSClient.LOG, fis[0], fis[1]); } } - return new ShortCircuitReplicaInfo(replica); case ERROR_UNSUPPORTED: if (!resp.hasShortCircuitAccessVersion()) { LOG.warn("short-circuit read access is disabled for " + diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java index abcd847dbfb51..29bb60427a26c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java @@ -24,6 +24,8 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_RETRIES_DEFAULT; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_RETRIES_KEY; +import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_DEFAULT; +import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_BLOCK_WRITE_RETRIES_DEFAULT; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_BLOCK_WRITE_RETRIES_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_CACHED_CONN_RETRY_DEFAULT; @@ -305,6 +307,7 @@ public static class Conf { final int nCachedConnRetry; final int nBlockWriteRetry; final int nBlockWriteLocateFollowingRetry; + final int blockWriteLocateFollowingInitialDelayMs; final long defaultBlockSize; final long prefetchSize; final short defaultReplication; @@ -337,6 +340,8 @@ public static class Conf { final long shortCircuitCacheStaleThresholdMs; final long keyProviderCacheExpiryMs; + public BlockReaderFactory.FailureInjector brfFailureInjector = + new BlockReaderFactory.FailureInjector(); public Conf(Configuration conf) { // The hdfsTimeout is currently the same as the ipc timeout @@ -414,6 +419,9 @@ public Conf(Configuration conf) { nBlockWriteLocateFollowingRetry = conf.getInt( DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_RETRIES_KEY, DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_RETRIES_DEFAULT); + blockWriteLocateFollowingInitialDelayMs = conf.getInt( + DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_KEY, + DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_DEFAULT); uMask = FsPermission.getUMask(conf); connectToDnViaHostname = conf.getBoolean(DFS_CLIENT_USE_DN_HOSTNAME, DFS_CLIENT_USE_DN_HOSTNAME_DEFAULT); @@ -564,6 +572,11 @@ private DataChecksum createChecksum(ChecksumOpt userOpt) { } return dataChecksum; } + + @VisibleForTesting + public int getBlockWriteLocateFollowingInitialDelayMs() { + return blockWriteLocateFollowingInitialDelayMs; + } } public Conf getConf() { @@ -872,7 +885,7 @@ void updateLastLeaseRenewal() { if (filesBeingWritten.isEmpty()) { return; } - lastLeaseRenewal = Time.now(); + lastLeaseRenewal = Time.monotonicNow(); } } @@ -889,7 +902,7 @@ boolean renewLease() throws IOException { return true; } catch (IOException e) { // Abort if the lease has already expired. - final long elapsed = Time.now() - getLastLeaseRenewal(); + final long elapsed = Time.monotonicNow() - getLastLeaseRenewal(); if (elapsed > HdfsConstants.LEASE_HARDLIMIT_PERIOD) { LOG.warn("Failed to renew lease for " + clientName + " for " + (elapsed/1000) + " seconds (>= hard-limit =" @@ -1007,7 +1020,7 @@ public long getBlockSize(String f) throws IOException { * @see ClientProtocol#getServerDefaults() */ public FsServerDefaults getServerDefaults() throws IOException { - long now = Time.now(); + long now = Time.monotonicNow(); if (now - serverDefaultsLastUpdate > SERVER_DEFAULTS_VALIDITY_PERIOD) { serverDefaults = namenode.getServerDefaults(); serverDefaultsLastUpdate = now; @@ -1814,10 +1827,10 @@ private DFSOutputStream callAppend(String src, int buffersize, try { LastBlockWithStatus blkWithStatus = namenode.append(src, clientName, new EnumSetWritable<>(flag, CreateFlag.class)); - return DFSOutputStream.newStreamForAppend(this, src, - flag.contains(CreateFlag.NEW_BLOCK), - buffersize, progress, blkWithStatus.getLastBlock(), - blkWithStatus.getFileStatus(), dfsClientConf.createChecksum(), favoredNodes); + return DFSOutputStream.newStreamForAppend(this, src, flag, buffersize, + progress, blkWithStatus.getLastBlock(), + blkWithStatus.getFileStatus(), dfsClientConf.createChecksum(), + favoredNodes); } catch(RemoteException re) { throw re.unwrapRemoteException(AccessControlException.class, FileNotFoundException.class, @@ -2220,6 +2233,9 @@ public MD5MD5CRC32FileChecksum getFileChecksum(String src, long length) // get block checksum for each block long remaining = length; + if (src.contains(HdfsConstants.SEPARATOR_DOT_SNAPSHOT_DIR_SEPARATOR)) { + remaining = Math.min(length, blockLocations.getFileLength()); + } for(int i = 0; i < locatedblocks.size() && remaining > 0; i++) { if (refetchBlocks) { // refetch to get fresh tokens blockLocations = callGetBlockLocations(namenode, src, 0, length); @@ -2824,12 +2840,12 @@ public RemoteIterator listCachePools() throws IOException { /** * Save namespace image. * - * @see ClientProtocol#saveNamespace() + * @see ClientProtocol#saveNamespace(long, long) */ - void saveNamespace() throws AccessControlException, IOException { + boolean saveNamespace(long timeWindow, long txGap) throws IOException { TraceScope scope = Trace.startSpan("saveNamespace", traceSampler); try { - namenode.saveNamespace(); + return namenode.saveNamespace(timeWindow, txGap); } catch(RemoteException re) { throw re.unwrapRemoteException(AccessControlException.class); } finally { @@ -3084,6 +3100,7 @@ void setQuotaByStorageType(String src, StorageType type, long quota) throw new IllegalArgumentException("Don't support Quota for storage type : " + type.toString()); } + TraceScope scope = getPathTraceScope("setQuotaByStorageType", src); try { namenode.setQuota(src, HdfsConstants.QUOTA_DONT_SET, quota, type); } catch (RemoteException re) { @@ -3092,6 +3109,8 @@ void setQuotaByStorageType(String src, StorageType type, long quota) QuotaByStorageTypeExceededException.class, UnresolvedPathException.class, SnapshotAccessControlException.class); + } finally { + scope.close(); } } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java index 975f023f272b5..610932a638ccb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java @@ -201,6 +201,8 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final long DFS_NAMENODE_CHECKPOINT_TXNS_DEFAULT = 1000000; public static final String DFS_NAMENODE_CHECKPOINT_MAX_RETRIES_KEY = "dfs.namenode.checkpoint.max-retries"; public static final int DFS_NAMENODE_CHECKPOINT_MAX_RETRIES_DEFAULT = 3; + public static final String DFS_NAMENODE_MISSING_CHECKPOINT_PERIODS_BEFORE_SHUTDOWN_KEY = "dfs.namenode.missing.checkpoint.periods.before.shutdown"; + public static final int DFS_NAMENODE_MISSING_CHECKPOINT_PERIODS_BEFORE_SHUTDONW_DEFAULT = 3; public static final String DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY = "dfs.namenode.heartbeat.recheck-interval"; public static final int DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_DEFAULT = 5*60*1000; public static final String DFS_NAMENODE_TOLERATE_HEARTBEAT_MULTIPLIER_KEY = "dfs.namenode.tolerate.heartbeat.multiplier"; @@ -227,8 +229,6 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final int DFS_NAMENODE_REPLICATION_STREAMS_HARD_LIMIT_DEFAULT = 4; public static final String DFS_WEBHDFS_AUTHENTICATION_FILTER_KEY = "dfs.web.authentication.filter"; public static final String DFS_WEBHDFS_AUTHENTICATION_FILTER_DEFAULT = AuthFilter.class.getName(); - public static final String DFS_WEBHDFS_ENABLED_KEY = "dfs.webhdfs.enabled"; - public static final boolean DFS_WEBHDFS_ENABLED_DEFAULT = true; public static final String DFS_WEBHDFS_USER_PATTERN_KEY = "dfs.webhdfs.user.provider.user.pattern"; public static final String DFS_WEBHDFS_USER_PATTERN_DEFAULT = "^[A-Za-z_][A-Za-z0-9._-]*[$]?$"; public static final String DFS_PERMISSIONS_ENABLED_KEY = "dfs.permissions.enabled"; @@ -399,6 +399,9 @@ public class DFSConfigKeys extends CommonConfigurationKeys { // Much code in hdfs is not yet updated to use these keys. public static final String DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_RETRIES_KEY = "dfs.client.block.write.locateFollowingBlock.retries"; public static final int DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_RETRIES_DEFAULT = 5; + // the initial delay (unit is ms) for locateFollowingBlock, the delay time will increase exponentially(double) for each retry. + public static final String DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_KEY = "dfs.client.block.write.locateFollowingBlock.initial.delay.ms"; + public static final int DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_DEFAULT = 400; public static final String DFS_CLIENT_BLOCK_WRITE_RETRIES_KEY = "dfs.client.block.write.retries"; public static final int DFS_CLIENT_BLOCK_WRITE_RETRIES_DEFAULT = 3; public static final String DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_KEY = "dfs.client.max.block.acquire.failures"; @@ -453,8 +456,10 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final long DFS_NAMENODE_PATH_BASED_CACHE_RETRY_INTERVAL_MS_DEFAULT = 30000L; public static final String DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY = "dfs.namenode.decommission.interval"; public static final int DFS_NAMENODE_DECOMMISSION_INTERVAL_DEFAULT = 30; - public static final String DFS_NAMENODE_DECOMMISSION_NODES_PER_INTERVAL_KEY = "dfs.namenode.decommission.nodes.per.interval"; - public static final int DFS_NAMENODE_DECOMMISSION_NODES_PER_INTERVAL_DEFAULT = 5; + public static final String DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY = "dfs.namenode.decommission.blocks.per.interval"; + public static final int DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT = 500000; + public static final String DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES = "dfs.namenode.decommission.max.concurrent.tracked.nodes"; + public static final int DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT = 100; public static final String DFS_NAMENODE_HANDLER_COUNT_KEY = "dfs.namenode.handler.count"; public static final int DFS_NAMENODE_HANDLER_COUNT_DEFAULT = 10; public static final String DFS_NAMENODE_SERVICE_HANDLER_COUNT_KEY = "dfs.namenode.service.handler.count"; @@ -472,6 +477,7 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final String DFS_DATANODE_IPC_ADDRESS_DEFAULT = "0.0.0.0:" + DFS_DATANODE_IPC_DEFAULT_PORT; public static final String DFS_DATANODE_MIN_SUPPORTED_NAMENODE_VERSION_KEY = "dfs.datanode.min.supported.namenode.version"; public static final String DFS_DATANODE_MIN_SUPPORTED_NAMENODE_VERSION_DEFAULT = "3.0.0-SNAPSHOT"; + public static final String DFS_NAMENODE_INODE_ATTRIBUTES_PROVIDER_KEY = "dfs.namenode.inode.attributes.provider.class"; public static final String DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY = "dfs.block.access.token.enable"; public static final boolean DFS_BLOCK_ACCESS_TOKEN_ENABLE_DEFAULT = false; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java old mode 100644 new mode 100755 index dc2f67433c209..933d8e6a78677 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java @@ -17,30 +17,12 @@ */ package org.apache.hadoop.hdfs; -import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status.SUCCESS; - -import java.io.BufferedOutputStream; -import java.io.DataInputStream; -import java.io.DataOutputStream; import java.io.FileNotFoundException; import java.io.IOException; -import java.io.InputStream; import java.io.InterruptedIOException; -import java.io.OutputStream; -import java.net.InetAddress; -import java.net.InetSocketAddress; import java.net.Socket; -import java.nio.BufferOverflowException; import java.nio.channels.ClosedChannelException; -import java.util.ArrayList; -import java.util.Arrays; import java.util.EnumSet; -import java.util.HashSet; -import java.util.LinkedList; -import java.util.List; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.atomic.AtomicBoolean; -import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import org.apache.hadoop.HadoopIllegalArgumentException; @@ -53,61 +35,37 @@ import org.apache.hadoop.fs.FileEncryptionInfo; import org.apache.hadoop.fs.ParentNotDirectoryException; import org.apache.hadoop.fs.permission.FsPermission; -import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.fs.Syncable; import org.apache.hadoop.hdfs.client.HdfsDataOutputStream; import org.apache.hadoop.hdfs.client.HdfsDataOutputStream.SyncFlag; -import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy; import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; -import org.apache.hadoop.hdfs.protocol.HdfsConstants; import org.apache.hadoop.hdfs.protocol.HdfsFileStatus; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.protocol.NSQuotaExceededException; import org.apache.hadoop.hdfs.protocol.SnapshotAccessControlException; import org.apache.hadoop.hdfs.protocol.UnresolvedPathException; -import org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage; -import org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtocol; -import org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil; -import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair; -import org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException; import org.apache.hadoop.hdfs.protocol.datatransfer.PacketHeader; -import org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck; -import org.apache.hadoop.hdfs.protocol.datatransfer.Sender; -import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.BlockOpResponseProto; -import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status; -import org.apache.hadoop.hdfs.protocolPB.PBHelper; import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier; -import org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException; -import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite; import org.apache.hadoop.hdfs.server.datanode.CachingStrategy; -import org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException; import org.apache.hadoop.hdfs.server.namenode.RetryStartFileException; import org.apache.hadoop.hdfs.server.namenode.SafeModeException; import org.apache.hadoop.hdfs.util.ByteArrayManager; import org.apache.hadoop.io.EnumSetWritable; -import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.ipc.RemoteException; -import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.AccessControlException; import org.apache.hadoop.security.token.Token; -import org.apache.hadoop.util.Daemon; import org.apache.hadoop.util.DataChecksum; import org.apache.hadoop.util.DataChecksum.Type; import org.apache.hadoop.util.Progressable; import org.apache.hadoop.util.Time; -import org.apache.htrace.Span; +import org.apache.htrace.Sampler; import org.apache.htrace.Trace; import org.apache.htrace.TraceScope; import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; -import com.google.common.cache.CacheBuilder; -import com.google.common.cache.CacheLoader; -import com.google.common.cache.LoadingCache; -import com.google.common.cache.RemovalListener; -import com.google.common.cache.RemovalNotification; /**************************************************************** @@ -119,19 +77,11 @@ * is typically 512 bytes and has an associated checksum with it. * * When a client application fills up the currentPacket, it is - * enqueued into dataQueue. The DataStreamer thread picks up - * packets from the dataQueue, sends it to the first datanode in - * the pipeline and moves it from the dataQueue to the ackQueue. - * The ResponseProcessor receives acks from the datanodes. When an - * successful ack for a packet is received from all datanodes, the - * ResponseProcessor removes the corresponding packet from the - * ackQueue. + * enqueued into the dataQueue of DataStreamer. DataStreamer is a + * thread that picks up packets from the dataQueue and sends it to + * the first datanode in the pipeline. * - * In case of error, all outstanding packets and moved from - * ackQueue. A new pipeline is setup by eliminating the bad - * datanode from the original pipeline. The DataStreamer now - * starts sending packets from the dataQueue. -****************************************************************/ + ****************************************************************/ @InterfaceAudience.Private public class DFSOutputStream extends FSOutputSummer implements Syncable, CanSetDropBehind { @@ -146,49 +96,29 @@ public class DFSOutputStream extends FSOutputSummer CryptoProtocolVersion.supported(); private final DFSClient dfsClient; - private final long dfsclientSlowLogThresholdMs; private final ByteArrayManager byteArrayManager; - private Socket s; // closed is accessed by different threads under different locks. private volatile boolean closed = false; - private String src; + private final String src; private final long fileId; private final long blockSize; - /** Only for DataTransferProtocol.writeBlock(..) */ - private final DataChecksum checksum4WriteBlock; - private final int bytesPerChecksum; - - // both dataQueue and ackQueue are protected by dataQueue lock - private final LinkedList dataQueue = new LinkedList(); - private final LinkedList ackQueue = new LinkedList(); - private Packet currentPacket = null; + private final int bytesPerChecksum; + + private DFSPacket currentPacket = null; private DataStreamer streamer; - private long currentSeqno = 0; - private long lastQueuedSeqno = -1; - private long lastAckedSeqno = -1; - private long bytesCurBlock = 0; // bytes written in current block private int packetSize = 0; // write packet size, not including the header. private int chunksPerPacket = 0; - private final AtomicReference lastException = new AtomicReference(); - private long artificialSlowdown = 0; private long lastFlushOffset = 0; // offset when flush was invoked - //persist blocks on namenode - private final AtomicBoolean persistBlocks = new AtomicBoolean(false); - private volatile boolean appendChunk = false; // appending to existing partial block private long initialFileSize = 0; // at time of file open - private final Progressable progress; private final short blockReplication; // replication factor of file private boolean shouldSyncBlock = false; // force blocks to disk upon close private final AtomicReference cachingStrategy; - private boolean failPacket = false; private FileEncryptionInfo fileEncryptionInfo; - private static final BlockStoragePolicySuite blockStoragePolicySuite = - BlockStoragePolicySuite.createDefaultSuite(); /** Use {@link ByteArrayManager} to create buffer for non-heartbeat packets.*/ - private Packet createPacket(int packetSize, int chunksPerPkt, long offsetInBlock, - long seqno) throws InterruptedIOException { + private DFSPacket createPacket(int packetSize, int chunksPerPkt, long offsetInBlock, + long seqno, boolean lastPacketInBlock) throws InterruptedIOException { final byte[] buf; final int bufferSize = PacketHeader.PKT_MAX_HEADER_LEN + packetSize; @@ -201,1465 +131,14 @@ private Packet createPacket(int packetSize, int chunksPerPkt, long offsetInBlock throw iioe; } - return new Packet(buf, chunksPerPkt, offsetInBlock, seqno, getChecksumSize()); - } - - /** - * For heartbeat packets, create buffer directly by new byte[] - * since heartbeats should not be blocked. - */ - private Packet createHeartbeatPacket() throws InterruptedIOException { - final byte[] buf = new byte[PacketHeader.PKT_MAX_HEADER_LEN]; - return new Packet(buf, 0, 0, Packet.HEART_BEAT_SEQNO, getChecksumSize()); - } - - private static class Packet { - private static final long HEART_BEAT_SEQNO = -1L; - final long seqno; // sequencenumber of buffer in block - final long offsetInBlock; // offset in block - boolean syncBlock; // this packet forces the current block to disk - int numChunks; // number of chunks currently in packet - final int maxChunks; // max chunks in packet - private byte[] buf; - private boolean lastPacketInBlock; // is this the last packet in block? - - /** - * buf is pointed into like follows: - * (C is checksum data, D is payload data) - * - * [_________CCCCCCCCC________________DDDDDDDDDDDDDDDD___] - * ^ ^ ^ ^ - * | checksumPos dataStart dataPos - * checksumStart - * - * Right before sending, we move the checksum data to immediately precede - * the actual data, and then insert the header into the buffer immediately - * preceding the checksum data, so we make sure to keep enough space in - * front of the checksum data to support the largest conceivable header. - */ - int checksumStart; - int checksumPos; - final int dataStart; - int dataPos; - - /** - * Create a new packet. - * - * @param chunksPerPkt maximum number of chunks per packet. - * @param offsetInBlock offset in bytes into the HDFS block. - */ - private Packet(byte[] buf, int chunksPerPkt, long offsetInBlock, long seqno, - int checksumSize) { - this.lastPacketInBlock = false; - this.numChunks = 0; - this.offsetInBlock = offsetInBlock; - this.seqno = seqno; - - this.buf = buf; - - checksumStart = PacketHeader.PKT_MAX_HEADER_LEN; - checksumPos = checksumStart; - dataStart = checksumStart + (chunksPerPkt * checksumSize); - dataPos = dataStart; - maxChunks = chunksPerPkt; - } - - synchronized void writeData(byte[] inarray, int off, int len) - throws ClosedChannelException { - checkBuffer(); - if (dataPos + len > buf.length) { - throw new BufferOverflowException(); - } - System.arraycopy(inarray, off, buf, dataPos, len); - dataPos += len; - } - - synchronized void writeChecksum(byte[] inarray, int off, int len) - throws ClosedChannelException { - checkBuffer(); - if (len == 0) { - return; - } - if (checksumPos + len > dataStart) { - throw new BufferOverflowException(); - } - System.arraycopy(inarray, off, buf, checksumPos, len); - checksumPos += len; - } - - /** - * Write the full packet, including the header, to the given output stream. - */ - synchronized void writeTo(DataOutputStream stm) throws IOException { - checkBuffer(); - - final int dataLen = dataPos - dataStart; - final int checksumLen = checksumPos - checksumStart; - final int pktLen = HdfsConstants.BYTES_IN_INTEGER + dataLen + checksumLen; - - PacketHeader header = new PacketHeader( - pktLen, offsetInBlock, seqno, lastPacketInBlock, dataLen, syncBlock); - - if (checksumPos != dataStart) { - // Move the checksum to cover the gap. This can happen for the last - // packet or during an hflush/hsync call. - System.arraycopy(buf, checksumStart, buf, - dataStart - checksumLen , checksumLen); - checksumPos = dataStart; - checksumStart = checksumPos - checksumLen; - } - - final int headerStart = checksumStart - header.getSerializedSize(); - assert checksumStart + 1 >= header.getSerializedSize(); - assert checksumPos == dataStart; - assert headerStart >= 0; - assert headerStart + header.getSerializedSize() == checksumStart; - - // Copy the header data into the buffer immediately preceding the checksum - // data. - System.arraycopy(header.getBytes(), 0, buf, headerStart, - header.getSerializedSize()); - - // corrupt the data for testing. - if (DFSClientFaultInjector.get().corruptPacket()) { - buf[headerStart+header.getSerializedSize() + checksumLen + dataLen-1] ^= 0xff; - } - - // Write the now contiguous full packet to the output stream. - stm.write(buf, headerStart, header.getSerializedSize() + checksumLen + dataLen); - - // undo corruption. - if (DFSClientFaultInjector.get().uncorruptPacket()) { - buf[headerStart+header.getSerializedSize() + checksumLen + dataLen-1] ^= 0xff; - } - } - - private synchronized void checkBuffer() throws ClosedChannelException { - if (buf == null) { - throw new ClosedChannelException(); - } - } - - private synchronized void releaseBuffer(ByteArrayManager bam) { - bam.release(buf); - buf = null; - } - - // get the packet's last byte's offset in the block - synchronized long getLastByteOffsetBlock() { - return offsetInBlock + dataPos - dataStart; - } - - /** - * Check if this packet is a heart beat packet - * @return true if the sequence number is HEART_BEAT_SEQNO - */ - private boolean isHeartbeatPacket() { - return seqno == HEART_BEAT_SEQNO; - } - - @Override - public String toString() { - return "packet seqno: " + this.seqno + - " offsetInBlock: " + this.offsetInBlock + - " lastPacketInBlock: " + this.lastPacketInBlock + - " lastByteOffsetInBlock: " + this.getLastByteOffsetBlock(); - } - } - - // - // The DataStreamer class is responsible for sending data packets to the - // datanodes in the pipeline. It retrieves a new blockid and block locations - // from the namenode, and starts streaming packets to the pipeline of - // Datanodes. Every packet has a sequence number associated with - // it. When all the packets for a block are sent out and acks for each - // if them are received, the DataStreamer closes the current block. - // - class DataStreamer extends Daemon { - private volatile boolean streamerClosed = false; - private ExtendedBlock block; // its length is number of bytes acked - private Token accessToken; - private DataOutputStream blockStream; - private DataInputStream blockReplyStream; - private ResponseProcessor response = null; - private volatile DatanodeInfo[] nodes = null; // list of targets for current block - private volatile StorageType[] storageTypes = null; - private volatile String[] storageIDs = null; - private final LoadingCache excludedNodes = - CacheBuilder.newBuilder() - .expireAfterWrite( - dfsClient.getConf().excludedNodesCacheExpiry, - TimeUnit.MILLISECONDS) - .removalListener(new RemovalListener() { - @Override - public void onRemoval( - RemovalNotification notification) { - DFSClient.LOG.info("Removing node " + - notification.getKey() + " from the excluded nodes list"); - } - }) - .build(new CacheLoader() { - @Override - public DatanodeInfo load(DatanodeInfo key) throws Exception { - return key; - } - }); - private String[] favoredNodes; - volatile boolean hasError = false; - volatile int errorIndex = -1; - // Restarting node index - AtomicInteger restartingNodeIndex = new AtomicInteger(-1); - private long restartDeadline = 0; // Deadline of DN restart - private BlockConstructionStage stage; // block construction stage - private long bytesSent = 0; // number of bytes that've been sent - private final boolean isLazyPersistFile; - - /** Nodes have been used in the pipeline before and have failed. */ - private final List failed = new ArrayList(); - /** The last ack sequence number before pipeline failure. */ - private long lastAckedSeqnoBeforeFailure = -1; - private int pipelineRecoveryCount = 0; - /** Has the current block been hflushed? */ - private boolean isHflushed = false; - /** Append on an existing block? */ - private final boolean isAppend; - - private final Span traceSpan; - - /** - * construction with tracing info - */ - private DataStreamer(HdfsFileStatus stat, ExtendedBlock block, Span span) { - isAppend = false; - isLazyPersistFile = isLazyPersist(stat); - this.block = block; - stage = BlockConstructionStage.PIPELINE_SETUP_CREATE; - traceSpan = span; - } - - /** - * Construct a data streamer for appending to the last partial block - * @param lastBlock last block of the file to be appended - * @param stat status of the file to be appended - * @param bytesPerChecksum number of bytes per checksum - * @throws IOException if error occurs - */ - private DataStreamer(LocatedBlock lastBlock, HdfsFileStatus stat, - int bytesPerChecksum, Span span) throws IOException { - isAppend = true; - stage = BlockConstructionStage.PIPELINE_SETUP_APPEND; - traceSpan = span; - block = lastBlock.getBlock(); - bytesSent = block.getNumBytes(); - accessToken = lastBlock.getBlockToken(); - isLazyPersistFile = isLazyPersist(stat); - long usedInLastBlock = stat.getLen() % blockSize; - int freeInLastBlock = (int)(blockSize - usedInLastBlock); - - // calculate the amount of free space in the pre-existing - // last crc chunk - int usedInCksum = (int)(stat.getLen() % bytesPerChecksum); - int freeInCksum = bytesPerChecksum - usedInCksum; - - // if there is space in the last block, then we have to - // append to that block - if (freeInLastBlock == blockSize) { - throw new IOException("The last block for file " + - src + " is full."); - } - - if (usedInCksum > 0 && freeInCksum > 0) { - // if there is space in the last partial chunk, then - // setup in such a way that the next packet will have only - // one chunk that fills up the partial chunk. - // - computePacketChunkSize(0, freeInCksum); - setChecksumBufSize(freeInCksum); - appendChunk = true; - } else { - // if the remaining space in the block is smaller than - // that expected size of of a packet, then create - // smaller size packet. - // - computePacketChunkSize(Math.min(dfsClient.getConf().writePacketSize, freeInLastBlock), - bytesPerChecksum); - } - - // setup pipeline to append to the last block XXX retries?? - setPipeline(lastBlock); - errorIndex = -1; // no errors yet. - if (nodes.length < 1) { - throw new IOException("Unable to retrieve blocks locations " + - " for last block " + block + - "of file " + src); - - } - } - - private void setPipeline(LocatedBlock lb) { - setPipeline(lb.getLocations(), lb.getStorageTypes(), lb.getStorageIDs()); - } - private void setPipeline(DatanodeInfo[] nodes, StorageType[] storageTypes, - String[] storageIDs) { - this.nodes = nodes; - this.storageTypes = storageTypes; - this.storageIDs = storageIDs; - } - - private void setFavoredNodes(String[] favoredNodes) { - this.favoredNodes = favoredNodes; - } - - /** - * Initialize for data streaming - */ - private void initDataStreaming() { - this.setName("DataStreamer for file " + src + - " block " + block); - response = new ResponseProcessor(nodes); - response.start(); - stage = BlockConstructionStage.DATA_STREAMING; - } - - private void endBlock() { - if(DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Closing old block " + block); - } - this.setName("DataStreamer for file " + src); - closeResponder(); - closeStream(); - setPipeline(null, null, null); - stage = BlockConstructionStage.PIPELINE_SETUP_CREATE; - } - - /* - * streamer thread is the only thread that opens streams to datanode, - * and closes them. Any error recovery is also done by this thread. - */ - @Override - public void run() { - long lastPacket = Time.now(); - TraceScope traceScope = null; - if (traceSpan != null) { - traceScope = Trace.continueSpan(traceSpan); - } - while (!streamerClosed && dfsClient.clientRunning) { - - // if the Responder encountered an error, shutdown Responder - if (hasError && response != null) { - try { - response.close(); - response.join(); - response = null; - } catch (InterruptedException e) { - DFSClient.LOG.warn("Caught exception ", e); - } - } - - Packet one; - try { - // process datanode IO errors if any - boolean doSleep = false; - if (hasError && (errorIndex >= 0 || restartingNodeIndex.get() >= 0)) { - doSleep = processDatanodeError(); - } - - synchronized (dataQueue) { - // wait for a packet to be sent. - long now = Time.now(); - while ((!streamerClosed && !hasError && dfsClient.clientRunning - && dataQueue.size() == 0 && - (stage != BlockConstructionStage.DATA_STREAMING || - stage == BlockConstructionStage.DATA_STREAMING && - now - lastPacket < dfsClient.getConf().socketTimeout/2)) || doSleep ) { - long timeout = dfsClient.getConf().socketTimeout/2 - (now-lastPacket); - timeout = timeout <= 0 ? 1000 : timeout; - timeout = (stage == BlockConstructionStage.DATA_STREAMING)? - timeout : 1000; - try { - dataQueue.wait(timeout); - } catch (InterruptedException e) { - DFSClient.LOG.warn("Caught exception ", e); - } - doSleep = false; - now = Time.now(); - } - if (streamerClosed || hasError || !dfsClient.clientRunning) { - continue; - } - // get packet to be sent. - if (dataQueue.isEmpty()) { - one = createHeartbeatPacket(); - } else { - one = dataQueue.getFirst(); // regular data packet - } - } - assert one != null; - - // get new block from namenode. - if (stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) { - if(DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Allocating new block"); - } - setPipeline(nextBlockOutputStream()); - initDataStreaming(); - } else if (stage == BlockConstructionStage.PIPELINE_SETUP_APPEND) { - if(DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Append to block " + block); - } - setupPipelineForAppendOrRecovery(); - initDataStreaming(); - } - - long lastByteOffsetInBlock = one.getLastByteOffsetBlock(); - if (lastByteOffsetInBlock > blockSize) { - throw new IOException("BlockSize " + blockSize + - " is smaller than data size. " + - " Offset of packet in block " + - lastByteOffsetInBlock + - " Aborting file " + src); - } - - if (one.lastPacketInBlock) { - // wait for all data packets have been successfully acked - synchronized (dataQueue) { - while (!streamerClosed && !hasError && - ackQueue.size() != 0 && dfsClient.clientRunning) { - try { - // wait for acks to arrive from datanodes - dataQueue.wait(1000); - } catch (InterruptedException e) { - DFSClient.LOG.warn("Caught exception ", e); - } - } - } - if (streamerClosed || hasError || !dfsClient.clientRunning) { - continue; - } - stage = BlockConstructionStage.PIPELINE_CLOSE; - } - - // send the packet - synchronized (dataQueue) { - // move packet from dataQueue to ackQueue - if (!one.isHeartbeatPacket()) { - dataQueue.removeFirst(); - ackQueue.addLast(one); - dataQueue.notifyAll(); - } - } - - if (DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("DataStreamer block " + block + - " sending packet " + one); - } - - // write out data to remote datanode - try { - one.writeTo(blockStream); - blockStream.flush(); - } catch (IOException e) { - // HDFS-3398 treat primary DN is down since client is unable to - // write to primary DN. If a failed or restarting node has already - // been recorded by the responder, the following call will have no - // effect. Pipeline recovery can handle only one node error at a - // time. If the primary node fails again during the recovery, it - // will be taken out then. - tryMarkPrimaryDatanodeFailed(); - throw e; - } - lastPacket = Time.now(); - - // update bytesSent - long tmpBytesSent = one.getLastByteOffsetBlock(); - if (bytesSent < tmpBytesSent) { - bytesSent = tmpBytesSent; - } - - if (streamerClosed || hasError || !dfsClient.clientRunning) { - continue; - } - - // Is this block full? - if (one.lastPacketInBlock) { - // wait for the close packet has been acked - synchronized (dataQueue) { - while (!streamerClosed && !hasError && - ackQueue.size() != 0 && dfsClient.clientRunning) { - dataQueue.wait(1000);// wait for acks to arrive from datanodes - } - } - if (streamerClosed || hasError || !dfsClient.clientRunning) { - continue; - } - - endBlock(); - } - if (progress != null) { progress.progress(); } - - // This is used by unit test to trigger race conditions. - if (artificialSlowdown != 0 && dfsClient.clientRunning) { - Thread.sleep(artificialSlowdown); - } - } catch (Throwable e) { - // Log warning if there was a real error. - if (restartingNodeIndex.get() == -1) { - DFSClient.LOG.warn("DataStreamer Exception", e); - } - if (e instanceof IOException) { - setLastException((IOException)e); - } else { - setLastException(new IOException("DataStreamer Exception: ",e)); - } - hasError = true; - if (errorIndex == -1 && restartingNodeIndex.get() == -1) { - // Not a datanode issue - streamerClosed = true; - } - } - } - if (traceScope != null) { - traceScope.close(); - } - closeInternal(); - } - - private void closeInternal() { - closeResponder(); // close and join - closeStream(); - streamerClosed = true; - setClosed(); - synchronized (dataQueue) { - dataQueue.notifyAll(); - } - } - - /* - * close both streamer and DFSOutputStream, should be called only - * by an external thread and only after all data to be sent has - * been flushed to datanode. - * - * Interrupt this data streamer if force is true - * - * @param force if this data stream is forced to be closed - */ - void close(boolean force) { - streamerClosed = true; - synchronized (dataQueue) { - dataQueue.notifyAll(); - } - if (force) { - this.interrupt(); - } - } - - private void closeResponder() { - if (response != null) { - try { - response.close(); - response.join(); - } catch (InterruptedException e) { - DFSClient.LOG.warn("Caught exception ", e); - } finally { - response = null; - } - } - } - - private void closeStream() { - if (blockStream != null) { - try { - blockStream.close(); - } catch (IOException e) { - setLastException(e); - } finally { - blockStream = null; - } - } - if (blockReplyStream != null) { - try { - blockReplyStream.close(); - } catch (IOException e) { - setLastException(e); - } finally { - blockReplyStream = null; - } - } - if (null != s) { - try { - s.close(); - } catch (IOException e) { - setLastException(e); - } finally { - s = null; - } - } - } - - // The following synchronized methods are used whenever - // errorIndex or restartingNodeIndex is set. This is because - // check & set needs to be atomic. Simply reading variables - // does not require a synchronization. When responder is - // not running (e.g. during pipeline recovery), there is no - // need to use these methods. - - /** Set the error node index. Called by responder */ - synchronized void setErrorIndex(int idx) { - errorIndex = idx; - } - - /** Set the restarting node index. Called by responder */ - synchronized void setRestartingNodeIndex(int idx) { - restartingNodeIndex.set(idx); - // If the data streamer has already set the primary node - // bad, clear it. It is likely that the write failed due to - // the DN shutdown. Even if it was a real failure, the pipeline - // recovery will take care of it. - errorIndex = -1; - } - - /** - * This method is used when no explicit error report was received, - * but something failed. When the primary node is a suspect or - * unsure about the cause, the primary node is marked as failed. - */ - synchronized void tryMarkPrimaryDatanodeFailed() { - // There should be no existing error and no ongoing restart. - if ((errorIndex == -1) && (restartingNodeIndex.get() == -1)) { - errorIndex = 0; - } - } - - /** - * Examine whether it is worth waiting for a node to restart. - * @param index the node index - */ - boolean shouldWaitForRestart(int index) { - // Only one node in the pipeline. - if (nodes.length == 1) { - return true; - } - - // Is it a local node? - InetAddress addr = null; - try { - addr = InetAddress.getByName(nodes[index].getIpAddr()); - } catch (java.net.UnknownHostException e) { - // we are passing an ip address. this should not happen. - assert false; - } - - if (addr != null && NetUtils.isLocalAddress(addr)) { - return true; - } - return false; - } - - // - // Processes responses from the datanodes. A packet is removed - // from the ackQueue when its response arrives. - // - private class ResponseProcessor extends Daemon { - - private volatile boolean responderClosed = false; - private DatanodeInfo[] targets = null; - private boolean isLastPacketInBlock = false; - - ResponseProcessor (DatanodeInfo[] targets) { - this.targets = targets; - } - - @Override - public void run() { - - setName("ResponseProcessor for block " + block); - PipelineAck ack = new PipelineAck(); - - while (!responderClosed && dfsClient.clientRunning && !isLastPacketInBlock) { - // process responses from datanodes. - try { - // read an ack from the pipeline - long begin = Time.monotonicNow(); - ack.readFields(blockReplyStream); - long duration = Time.monotonicNow() - begin; - if (duration > dfsclientSlowLogThresholdMs - && ack.getSeqno() != Packet.HEART_BEAT_SEQNO) { - DFSClient.LOG - .warn("Slow ReadProcessor read fields took " + duration - + "ms (threshold=" + dfsclientSlowLogThresholdMs + "ms); ack: " - + ack + ", targets: " + Arrays.asList(targets)); - } else if (DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("DFSClient " + ack); - } - - long seqno = ack.getSeqno(); - // processes response status from datanodes. - for (int i = ack.getNumOfReplies()-1; i >=0 && dfsClient.clientRunning; i--) { - final Status reply = PipelineAck.getStatusFromHeader(ack - .getReply(i)); - // Restart will not be treated differently unless it is - // the local node or the only one in the pipeline. - if (PipelineAck.isRestartOOBStatus(reply) && - shouldWaitForRestart(i)) { - restartDeadline = dfsClient.getConf().datanodeRestartTimeout + - Time.now(); - setRestartingNodeIndex(i); - String message = "A datanode is restarting: " + targets[i]; - DFSClient.LOG.info(message); - throw new IOException(message); - } - // node error - if (reply != SUCCESS) { - setErrorIndex(i); // first bad datanode - throw new IOException("Bad response " + reply + - " for block " + block + - " from datanode " + - targets[i]); - } - } - - assert seqno != PipelineAck.UNKOWN_SEQNO : - "Ack for unknown seqno should be a failed ack: " + ack; - if (seqno == Packet.HEART_BEAT_SEQNO) { // a heartbeat ack - continue; - } - - // a success ack for a data packet - Packet one; - synchronized (dataQueue) { - one = ackQueue.getFirst(); - } - if (one.seqno != seqno) { - throw new IOException("ResponseProcessor: Expecting seqno " + - " for block " + block + - one.seqno + " but received " + seqno); - } - isLastPacketInBlock = one.lastPacketInBlock; - - // Fail the packet write for testing in order to force a - // pipeline recovery. - if (DFSClientFaultInjector.get().failPacket() && - isLastPacketInBlock) { - failPacket = true; - throw new IOException( - "Failing the last packet for testing."); - } - - // update bytesAcked - block.setNumBytes(one.getLastByteOffsetBlock()); - - synchronized (dataQueue) { - lastAckedSeqno = seqno; - ackQueue.removeFirst(); - dataQueue.notifyAll(); - - one.releaseBuffer(byteArrayManager); - } - } catch (Exception e) { - if (!responderClosed) { - if (e instanceof IOException) { - setLastException((IOException)e); - } - hasError = true; - // If no explicit error report was received, mark the primary - // node as failed. - tryMarkPrimaryDatanodeFailed(); - synchronized (dataQueue) { - dataQueue.notifyAll(); - } - if (restartingNodeIndex.get() == -1) { - DFSClient.LOG.warn("DFSOutputStream ResponseProcessor exception " - + " for block " + block, e); - } - responderClosed = true; - } - } - } - } - - void close() { - responderClosed = true; - this.interrupt(); - } - } - - // If this stream has encountered any errors so far, shutdown - // threads and mark stream as closed. Returns true if we should - // sleep for a while after returning from this call. - // - private boolean processDatanodeError() throws IOException { - if (response != null) { - DFSClient.LOG.info("Error Recovery for " + block + - " waiting for responder to exit. "); - return true; - } - closeStream(); - - // move packets from ack queue to front of the data queue - synchronized (dataQueue) { - dataQueue.addAll(0, ackQueue); - ackQueue.clear(); - } - - // Record the new pipeline failure recovery. - if (lastAckedSeqnoBeforeFailure != lastAckedSeqno) { - lastAckedSeqnoBeforeFailure = lastAckedSeqno; - pipelineRecoveryCount = 1; - } else { - // If we had to recover the pipeline five times in a row for the - // same packet, this client likely has corrupt data or corrupting - // during transmission. - if (++pipelineRecoveryCount > 5) { - DFSClient.LOG.warn("Error recovering pipeline for writing " + - block + ". Already retried 5 times for the same packet."); - lastException.set(new IOException("Failing write. Tried pipeline " + - "recovery 5 times without success.")); - streamerClosed = true; - return false; - } - } - boolean doSleep = setupPipelineForAppendOrRecovery(); - - if (!streamerClosed && dfsClient.clientRunning) { - if (stage == BlockConstructionStage.PIPELINE_CLOSE) { - - // If we had an error while closing the pipeline, we go through a fast-path - // where the BlockReceiver does not run. Instead, the DataNode just finalizes - // the block immediately during the 'connect ack' process. So, we want to pull - // the end-of-block packet from the dataQueue, since we don't actually have - // a true pipeline to send it over. - // - // We also need to set lastAckedSeqno to the end-of-block Packet's seqno, so that - // a client waiting on close() will be aware that the flush finished. - synchronized (dataQueue) { - Packet endOfBlockPacket = dataQueue.remove(); // remove the end of block packet - assert endOfBlockPacket.lastPacketInBlock; - assert lastAckedSeqno == endOfBlockPacket.seqno - 1; - lastAckedSeqno = endOfBlockPacket.seqno; - dataQueue.notifyAll(); - } - endBlock(); - } else { - initDataStreaming(); - } - } - - return doSleep; - } - - private void setHflush() { - isHflushed = true; - } - - private int findNewDatanode(final DatanodeInfo[] original - ) throws IOException { - if (nodes.length != original.length + 1) { - throw new IOException( - new StringBuilder() - .append("Failed to replace a bad datanode on the existing pipeline ") - .append("due to no more good datanodes being available to try. ") - .append("(Nodes: current=").append(Arrays.asList(nodes)) - .append(", original=").append(Arrays.asList(original)).append("). ") - .append("The current failed datanode replacement policy is ") - .append(dfsClient.dtpReplaceDatanodeOnFailure).append(", and ") - .append("a client may configure this via '") - .append(DFSConfigKeys.DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_POLICY_KEY) - .append("' in its configuration.") - .toString()); - } - for(int i = 0; i < nodes.length; i++) { - int j = 0; - for(; j < original.length && !nodes[i].equals(original[j]); j++); - if (j == original.length) { - return i; - } - } - throw new IOException("Failed: new datanode not found: nodes=" - + Arrays.asList(nodes) + ", original=" + Arrays.asList(original)); - } - - private void addDatanode2ExistingPipeline() throws IOException { - if (DataTransferProtocol.LOG.isDebugEnabled()) { - DataTransferProtocol.LOG.debug("lastAckedSeqno = " + lastAckedSeqno); - } - /* - * Is data transfer necessary? We have the following cases. - * - * Case 1: Failure in Pipeline Setup - * - Append - * + Transfer the stored replica, which may be a RBW or a finalized. - * - Create - * + If no data, then no transfer is required. - * + If there are data written, transfer RBW. This case may happens - * when there are streaming failure earlier in this pipeline. - * - * Case 2: Failure in Streaming - * - Append/Create: - * + transfer RBW - * - * Case 3: Failure in Close - * - Append/Create: - * + no transfer, let NameNode replicates the block. - */ - if (!isAppend && lastAckedSeqno < 0 - && stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) { - //no data have been written - return; - } else if (stage == BlockConstructionStage.PIPELINE_CLOSE - || stage == BlockConstructionStage.PIPELINE_CLOSE_RECOVERY) { - //pipeline is closing - return; - } - - //get a new datanode - final DatanodeInfo[] original = nodes; - final LocatedBlock lb = dfsClient.namenode.getAdditionalDatanode( - src, fileId, block, nodes, storageIDs, - failed.toArray(new DatanodeInfo[failed.size()]), - 1, dfsClient.clientName); - setPipeline(lb); - - //find the new datanode - final int d = findNewDatanode(original); - - //transfer replica - final DatanodeInfo src = d == 0? nodes[1]: nodes[d - 1]; - final DatanodeInfo[] targets = {nodes[d]}; - final StorageType[] targetStorageTypes = {storageTypes[d]}; - transfer(src, targets, targetStorageTypes, lb.getBlockToken()); - } - - private void transfer(final DatanodeInfo src, final DatanodeInfo[] targets, - final StorageType[] targetStorageTypes, - final Token blockToken) throws IOException { - //transfer replica to the new datanode - Socket sock = null; - DataOutputStream out = null; - DataInputStream in = null; - try { - sock = createSocketForPipeline(src, 2, dfsClient); - final long writeTimeout = dfsClient.getDatanodeWriteTimeout(2); - - OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout); - InputStream unbufIn = NetUtils.getInputStream(sock); - IOStreamPair saslStreams = dfsClient.saslClient.socketSend(sock, - unbufOut, unbufIn, dfsClient, blockToken, src); - unbufOut = saslStreams.out; - unbufIn = saslStreams.in; - out = new DataOutputStream(new BufferedOutputStream(unbufOut, - HdfsConstants.SMALL_BUFFER_SIZE)); - in = new DataInputStream(unbufIn); - - //send the TRANSFER_BLOCK request - new Sender(out).transferBlock(block, blockToken, dfsClient.clientName, - targets, targetStorageTypes); - out.flush(); - - //ack - BlockOpResponseProto response = - BlockOpResponseProto.parseFrom(PBHelper.vintPrefixed(in)); - if (SUCCESS != response.getStatus()) { - throw new IOException("Failed to add a datanode"); - } - } finally { - IOUtils.closeStream(in); - IOUtils.closeStream(out); - IOUtils.closeSocket(sock); - } - } - - /** - * Open a DataOutputStream to a DataNode pipeline so that - * it can be written to. - * This happens when a file is appended or data streaming fails - * It keeps on trying until a pipeline is setup - */ - private boolean setupPipelineForAppendOrRecovery() throws IOException { - // check number of datanodes - if (nodes == null || nodes.length == 0) { - String msg = "Could not get block locations. " + "Source file \"" - + src + "\" - Aborting..."; - DFSClient.LOG.warn(msg); - setLastException(new IOException(msg)); - streamerClosed = true; - return false; - } - - boolean success = false; - long newGS = 0L; - while (!success && !streamerClosed && dfsClient.clientRunning) { - // Sleep before reconnect if a dn is restarting. - // This process will be repeated until the deadline or the datanode - // starts back up. - if (restartingNodeIndex.get() >= 0) { - // 4 seconds or the configured deadline period, whichever is shorter. - // This is the retry interval and recovery will be retried in this - // interval until timeout or success. - long delay = Math.min(dfsClient.getConf().datanodeRestartTimeout, - 4000L); - try { - Thread.sleep(delay); - } catch (InterruptedException ie) { - lastException.set(new IOException("Interrupted while waiting for " + - "datanode to restart. " + nodes[restartingNodeIndex.get()])); - streamerClosed = true; - return false; - } - } - boolean isRecovery = hasError; - // remove bad datanode from list of datanodes. - // If errorIndex was not set (i.e. appends), then do not remove - // any datanodes - // - if (errorIndex >= 0) { - StringBuilder pipelineMsg = new StringBuilder(); - for (int j = 0; j < nodes.length; j++) { - pipelineMsg.append(nodes[j]); - if (j < nodes.length - 1) { - pipelineMsg.append(", "); - } - } - if (nodes.length <= 1) { - lastException.set(new IOException("All datanodes " + pipelineMsg - + " are bad. Aborting...")); - streamerClosed = true; - return false; - } - DFSClient.LOG.warn("Error Recovery for block " + block + - " in pipeline " + pipelineMsg + - ": bad datanode " + nodes[errorIndex]); - failed.add(nodes[errorIndex]); - - DatanodeInfo[] newnodes = new DatanodeInfo[nodes.length-1]; - arraycopy(nodes, newnodes, errorIndex); - - final StorageType[] newStorageTypes = new StorageType[newnodes.length]; - arraycopy(storageTypes, newStorageTypes, errorIndex); - - final String[] newStorageIDs = new String[newnodes.length]; - arraycopy(storageIDs, newStorageIDs, errorIndex); - - setPipeline(newnodes, newStorageTypes, newStorageIDs); - - // Just took care of a node error while waiting for a node restart - if (restartingNodeIndex.get() >= 0) { - // If the error came from a node further away than the restarting - // node, the restart must have been complete. - if (errorIndex > restartingNodeIndex.get()) { - restartingNodeIndex.set(-1); - } else if (errorIndex < restartingNodeIndex.get()) { - // the node index has shifted. - restartingNodeIndex.decrementAndGet(); - } else { - // this shouldn't happen... - assert false; - } - } - - if (restartingNodeIndex.get() == -1) { - hasError = false; - } - lastException.set(null); - errorIndex = -1; - } - - // Check if replace-datanode policy is satisfied. - if (dfsClient.dtpReplaceDatanodeOnFailure.satisfy(blockReplication, - nodes, isAppend, isHflushed)) { - try { - addDatanode2ExistingPipeline(); - } catch(IOException ioe) { - if (!dfsClient.dtpReplaceDatanodeOnFailure.isBestEffort()) { - throw ioe; - } - DFSClient.LOG.warn("Failed to replace datanode." - + " Continue with the remaining datanodes since " - + DFSConfigKeys.DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_BEST_EFFORT_KEY - + " is set to true.", ioe); - } - } - - // get a new generation stamp and an access token - LocatedBlock lb = dfsClient.namenode.updateBlockForPipeline(block, dfsClient.clientName); - newGS = lb.getBlock().getGenerationStamp(); - accessToken = lb.getBlockToken(); - - // set up the pipeline again with the remaining nodes - if (failPacket) { // for testing - success = createBlockOutputStream(nodes, storageTypes, newGS, isRecovery); - failPacket = false; - try { - // Give DNs time to send in bad reports. In real situations, - // good reports should follow bad ones, if client committed - // with those nodes. - Thread.sleep(2000); - } catch (InterruptedException ie) {} - } else { - success = createBlockOutputStream(nodes, storageTypes, newGS, isRecovery); - } - - if (restartingNodeIndex.get() >= 0) { - assert hasError == true; - // check errorIndex set above - if (errorIndex == restartingNodeIndex.get()) { - // ignore, if came from the restarting node - errorIndex = -1; - } - // still within the deadline - if (Time.now() < restartDeadline) { - continue; // with in the deadline - } - // expired. declare the restarting node dead - restartDeadline = 0; - int expiredNodeIndex = restartingNodeIndex.get(); - restartingNodeIndex.set(-1); - DFSClient.LOG.warn("Datanode did not restart in time: " + - nodes[expiredNodeIndex]); - // Mark the restarting node as failed. If there is any other failed - // node during the last pipeline construction attempt, it will not be - // overwritten/dropped. In this case, the restarting node will get - // excluded in the following attempt, if it still does not come up. - if (errorIndex == -1) { - errorIndex = expiredNodeIndex; - } - // From this point on, normal pipeline recovery applies. - } - } // while - - if (success) { - // update pipeline at the namenode - ExtendedBlock newBlock = new ExtendedBlock( - block.getBlockPoolId(), block.getBlockId(), block.getNumBytes(), newGS); - dfsClient.namenode.updatePipeline(dfsClient.clientName, block, newBlock, - nodes, storageIDs); - // update client side generation stamp - block = newBlock; - } - return false; // do not sleep, continue processing - } - - /** - * Open a DataOutputStream to a DataNode so that it can be written to. - * This happens when a file is created and each time a new block is allocated. - * Must get block ID and the IDs of the destinations from the namenode. - * Returns the list of target datanodes. - */ - private LocatedBlock nextBlockOutputStream() throws IOException { - LocatedBlock lb = null; - DatanodeInfo[] nodes = null; - StorageType[] storageTypes = null; - int count = dfsClient.getConf().nBlockWriteRetry; - boolean success = false; - ExtendedBlock oldBlock = block; - do { - hasError = false; - lastException.set(null); - errorIndex = -1; - success = false; - - long startTime = Time.now(); - DatanodeInfo[] excluded = - excludedNodes.getAllPresent(excludedNodes.asMap().keySet()) - .keySet() - .toArray(new DatanodeInfo[0]); - block = oldBlock; - lb = locateFollowingBlock(startTime, - excluded.length > 0 ? excluded : null); - block = lb.getBlock(); - block.setNumBytes(0); - bytesSent = 0; - accessToken = lb.getBlockToken(); - nodes = lb.getLocations(); - storageTypes = lb.getStorageTypes(); - - // - // Connect to first DataNode in the list. - // - success = createBlockOutputStream(nodes, storageTypes, 0L, false); - - if (!success) { - DFSClient.LOG.info("Abandoning " + block); - dfsClient.namenode.abandonBlock(block, fileId, src, - dfsClient.clientName); - block = null; - DFSClient.LOG.info("Excluding datanode " + nodes[errorIndex]); - excludedNodes.put(nodes[errorIndex], nodes[errorIndex]); - } - } while (!success && --count >= 0); - - if (!success) { - throw new IOException("Unable to create new block."); - } - return lb; - } - - // connects to the first datanode in the pipeline - // Returns true if success, otherwise return failure. - // - private boolean createBlockOutputStream(DatanodeInfo[] nodes, - StorageType[] nodeStorageTypes, long newGS, boolean recoveryFlag) { - if (nodes.length == 0) { - DFSClient.LOG.info("nodes are empty for write pipeline of block " - + block); - return false; - } - Status pipelineStatus = SUCCESS; - String firstBadLink = ""; - boolean checkRestart = false; - if (DFSClient.LOG.isDebugEnabled()) { - for (int i = 0; i < nodes.length; i++) { - DFSClient.LOG.debug("pipeline = " + nodes[i]); - } - } - - // persist blocks on namenode on next flush - persistBlocks.set(true); - - int refetchEncryptionKey = 1; - while (true) { - boolean result = false; - DataOutputStream out = null; - try { - assert null == s : "Previous socket unclosed"; - assert null == blockReplyStream : "Previous blockReplyStream unclosed"; - s = createSocketForPipeline(nodes[0], nodes.length, dfsClient); - long writeTimeout = dfsClient.getDatanodeWriteTimeout(nodes.length); - - OutputStream unbufOut = NetUtils.getOutputStream(s, writeTimeout); - InputStream unbufIn = NetUtils.getInputStream(s); - IOStreamPair saslStreams = dfsClient.saslClient.socketSend(s, - unbufOut, unbufIn, dfsClient, accessToken, nodes[0]); - unbufOut = saslStreams.out; - unbufIn = saslStreams.in; - out = new DataOutputStream(new BufferedOutputStream(unbufOut, - HdfsConstants.SMALL_BUFFER_SIZE)); - blockReplyStream = new DataInputStream(unbufIn); - - // - // Xmit header info to datanode - // - - BlockConstructionStage bcs = recoveryFlag? stage.getRecoveryStage(): stage; - - // We cannot change the block length in 'block' as it counts the number - // of bytes ack'ed. - ExtendedBlock blockCopy = new ExtendedBlock(block); - blockCopy.setNumBytes(blockSize); - - boolean[] targetPinnings = getPinnings(nodes, true); - // send the request - new Sender(out).writeBlock(blockCopy, nodeStorageTypes[0], accessToken, - dfsClient.clientName, nodes, nodeStorageTypes, null, bcs, - nodes.length, block.getNumBytes(), bytesSent, newGS, - checksum4WriteBlock, cachingStrategy.get(), isLazyPersistFile, - (targetPinnings == null ? false : targetPinnings[0]), targetPinnings); - - // receive ack for connect - BlockOpResponseProto resp = BlockOpResponseProto.parseFrom( - PBHelper.vintPrefixed(blockReplyStream)); - pipelineStatus = resp.getStatus(); - firstBadLink = resp.getFirstBadLink(); - - // Got an restart OOB ack. - // If a node is already restarting, this status is not likely from - // the same node. If it is from a different node, it is not - // from the local datanode. Thus it is safe to treat this as a - // regular node error. - if (PipelineAck.isRestartOOBStatus(pipelineStatus) && - restartingNodeIndex.get() == -1) { - checkRestart = true; - throw new IOException("A datanode is restarting."); - } - - String logInfo = "ack with firstBadLink as " + firstBadLink; - DataTransferProtoUtil.checkBlockOpStatus(resp, logInfo); - - assert null == blockStream : "Previous blockStream unclosed"; - blockStream = out; - result = true; // success - restartingNodeIndex.set(-1); - hasError = false; - } catch (IOException ie) { - if (restartingNodeIndex.get() == -1) { - DFSClient.LOG.info("Exception in createBlockOutputStream", ie); - } - if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) { - DFSClient.LOG.info("Will fetch a new encryption key and retry, " - + "encryption key was invalid when connecting to " - + nodes[0] + " : " + ie); - // The encryption key used is invalid. - refetchEncryptionKey--; - dfsClient.clearDataEncryptionKey(); - // Don't close the socket/exclude this node just yet. Try again with - // a new encryption key. - continue; - } - - // find the datanode that matches - if (firstBadLink.length() != 0) { - for (int i = 0; i < nodes.length; i++) { - // NB: Unconditionally using the xfer addr w/o hostname - if (firstBadLink.equals(nodes[i].getXferAddr())) { - errorIndex = i; - break; - } - } - } else { - assert checkRestart == false; - errorIndex = 0; - } - // Check whether there is a restart worth waiting for. - if (checkRestart && shouldWaitForRestart(errorIndex)) { - restartDeadline = dfsClient.getConf().datanodeRestartTimeout + - Time.now(); - restartingNodeIndex.set(errorIndex); - errorIndex = -1; - DFSClient.LOG.info("Waiting for the datanode to be restarted: " + - nodes[restartingNodeIndex.get()]); - } - hasError = true; - setLastException(ie); - result = false; // error - } finally { - if (!result) { - IOUtils.closeSocket(s); - s = null; - IOUtils.closeStream(out); - out = null; - IOUtils.closeStream(blockReplyStream); - blockReplyStream = null; - } - } - return result; - } - } - - private boolean[] getPinnings(DatanodeInfo[] nodes, boolean shouldLog) { - if (favoredNodes == null) { - return null; - } else { - boolean[] pinnings = new boolean[nodes.length]; - HashSet favoredSet = - new HashSet(Arrays.asList(favoredNodes)); - for (int i = 0; i < nodes.length; i++) { - pinnings[i] = favoredSet.remove(nodes[i].getXferAddrWithHostname()); - if (DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug(nodes[i].getXferAddrWithHostname() + - " was chosen by name node (favored=" + pinnings[i] + - ")."); - } - } - if (shouldLog && !favoredSet.isEmpty()) { - // There is one or more favored nodes that were not allocated. - DFSClient.LOG.warn( - "These favored nodes were specified but not chosen: " + - favoredSet + - " Specified favored nodes: " + Arrays.toString(favoredNodes)); - - } - return pinnings; - } - } - - private LocatedBlock locateFollowingBlock(long start, - DatanodeInfo[] excludedNodes) throws IOException { - int retries = dfsClient.getConf().nBlockWriteLocateFollowingRetry; - long sleeptime = 400; - while (true) { - long localstart = Time.now(); - while (true) { - try { - return dfsClient.namenode.addBlock(src, dfsClient.clientName, - block, excludedNodes, fileId, favoredNodes); - } catch (RemoteException e) { - IOException ue = - e.unwrapRemoteException(FileNotFoundException.class, - AccessControlException.class, - NSQuotaExceededException.class, - DSQuotaExceededException.class, - UnresolvedPathException.class); - if (ue != e) { - throw ue; // no need to retry these exceptions - } - - - if (NotReplicatedYetException.class.getName(). - equals(e.getClassName())) { - if (retries == 0) { - throw e; - } else { - --retries; - DFSClient.LOG.info("Exception while adding a block", e); - if (Time.now() - localstart > 5000) { - DFSClient.LOG.info("Waiting for replication for " - + (Time.now() - localstart) / 1000 - + " seconds"); - } - try { - DFSClient.LOG.warn("NotReplicatedYetException sleeping " + src - + " retries left " + retries); - Thread.sleep(sleeptime); - sleeptime *= 2; - } catch (InterruptedException ie) { - DFSClient.LOG.warn("Caught exception ", ie); - } - } - } else { - throw e; - } - - } - } - } - } - - ExtendedBlock getBlock() { - return block; - } - - DatanodeInfo[] getNodes() { - return nodes; - } - - Token getBlockToken() { - return accessToken; - } - - private void setLastException(IOException e) { - lastException.compareAndSet(null, e); - } - } - - /** - * Create a socket for a write pipeline - * @param first the first datanode - * @param length the pipeline length - * @param client client - * @return the socket connected to the first datanode - */ - static Socket createSocketForPipeline(final DatanodeInfo first, - final int length, final DFSClient client) throws IOException { - final String dnAddr = first.getXferAddr( - client.getConf().connectToDnViaHostname); - if (DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Connecting to datanode " + dnAddr); - } - final InetSocketAddress isa = NetUtils.createSocketAddr(dnAddr); - final Socket sock = client.socketFactory.createSocket(); - final int timeout = client.getDatanodeReadTimeout(length); - NetUtils.connect(sock, isa, client.getRandomLocalInterfaceAddr(), client.getConf().socketTimeout); - sock.setSoTimeout(timeout); - sock.setSendBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE); - if(DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Send buf size " + sock.getSendBufferSize()); - } - return sock; + return new DFSPacket(buf, chunksPerPkt, offsetInBlock, seqno, + getChecksumSize(), lastPacketInBlock); } @Override protected void checkClosed() throws IOException { if (isClosed()) { - IOException e = lastException.get(); + IOException e = streamer.getLastException().get(); throw e != null ? e : new ClosedChannelException(); } } @@ -1669,7 +148,7 @@ protected void checkClosed() throws IOException { // @VisibleForTesting public synchronized DatanodeInfo[] getPipeline() { - if (streamer == null) { + if (streamer.streamerClosed()) { return null; } DatanodeInfo[] currentNodes = streamer.getNodes(); @@ -1689,7 +168,7 @@ public synchronized DatanodeInfo[] getPipeline() { */ private static DataChecksum getChecksum4Compute(DataChecksum checksum, HdfsFileStatus stat) { - if (isLazyPersist(stat) && stat.getReplication() == 1) { + if (DataStreamer.isLazyPersist(stat) && stat.getReplication() == 1) { // do not compute checksum for writing to single replica to memory return DataChecksum.newDataChecksum(Type.NULL, checksum.getBytesPerChecksum()); @@ -1706,7 +185,6 @@ private DFSOutputStream(DFSClient dfsClient, String src, Progressable progress, this.blockSize = stat.getBlockSize(); this.blockReplication = stat.getReplication(); this.fileEncryptionInfo = stat.getFileEncryptionInfo(); - this.progress = progress; this.cachingStrategy = new AtomicReference( dfsClient.getDefaultWriteCachingStrategy()); if ((progress != null) && DFSClient.LOG.isDebugEnabled()) { @@ -1724,10 +202,6 @@ private DFSOutputStream(DFSClient dfsClient, String src, Progressable progress, + DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY + " (=" + bytesPerChecksum + ") must divide block size (=" + blockSize + ")."); } - this.checksum4WriteBlock = checksum; - - this.dfsclientSlowLogThresholdMs = - dfsClient.getConf().dfsclientSlowIoWarningThresholdMs; this.byteArrayManager = dfsClient.getClientContext().getByteArrayManager(); } @@ -1740,11 +214,8 @@ private DFSOutputStream(DFSClient dfsClient, String src, HdfsFileStatus stat, computePacketChunkSize(dfsClient.getConf().writePacketSize, bytesPerChecksum); - Span traceSpan = null; - if (Trace.isTracing()) { - traceSpan = Trace.startSpan(this.getClass().getSimpleName()).detach(); - } - streamer = new DataStreamer(stat, null, traceSpan); + streamer = new DataStreamer(stat, null, dfsClient, src, progress, checksum, + cachingStrategy, byteArrayManager); if (favoredNodes != null && favoredNodes.length != 0) { streamer.setFavoredNodes(favoredNodes); } @@ -1754,95 +225,138 @@ static DFSOutputStream newStreamForCreate(DFSClient dfsClient, String src, FsPermission masked, EnumSet flag, boolean createParent, short replication, long blockSize, Progressable progress, int buffersize, DataChecksum checksum, String[] favoredNodes) throws IOException { - HdfsFileStatus stat = null; - - // Retry the create if we get a RetryStartFileException up to a maximum - // number of times - boolean shouldRetry = true; - int retryCount = CREATE_RETRY_COUNT; - while (shouldRetry) { - shouldRetry = false; - try { - stat = dfsClient.namenode.create(src, masked, dfsClient.clientName, - new EnumSetWritable(flag), createParent, replication, - blockSize, SUPPORTED_CRYPTO_VERSIONS); - break; - } catch (RemoteException re) { - IOException e = re.unwrapRemoteException( - AccessControlException.class, - DSQuotaExceededException.class, - FileAlreadyExistsException.class, - FileNotFoundException.class, - ParentNotDirectoryException.class, - NSQuotaExceededException.class, - RetryStartFileException.class, - SafeModeException.class, - UnresolvedPathException.class, - SnapshotAccessControlException.class, - UnknownCryptoProtocolVersionException.class); - if (e instanceof RetryStartFileException) { - if (retryCount > 0) { - shouldRetry = true; - retryCount--; + TraceScope scope = + dfsClient.getPathTraceScope("newStreamForCreate", src); + try { + HdfsFileStatus stat = null; + + // Retry the create if we get a RetryStartFileException up to a maximum + // number of times + boolean shouldRetry = true; + int retryCount = CREATE_RETRY_COUNT; + while (shouldRetry) { + shouldRetry = false; + try { + stat = dfsClient.namenode.create(src, masked, dfsClient.clientName, + new EnumSetWritable(flag), createParent, replication, + blockSize, SUPPORTED_CRYPTO_VERSIONS); + break; + } catch (RemoteException re) { + IOException e = re.unwrapRemoteException( + AccessControlException.class, + DSQuotaExceededException.class, + FileAlreadyExistsException.class, + FileNotFoundException.class, + ParentNotDirectoryException.class, + NSQuotaExceededException.class, + RetryStartFileException.class, + SafeModeException.class, + UnresolvedPathException.class, + SnapshotAccessControlException.class, + UnknownCryptoProtocolVersionException.class); + if (e instanceof RetryStartFileException) { + if (retryCount > 0) { + shouldRetry = true; + retryCount--; + } else { + throw new IOException("Too many retries because of encryption" + + " zone operations", e); + } } else { - throw new IOException("Too many retries because of encryption" + - " zone operations", e); + throw e; } - } else { - throw e; } } + Preconditions.checkNotNull(stat, "HdfsFileStatus should not be null!"); + final DFSOutputStream out = new DFSOutputStream(dfsClient, src, stat, + flag, progress, checksum, favoredNodes); + out.start(); + return out; + } finally { + scope.close(); } - Preconditions.checkNotNull(stat, "HdfsFileStatus should not be null!"); - final DFSOutputStream out = new DFSOutputStream(dfsClient, src, stat, - flag, progress, checksum, favoredNodes); - out.start(); - return out; } /** Construct a new output stream for append. */ - private DFSOutputStream(DFSClient dfsClient, String src, boolean toNewBlock, - Progressable progress, LocatedBlock lastBlock, HdfsFileStatus stat, - DataChecksum checksum) throws IOException { + private DFSOutputStream(DFSClient dfsClient, String src, + EnumSet flags, Progressable progress, LocatedBlock lastBlock, + HdfsFileStatus stat, DataChecksum checksum) throws IOException { this(dfsClient, src, progress, stat, checksum); initialFileSize = stat.getLen(); // length of file when opened + this.shouldSyncBlock = flags.contains(CreateFlag.SYNC_BLOCK); - Span traceSpan = null; - if (Trace.isTracing()) { - traceSpan = Trace.startSpan(this.getClass().getSimpleName()).detach(); - } + boolean toNewBlock = flags.contains(CreateFlag.NEW_BLOCK); + + this.fileEncryptionInfo = stat.getFileEncryptionInfo(); // The last partial block of the file has to be filled. if (!toNewBlock && lastBlock != null) { // indicate that we are appending to an existing block - bytesCurBlock = lastBlock.getBlockSize(); - streamer = new DataStreamer(lastBlock, stat, bytesPerChecksum, traceSpan); + streamer = new DataStreamer(lastBlock, stat, dfsClient, src, progress, checksum, + cachingStrategy, byteArrayManager); + streamer.setBytesCurBlock(lastBlock.getBlockSize()); + adjustPacketChunkSize(stat); + streamer.setPipelineInConstruction(lastBlock); } else { computePacketChunkSize(dfsClient.getConf().writePacketSize, bytesPerChecksum); - streamer = new DataStreamer(stat, - lastBlock != null ? lastBlock.getBlock() : null, traceSpan); + streamer = new DataStreamer(stat, lastBlock != null ? lastBlock.getBlock() : null, + dfsClient, src, progress, checksum, cachingStrategy, byteArrayManager); + } + } + + private void adjustPacketChunkSize(HdfsFileStatus stat) throws IOException{ + + long usedInLastBlock = stat.getLen() % blockSize; + int freeInLastBlock = (int)(blockSize - usedInLastBlock); + + // calculate the amount of free space in the pre-existing + // last crc chunk + int usedInCksum = (int)(stat.getLen() % bytesPerChecksum); + int freeInCksum = bytesPerChecksum - usedInCksum; + + // if there is space in the last block, then we have to + // append to that block + if (freeInLastBlock == blockSize) { + throw new IOException("The last block for file " + + src + " is full."); + } + + if (usedInCksum > 0 && freeInCksum > 0) { + // if there is space in the last partial chunk, then + // setup in such a way that the next packet will have only + // one chunk that fills up the partial chunk. + // + computePacketChunkSize(0, freeInCksum); + setChecksumBufSize(freeInCksum); + streamer.setAppendChunk(true); + } else { + // if the remaining space in the block is smaller than + // that expected size of of a packet, then create + // smaller size packet. + // + computePacketChunkSize(Math.min(dfsClient.getConf().writePacketSize, freeInLastBlock), + bytesPerChecksum); } - this.fileEncryptionInfo = stat.getFileEncryptionInfo(); } static DFSOutputStream newStreamForAppend(DFSClient dfsClient, String src, - boolean toNewBlock, int bufferSize, Progressable progress, + EnumSet flags, int bufferSize, Progressable progress, LocatedBlock lastBlock, HdfsFileStatus stat, DataChecksum checksum, String[] favoredNodes) throws IOException { - final DFSOutputStream out = new DFSOutputStream(dfsClient, src, toNewBlock, - progress, lastBlock, stat, checksum); - if (favoredNodes != null && favoredNodes.length != 0) { - out.streamer.setFavoredNodes(favoredNodes); + TraceScope scope = + dfsClient.getPathTraceScope("newStreamForAppend", src); + try { + final DFSOutputStream out = new DFSOutputStream(dfsClient, src, flags, + progress, lastBlock, stat, checksum); + if (favoredNodes != null && favoredNodes.length != 0) { + out.streamer.setFavoredNodes(favoredNodes); + } + out.start(); + return out; + } finally { + scope.close(); } - out.start(); - return out; - } - - private static boolean isLazyPersist(HdfsFileStatus stat) { - final BlockStoragePolicy p = blockStoragePolicySuite.getPolicy( - HdfsConstants.MEMORY_STORAGE_POLICY_NAME); - return p != null && stat.getStoragePolicy() == p.getId(); } private void computePacketChunkSize(int psize, int csize) { @@ -1858,49 +372,21 @@ private void computePacketChunkSize(int psize, int csize) { } } - private void queueCurrentPacket() { - synchronized (dataQueue) { - if (currentPacket == null) return; - dataQueue.addLast(currentPacket); - lastQueuedSeqno = currentPacket.seqno; - if (DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Queued packet " + currentPacket.seqno); - } - currentPacket = null; - dataQueue.notifyAll(); - } - } - - private void waitAndQueueCurrentPacket() throws IOException { - synchronized (dataQueue) { - try { - // If queue is full, then wait till we have enough space - while (!isClosed() && dataQueue.size() + ackQueue.size() > dfsClient.getConf().writeMaxPackets) { - try { - dataQueue.wait(); - } catch (InterruptedException e) { - // If we get interrupted while waiting to queue data, we still need to get rid - // of the current packet. This is because we have an invariant that if - // currentPacket gets full, it will get queued before the next writeChunk. - // - // Rather than wait around for space in the queue, we should instead try to - // return to the caller as soon as possible, even though we slightly overrun - // the MAX_PACKETS length. - Thread.currentThread().interrupt(); - break; - } - } - checkClosed(); - queueCurrentPacket(); - } catch (ClosedChannelException e) { - } - } - } - // @see FSOutputSummer#writeChunk() @Override protected synchronized void writeChunk(byte[] b, int offset, int len, byte[] checksum, int ckoff, int cklen) throws IOException { + TraceScope scope = + dfsClient.getPathTraceScope("DFSOutputStream#writeChunk", src); + try { + writeChunkImpl(b, offset, len, checksum, ckoff, cklen); + } finally { + scope.close(); + } + } + + private synchronized void writeChunkImpl(byte[] b, int offset, int len, + byte[] checksum, int ckoff, int cklen) throws IOException { dfsClient.checkOpen(); checkClosed(); @@ -1916,58 +402,62 @@ protected synchronized void writeChunk(byte[] b, int offset, int len, if (currentPacket == null) { currentPacket = createPacket(packetSize, chunksPerPacket, - bytesCurBlock, currentSeqno++); + streamer.getBytesCurBlock(), streamer.getAndIncCurrentSeqno(), false); if (DFSClient.LOG.isDebugEnabled()) { DFSClient.LOG.debug("DFSClient writeChunk allocating new packet seqno=" + - currentPacket.seqno + + currentPacket.getSeqno() + ", src=" + src + ", packetSize=" + packetSize + ", chunksPerPacket=" + chunksPerPacket + - ", bytesCurBlock=" + bytesCurBlock); + ", bytesCurBlock=" + streamer.getBytesCurBlock()); } } currentPacket.writeChecksum(checksum, ckoff, cklen); currentPacket.writeData(b, offset, len); - currentPacket.numChunks++; - bytesCurBlock += len; + currentPacket.incNumChunks(); + streamer.incBytesCurBlock(len); // If packet is full, enqueue it for transmission // - if (currentPacket.numChunks == currentPacket.maxChunks || - bytesCurBlock == blockSize) { + if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() || + streamer.getBytesCurBlock() == blockSize) { if (DFSClient.LOG.isDebugEnabled()) { DFSClient.LOG.debug("DFSClient writeChunk packet full seqno=" + - currentPacket.seqno + + currentPacket.getSeqno() + ", src=" + src + - ", bytesCurBlock=" + bytesCurBlock + + ", bytesCurBlock=" + streamer.getBytesCurBlock() + ", blockSize=" + blockSize + - ", appendChunk=" + appendChunk); + ", appendChunk=" + streamer.getAppendChunk()); } - waitAndQueueCurrentPacket(); + streamer.waitAndQueuePacket(currentPacket); + currentPacket = null; // If the reopened file did not end at chunk boundary and the above // write filled up its partial chunk. Tell the summer to generate full // crc chunks from now on. - if (appendChunk && bytesCurBlock%bytesPerChecksum == 0) { - appendChunk = false; + if (streamer.getAppendChunk() && + streamer.getBytesCurBlock() % bytesPerChecksum == 0) { + streamer.setAppendChunk(false); resetChecksumBufSize(); } - if (!appendChunk) { - int psize = Math.min((int)(blockSize-bytesCurBlock), dfsClient.getConf().writePacketSize); + if (!streamer.getAppendChunk()) { + int psize = Math.min((int)(blockSize-streamer.getBytesCurBlock()), + dfsClient.getConf().writePacketSize); computePacketChunkSize(psize, bytesPerChecksum); } // // if encountering a block boundary, send an empty packet to // indicate the end of block and reset bytesCurBlock. // - if (bytesCurBlock == blockSize) { - currentPacket = createPacket(0, 0, bytesCurBlock, currentSeqno++); - currentPacket.lastPacketInBlock = true; - currentPacket.syncBlock = shouldSyncBlock; - waitAndQueueCurrentPacket(); - bytesCurBlock = 0; + if (streamer.getBytesCurBlock() == blockSize) { + currentPacket = createPacket(0, 0, streamer.getBytesCurBlock(), + streamer.getAndIncCurrentSeqno(), true); + currentPacket.setSyncBlock(shouldSyncBlock); + streamer.waitAndQueuePacket(currentPacket); + currentPacket = null; + streamer.setBytesCurBlock(0); lastFlushOffset = 0; } } @@ -1985,12 +475,24 @@ protected synchronized void writeChunk(byte[] b, int offset, int len, */ @Override public void hflush() throws IOException { - flushOrSync(false, EnumSet.noneOf(SyncFlag.class)); + TraceScope scope = + dfsClient.getPathTraceScope("hflush", src); + try { + flushOrSync(false, EnumSet.noneOf(SyncFlag.class)); + } finally { + scope.close(); + } } @Override public void hsync() throws IOException { - hsync(EnumSet.noneOf(SyncFlag.class)); + TraceScope scope = + dfsClient.getPathTraceScope("hsync", src); + try { + flushOrSync(true, EnumSet.noneOf(SyncFlag.class)); + } finally { + scope.close(); + } } /** @@ -2007,7 +509,13 @@ public void hsync() throws IOException { * whether or not to update the block length in NameNode. */ public void hsync(EnumSet syncFlags) throws IOException { - flushOrSync(true, syncFlags); + TraceScope scope = + dfsClient.getPathTraceScope("hsync", src); + try { + flushOrSync(true, syncFlags); + } finally { + scope.close(); + } } /** @@ -2039,30 +547,30 @@ private void flushOrSync(boolean isSync, EnumSet syncFlags) if (DFSClient.LOG.isDebugEnabled()) { DFSClient.LOG.debug("DFSClient flush(): " - + " bytesCurBlock=" + bytesCurBlock + + " bytesCurBlock=" + streamer.getBytesCurBlock() + " lastFlushOffset=" + lastFlushOffset + " createNewBlock=" + endBlock); } // Flush only if we haven't already flushed till this offset. - if (lastFlushOffset != bytesCurBlock) { - assert bytesCurBlock > lastFlushOffset; + if (lastFlushOffset != streamer.getBytesCurBlock()) { + assert streamer.getBytesCurBlock() > lastFlushOffset; // record the valid offset of this flush - lastFlushOffset = bytesCurBlock; + lastFlushOffset = streamer.getBytesCurBlock(); if (isSync && currentPacket == null && !endBlock) { // Nothing to send right now, // but sync was requested. // Send an empty packet if we do not end the block right now currentPacket = createPacket(packetSize, chunksPerPacket, - bytesCurBlock, currentSeqno++); + streamer.getBytesCurBlock(), streamer.getAndIncCurrentSeqno(), false); } } else { - if (isSync && bytesCurBlock > 0 && !endBlock) { + if (isSync && streamer.getBytesCurBlock() > 0 && !endBlock) { // Nothing to send right now, // and the block was partially written, // and sync was requested. // So send an empty sync packet if we do not end the block right now currentPacket = createPacket(packetSize, chunksPerPacket, - bytesCurBlock, currentSeqno++); + streamer.getBytesCurBlock(), streamer.getAndIncCurrentSeqno(), false); } else if (currentPacket != null) { // just discard the current packet since it is already been sent. currentPacket.releaseBuffer(byteArrayManager); @@ -2070,41 +578,43 @@ private void flushOrSync(boolean isSync, EnumSet syncFlags) } } if (currentPacket != null) { - currentPacket.syncBlock = isSync; - waitAndQueueCurrentPacket(); + currentPacket.setSyncBlock(isSync); + streamer.waitAndQueuePacket(currentPacket); + currentPacket = null; } - if (endBlock && bytesCurBlock > 0) { + if (endBlock && streamer.getBytesCurBlock() > 0) { // Need to end the current block, thus send an empty packet to // indicate this is the end of the block and reset bytesCurBlock - currentPacket = createPacket(0, 0, bytesCurBlock, currentSeqno++); - currentPacket.lastPacketInBlock = true; - currentPacket.syncBlock = shouldSyncBlock || isSync; - waitAndQueueCurrentPacket(); - bytesCurBlock = 0; + currentPacket = createPacket(0, 0, streamer.getBytesCurBlock(), + streamer.getAndIncCurrentSeqno(), true); + currentPacket.setSyncBlock(shouldSyncBlock || isSync); + streamer.waitAndQueuePacket(currentPacket); + currentPacket = null; + streamer.setBytesCurBlock(0); lastFlushOffset = 0; } else { // Restore state of stream. Record the last flush offset // of the last full chunk that was flushed. - bytesCurBlock -= numKept; + streamer.setBytesCurBlock(streamer.getBytesCurBlock() - numKept); } - toWaitFor = lastQueuedSeqno; + toWaitFor = streamer.getLastQueuedSeqno(); } // end synchronized - waitForAckedSeqno(toWaitFor); + streamer.waitForAckedSeqno(toWaitFor); // update the block length first time irrespective of flag - if (updateLength || persistBlocks.get()) { + if (updateLength || streamer.getPersistBlocks().get()) { synchronized (this) { - if (streamer != null && streamer.block != null) { - lastBlockLength = streamer.block.getNumBytes(); + if (!streamer.streamerClosed() && streamer.getBlock() != null) { + lastBlockLength = streamer.getBlock().getNumBytes(); } } } // If 1) any new blocks were allocated since the last flush, or 2) to // update length in NN is required, then persist block locations on // namenode. - if (persistBlocks.getAndSet(false) || updateLength) { + if (streamer.getPersistBlocks().getAndSet(false) || updateLength) { try { dfsClient.namenode.fsync(src, fileId, dfsClient.clientName, lastBlockLength); @@ -2121,7 +631,7 @@ private void flushOrSync(boolean isSync, EnumSet syncFlags) } synchronized(this) { - if (streamer != null) { + if (!streamer.streamerClosed()) { streamer.setHflush(); } } @@ -2134,7 +644,7 @@ private void flushOrSync(boolean isSync, EnumSet syncFlags) DFSClient.LOG.warn("Error while syncing", e); synchronized (this) { if (!isClosed()) { - lastException.set(new IOException("IOException flush: " + e)); + streamer.getLastException().set(new IOException("IOException flush: " + e)); closeThreads(true); } } @@ -2159,7 +669,7 @@ public synchronized int getNumCurrentReplicas() throws IOException { public synchronized int getCurrentBlockReplication() throws IOException { dfsClient.checkOpen(); checkClosed(); - if (streamer == null) { + if (streamer.streamerClosed()) { return blockReplication; // no pipeline, return repl factor of file } DatanodeInfo[] currentNodes = streamer.getNodes(); @@ -2181,42 +691,12 @@ private void flushInternal() throws IOException { // // If there is data in the current buffer, send it across // - queueCurrentPacket(); - toWaitFor = lastQueuedSeqno; + streamer.queuePacket(currentPacket); + currentPacket = null; + toWaitFor = streamer.getLastQueuedSeqno(); } - waitForAckedSeqno(toWaitFor); - } - - private void waitForAckedSeqno(long seqno) throws IOException { - if (DFSClient.LOG.isDebugEnabled()) { - DFSClient.LOG.debug("Waiting for ack for: " + seqno); - } - long begin = Time.monotonicNow(); - try { - synchronized (dataQueue) { - while (!isClosed()) { - checkClosed(); - if (lastAckedSeqno >= seqno) { - break; - } - try { - dataQueue.wait(1000); // when we receive an ack, we notify on - // dataQueue - } catch (InterruptedException ie) { - throw new InterruptedIOException( - "Interrupted while waiting for data to be acknowledged by pipeline"); - } - } - } - checkClosed(); - } catch (ClosedChannelException e) { - } - long duration = Time.monotonicNow() - begin; - if (duration > dfsclientSlowLogThresholdMs) { - DFSClient.LOG.warn("Slow waitForAckedSeqno took " + duration - + "ms (threshold=" + dfsclientSlowLogThresholdMs + "ms)"); - } + streamer.waitForAckedSeqno(toWaitFor); } private synchronized void start() { @@ -2238,22 +718,12 @@ synchronized void abort() throws IOException { } boolean isClosed() { - return closed; + return closed || streamer.streamerClosed(); } void setClosed() { closed = true; - synchronized (dataQueue) { - releaseBuffer(dataQueue, byteArrayManager); - releaseBuffer(ackQueue, byteArrayManager); - } - } - - private static void releaseBuffer(List packets, ByteArrayManager bam) { - for(Packet p : packets) { - p.releaseBuffer(bam); - } - packets.clear(); + streamer.release(); } // shutdown datastreamer and responseprocessor threads. @@ -2262,14 +732,11 @@ private void closeThreads(boolean force) throws IOException { try { streamer.close(force); streamer.join(); - if (s != null) { - s.close(); - } + streamer.closeSocket(); } catch (InterruptedException e) { throw new IOException("Failed to shutdown streamer"); } finally { - streamer = null; - s = null; + streamer.setSocketToNull(); setClosed(); } } @@ -2280,8 +747,18 @@ private void closeThreads(boolean force) throws IOException { */ @Override public synchronized void close() throws IOException { + TraceScope scope = + dfsClient.getPathTraceScope("DFSOutputStream#close", src); + try { + closeImpl(); + } finally { + scope.close(); + } + } + + private synchronized void closeImpl() throws IOException { if (isClosed()) { - IOException e = lastException.getAndSet(null); + IOException e = streamer.getLastException().getAndSet(null); if (e == null) return; else @@ -2292,21 +769,27 @@ public synchronized void close() throws IOException { flushBuffer(); // flush from all upper layers if (currentPacket != null) { - waitAndQueueCurrentPacket(); + streamer.waitAndQueuePacket(currentPacket); + currentPacket = null; } - if (bytesCurBlock != 0) { + if (streamer.getBytesCurBlock() != 0) { // send an empty packet to mark the end of the block - currentPacket = createPacket(0, 0, bytesCurBlock, currentSeqno++); - currentPacket.lastPacketInBlock = true; - currentPacket.syncBlock = shouldSyncBlock; + currentPacket = createPacket(0, 0, streamer.getBytesCurBlock(), + streamer.getAndIncCurrentSeqno(), true); + currentPacket.setSyncBlock(shouldSyncBlock); } flushInternal(); // flush all data to Datanodes // get last block before destroying the streamer ExtendedBlock lastBlock = streamer.getBlock(); closeThreads(false); - completeFile(lastBlock); + TraceScope scope = Trace.startSpan("completeFile", Sampler.NEVER); + try { + completeFile(lastBlock); + } finally { + scope.close(); + } dfsClient.endFileLease(fileId); } catch (ClosedChannelException e) { } finally { @@ -2317,8 +800,9 @@ public synchronized void close() throws IOException { // should be called holding (this) lock since setTestFilename() may // be called during unit tests private void completeFile(ExtendedBlock last) throws IOException { - long localstart = Time.now(); - long localTimeout = 400; + long localstart = Time.monotonicNow(); + long sleeptime = dfsClient.getConf(). + blockWriteLocateFollowingInitialDelayMs; boolean fileComplete = false; int retries = dfsClient.getConf().nBlockWriteLocateFollowingRetry; while (!fileComplete) { @@ -2326,8 +810,9 @@ private void completeFile(ExtendedBlock last) throws IOException { dfsClient.namenode.complete(src, dfsClient.clientName, last, fileId); if (!fileComplete) { final int hdfsTimeout = dfsClient.getHdfsTimeout(); - if (!dfsClient.clientRunning || - (hdfsTimeout > 0 && localstart + hdfsTimeout < Time.now())) { + if (!dfsClient.clientRunning + || (hdfsTimeout > 0 + && localstart + hdfsTimeout < Time.monotonicNow())) { String msg = "Unable to close file because dfsclient " + " was unable to contact the HDFS servers." + " clientRunning " + dfsClient.clientRunning + @@ -2341,9 +826,9 @@ private void completeFile(ExtendedBlock last) throws IOException { + " does not have enough number of replicas."); } retries--; - Thread.sleep(localTimeout); - localTimeout *= 2; - if (Time.now() - localstart > 5000) { + Thread.sleep(sleeptime); + sleeptime *= 2; + if (Time.monotonicNow() - localstart > 5000) { DFSClient.LOG.info("Could not complete " + src + " retrying..."); } } catch (InterruptedException ie) { @@ -2355,7 +840,7 @@ private void completeFile(ExtendedBlock last) throws IOException { @VisibleForTesting public void setArtificialSlowdown(long period) { - artificialSlowdown = period; + streamer.setArtificialSlowdown(period); } @VisibleForTesting @@ -2364,10 +849,6 @@ public synchronized void setChunksPerPacket(int value) { packetSize = (bytesPerChecksum + getChecksumSize()) * chunksPerPacket; } - synchronized void setTestFilename(String newname) { - src = newname; - } - /** * Returns the size of a file as it was when this stream was opened */ @@ -2410,9 +891,4 @@ ExtendedBlock getBlock() { public long getFileId() { return fileId; } - - private static void arraycopy(T[] srcs, T[] dsts, int skipIndex) { - System.arraycopy(srcs, 0, dsts, 0, skipIndex); - System.arraycopy(srcs, skipIndex+1, dsts, skipIndex, dsts.length-skipIndex); - } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java new file mode 100755 index 0000000000000..7e7f7801b70da --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSPacket.java @@ -0,0 +1,343 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs; + +import java.io.DataOutputStream; +import java.io.IOException; +import java.nio.BufferOverflowException; +import java.nio.channels.ClosedChannelException; +import java.util.Arrays; + +import org.apache.hadoop.hdfs.protocol.HdfsConstants; +import org.apache.hadoop.hdfs.protocol.datatransfer.PacketHeader; +import org.apache.hadoop.hdfs.util.ByteArrayManager; +import org.apache.htrace.Span; + +/**************************************************************** + * DFSPacket is used by DataStreamer and DFSOutputStream. + * DFSOutputStream generates packets and then ask DatStreamer + * to send them to datanodes. + ****************************************************************/ + +class DFSPacket { + public static final long HEART_BEAT_SEQNO = -1L; + private static long[] EMPTY = new long[0]; + private final long seqno; // sequence number of buffer in block + private final long offsetInBlock; // offset in block + private boolean syncBlock; // this packet forces the current block to disk + private int numChunks; // number of chunks currently in packet + private final int maxChunks; // max chunks in packet + private byte[] buf; + private final boolean lastPacketInBlock; // is this the last packet in block? + + /** + * buf is pointed into like follows: + * (C is checksum data, D is payload data) + * + * [_________CCCCCCCCC________________DDDDDDDDDDDDDDDD___] + * ^ ^ ^ ^ + * | checksumPos dataStart dataPos + * checksumStart + * + * Right before sending, we move the checksum data to immediately precede + * the actual data, and then insert the header into the buffer immediately + * preceding the checksum data, so we make sure to keep enough space in + * front of the checksum data to support the largest conceivable header. + */ + private int checksumStart; + private int checksumPos; + private final int dataStart; + private int dataPos; + private long[] traceParents = EMPTY; + private int traceParentsUsed; + private Span span; + + /** + * Create a new packet. + * + * @param buf the buffer storing data and checksums + * @param chunksPerPkt maximum number of chunks per packet. + * @param offsetInBlock offset in bytes into the HDFS block. + * @param seqno the sequence number of this packet + * @param checksumSize the size of checksum + * @param lastPacketInBlock if this is the last packet + */ + DFSPacket(byte[] buf, int chunksPerPkt, long offsetInBlock, long seqno, + int checksumSize, boolean lastPacketInBlock) { + this.lastPacketInBlock = lastPacketInBlock; + this.numChunks = 0; + this.offsetInBlock = offsetInBlock; + this.seqno = seqno; + + this.buf = buf; + + checksumStart = PacketHeader.PKT_MAX_HEADER_LEN; + checksumPos = checksumStart; + dataStart = checksumStart + (chunksPerPkt * checksumSize); + dataPos = dataStart; + maxChunks = chunksPerPkt; + } + + /** + * Write data to this packet. + * + * @param inarray input array of data + * @param off the offset of data to write + * @param len the length of data to write + * @throws ClosedChannelException + */ + synchronized void writeData(byte[] inarray, int off, int len) + throws ClosedChannelException { + checkBuffer(); + if (dataPos + len > buf.length) { + throw new BufferOverflowException(); + } + System.arraycopy(inarray, off, buf, dataPos, len); + dataPos += len; + } + + /** + * Write checksums to this packet + * + * @param inarray input array of checksums + * @param off the offset of checksums to write + * @param len the length of checksums to write + * @throws ClosedChannelException + */ + synchronized void writeChecksum(byte[] inarray, int off, int len) + throws ClosedChannelException { + checkBuffer(); + if (len == 0) { + return; + } + if (checksumPos + len > dataStart) { + throw new BufferOverflowException(); + } + System.arraycopy(inarray, off, buf, checksumPos, len); + checksumPos += len; + } + + /** + * Write the full packet, including the header, to the given output stream. + * + * @param stm + * @throws IOException + */ + synchronized void writeTo(DataOutputStream stm) throws IOException { + checkBuffer(); + + final int dataLen = dataPos - dataStart; + final int checksumLen = checksumPos - checksumStart; + final int pktLen = HdfsConstants.BYTES_IN_INTEGER + dataLen + checksumLen; + + PacketHeader header = new PacketHeader( + pktLen, offsetInBlock, seqno, lastPacketInBlock, dataLen, syncBlock); + + if (checksumPos != dataStart) { + // Move the checksum to cover the gap. This can happen for the last + // packet or during an hflush/hsync call. + System.arraycopy(buf, checksumStart, buf, + dataStart - checksumLen , checksumLen); + checksumPos = dataStart; + checksumStart = checksumPos - checksumLen; + } + + final int headerStart = checksumStart - header.getSerializedSize(); + assert checksumStart + 1 >= header.getSerializedSize(); + assert headerStart >= 0; + assert headerStart + header.getSerializedSize() == checksumStart; + + // Copy the header data into the buffer immediately preceding the checksum + // data. + System.arraycopy(header.getBytes(), 0, buf, headerStart, + header.getSerializedSize()); + + // corrupt the data for testing. + if (DFSClientFaultInjector.get().corruptPacket()) { + buf[headerStart+header.getSerializedSize() + checksumLen + dataLen-1] ^= 0xff; + } + + // Write the now contiguous full packet to the output stream. + stm.write(buf, headerStart, header.getSerializedSize() + checksumLen + dataLen); + + // undo corruption. + if (DFSClientFaultInjector.get().uncorruptPacket()) { + buf[headerStart+header.getSerializedSize() + checksumLen + dataLen-1] ^= 0xff; + } + } + + private synchronized void checkBuffer() throws ClosedChannelException { + if (buf == null) { + throw new ClosedChannelException(); + } + } + + /** + * Release the buffer in this packet to ByteArrayManager. + * + * @param bam + */ + synchronized void releaseBuffer(ByteArrayManager bam) { + bam.release(buf); + buf = null; + } + + /** + * get the packet's last byte's offset in the block + * + * @return the packet's last byte's offset in the block + */ + synchronized long getLastByteOffsetBlock() { + return offsetInBlock + dataPos - dataStart; + } + + /** + * Check if this packet is a heart beat packet + * + * @return true if the sequence number is HEART_BEAT_SEQNO + */ + boolean isHeartbeatPacket() { + return seqno == HEART_BEAT_SEQNO; + } + + /** + * check if this packet is the last packet in block + * + * @return true if the packet is the last packet + */ + boolean isLastPacketInBlock(){ + return lastPacketInBlock; + } + + /** + * get sequence number of this packet + * + * @return the sequence number of this packet + */ + long getSeqno(){ + return seqno; + } + + /** + * get the number of chunks this packet contains + * + * @return the number of chunks in this packet + */ + synchronized int getNumChunks(){ + return numChunks; + } + + /** + * increase the number of chunks by one + */ + synchronized void incNumChunks(){ + numChunks++; + } + + /** + * get the maximum number of packets + * + * @return the maximum number of packets + */ + int getMaxChunks(){ + return maxChunks; + } + + /** + * set if to sync block + * + * @param syncBlock if to sync block + */ + synchronized void setSyncBlock(boolean syncBlock){ + this.syncBlock = syncBlock; + } + + @Override + public String toString() { + return "packet seqno: " + this.seqno + + " offsetInBlock: " + this.offsetInBlock + + " lastPacketInBlock: " + this.lastPacketInBlock + + " lastByteOffsetInBlock: " + this.getLastByteOffsetBlock(); + } + + /** + * Add a trace parent span for this packet.

+ * + * Trace parent spans for a packet are the trace spans responsible for + * adding data to that packet. We store them as an array of longs for + * efficiency.

+ * + * Protected by the DFSOutputStream dataQueue lock. + */ + public void addTraceParent(Span span) { + if (span == null) { + return; + } + addTraceParent(span.getSpanId()); + } + + public void addTraceParent(long id) { + if (traceParentsUsed == traceParents.length) { + int newLength = (traceParents.length == 0) ? 8 : + traceParents.length * 2; + traceParents = Arrays.copyOf(traceParents, newLength); + } + traceParents[traceParentsUsed] = id; + traceParentsUsed++; + } + + /** + * Get the trace parent spans for this packet.

+ * + * Will always be non-null.

+ * + * Protected by the DFSOutputStream dataQueue lock. + */ + public long[] getTraceParents() { + // Remove duplicates from the array. + int len = traceParentsUsed; + Arrays.sort(traceParents, 0, len); + int i = 0, j = 0; + long prevVal = 0; // 0 is not a valid span id + while (true) { + if (i == len) { + break; + } + long val = traceParents[i]; + if (val != prevVal) { + traceParents[j] = val; + j++; + prevVal = val; + } + i++; + } + if (j < traceParents.length) { + traceParents = Arrays.copyOf(traceParents, j); + traceParentsUsed = traceParents.length; + } + return traceParents; + } + + public void setTraceSpan(Span span) { + this.span = span; + } + + public Span getTraceSpan() { + return span; + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java new file mode 100644 index 0000000000000..9c437babe8c3c --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java @@ -0,0 +1,1754 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs; + +import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status.SUCCESS; + +import java.io.BufferedOutputStream; +import java.io.DataInputStream; +import java.io.DataOutputStream; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.io.InterruptedIOException; +import java.io.OutputStream; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.net.Socket; +import java.nio.channels.ClosedChannelException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import org.apache.hadoop.fs.StorageType; +import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy; +import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException; +import org.apache.hadoop.hdfs.protocol.DatanodeInfo; +import org.apache.hadoop.hdfs.protocol.ExtendedBlock; +import org.apache.hadoop.hdfs.protocol.HdfsConstants; +import org.apache.hadoop.hdfs.protocol.HdfsFileStatus; +import org.apache.hadoop.hdfs.protocol.LocatedBlock; +import org.apache.hadoop.hdfs.protocol.NSQuotaExceededException; +import org.apache.hadoop.hdfs.protocol.QuotaExceededException; +import org.apache.hadoop.hdfs.protocol.UnresolvedPathException; +import org.apache.hadoop.hdfs.protocol.datatransfer.BlockConstructionStage; +import org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil; +import org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtocol; +import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair; +import org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException; +import org.apache.hadoop.hdfs.protocol.datatransfer.PacketHeader; +import org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck; +import org.apache.hadoop.hdfs.protocol.datatransfer.Sender; +import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.BlockOpResponseProto; +import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status; +import org.apache.hadoop.hdfs.protocolPB.PBHelper; +import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier; +import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite; +import org.apache.hadoop.hdfs.server.datanode.CachingStrategy; +import org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException; +import org.apache.hadoop.hdfs.util.ByteArrayManager; +import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.ipc.RemoteException; +import org.apache.hadoop.net.NetUtils; +import org.apache.hadoop.security.AccessControlException; +import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.util.Daemon; +import org.apache.hadoop.util.DataChecksum; +import org.apache.hadoop.util.Progressable; +import org.apache.hadoop.util.Time; +import org.apache.htrace.NullScope; +import org.apache.htrace.Sampler; +import org.apache.htrace.Span; +import org.apache.htrace.Trace; +import org.apache.htrace.TraceInfo; +import org.apache.htrace.TraceScope; +import com.google.common.cache.CacheBuilder; +import com.google.common.cache.CacheLoader; +import com.google.common.cache.LoadingCache; +import com.google.common.cache.RemovalListener; +import com.google.common.cache.RemovalNotification; + +/********************************************************************* + * + * The DataStreamer class is responsible for sending data packets to the + * datanodes in the pipeline. It retrieves a new blockid and block locations + * from the namenode, and starts streaming packets to the pipeline of + * Datanodes. Every packet has a sequence number associated with + * it. When all the packets for a block are sent out and acks for each + * if them are received, the DataStreamer closes the current block. + * + * The DataStreamer thread picks up packets from the dataQueue, sends it to + * the first datanode in the pipeline and moves it from the dataQueue to the + * ackQueue. The ResponseProcessor receives acks from the datanodes. When an + * successful ack for a packet is received from all datanodes, the + * ResponseProcessor removes the corresponding packet from the ackQueue. + * + * In case of error, all outstanding packets are moved from ackQueue. A new + * pipeline is setup by eliminating the bad datanode from the original + * pipeline. The DataStreamer now starts sending packets from the dataQueue. + * + *********************************************************************/ + +class DataStreamer extends Daemon { + /** + * Create a socket for a write pipeline + * + * @param first the first datanode + * @param length the pipeline length + * @param client client + * @return the socket connected to the first datanode + */ + static Socket createSocketForPipeline(final DatanodeInfo first, + final int length, final DFSClient client) throws IOException { + final String dnAddr = first.getXferAddr( + client.getConf().connectToDnViaHostname); + if (DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Connecting to datanode " + dnAddr); + } + final InetSocketAddress isa = NetUtils.createSocketAddr(dnAddr); + final Socket sock = client.socketFactory.createSocket(); + final int timeout = client.getDatanodeReadTimeout(length); + NetUtils.connect(sock, isa, client.getRandomLocalInterfaceAddr(), client.getConf().socketTimeout); + sock.setSoTimeout(timeout); + sock.setSendBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE); + if(DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Send buf size " + sock.getSendBufferSize()); + } + return sock; + } + + /** + * if this file is lazy persist + * + * @param stat the HdfsFileStatus of a file + * @return if this file is lazy persist + */ + static boolean isLazyPersist(HdfsFileStatus stat) { + final BlockStoragePolicy p = blockStoragePolicySuite.getPolicy( + HdfsConstants.MEMORY_STORAGE_POLICY_NAME); + return p != null && stat.getStoragePolicy() == p.getId(); + } + + /** + * release a list of packets to ByteArrayManager + * + * @param packets packets to be release + * @param bam ByteArrayManager + */ + private static void releaseBuffer(List packets, ByteArrayManager bam) { + for(DFSPacket p : packets) { + p.releaseBuffer(bam); + } + packets.clear(); + } + + private volatile boolean streamerClosed = false; + private ExtendedBlock block; // its length is number of bytes acked + private Token accessToken; + private DataOutputStream blockStream; + private DataInputStream blockReplyStream; + private ResponseProcessor response = null; + private volatile DatanodeInfo[] nodes = null; // list of targets for current block + private volatile StorageType[] storageTypes = null; + private volatile String[] storageIDs = null; + private String[] favoredNodes; + volatile boolean hasError = false; + volatile int errorIndex = -1; + // Restarting node index + AtomicInteger restartingNodeIndex = new AtomicInteger(-1); + private long restartDeadline = 0; // Deadline of DN restart + private BlockConstructionStage stage; // block construction stage + private long bytesSent = 0; // number of bytes that've been sent + private final boolean isLazyPersistFile; + + /** Nodes have been used in the pipeline before and have failed. */ + private final List failed = new ArrayList<>(); + /** The last ack sequence number before pipeline failure. */ + private long lastAckedSeqnoBeforeFailure = -1; + private int pipelineRecoveryCount = 0; + /** Has the current block been hflushed? */ + private boolean isHflushed = false; + /** Append on an existing block? */ + private boolean isAppend; + + private long currentSeqno = 0; + private long lastQueuedSeqno = -1; + private long lastAckedSeqno = -1; + private long bytesCurBlock = 0; // bytes written in current block + private final AtomicReference lastException = new AtomicReference<>(); + private Socket s; + + private final DFSClient dfsClient; + private final String src; + /** Only for DataTransferProtocol.writeBlock(..) */ + private final DataChecksum checksum4WriteBlock; + private final Progressable progress; + private final HdfsFileStatus stat; + // appending to existing partial block + private volatile boolean appendChunk = false; + // both dataQueue and ackQueue are protected by dataQueue lock + private final LinkedList dataQueue = new LinkedList<>(); + private final LinkedList ackQueue = new LinkedList<>(); + private final AtomicReference cachingStrategy; + private final ByteArrayManager byteArrayManager; + private static final BlockStoragePolicySuite blockStoragePolicySuite = + BlockStoragePolicySuite.createDefaultSuite(); + //persist blocks on namenode + private final AtomicBoolean persistBlocks = new AtomicBoolean(false); + private boolean failPacket = false; + private final long dfsclientSlowLogThresholdMs; + private long artificialSlowdown = 0; + + private final LoadingCache excludedNodes; + + private DataStreamer(HdfsFileStatus stat, DFSClient dfsClient, String src, + Progressable progress, DataChecksum checksum, + AtomicReference cachingStrategy, + ByteArrayManager byteArrayManage){ + this.dfsClient = dfsClient; + this.src = src; + this.progress = progress; + this.stat = stat; + this.checksum4WriteBlock = checksum; + this.cachingStrategy = cachingStrategy; + this.byteArrayManager = byteArrayManage; + isLazyPersistFile = isLazyPersist(stat); + this.dfsclientSlowLogThresholdMs = + dfsClient.getConf().dfsclientSlowIoWarningThresholdMs; + excludedNodes = initExcludedNodes(); + } + + /** + * construction with tracing info + */ + DataStreamer(HdfsFileStatus stat, ExtendedBlock block, DFSClient dfsClient, + String src, Progressable progress, DataChecksum checksum, + AtomicReference cachingStrategy, + ByteArrayManager byteArrayManage) { + this(stat, dfsClient, src, progress, checksum, cachingStrategy, + byteArrayManage); + isAppend = false; + this.block = block; + stage = BlockConstructionStage.PIPELINE_SETUP_CREATE; + } + + /** + * Construct a data streamer for appending to the last partial block + * @param lastBlock last block of the file to be appended + * @param stat status of the file to be appended + * @throws IOException if error occurs + */ + DataStreamer(LocatedBlock lastBlock, HdfsFileStatus stat, DFSClient dfsClient, + String src, Progressable progress, DataChecksum checksum, + AtomicReference cachingStrategy, + ByteArrayManager byteArrayManage) throws IOException { + this(stat, dfsClient, src, progress, checksum, cachingStrategy, + byteArrayManage); + isAppend = true; + stage = BlockConstructionStage.PIPELINE_SETUP_APPEND; + block = lastBlock.getBlock(); + bytesSent = block.getNumBytes(); + accessToken = lastBlock.getBlockToken(); + } + + /** + * Set pipeline in construction + * + * @param lastBlock the last block of a file + * @throws IOException + */ + void setPipelineInConstruction(LocatedBlock lastBlock) throws IOException{ + // setup pipeline to append to the last block XXX retries?? + setPipeline(lastBlock); + errorIndex = -1; // no errors yet. + if (nodes.length < 1) { + throw new IOException("Unable to retrieve blocks locations " + + " for last block " + block + + "of file " + src); + } + } + + private void setPipeline(LocatedBlock lb) { + setPipeline(lb.getLocations(), lb.getStorageTypes(), lb.getStorageIDs()); + } + + private void setPipeline(DatanodeInfo[] nodes, StorageType[] storageTypes, + String[] storageIDs) { + this.nodes = nodes; + this.storageTypes = storageTypes; + this.storageIDs = storageIDs; + } + + /** + * Set favored nodes + * + * @param favoredNodes favored nodes + */ + void setFavoredNodes(String[] favoredNodes) { + this.favoredNodes = favoredNodes; + } + + /** + * Initialize for data streaming + */ + private void initDataStreaming() { + this.setName("DataStreamer for file " + src + + " block " + block); + response = new ResponseProcessor(nodes); + response.start(); + stage = BlockConstructionStage.DATA_STREAMING; + } + + private void endBlock() { + if(DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Closing old block " + block); + } + this.setName("DataStreamer for file " + src); + closeResponder(); + closeStream(); + setPipeline(null, null, null); + stage = BlockConstructionStage.PIPELINE_SETUP_CREATE; + } + + /* + * streamer thread is the only thread that opens streams to datanode, + * and closes them. Any error recovery is also done by this thread. + */ + @Override + public void run() { + long lastPacket = Time.monotonicNow(); + TraceScope scope = NullScope.INSTANCE; + while (!streamerClosed && dfsClient.clientRunning) { + // if the Responder encountered an error, shutdown Responder + if (hasError && response != null) { + try { + response.close(); + response.join(); + response = null; + } catch (InterruptedException e) { + DFSClient.LOG.warn("Caught exception ", e); + } + } + + DFSPacket one; + try { + // process datanode IO errors if any + boolean doSleep = false; + if (hasError && (errorIndex >= 0 || restartingNodeIndex.get() >= 0)) { + doSleep = processDatanodeError(); + } + + synchronized (dataQueue) { + // wait for a packet to be sent. + long now = Time.monotonicNow(); + while ((!streamerClosed && !hasError && dfsClient.clientRunning + && dataQueue.size() == 0 && + (stage != BlockConstructionStage.DATA_STREAMING || + stage == BlockConstructionStage.DATA_STREAMING && + now - lastPacket < dfsClient.getConf().socketTimeout/2)) || doSleep ) { + long timeout = dfsClient.getConf().socketTimeout/2 - (now-lastPacket); + timeout = timeout <= 0 ? 1000 : timeout; + timeout = (stage == BlockConstructionStage.DATA_STREAMING)? + timeout : 1000; + try { + dataQueue.wait(timeout); + } catch (InterruptedException e) { + DFSClient.LOG.warn("Caught exception ", e); + } + doSleep = false; + now = Time.monotonicNow(); + } + if (streamerClosed || hasError || !dfsClient.clientRunning) { + continue; + } + // get packet to be sent. + if (dataQueue.isEmpty()) { + one = createHeartbeatPacket(); + assert one != null; + } else { + one = dataQueue.getFirst(); // regular data packet + long parents[] = one.getTraceParents(); + if (parents.length > 0) { + scope = Trace.startSpan("dataStreamer", new TraceInfo(0, parents[0])); + // TODO: use setParents API once it's available from HTrace 3.2 + // scope = Trace.startSpan("dataStreamer", Sampler.ALWAYS); + // scope.getSpan().setParents(parents); + } + } + } + + // get new block from namenode. + if (stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) { + if(DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Allocating new block"); + } + setPipeline(nextBlockOutputStream()); + initDataStreaming(); + } else if (stage == BlockConstructionStage.PIPELINE_SETUP_APPEND) { + if(DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Append to block " + block); + } + setupPipelineForAppendOrRecovery(); + initDataStreaming(); + } + + long lastByteOffsetInBlock = one.getLastByteOffsetBlock(); + if (lastByteOffsetInBlock > stat.getBlockSize()) { + throw new IOException("BlockSize " + stat.getBlockSize() + + " is smaller than data size. " + + " Offset of packet in block " + + lastByteOffsetInBlock + + " Aborting file " + src); + } + + if (one.isLastPacketInBlock()) { + // wait for all data packets have been successfully acked + synchronized (dataQueue) { + while (!streamerClosed && !hasError && + ackQueue.size() != 0 && dfsClient.clientRunning) { + try { + // wait for acks to arrive from datanodes + dataQueue.wait(1000); + } catch (InterruptedException e) { + DFSClient.LOG.warn("Caught exception ", e); + } + } + } + if (streamerClosed || hasError || !dfsClient.clientRunning) { + continue; + } + stage = BlockConstructionStage.PIPELINE_CLOSE; + } + + // send the packet + Span span = null; + synchronized (dataQueue) { + // move packet from dataQueue to ackQueue + if (!one.isHeartbeatPacket()) { + span = scope.detach(); + one.setTraceSpan(span); + dataQueue.removeFirst(); + ackQueue.addLast(one); + dataQueue.notifyAll(); + } + } + + if (DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("DataStreamer block " + block + + " sending packet " + one); + } + + // write out data to remote datanode + TraceScope writeScope = Trace.startSpan("writeTo", span); + try { + one.writeTo(blockStream); + blockStream.flush(); + } catch (IOException e) { + // HDFS-3398 treat primary DN is down since client is unable to + // write to primary DN. If a failed or restarting node has already + // been recorded by the responder, the following call will have no + // effect. Pipeline recovery can handle only one node error at a + // time. If the primary node fails again during the recovery, it + // will be taken out then. + tryMarkPrimaryDatanodeFailed(); + throw e; + } finally { + writeScope.close(); + } + lastPacket = Time.monotonicNow(); + + // update bytesSent + long tmpBytesSent = one.getLastByteOffsetBlock(); + if (bytesSent < tmpBytesSent) { + bytesSent = tmpBytesSent; + } + + if (streamerClosed || hasError || !dfsClient.clientRunning) { + continue; + } + + // Is this block full? + if (one.isLastPacketInBlock()) { + // wait for the close packet has been acked + synchronized (dataQueue) { + while (!streamerClosed && !hasError && + ackQueue.size() != 0 && dfsClient.clientRunning) { + dataQueue.wait(1000);// wait for acks to arrive from datanodes + } + } + if (streamerClosed || hasError || !dfsClient.clientRunning) { + continue; + } + + endBlock(); + } + if (progress != null) { progress.progress(); } + + // This is used by unit test to trigger race conditions. + if (artificialSlowdown != 0 && dfsClient.clientRunning) { + Thread.sleep(artificialSlowdown); + } + } catch (Throwable e) { + // Log warning if there was a real error. + if (restartingNodeIndex.get() == -1) { + // Since their messages are descriptive enough, do not always + // log a verbose stack-trace WARN for quota exceptions. + if (e instanceof QuotaExceededException) { + DFSClient.LOG.debug("DataStreamer Quota Exception", e); + } else { + DFSClient.LOG.warn("DataStreamer Exception", e); + } + } + if (e instanceof IOException) { + setLastException((IOException)e); + } else { + setLastException(new IOException("DataStreamer Exception: ",e)); + } + hasError = true; + if (errorIndex == -1 && restartingNodeIndex.get() == -1) { + // Not a datanode issue + streamerClosed = true; + } + } finally { + scope.close(); + } + } + closeInternal(); + } + + private void closeInternal() { + closeResponder(); // close and join + closeStream(); + streamerClosed = true; + release(); + synchronized (dataQueue) { + dataQueue.notifyAll(); + } + } + + /** + * release the DFSPackets in the two queues + * + */ + void release() { + synchronized (dataQueue) { + releaseBuffer(dataQueue, byteArrayManager); + releaseBuffer(ackQueue, byteArrayManager); + } + } + + /** + * wait for the ack of seqno + * + * @param seqno the sequence number to be acked + * @throws IOException + */ + void waitForAckedSeqno(long seqno) throws IOException { + TraceScope scope = Trace.startSpan("waitForAckedSeqno", Sampler.NEVER); + try { + if (DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Waiting for ack for: " + seqno); + } + long begin = Time.monotonicNow(); + try { + synchronized (dataQueue) { + while (!streamerClosed) { + checkClosed(); + if (lastAckedSeqno >= seqno) { + break; + } + try { + dataQueue.wait(1000); // when we receive an ack, we notify on + // dataQueue + } catch (InterruptedException ie) { + throw new InterruptedIOException( + "Interrupted while waiting for data to be acknowledged by pipeline"); + } + } + } + checkClosed(); + } catch (ClosedChannelException e) { + } + long duration = Time.monotonicNow() - begin; + if (duration > dfsclientSlowLogThresholdMs) { + DFSClient.LOG.warn("Slow waitForAckedSeqno took " + duration + + "ms (threshold=" + dfsclientSlowLogThresholdMs + "ms)"); + } + } finally { + scope.close(); + } + } + + /** + * wait for space of dataQueue and queue the packet + * + * @param packet the DFSPacket to be queued + * @throws IOException + */ + void waitAndQueuePacket(DFSPacket packet) throws IOException { + synchronized (dataQueue) { + try { + // If queue is full, then wait till we have enough space + boolean firstWait = true; + try { + while (!streamerClosed && dataQueue.size() + ackQueue.size() > + dfsClient.getConf().writeMaxPackets) { + if (firstWait) { + Span span = Trace.currentSpan(); + if (span != null) { + span.addTimelineAnnotation("dataQueue.wait"); + } + firstWait = false; + } + try { + dataQueue.wait(); + } catch (InterruptedException e) { + // If we get interrupted while waiting to queue data, we still need to get rid + // of the current packet. This is because we have an invariant that if + // currentPacket gets full, it will get queued before the next writeChunk. + // + // Rather than wait around for space in the queue, we should instead try to + // return to the caller as soon as possible, even though we slightly overrun + // the MAX_PACKETS length. + Thread.currentThread().interrupt(); + break; + } + } + } finally { + Span span = Trace.currentSpan(); + if ((span != null) && (!firstWait)) { + span.addTimelineAnnotation("end.wait"); + } + } + checkClosed(); + queuePacket(packet); + } catch (ClosedChannelException e) { + } + } + } + + /* + * close the streamer, should be called only by an external thread + * and only after all data to be sent has been flushed to datanode. + * + * Interrupt this data streamer if force is true + * + * @param force if this data stream is forced to be closed + */ + void close(boolean force) { + streamerClosed = true; + synchronized (dataQueue) { + dataQueue.notifyAll(); + } + if (force) { + this.interrupt(); + } + } + + + private void checkClosed() throws IOException { + if (streamerClosed) { + IOException e = lastException.get(); + throw e != null ? e : new ClosedChannelException(); + } + } + + private void closeResponder() { + if (response != null) { + try { + response.close(); + response.join(); + } catch (InterruptedException e) { + DFSClient.LOG.warn("Caught exception ", e); + } finally { + response = null; + } + } + } + + private void closeStream() { + if (blockStream != null) { + try { + blockStream.close(); + } catch (IOException e) { + setLastException(e); + } finally { + blockStream = null; + } + } + if (blockReplyStream != null) { + try { + blockReplyStream.close(); + } catch (IOException e) { + setLastException(e); + } finally { + blockReplyStream = null; + } + } + if (null != s) { + try { + s.close(); + } catch (IOException e) { + setLastException(e); + } finally { + s = null; + } + } + } + + // The following synchronized methods are used whenever + // errorIndex or restartingNodeIndex is set. This is because + // check & set needs to be atomic. Simply reading variables + // does not require a synchronization. When responder is + // not running (e.g. during pipeline recovery), there is no + // need to use these methods. + + /** Set the error node index. Called by responder */ + synchronized void setErrorIndex(int idx) { + errorIndex = idx; + } + + /** Set the restarting node index. Called by responder */ + synchronized void setRestartingNodeIndex(int idx) { + restartingNodeIndex.set(idx); + // If the data streamer has already set the primary node + // bad, clear it. It is likely that the write failed due to + // the DN shutdown. Even if it was a real failure, the pipeline + // recovery will take care of it. + errorIndex = -1; + } + + /** + * This method is used when no explicit error report was received, + * but something failed. When the primary node is a suspect or + * unsure about the cause, the primary node is marked as failed. + */ + synchronized void tryMarkPrimaryDatanodeFailed() { + // There should be no existing error and no ongoing restart. + if ((errorIndex == -1) && (restartingNodeIndex.get() == -1)) { + errorIndex = 0; + } + } + + /** + * Examine whether it is worth waiting for a node to restart. + * @param index the node index + */ + boolean shouldWaitForRestart(int index) { + // Only one node in the pipeline. + if (nodes.length == 1) { + return true; + } + + // Is it a local node? + InetAddress addr = null; + try { + addr = InetAddress.getByName(nodes[index].getIpAddr()); + } catch (java.net.UnknownHostException e) { + // we are passing an ip address. this should not happen. + assert false; + } + + if (addr != null && NetUtils.isLocalAddress(addr)) { + return true; + } + return false; + } + + // + // Processes responses from the datanodes. A packet is removed + // from the ackQueue when its response arrives. + // + private class ResponseProcessor extends Daemon { + + private volatile boolean responderClosed = false; + private DatanodeInfo[] targets = null; + private boolean isLastPacketInBlock = false; + + ResponseProcessor (DatanodeInfo[] targets) { + this.targets = targets; + } + + @Override + public void run() { + + setName("ResponseProcessor for block " + block); + PipelineAck ack = new PipelineAck(); + + TraceScope scope = NullScope.INSTANCE; + while (!responderClosed && dfsClient.clientRunning && !isLastPacketInBlock) { + // process responses from datanodes. + try { + // read an ack from the pipeline + long begin = Time.monotonicNow(); + ack.readFields(blockReplyStream); + long duration = Time.monotonicNow() - begin; + if (duration > dfsclientSlowLogThresholdMs + && ack.getSeqno() != DFSPacket.HEART_BEAT_SEQNO) { + DFSClient.LOG + .warn("Slow ReadProcessor read fields took " + duration + + "ms (threshold=" + dfsclientSlowLogThresholdMs + "ms); ack: " + + ack + ", targets: " + Arrays.asList(targets)); + } else if (DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("DFSClient " + ack); + } + + long seqno = ack.getSeqno(); + // processes response status from datanodes. + for (int i = ack.getNumOfReplies()-1; i >=0 && dfsClient.clientRunning; i--) { + final Status reply = PipelineAck.getStatusFromHeader(ack + .getHeaderFlag(i)); + // Restart will not be treated differently unless it is + // the local node or the only one in the pipeline. + if (PipelineAck.isRestartOOBStatus(reply) && + shouldWaitForRestart(i)) { + restartDeadline = dfsClient.getConf().datanodeRestartTimeout + + Time.monotonicNow(); + setRestartingNodeIndex(i); + String message = "A datanode is restarting: " + targets[i]; + DFSClient.LOG.info(message); + throw new IOException(message); + } + // node error + if (reply != SUCCESS) { + setErrorIndex(i); // first bad datanode + throw new IOException("Bad response " + reply + + " for block " + block + + " from datanode " + + targets[i]); + } + } + + assert seqno != PipelineAck.UNKOWN_SEQNO : + "Ack for unknown seqno should be a failed ack: " + ack; + if (seqno == DFSPacket.HEART_BEAT_SEQNO) { // a heartbeat ack + continue; + } + + // a success ack for a data packet + DFSPacket one; + synchronized (dataQueue) { + one = ackQueue.getFirst(); + } + if (one.getSeqno() != seqno) { + throw new IOException("ResponseProcessor: Expecting seqno " + + " for block " + block + + one.getSeqno() + " but received " + seqno); + } + isLastPacketInBlock = one.isLastPacketInBlock(); + + // Fail the packet write for testing in order to force a + // pipeline recovery. + if (DFSClientFaultInjector.get().failPacket() && + isLastPacketInBlock) { + failPacket = true; + throw new IOException( + "Failing the last packet for testing."); + } + + // update bytesAcked + block.setNumBytes(one.getLastByteOffsetBlock()); + + synchronized (dataQueue) { + scope = Trace.continueSpan(one.getTraceSpan()); + one.setTraceSpan(null); + lastAckedSeqno = seqno; + ackQueue.removeFirst(); + dataQueue.notifyAll(); + + one.releaseBuffer(byteArrayManager); + } + } catch (Exception e) { + if (!responderClosed) { + if (e instanceof IOException) { + setLastException((IOException)e); + } + hasError = true; + // If no explicit error report was received, mark the primary + // node as failed. + tryMarkPrimaryDatanodeFailed(); + synchronized (dataQueue) { + dataQueue.notifyAll(); + } + if (restartingNodeIndex.get() == -1) { + DFSClient.LOG.warn("DataStreamer ResponseProcessor exception " + + " for block " + block, e); + } + responderClosed = true; + } + } finally { + scope.close(); + } + } + } + + void close() { + responderClosed = true; + this.interrupt(); + } + } + + // If this stream has encountered any errors so far, shutdown + // threads and mark stream as closed. Returns true if we should + // sleep for a while after returning from this call. + // + private boolean processDatanodeError() throws IOException { + if (response != null) { + DFSClient.LOG.info("Error Recovery for " + block + + " waiting for responder to exit. "); + return true; + } + closeStream(); + + // move packets from ack queue to front of the data queue + synchronized (dataQueue) { + dataQueue.addAll(0, ackQueue); + ackQueue.clear(); + } + + // Record the new pipeline failure recovery. + if (lastAckedSeqnoBeforeFailure != lastAckedSeqno) { + lastAckedSeqnoBeforeFailure = lastAckedSeqno; + pipelineRecoveryCount = 1; + } else { + // If we had to recover the pipeline five times in a row for the + // same packet, this client likely has corrupt data or corrupting + // during transmission. + if (++pipelineRecoveryCount > 5) { + DFSClient.LOG.warn("Error recovering pipeline for writing " + + block + ". Already retried 5 times for the same packet."); + lastException.set(new IOException("Failing write. Tried pipeline " + + "recovery 5 times without success.")); + streamerClosed = true; + return false; + } + } + boolean doSleep = setupPipelineForAppendOrRecovery(); + + if (!streamerClosed && dfsClient.clientRunning) { + if (stage == BlockConstructionStage.PIPELINE_CLOSE) { + + // If we had an error while closing the pipeline, we go through a fast-path + // where the BlockReceiver does not run. Instead, the DataNode just finalizes + // the block immediately during the 'connect ack' process. So, we want to pull + // the end-of-block packet from the dataQueue, since we don't actually have + // a true pipeline to send it over. + // + // We also need to set lastAckedSeqno to the end-of-block Packet's seqno, so that + // a client waiting on close() will be aware that the flush finished. + synchronized (dataQueue) { + DFSPacket endOfBlockPacket = dataQueue.remove(); // remove the end of block packet + Span span = endOfBlockPacket.getTraceSpan(); + if (span != null) { + // Close any trace span associated with this Packet + TraceScope scope = Trace.continueSpan(span); + scope.close(); + } + assert endOfBlockPacket.isLastPacketInBlock(); + assert lastAckedSeqno == endOfBlockPacket.getSeqno() - 1; + lastAckedSeqno = endOfBlockPacket.getSeqno(); + dataQueue.notifyAll(); + } + endBlock(); + } else { + initDataStreaming(); + } + } + + return doSleep; + } + + void setHflush() { + isHflushed = true; + } + + private int findNewDatanode(final DatanodeInfo[] original + ) throws IOException { + if (nodes.length != original.length + 1) { + throw new IOException( + new StringBuilder() + .append("Failed to replace a bad datanode on the existing pipeline ") + .append("due to no more good datanodes being available to try. ") + .append("(Nodes: current=").append(Arrays.asList(nodes)) + .append(", original=").append(Arrays.asList(original)).append("). ") + .append("The current failed datanode replacement policy is ") + .append(dfsClient.dtpReplaceDatanodeOnFailure).append(", and ") + .append("a client may configure this via '") + .append(DFSConfigKeys.DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_POLICY_KEY) + .append("' in its configuration.") + .toString()); + } + for(int i = 0; i < nodes.length; i++) { + int j = 0; + for(; j < original.length && !nodes[i].equals(original[j]); j++); + if (j == original.length) { + return i; + } + } + throw new IOException("Failed: new datanode not found: nodes=" + + Arrays.asList(nodes) + ", original=" + Arrays.asList(original)); + } + + private void addDatanode2ExistingPipeline() throws IOException { + if (DataTransferProtocol.LOG.isDebugEnabled()) { + DataTransferProtocol.LOG.debug("lastAckedSeqno = " + lastAckedSeqno); + } + /* + * Is data transfer necessary? We have the following cases. + * + * Case 1: Failure in Pipeline Setup + * - Append + * + Transfer the stored replica, which may be a RBW or a finalized. + * - Create + * + If no data, then no transfer is required. + * + If there are data written, transfer RBW. This case may happens + * when there are streaming failure earlier in this pipeline. + * + * Case 2: Failure in Streaming + * - Append/Create: + * + transfer RBW + * + * Case 3: Failure in Close + * - Append/Create: + * + no transfer, let NameNode replicates the block. + */ + if (!isAppend && lastAckedSeqno < 0 + && stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) { + //no data have been written + return; + } else if (stage == BlockConstructionStage.PIPELINE_CLOSE + || stage == BlockConstructionStage.PIPELINE_CLOSE_RECOVERY) { + //pipeline is closing + return; + } + + //get a new datanode + final DatanodeInfo[] original = nodes; + final LocatedBlock lb = dfsClient.namenode.getAdditionalDatanode( + src, stat.getFileId(), block, nodes, storageIDs, + failed.toArray(new DatanodeInfo[failed.size()]), + 1, dfsClient.clientName); + setPipeline(lb); + + //find the new datanode + final int d = findNewDatanode(original); + + //transfer replica + final DatanodeInfo src = d == 0? nodes[1]: nodes[d - 1]; + final DatanodeInfo[] targets = {nodes[d]}; + final StorageType[] targetStorageTypes = {storageTypes[d]}; + transfer(src, targets, targetStorageTypes, lb.getBlockToken()); + } + + private void transfer(final DatanodeInfo src, final DatanodeInfo[] targets, + final StorageType[] targetStorageTypes, + final Token blockToken) throws IOException { + //transfer replica to the new datanode + Socket sock = null; + DataOutputStream out = null; + DataInputStream in = null; + try { + sock = createSocketForPipeline(src, 2, dfsClient); + final long writeTimeout = dfsClient.getDatanodeWriteTimeout(2); + + OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout); + InputStream unbufIn = NetUtils.getInputStream(sock); + IOStreamPair saslStreams = dfsClient.saslClient.socketSend(sock, + unbufOut, unbufIn, dfsClient, blockToken, src); + unbufOut = saslStreams.out; + unbufIn = saslStreams.in; + out = new DataOutputStream(new BufferedOutputStream(unbufOut, + HdfsConstants.SMALL_BUFFER_SIZE)); + in = new DataInputStream(unbufIn); + + //send the TRANSFER_BLOCK request + new Sender(out).transferBlock(block, blockToken, dfsClient.clientName, + targets, targetStorageTypes); + out.flush(); + + //ack + BlockOpResponseProto response = + BlockOpResponseProto.parseFrom(PBHelper.vintPrefixed(in)); + if (SUCCESS != response.getStatus()) { + throw new IOException("Failed to add a datanode"); + } + } finally { + IOUtils.closeStream(in); + IOUtils.closeStream(out); + IOUtils.closeSocket(sock); + } + } + + /** + * Open a DataStreamer to a DataNode pipeline so that + * it can be written to. + * This happens when a file is appended or data streaming fails + * It keeps on trying until a pipeline is setup + */ + private boolean setupPipelineForAppendOrRecovery() throws IOException { + // check number of datanodes + if (nodes == null || nodes.length == 0) { + String msg = "Could not get block locations. " + "Source file \"" + + src + "\" - Aborting..."; + DFSClient.LOG.warn(msg); + setLastException(new IOException(msg)); + streamerClosed = true; + return false; + } + + boolean success = false; + long newGS = 0L; + while (!success && !streamerClosed && dfsClient.clientRunning) { + // Sleep before reconnect if a dn is restarting. + // This process will be repeated until the deadline or the datanode + // starts back up. + if (restartingNodeIndex.get() >= 0) { + // 4 seconds or the configured deadline period, whichever is shorter. + // This is the retry interval and recovery will be retried in this + // interval until timeout or success. + long delay = Math.min(dfsClient.getConf().datanodeRestartTimeout, + 4000L); + try { + Thread.sleep(delay); + } catch (InterruptedException ie) { + lastException.set(new IOException("Interrupted while waiting for " + + "datanode to restart. " + nodes[restartingNodeIndex.get()])); + streamerClosed = true; + return false; + } + } + boolean isRecovery = hasError; + // remove bad datanode from list of datanodes. + // If errorIndex was not set (i.e. appends), then do not remove + // any datanodes + // + if (errorIndex >= 0) { + StringBuilder pipelineMsg = new StringBuilder(); + for (int j = 0; j < nodes.length; j++) { + pipelineMsg.append(nodes[j]); + if (j < nodes.length - 1) { + pipelineMsg.append(", "); + } + } + if (nodes.length <= 1) { + lastException.set(new IOException("All datanodes " + pipelineMsg + + " are bad. Aborting...")); + streamerClosed = true; + return false; + } + DFSClient.LOG.warn("Error Recovery for block " + block + + " in pipeline " + pipelineMsg + + ": bad datanode " + nodes[errorIndex]); + failed.add(nodes[errorIndex]); + + DatanodeInfo[] newnodes = new DatanodeInfo[nodes.length-1]; + arraycopy(nodes, newnodes, errorIndex); + + final StorageType[] newStorageTypes = new StorageType[newnodes.length]; + arraycopy(storageTypes, newStorageTypes, errorIndex); + + final String[] newStorageIDs = new String[newnodes.length]; + arraycopy(storageIDs, newStorageIDs, errorIndex); + + setPipeline(newnodes, newStorageTypes, newStorageIDs); + + // Just took care of a node error while waiting for a node restart + if (restartingNodeIndex.get() >= 0) { + // If the error came from a node further away than the restarting + // node, the restart must have been complete. + if (errorIndex > restartingNodeIndex.get()) { + restartingNodeIndex.set(-1); + } else if (errorIndex < restartingNodeIndex.get()) { + // the node index has shifted. + restartingNodeIndex.decrementAndGet(); + } else { + // this shouldn't happen... + assert false; + } + } + + if (restartingNodeIndex.get() == -1) { + hasError = false; + } + lastException.set(null); + errorIndex = -1; + } + + // Check if replace-datanode policy is satisfied. + if (dfsClient.dtpReplaceDatanodeOnFailure.satisfy(stat.getReplication(), + nodes, isAppend, isHflushed)) { + try { + addDatanode2ExistingPipeline(); + } catch(IOException ioe) { + if (!dfsClient.dtpReplaceDatanodeOnFailure.isBestEffort()) { + throw ioe; + } + DFSClient.LOG.warn("Failed to replace datanode." + + " Continue with the remaining datanodes since " + + DFSConfigKeys.DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_BEST_EFFORT_KEY + + " is set to true.", ioe); + } + } + + // get a new generation stamp and an access token + LocatedBlock lb = dfsClient.namenode.updateBlockForPipeline(block, dfsClient.clientName); + newGS = lb.getBlock().getGenerationStamp(); + accessToken = lb.getBlockToken(); + + // set up the pipeline again with the remaining nodes + if (failPacket) { // for testing + success = createBlockOutputStream(nodes, storageTypes, newGS, isRecovery); + failPacket = false; + try { + // Give DNs time to send in bad reports. In real situations, + // good reports should follow bad ones, if client committed + // with those nodes. + Thread.sleep(2000); + } catch (InterruptedException ie) {} + } else { + success = createBlockOutputStream(nodes, storageTypes, newGS, isRecovery); + } + + if (restartingNodeIndex.get() >= 0) { + assert hasError == true; + // check errorIndex set above + if (errorIndex == restartingNodeIndex.get()) { + // ignore, if came from the restarting node + errorIndex = -1; + } + // still within the deadline + if (Time.monotonicNow() < restartDeadline) { + continue; // with in the deadline + } + // expired. declare the restarting node dead + restartDeadline = 0; + int expiredNodeIndex = restartingNodeIndex.get(); + restartingNodeIndex.set(-1); + DFSClient.LOG.warn("Datanode did not restart in time: " + + nodes[expiredNodeIndex]); + // Mark the restarting node as failed. If there is any other failed + // node during the last pipeline construction attempt, it will not be + // overwritten/dropped. In this case, the restarting node will get + // excluded in the following attempt, if it still does not come up. + if (errorIndex == -1) { + errorIndex = expiredNodeIndex; + } + // From this point on, normal pipeline recovery applies. + } + } // while + + if (success) { + // update pipeline at the namenode + ExtendedBlock newBlock = new ExtendedBlock( + block.getBlockPoolId(), block.getBlockId(), block.getNumBytes(), newGS); + dfsClient.namenode.updatePipeline(dfsClient.clientName, block, newBlock, + nodes, storageIDs); + // update client side generation stamp + block = newBlock; + } + return false; // do not sleep, continue processing + } + + /** + * Open a DataStreamer to a DataNode so that it can be written to. + * This happens when a file is created and each time a new block is allocated. + * Must get block ID and the IDs of the destinations from the namenode. + * Returns the list of target datanodes. + */ + private LocatedBlock nextBlockOutputStream() throws IOException { + LocatedBlock lb = null; + DatanodeInfo[] nodes = null; + StorageType[] storageTypes = null; + int count = dfsClient.getConf().nBlockWriteRetry; + boolean success = false; + ExtendedBlock oldBlock = block; + do { + hasError = false; + lastException.set(null); + errorIndex = -1; + success = false; + + DatanodeInfo[] excluded = + excludedNodes.getAllPresent(excludedNodes.asMap().keySet()) + .keySet() + .toArray(new DatanodeInfo[0]); + block = oldBlock; + lb = locateFollowingBlock(excluded.length > 0 ? excluded : null); + block = lb.getBlock(); + block.setNumBytes(0); + bytesSent = 0; + accessToken = lb.getBlockToken(); + nodes = lb.getLocations(); + storageTypes = lb.getStorageTypes(); + + // + // Connect to first DataNode in the list. + // + success = createBlockOutputStream(nodes, storageTypes, 0L, false); + + if (!success) { + DFSClient.LOG.info("Abandoning " + block); + dfsClient.namenode.abandonBlock(block, stat.getFileId(), src, + dfsClient.clientName); + block = null; + DFSClient.LOG.info("Excluding datanode " + nodes[errorIndex]); + excludedNodes.put(nodes[errorIndex], nodes[errorIndex]); + } + } while (!success && --count >= 0); + + if (!success) { + throw new IOException("Unable to create new block."); + } + return lb; + } + + // connects to the first datanode in the pipeline + // Returns true if success, otherwise return failure. + // + private boolean createBlockOutputStream(DatanodeInfo[] nodes, + StorageType[] nodeStorageTypes, long newGS, boolean recoveryFlag) { + if (nodes.length == 0) { + DFSClient.LOG.info("nodes are empty for write pipeline of block " + + block); + return false; + } + Status pipelineStatus = SUCCESS; + String firstBadLink = ""; + boolean checkRestart = false; + if (DFSClient.LOG.isDebugEnabled()) { + for (int i = 0; i < nodes.length; i++) { + DFSClient.LOG.debug("pipeline = " + nodes[i]); + } + } + + // persist blocks on namenode on next flush + persistBlocks.set(true); + + int refetchEncryptionKey = 1; + while (true) { + boolean result = false; + DataOutputStream out = null; + try { + assert null == s : "Previous socket unclosed"; + assert null == blockReplyStream : "Previous blockReplyStream unclosed"; + s = createSocketForPipeline(nodes[0], nodes.length, dfsClient); + long writeTimeout = dfsClient.getDatanodeWriteTimeout(nodes.length); + + OutputStream unbufOut = NetUtils.getOutputStream(s, writeTimeout); + InputStream unbufIn = NetUtils.getInputStream(s); + IOStreamPair saslStreams = dfsClient.saslClient.socketSend(s, + unbufOut, unbufIn, dfsClient, accessToken, nodes[0]); + unbufOut = saslStreams.out; + unbufIn = saslStreams.in; + out = new DataOutputStream(new BufferedOutputStream(unbufOut, + HdfsConstants.SMALL_BUFFER_SIZE)); + blockReplyStream = new DataInputStream(unbufIn); + + // + // Xmit header info to datanode + // + + BlockConstructionStage bcs = recoveryFlag? stage.getRecoveryStage(): stage; + + // We cannot change the block length in 'block' as it counts the number + // of bytes ack'ed. + ExtendedBlock blockCopy = new ExtendedBlock(block); + blockCopy.setNumBytes(stat.getBlockSize()); + + boolean[] targetPinnings = getPinnings(nodes, true); + // send the request + new Sender(out).writeBlock(blockCopy, nodeStorageTypes[0], accessToken, + dfsClient.clientName, nodes, nodeStorageTypes, null, bcs, + nodes.length, block.getNumBytes(), bytesSent, newGS, + checksum4WriteBlock, cachingStrategy.get(), isLazyPersistFile, + (targetPinnings == null ? false : targetPinnings[0]), targetPinnings); + + // receive ack for connect + BlockOpResponseProto resp = BlockOpResponseProto.parseFrom( + PBHelper.vintPrefixed(blockReplyStream)); + pipelineStatus = resp.getStatus(); + firstBadLink = resp.getFirstBadLink(); + + // Got an restart OOB ack. + // If a node is already restarting, this status is not likely from + // the same node. If it is from a different node, it is not + // from the local datanode. Thus it is safe to treat this as a + // regular node error. + if (PipelineAck.isRestartOOBStatus(pipelineStatus) && + restartingNodeIndex.get() == -1) { + checkRestart = true; + throw new IOException("A datanode is restarting."); + } + + String logInfo = "ack with firstBadLink as " + firstBadLink; + DataTransferProtoUtil.checkBlockOpStatus(resp, logInfo); + + assert null == blockStream : "Previous blockStream unclosed"; + blockStream = out; + result = true; // success + restartingNodeIndex.set(-1); + hasError = false; + } catch (IOException ie) { + if (restartingNodeIndex.get() == -1) { + DFSClient.LOG.info("Exception in createBlockOutputStream", ie); + } + if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) { + DFSClient.LOG.info("Will fetch a new encryption key and retry, " + + "encryption key was invalid when connecting to " + + nodes[0] + " : " + ie); + // The encryption key used is invalid. + refetchEncryptionKey--; + dfsClient.clearDataEncryptionKey(); + // Don't close the socket/exclude this node just yet. Try again with + // a new encryption key. + continue; + } + + // find the datanode that matches + if (firstBadLink.length() != 0) { + for (int i = 0; i < nodes.length; i++) { + // NB: Unconditionally using the xfer addr w/o hostname + if (firstBadLink.equals(nodes[i].getXferAddr())) { + errorIndex = i; + break; + } + } + } else { + assert checkRestart == false; + errorIndex = 0; + } + // Check whether there is a restart worth waiting for. + if (checkRestart && shouldWaitForRestart(errorIndex)) { + restartDeadline = dfsClient.getConf().datanodeRestartTimeout + + Time.monotonicNow(); + restartingNodeIndex.set(errorIndex); + errorIndex = -1; + DFSClient.LOG.info("Waiting for the datanode to be restarted: " + + nodes[restartingNodeIndex.get()]); + } + hasError = true; + setLastException(ie); + result = false; // error + } finally { + if (!result) { + IOUtils.closeSocket(s); + s = null; + IOUtils.closeStream(out); + out = null; + IOUtils.closeStream(blockReplyStream); + blockReplyStream = null; + } + } + return result; + } + } + + private boolean[] getPinnings(DatanodeInfo[] nodes, boolean shouldLog) { + if (favoredNodes == null) { + return null; + } else { + boolean[] pinnings = new boolean[nodes.length]; + HashSet favoredSet = + new HashSet(Arrays.asList(favoredNodes)); + for (int i = 0; i < nodes.length; i++) { + pinnings[i] = favoredSet.remove(nodes[i].getXferAddrWithHostname()); + if (DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug(nodes[i].getXferAddrWithHostname() + + " was chosen by name node (favored=" + pinnings[i] + + ")."); + } + } + if (shouldLog && !favoredSet.isEmpty()) { + // There is one or more favored nodes that were not allocated. + DFSClient.LOG.warn( + "These favored nodes were specified but not chosen: " + + favoredSet + + " Specified favored nodes: " + Arrays.toString(favoredNodes)); + + } + return pinnings; + } + } + + private LocatedBlock locateFollowingBlock(DatanodeInfo[] excludedNodes) + throws IOException { + int retries = dfsClient.getConf().nBlockWriteLocateFollowingRetry; + long sleeptime = dfsClient.getConf(). + blockWriteLocateFollowingInitialDelayMs; + while (true) { + long localstart = Time.monotonicNow(); + while (true) { + try { + return dfsClient.namenode.addBlock(src, dfsClient.clientName, + block, excludedNodes, stat.getFileId(), favoredNodes); + } catch (RemoteException e) { + IOException ue = + e.unwrapRemoteException(FileNotFoundException.class, + AccessControlException.class, + NSQuotaExceededException.class, + DSQuotaExceededException.class, + UnresolvedPathException.class); + if (ue != e) { + throw ue; // no need to retry these exceptions + } + + + if (NotReplicatedYetException.class.getName(). + equals(e.getClassName())) { + if (retries == 0) { + throw e; + } else { + --retries; + DFSClient.LOG.info("Exception while adding a block", e); + long elapsed = Time.monotonicNow() - localstart; + if (elapsed > 5000) { + DFSClient.LOG.info("Waiting for replication for " + + (elapsed / 1000) + " seconds"); + } + try { + DFSClient.LOG.warn("NotReplicatedYetException sleeping " + src + + " retries left " + retries); + Thread.sleep(sleeptime); + sleeptime *= 2; + } catch (InterruptedException ie) { + DFSClient.LOG.warn("Caught exception ", ie); + } + } + } else { + throw e; + } + + } + } + } + } + + /** + * get the block this streamer is writing to + * + * @return the block this streamer is writing to + */ + ExtendedBlock getBlock() { + return block; + } + + /** + * return the target datanodes in the pipeline + * + * @return the target datanodes in the pipeline + */ + DatanodeInfo[] getNodes() { + return nodes; + } + + /** + * return the token of the block + * + * @return the token of the block + */ + Token getBlockToken() { + return accessToken; + } + + /** + * set last exception + * + * @param e an exception + */ + void setLastException(IOException e) { + lastException.compareAndSet(null, e); + } + + /** + * Put a packet to the data queue + * + * @param packet the packet to be put into the data queued + */ + void queuePacket(DFSPacket packet) { + synchronized (dataQueue) { + if (packet == null) return; + packet.addTraceParent(Trace.currentSpan()); + dataQueue.addLast(packet); + lastQueuedSeqno = packet.getSeqno(); + if (DFSClient.LOG.isDebugEnabled()) { + DFSClient.LOG.debug("Queued packet " + packet.getSeqno()); + } + dataQueue.notifyAll(); + } + } + + /** + * For heartbeat packets, create buffer directly by new byte[] + * since heartbeats should not be blocked. + */ + private DFSPacket createHeartbeatPacket() throws InterruptedIOException { + final byte[] buf = new byte[PacketHeader.PKT_MAX_HEADER_LEN]; + return new DFSPacket(buf, 0, 0, DFSPacket.HEART_BEAT_SEQNO, 0, false); + } + + private LoadingCache initExcludedNodes() { + return CacheBuilder.newBuilder().expireAfterWrite( + dfsClient.getConf().excludedNodesCacheExpiry, TimeUnit.MILLISECONDS) + .removalListener(new RemovalListener() { + @Override + public void onRemoval( + RemovalNotification notification) { + DFSClient.LOG.info("Removing node " + notification.getKey() + + " from the excluded nodes list"); + } + }).build(new CacheLoader() { + @Override + public DatanodeInfo load(DatanodeInfo key) throws Exception { + return key; + } + }); + } + + private static void arraycopy(T[] srcs, T[] dsts, int skipIndex) { + System.arraycopy(srcs, 0, dsts, 0, skipIndex); + System.arraycopy(srcs, skipIndex+1, dsts, skipIndex, dsts.length-skipIndex); + } + + /** + * check if to persist blocks on namenode + * + * @return if to persist blocks on namenode + */ + AtomicBoolean getPersistBlocks(){ + return persistBlocks; + } + + /** + * check if to append a chunk + * + * @param appendChunk if to append a chunk + */ + void setAppendChunk(boolean appendChunk){ + this.appendChunk = appendChunk; + } + + /** + * get if to append a chunk + * + * @return if to append a chunk + */ + boolean getAppendChunk(){ + return appendChunk; + } + + /** + * get the last exception + * + * @return the last exception + */ + AtomicReference getLastException(){ + return lastException; + } + + /** + * get the socket connecting to the first datanode in pipeline + * + * @return socket connecting to the first datanode in pipeline + */ + Socket getSocket() { + return s; + } + + /** + * set socket to null + */ + void setSocketToNull() { + this.s = null; + } + + /** + * return current sequence number and then increase it by 1 + * + * @return current sequence number before increasing + */ + long getAndIncCurrentSeqno() { + long old = this.currentSeqno; + this.currentSeqno++; + return old; + } + + /** + * get last queued sequence number + * + * @return last queued sequence number + */ + long getLastQueuedSeqno() { + return lastQueuedSeqno; + } + + /** + * get the number of bytes of current block + * + * @return the number of bytes of current block + */ + long getBytesCurBlock() { + return bytesCurBlock; + } + + /** + * set the bytes of current block that have been written + * + * @param bytesCurBlock bytes of current block that have been written + */ + void setBytesCurBlock(long bytesCurBlock) { + this.bytesCurBlock = bytesCurBlock; + } + + /** + * increase bytes of current block by len. + * + * @param len how many bytes to increase to current block + */ + void incBytesCurBlock(long len) { + this.bytesCurBlock += len; + } + + /** + * set artificial slow down for unit test + * + * @param period artificial slow down + */ + void setArtificialSlowdown(long period) { + this.artificialSlowdown = period; + } + + /** + * if this streamer is to terminate + * + * @return if this streamer is to terminate + */ + boolean streamerClosed(){ + return streamerClosed; + } + + void closeSocket() throws IOException { + if (s != null) { + s.close(); + } + } +} \ No newline at end of file diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java index e6bbd924a5773..432e4efa17d80 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java @@ -68,6 +68,7 @@ import org.apache.hadoop.hdfs.protocol.CacheDirectiveInfo; import org.apache.hadoop.hdfs.protocol.CachePoolEntry; import org.apache.hadoop.hdfs.protocol.CachePoolInfo; +import org.apache.hadoop.hdfs.protocol.ClientProtocol; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.DirectoryListing; import org.apache.hadoop.hdfs.protocol.EncryptionZone; @@ -1181,13 +1182,24 @@ public boolean setSafeMode(HdfsConstants.SafeModeAction action, /** * Save namespace image. - * - * @see org.apache.hadoop.hdfs.protocol.ClientProtocol#saveNamespace() + * + * @param timeWindow NameNode can ignore this command if the latest + * checkpoint was done within the given time period (in + * seconds). + * @return true if a new checkpoint has been made + * @see ClientProtocol#saveNamespace(long, long) */ - public void saveNamespace() throws AccessControlException, IOException { - dfs.saveNamespace(); + public boolean saveNamespace(long timeWindow, long txGap) throws IOException { + return dfs.saveNamespace(timeWindow, txGap); } - + + /** + * Save namespace image. NameNode always does the checkpoint. + */ + public void saveNamespace() throws IOException { + saveNamespace(0, 0); + } + /** * Rolls the edit log on the active NameNode. * Requires super-user privileges. @@ -1226,7 +1238,7 @@ public void finalizeUpgrade() throws IOException { } /** - * Rolling upgrade: start/finalize/query. + * Rolling upgrade: prepare/finalize/query. */ public RollingUpgradeInfo rollingUpgrade(RollingUpgradeAction action) throws IOException { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java index 8f2966ac47bcb..29a26678cbf42 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java @@ -139,7 +139,7 @@ private static void addDeprecatedKeys() { new DeprecationDelta("dfs.federation.nameservice.id", DFSConfigKeys.DFS_NAMESERVICE_ID), new DeprecationDelta("dfs.client.file-block-storage-locations.timeout", - DFSConfigKeys.DFS_CLIENT_FILE_BLOCK_STORAGE_LOCATIONS_TIMEOUT_MS) + DFSConfigKeys.DFS_CLIENT_FILE_BLOCK_STORAGE_LOCATIONS_TIMEOUT_MS), }); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java index f8f337c16f785..e76750196e296 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java @@ -278,7 +278,7 @@ synchronized String getDaemonName() { /** Is the empty period longer than the grace period? */ private synchronized boolean isRenewerExpired() { return emptyTime != Long.MAX_VALUE - && Time.now() - emptyTime > gracePeriod; + && Time.monotonicNow() - emptyTime > gracePeriod; } synchronized void put(final long inodeId, final DFSOutputStream out, @@ -346,7 +346,7 @@ void closeFile(final long inodeId, final DFSClient dfsc) { } } //discover the first time that all file-being-written maps are empty. - emptyTime = Time.now(); + emptyTime = Time.monotonicNow(); } } } @@ -361,7 +361,7 @@ synchronized void closeClient(final DFSClient dfsc) { } if (emptyTime == Long.MAX_VALUE) { //discover the first time that the client list is empty. - emptyTime = Time.now(); + emptyTime = Time.monotonicNow(); } } @@ -434,9 +434,9 @@ public int compare(final DFSClient left, final DFSClient right) { * when the lease period is half over. */ private void run(final int id) throws InterruptedException { - for(long lastRenewed = Time.now(); !Thread.interrupted(); + for(long lastRenewed = Time.monotonicNow(); !Thread.interrupted(); Thread.sleep(getSleepPeriod())) { - final long elapsed = Time.now() - lastRenewed; + final long elapsed = Time.monotonicNow() - lastRenewed; if (elapsed >= getRenewalTime()) { try { renew(); @@ -444,7 +444,7 @@ private void run(final int id) throws InterruptedException { LOG.debug("Lease renewer daemon for " + clientsString() + " with renew id " + id + " executed"); } - lastRenewed = Time.now(); + lastRenewed = Time.monotonicNow(); } catch (SocketTimeoutException ie) { LOG.warn("Failed to renew lease for " + clientsString() + " for " + (elapsed/1000) + " seconds. Aborting ...", ie); @@ -479,7 +479,7 @@ private void run(final int id) throws InterruptedException { // registered with this renewer, stop the daemon after the grace // period. if (!clientsRunning() && emptyTime == Long.MAX_VALUE) { - emptyTime = Time.now(); + emptyTime = Time.monotonicNow(); } } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java index 628c61030f1fa..ce96ac9fa3009 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java @@ -30,6 +30,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.ReadOption; import org.apache.hadoop.hdfs.net.Peer; +import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil; @@ -351,7 +352,8 @@ private RemoteBlockReader(String file, String bpid, long blockId, long startOffset, long firstChunkOffset, long bytesToRead, Peer peer, DatanodeID datanodeID, PeerCache peerCache) { // Path is used only for printing block and file information in debug - super(new Path("/blk_" + blockId + ":" + bpid + ":of:"+ file)/*too non path-like?*/, + super(new Path("/" + Block.BLOCK_FILE_PREFIX + blockId + + ":" + bpid + ":of:"+ file)/*too non path-like?*/, 1, verifyChecksum, checksum.getChecksumSize() > 0? checksum : null, checksum.getBytesPerChecksum(), diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java index 4389714986f45..834546b66051d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockListAsLongs.java @@ -17,342 +17,495 @@ */ package org.apache.hadoop.hdfs.protocol; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; import java.util.Iterator; import java.util.List; -import java.util.Random; -import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportReplica; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState; import org.apache.hadoop.hdfs.server.datanode.Replica; +import com.google.common.base.Preconditions; +import com.google.protobuf.ByteString; +import com.google.protobuf.CodedInputStream; +import com.google.protobuf.CodedOutputStream; +import com.google.protobuf.WireFormat; -/** - * This class provides an interface for accessing list of blocks that - * has been implemented as long[]. - * This class is useful for block report. Rather than send block reports - * as a Block[] we can send it as a long[]. - * - * The structure of the array is as follows: - * 0: the length of the finalized replica list; - * 1: the length of the under-construction replica list; - * - followed by finalized replica list where each replica is represented by - * 3 longs: one for the blockId, one for the block length, and one for - * the generation stamp; - * - followed by the invalid replica represented with three -1s; - * - followed by the under-construction replica list where each replica is - * represented by 4 longs: three for the block id, length, generation - * stamp, and the fourth for the replica state. - */ @InterfaceAudience.Private @InterfaceStability.Evolving -public class BlockListAsLongs implements Iterable { +public abstract class BlockListAsLongs implements Iterable { + private final static int CHUNK_SIZE = 64*1024; // 64K + private static long[] EMPTY_LONGS = new long[]{0, 0}; + + public static BlockListAsLongs EMPTY = new BlockListAsLongs() { + @Override + public int getNumberOfBlocks() { + return 0; + } + @Override + public ByteString getBlocksBuffer() { + return ByteString.EMPTY; + } + @Override + public long[] getBlockListAsLongs() { + return EMPTY_LONGS; + } + @Override + public Iterator iterator() { + return Collections.emptyIterator(); + } + }; + /** - * A finalized block as 3 longs - * block-id and block length and generation stamp + * Prepare an instance to in-place decode the given ByteString buffer + * @param numBlocks - blocks in the buffer + * @param blocksBuf - ByteString encoded varints + * @return BlockListAsLongs */ - private static final int LONGS_PER_FINALIZED_BLOCK = 3; + public static BlockListAsLongs decodeBuffer(final int numBlocks, + final ByteString blocksBuf) { + return new BufferDecoder(numBlocks, blocksBuf); + } /** - * An under-construction block as 4 longs - * block-id and block length, generation stamp and replica state + * Prepare an instance to in-place decode the given ByteString buffers + * @param numBlocks - blocks in the buffers + * @param blocksBufs - list of ByteString encoded varints + * @return BlockListAsLongs */ - private static final int LONGS_PER_UC_BLOCK = 4; - - /** Number of longs in the header */ - private static final int HEADER_SIZE = 2; + public static BlockListAsLongs decodeBuffers(final int numBlocks, + final List blocksBufs) { + // this doesn't actually copy the data + return decodeBuffer(numBlocks, ByteString.copyFrom(blocksBufs)); + } /** - * Returns the index of the first long in blockList - * belonging to the specified block. - * The first long contains the block id. + * Prepare an instance to in-place decode the given list of Longs. Note + * it's much more efficient to decode ByteString buffers and only exists + * for compatibility. + * @param blocksList - list of longs + * @return BlockListAsLongs */ - private int index2BlockId(int blockIndex) { - if(blockIndex < 0 || blockIndex > getNumberOfBlocks()) - return -1; - int finalizedSize = getNumberOfFinalizedReplicas(); - if(blockIndex < finalizedSize) - return HEADER_SIZE + blockIndex * LONGS_PER_FINALIZED_BLOCK; - return HEADER_SIZE + (finalizedSize + 1) * LONGS_PER_FINALIZED_BLOCK - + (blockIndex - finalizedSize) * LONGS_PER_UC_BLOCK; + public static BlockListAsLongs decodeLongs(List blocksList) { + return blocksList.isEmpty() ? EMPTY : new LongsDecoder(blocksList); } - private final long[] blockList; - /** - * Create block report from finalized and under construction lists of blocks. - * - * @param finalized - list of finalized blocks - * @param uc - list of under construction blocks + * Prepare an instance to encode the collection of replicas into an + * efficient ByteString. + * @param replicas - replicas to encode + * @return BlockListAsLongs */ - public BlockListAsLongs(final List finalized, - final List uc) { - int finalizedSize = finalized == null ? 0 : finalized.size(); - int ucSize = uc == null ? 0 : uc.size(); - int len = HEADER_SIZE - + (finalizedSize + 1) * LONGS_PER_FINALIZED_BLOCK - + ucSize * LONGS_PER_UC_BLOCK; + public static BlockListAsLongs encode( + final Collection replicas) { + BlockListAsLongs.Builder builder = builder(); + for (Replica replica : replicas) { + builder.add(replica); + } + return builder.build(); + } - blockList = new long[len]; + public static BlockListAsLongs readFrom(InputStream is) throws IOException { + CodedInputStream cis = CodedInputStream.newInstance(is); + int numBlocks = -1; + ByteString blocksBuf = null; + while (!cis.isAtEnd()) { + int tag = cis.readTag(); + int field = WireFormat.getTagFieldNumber(tag); + switch(field) { + case 0: + break; + case 1: + numBlocks = (int)cis.readInt32(); + break; + case 2: + blocksBuf = cis.readBytes(); + break; + default: + cis.skipField(tag); + break; + } + } + if (numBlocks != -1 && blocksBuf != null) { + return decodeBuffer(numBlocks, blocksBuf); + } + return null; + } - // set the header - blockList[0] = finalizedSize; - blockList[1] = ucSize; + public void writeTo(OutputStream os) throws IOException { + CodedOutputStream cos = CodedOutputStream.newInstance(os); + cos.writeInt32(1, getNumberOfBlocks()); + cos.writeBytes(2, getBlocksBuffer()); + cos.flush(); + } + + public static Builder builder() { + return new BlockListAsLongs.Builder(); + } - // set finalized blocks - for (int i = 0; i < finalizedSize; i++) { - setBlock(i, finalized.get(i)); - } + /** + * The number of blocks + * @return - the number of blocks + */ + abstract public int getNumberOfBlocks(); - // set invalid delimiting block - setDelimitingBlock(finalizedSize); + /** + * Very efficient encoding of the block report into a ByteString to avoid + * the overhead of protobuf repeating fields. Primitive repeating fields + * require re-allocs of an ArrayList and the associated (un)boxing + * overhead which puts pressure on GC. + * + * The structure of the buffer is as follows: + * - each replica is represented by 4 longs: + * blockId, block length, genstamp, replica state + * + * @return ByteString encoded block report + */ + abstract public ByteString getBlocksBuffer(); - // set under construction blocks - for (int i = 0; i < ucSize; i++) { - setBlock(finalizedSize + i, uc.get(i)); + /** + * List of ByteStrings that encode this block report + * + * @return ByteStrings + */ + public List getBlocksBuffers() { + final ByteString blocksBuf = getBlocksBuffer(); + final List buffers; + final int size = blocksBuf.size(); + if (size <= CHUNK_SIZE) { + buffers = Collections.singletonList(blocksBuf); + } else { + buffers = new ArrayList(); + for (int pos=0; pos < size; pos += CHUNK_SIZE) { + // this doesn't actually copy the data + buffers.add(blocksBuf.substring(pos, Math.min(pos+CHUNK_SIZE, size))); + } } + return buffers; } /** - * Create block report from a list of finalized blocks. Used by - * NNThroughputBenchmark. - * - * @param blocks - list of finalized blocks + * Convert block report to old-style list of longs. Only used to + * re-encode the block report when the DN detects an older NN. This is + * inefficient, but in practice a DN is unlikely to be upgraded first + * + * The structure of the array is as follows: + * 0: the length of the finalized replica list; + * 1: the length of the under-construction replica list; + * - followed by finalized replica list where each replica is represented by + * 3 longs: one for the blockId, one for the block length, and one for + * the generation stamp; + * - followed by the invalid replica represented with three -1s; + * - followed by the under-construction replica list where each replica is + * represented by 4 longs: three for the block id, length, generation + * stamp, and the fourth for the replica state. + * @return list of longs */ - public BlockListAsLongs(final List blocks) { - int finalizedSize = blocks == null ? 0 : blocks.size(); - int len = HEADER_SIZE - + (finalizedSize + 1) * LONGS_PER_FINALIZED_BLOCK; + abstract public long[] getBlockListAsLongs(); - blockList = new long[len]; + /** + * Returns a singleton iterator over blocks in the block report. Do not + * add the returned blocks to a collection. + * @return Iterator + */ + abstract public Iterator iterator(); - // set the header - blockList[0] = finalizedSize; - blockList[1] = 0; + public static class Builder { + private final ByteString.Output out; + private final CodedOutputStream cos; + private int numBlocks = 0; + private int numFinalized = 0; - // set finalized blocks - for (int i = 0; i < finalizedSize; i++) { - setBlock(i, blocks.get(i)); + Builder() { + out = ByteString.newOutput(64*1024); + cos = CodedOutputStream.newInstance(out); } - // set invalid delimiting block - setDelimitingBlock(finalizedSize); - } - - public BlockListAsLongs() { - this((long[])null); - } + public void add(Replica replica) { + try { + // zig-zag to reduce size of legacy blocks + cos.writeSInt64NoTag(replica.getBlockId()); + cos.writeRawVarint64(replica.getBytesOnDisk()); + cos.writeRawVarint64(replica.getGenerationStamp()); + ReplicaState state = replica.getState(); + // although state is not a 64-bit value, using a long varint to + // allow for future use of the upper bits + cos.writeRawVarint64(state.getValue()); + if (state == ReplicaState.FINALIZED) { + numFinalized++; + } + numBlocks++; + } catch (IOException ioe) { + // shouldn't happen, ByteString.Output doesn't throw IOE + throw new IllegalStateException(ioe); + } + } - /** - * Constructor - * @param iBlockList - BlockListALongs create from this long[] parameter - */ - public BlockListAsLongs(final long[] iBlockList) { - if (iBlockList == null) { - blockList = new long[HEADER_SIZE]; - return; + public int getNumberOfBlocks() { + return numBlocks; + } + + public BlockListAsLongs build() { + try { + cos.flush(); + } catch (IOException ioe) { + // shouldn't happen, ByteString.Output doesn't throw IOE + throw new IllegalStateException(ioe); + } + return new BufferDecoder(numBlocks, numFinalized, out.toByteString()); } - blockList = iBlockList; } - public long[] getBlockListAsLongs() { - return blockList; - } + // decode new-style ByteString buffer based block report + private static class BufferDecoder extends BlockListAsLongs { + // reserve upper bits for future use. decoding masks off these bits to + // allow compatibility for the current through future release that may + // start using the bits + private static long NUM_BYTES_MASK = (-1L) >>> (64 - 48); + private static long REPLICA_STATE_MASK = (-1L) >>> (64 - 4); - /** - * Iterates over blocks in the block report. - * Avoids object allocation on each iteration. - */ - @InterfaceAudience.Private - @InterfaceStability.Evolving - public class BlockReportIterator implements Iterator { - private int currentBlockIndex; - private final Block block; - private ReplicaState currentReplicaState; + private final ByteString buffer; + private final int numBlocks; + private int numFinalized; - BlockReportIterator() { - this.currentBlockIndex = 0; - this.block = new Block(); - this.currentReplicaState = null; + BufferDecoder(final int numBlocks, final ByteString buf) { + this(numBlocks, -1, buf); } - @Override - public boolean hasNext() { - return currentBlockIndex < getNumberOfBlocks(); + BufferDecoder(final int numBlocks, final int numFinalized, + final ByteString buf) { + this.numBlocks = numBlocks; + this.numFinalized = numFinalized; + this.buffer = buf; } @Override - public Block next() { - block.set(blockId(currentBlockIndex), - blockLength(currentBlockIndex), - blockGenerationStamp(currentBlockIndex)); - currentReplicaState = blockReplicaState(currentBlockIndex); - currentBlockIndex++; - return block; + public int getNumberOfBlocks() { + return numBlocks; } @Override - public void remove() { - throw new UnsupportedOperationException("Sorry. can't remove."); + public ByteString getBlocksBuffer() { + return buffer; } - /** - * Get the state of the current replica. - * The state corresponds to the replica returned - * by the latest {@link #next()}. - */ - public ReplicaState getCurrentReplicaState() { - return currentReplicaState; + @Override + public long[] getBlockListAsLongs() { + // terribly inefficient but only occurs if server tries to transcode + // an undecoded buffer into longs - ie. it will never happen but let's + // handle it anyway + if (numFinalized == -1) { + int n = 0; + for (Replica replica : this) { + if (replica.getState() == ReplicaState.FINALIZED) { + n++; + } + } + numFinalized = n; + } + int numUc = numBlocks - numFinalized; + int size = 2 + 3*(numFinalized+1) + 4*(numUc); + long[] longs = new long[size]; + longs[0] = numFinalized; + longs[1] = numUc; + + int idx = 2; + int ucIdx = idx + 3*numFinalized; + // delimiter block + longs[ucIdx++] = -1; + longs[ucIdx++] = -1; + longs[ucIdx++] = -1; + + for (BlockReportReplica block : this) { + switch (block.getState()) { + case FINALIZED: { + longs[idx++] = block.getBlockId(); + longs[idx++] = block.getNumBytes(); + longs[idx++] = block.getGenerationStamp(); + break; + } + default: { + longs[ucIdx++] = block.getBlockId(); + longs[ucIdx++] = block.getNumBytes(); + longs[ucIdx++] = block.getGenerationStamp(); + longs[ucIdx++] = block.getState().getValue(); + break; + } + } + } + return longs; } - } - - /** - * Returns an iterator over blocks in the block report. - */ - @Override - public Iterator iterator() { - return getBlockReportIterator(); - } - - /** - * Returns {@link BlockReportIterator}. - */ - public BlockReportIterator getBlockReportIterator() { - return new BlockReportIterator(); - } - /** - * The number of blocks - * @return - the number of blocks - */ - public int getNumberOfBlocks() { - assert blockList.length == HEADER_SIZE + - (blockList[0] + 1) * LONGS_PER_FINALIZED_BLOCK + - blockList[1] * LONGS_PER_UC_BLOCK : - "Number of blocks is inconcistent with the array length"; - return getNumberOfFinalizedReplicas() + getNumberOfUCReplicas(); - } - - /** - * Returns the number of finalized replicas in the block report. - */ - private int getNumberOfFinalizedReplicas() { - return (int)blockList[0]; - } - - /** - * Returns the number of under construction replicas in the block report. - */ - private int getNumberOfUCReplicas() { - return (int)blockList[1]; + @Override + public Iterator iterator() { + return new Iterator() { + final BlockReportReplica block = new BlockReportReplica(); + final CodedInputStream cis = buffer.newCodedInput(); + private int currentBlockIndex = 0; + + @Override + public boolean hasNext() { + return currentBlockIndex < numBlocks; + } + + @Override + public BlockReportReplica next() { + currentBlockIndex++; + try { + // zig-zag to reduce size of legacy blocks and mask off bits + // we don't (yet) understand + block.setBlockId(cis.readSInt64()); + block.setNumBytes(cis.readRawVarint64() & NUM_BYTES_MASK); + block.setGenerationStamp(cis.readRawVarint64()); + long state = cis.readRawVarint64() & REPLICA_STATE_MASK; + block.setState(ReplicaState.getState((int)state)); + } catch (IOException e) { + throw new IllegalStateException(e); + } + return block; + } + + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + }; + } } - /** - * Returns the id of the specified replica of the block report. - */ - private long blockId(int index) { - return blockList[index2BlockId(index)]; - } + // decode old style block report of longs + private static class LongsDecoder extends BlockListAsLongs { + private final List values; + private final int finalizedBlocks; + private final int numBlocks; - /** - * Returns the length of the specified replica of the block report. - */ - private long blockLength(int index) { - return blockList[index2BlockId(index) + 1]; - } + // set the header + LongsDecoder(List values) { + this.values = values.subList(2, values.size()); + this.finalizedBlocks = values.get(0).intValue(); + this.numBlocks = finalizedBlocks + values.get(1).intValue(); + } - /** - * Returns the generation stamp of the specified replica of the block report. - */ - private long blockGenerationStamp(int index) { - return blockList[index2BlockId(index) + 2]; - } + @Override + public int getNumberOfBlocks() { + return numBlocks; + } - /** - * Returns the state of the specified replica of the block report. - */ - private ReplicaState blockReplicaState(int index) { - if(index < getNumberOfFinalizedReplicas()) - return ReplicaState.FINALIZED; - return ReplicaState.getState((int)blockList[index2BlockId(index) + 3]); - } + @Override + public ByteString getBlocksBuffer() { + Builder builder = builder(); + for (Replica replica : this) { + builder.add(replica); + } + return builder.build().getBlocksBuffer(); + } - /** - * Corrupt the generation stamp of the block with the given index. - * Not meant to be used outside of tests. - */ - @VisibleForTesting - public long corruptBlockGSForTesting(final int blockIndex, Random rand) { - long oldGS = blockList[index2BlockId(blockIndex) + 2]; - while (blockList[index2BlockId(blockIndex) + 2] == oldGS) { - blockList[index2BlockId(blockIndex) + 2] = rand.nextInt(); + @Override + public long[] getBlockListAsLongs() { + long[] longs = new long[2+values.size()]; + longs[0] = finalizedBlocks; + longs[1] = numBlocks - finalizedBlocks; + for (int i=0; i < longs.length; i++) { + longs[i] = values.get(i); + } + return longs; } - return oldGS; - } - /** - * Corrupt the length of the block with the given index by truncation. - * Not meant to be used outside of tests. - */ - @VisibleForTesting - public long corruptBlockLengthForTesting(final int blockIndex, Random rand) { - long oldLength = blockList[index2BlockId(blockIndex) + 1]; - blockList[index2BlockId(blockIndex) + 1] = - rand.nextInt((int) oldLength - 1); - return oldLength; + @Override + public Iterator iterator() { + return new Iterator() { + private final BlockReportReplica block = new BlockReportReplica(); + final Iterator iter = values.iterator(); + private int currentBlockIndex = 0; + + @Override + public boolean hasNext() { + return currentBlockIndex < numBlocks; + } + + @Override + public BlockReportReplica next() { + if (currentBlockIndex == finalizedBlocks) { + // verify the presence of the delimiter block + readBlock(); + Preconditions.checkArgument(block.getBlockId() == -1 && + block.getNumBytes() == -1 && + block.getGenerationStamp() == -1, + "Invalid delimiter block"); + } + + readBlock(); + if (currentBlockIndex++ < finalizedBlocks) { + block.setState(ReplicaState.FINALIZED); + } else { + block.setState(ReplicaState.getState(iter.next().intValue())); + } + return block; + } + + private void readBlock() { + block.setBlockId(iter.next()); + block.setNumBytes(iter.next()); + block.setGenerationStamp(iter.next()); + } + + @Override + public void remove() { + throw new UnsupportedOperationException(); + } + }; + } } - /** - * Set the indexTh block - * @param index - the index of the block to set - * @param r - the block is set to the value of the this Replica - */ - private void setBlock(final int index, final Replica r) { - int pos = index2BlockId(index); - blockList[pos] = r.getBlockId(); - blockList[pos + 1] = r.getNumBytes(); - blockList[pos + 2] = r.getGenerationStamp(); - if(index < getNumberOfFinalizedReplicas()) - return; - assert r.getState() != ReplicaState.FINALIZED : - "Must be under-construction replica."; - blockList[pos + 3] = r.getState().getValue(); - } - - /** - * Set the indexTh block - * @param index - the index of the block to set - * @param b - the block is set to the value of the this Block - */ - private void setBlock(final int index, final Block b) { - int pos = index2BlockId(index); - blockList[pos] = b.getBlockId(); - blockList[pos + 1] = b.getNumBytes(); - blockList[pos + 2] = b.getGenerationStamp(); - } - - /** - * Set the invalid delimiting block between the finalized and - * the under-construction lists. - * The invalid block has all three fields set to -1. - * @param finalizedSzie - the size of the finalized list - */ - private void setDelimitingBlock(final int finalizedSzie) { - int idx = HEADER_SIZE + finalizedSzie * LONGS_PER_FINALIZED_BLOCK; - blockList[idx] = -1; - blockList[idx+1] = -1; - blockList[idx+2] = -1; - } - - public long getMaxGsInBlockList() { - long maxGs = -1; - Iterator iter = getBlockReportIterator(); - while (iter.hasNext()) { - Block b = iter.next(); - if (b.getGenerationStamp() > maxGs) { - maxGs = b.getGenerationStamp(); + @InterfaceAudience.Private + public static class BlockReportReplica extends Block implements Replica { + private ReplicaState state; + private BlockReportReplica() { + } + public BlockReportReplica(Block block) { + super(block); + if (block instanceof BlockReportReplica) { + this.state = ((BlockReportReplica)block).getState(); + } else { + this.state = ReplicaState.FINALIZED; } } - return maxGs; + public void setState(ReplicaState state) { + this.state = state; + } + @Override + public ReplicaState getState() { + return state; + } + @Override + public long getBytesOnDisk() { + return getNumBytes(); + } + @Override + public long getVisibleLength() { + throw new UnsupportedOperationException(); + } + @Override + public String getStorageUuid() { + throw new UnsupportedOperationException(); + } + @Override + public boolean isOnTransientStorage() { + throw new UnsupportedOperationException(); + } + @Override + public boolean equals(Object o) { + return super.equals(o); + } + @Override + public int hashCode() { + return super.hashCode(); + } } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java index d4fe9038e3dce..bafb02b8d4acc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java @@ -798,12 +798,17 @@ public boolean setSafeMode(HdfsConstants.SafeModeAction action, boolean isChecke *

* Saves current namespace into storage directories and reset edits log. * Requires superuser privilege and safe mode. - * - * @throws AccessControlException if the superuser privilege is violated. + * + * @param timeWindow NameNode does a checkpoint if the latest checkpoint was + * done beyond the given time period (in seconds). + * @param txGap NameNode does a checkpoint if the gap between the latest + * checkpoint and the latest transaction id is greater this gap. + * @return whether an extra checkpoint has been done + * * @throws IOException if image creation failed. */ @AtMostOnce - public void saveNamespace() throws AccessControlException, IOException; + public boolean saveNamespace(long timeWindow, long txGap) throws IOException; /** @@ -847,7 +852,7 @@ public boolean restoreFailedStorage(String arg) /** * Rolling upgrade operations. - * @param action either query, start or finailze. + * @param action either query, prepare or finalize. * @return rolling upgrade information. */ @Idempotent diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java index 779e3b905f1a9..f91696fb284ed 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java @@ -47,19 +47,23 @@ public class DatanodeID implements Comparable { private int infoSecurePort; // info server port private int ipcPort; // IPC server port private String xferAddr; - private int hashCode = -1; /** * UUID identifying a given datanode. For upgraded Datanodes this is the * same as the StorageID that was previously used by this Datanode. * For newly formatted Datanodes it is a UUID. */ - private String datanodeUuid = null; + private final String datanodeUuid; public DatanodeID(DatanodeID from) { + this(from.getDatanodeUuid(), from); + } + + @VisibleForTesting + public DatanodeID(String datanodeUuid, DatanodeID from) { this(from.getIpAddr(), from.getHostName(), - from.getDatanodeUuid(), + datanodeUuid, from.getXferPort(), from.getInfoPort(), from.getInfoSecurePort(), @@ -81,19 +85,24 @@ public DatanodeID(DatanodeID from) { */ public DatanodeID(String ipAddr, String hostName, String datanodeUuid, int xferPort, int infoPort, int infoSecurePort, int ipcPort) { - this.ipAddr = ipAddr; + setIpAndXferPort(ipAddr, xferPort); this.hostName = hostName; this.datanodeUuid = checkDatanodeUuid(datanodeUuid); - this.xferPort = xferPort; this.infoPort = infoPort; this.infoSecurePort = infoSecurePort; this.ipcPort = ipcPort; - updateXferAddrAndInvalidateHashCode(); } public void setIpAddr(String ipAddr) { + //updated during registration, preserve former xferPort + setIpAndXferPort(ipAddr, xferPort); + } + + private void setIpAndXferPort(String ipAddr, int xferPort) { + // build xferAddr string to reduce cost of frequent use this.ipAddr = ipAddr; - updateXferAddrAndInvalidateHashCode(); + this.xferPort = xferPort; + this.xferAddr = ipAddr + ":" + xferPort; } public void setPeerHostName(String peerHostName) { @@ -107,12 +116,6 @@ public String getDatanodeUuid() { return datanodeUuid; } - @VisibleForTesting - public void setDatanodeUuidForTesting(String datanodeUuid) { - this.datanodeUuid = datanodeUuid; - updateXferAddrAndInvalidateHashCode(); - } - private String checkDatanodeUuid(String uuid) { if (uuid == null || uuid.isEmpty()) { return null; @@ -242,11 +245,7 @@ public boolean equals(Object to) { @Override public int hashCode() { - if (hashCode == -1) { - int newHashCode = xferAddr.hashCode() ^ datanodeUuid.hashCode(); - hashCode = newHashCode & Integer.MAX_VALUE; - } - return hashCode; + return datanodeUuid.hashCode(); } @Override @@ -259,14 +258,12 @@ public String toString() { * Note that this does not update storageID. */ public void updateRegInfo(DatanodeID nodeReg) { - ipAddr = nodeReg.getIpAddr(); + setIpAndXferPort(nodeReg.getIpAddr(), nodeReg.getXferPort()); hostName = nodeReg.getHostName(); peerHostName = nodeReg.getPeerHostName(); - xferPort = nodeReg.getXferPort(); infoPort = nodeReg.getInfoPort(); infoSecurePort = nodeReg.getInfoSecurePort(); ipcPort = nodeReg.getIpcPort(); - updateXferAddrAndInvalidateHashCode(); } /** @@ -279,13 +276,4 @@ public void updateRegInfo(DatanodeID nodeReg) { public int compareTo(DatanodeID that) { return getXferAddr().compareTo(that.getXferAddr()); } - - // NOTE: mutable hash codes are dangerous, however this class chooses to - // use them. this method must be called when a value mutates that is used - // to compute the hash, equality, or comparison of instances. - private void updateXferAddrAndInvalidateHashCode() { - xferAddr = ipAddr + ":" + xferPort; - // can't compute new hash yet because uuid might still null... - hashCode = -1; - } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java index 9fcada734ae6d..5ded26b185232 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java @@ -49,6 +49,7 @@ public class DatanodeInfo extends DatanodeID implements Node { private long cacheCapacity; private long cacheUsed; private long lastUpdate; + private long lastUpdateMonotonic; private int xceiverCount; private String location = NetworkTopology.DEFAULT_RACK; private String softwareVersion; @@ -91,6 +92,7 @@ public DatanodeInfo(DatanodeInfo from) { this.cacheCapacity = from.getCacheCapacity(); this.cacheUsed = from.getCacheUsed(); this.lastUpdate = from.getLastUpdate(); + this.lastUpdateMonotonic = from.getLastUpdateMonotonic(); this.xceiverCount = from.getXceiverCount(); this.location = from.getNetworkLocation(); this.adminState = from.getAdminState(); @@ -105,6 +107,7 @@ public DatanodeInfo(DatanodeID nodeID) { this.cacheCapacity = 0L; this.cacheUsed = 0L; this.lastUpdate = 0L; + this.lastUpdateMonotonic = 0L; this.xceiverCount = 0; this.adminState = null; } @@ -117,13 +120,13 @@ public DatanodeInfo(DatanodeID nodeID, String location) { public DatanodeInfo(DatanodeID nodeID, String location, final long capacity, final long dfsUsed, final long remaining, final long blockPoolUsed, final long cacheCapacity, final long cacheUsed, - final long lastUpdate, final int xceiverCount, - final AdminStates adminState) { + final long lastUpdate, final long lastUpdateMonotonic, + final int xceiverCount, final AdminStates adminState) { this(nodeID.getIpAddr(), nodeID.getHostName(), nodeID.getDatanodeUuid(), nodeID.getXferPort(), nodeID.getInfoPort(), nodeID.getInfoSecurePort(), nodeID.getIpcPort(), capacity, dfsUsed, remaining, blockPoolUsed, - cacheCapacity, cacheUsed, lastUpdate, xceiverCount, location, - adminState); + cacheCapacity, cacheUsed, lastUpdate, lastUpdateMonotonic, + xceiverCount, location, adminState); } /** Constructor */ @@ -132,8 +135,9 @@ public DatanodeInfo(final String ipAddr, final String hostName, final int infoSecurePort, final int ipcPort, final long capacity, final long dfsUsed, final long remaining, final long blockPoolUsed, final long cacheCapacity, final long cacheUsed, - final long lastUpdate, final int xceiverCount, - final String networkLocation, final AdminStates adminState) { + final long lastUpdate, final long lastUpdateMonotonic, + final int xceiverCount, final String networkLocation, + final AdminStates adminState) { super(ipAddr, hostName, datanodeUuid, xferPort, infoPort, infoSecurePort, ipcPort); this.capacity = capacity; @@ -143,6 +147,7 @@ public DatanodeInfo(final String ipAddr, final String hostName, this.cacheCapacity = cacheCapacity; this.cacheUsed = cacheUsed; this.lastUpdate = lastUpdate; + this.lastUpdateMonotonic = lastUpdateMonotonic; this.xceiverCount = xceiverCount; this.location = networkLocation; this.adminState = adminState; @@ -223,9 +228,26 @@ public float getCacheRemainingPercent() { return DFSUtil.getPercentRemaining(getCacheRemaining(), cacheCapacity); } - /** The time when this information was accurate. */ + /** + * Get the last update timestamp. + * Return value is suitable for Date conversion. + */ public long getLastUpdate() { return lastUpdate; } + /** + * The time when this information was accurate.
+ * Ps: So return value is ideal for calculation of time differences. + * Should not be used to convert to Date. + */ + public long getLastUpdateMonotonic() { return lastUpdateMonotonic;} + + /** + * Set lastUpdate monotonic time + */ + public void setLastUpdateMonotonic(long lastUpdateMonotonic) { + this.lastUpdateMonotonic = lastUpdateMonotonic; + } + /** number of active connections */ public int getXceiverCount() { return xceiverCount; } @@ -437,7 +459,7 @@ public AdminStates getAdminState() { * @return true if the node is stale */ public boolean isStale(long staleInterval) { - return (Time.now() - lastUpdate) >= staleInterval; + return (Time.monotonicNow() - lastUpdateMonotonic) >= staleInterval; } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java index 0d521918484b3..e729869878f82 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java @@ -44,9 +44,9 @@ public class LocatedBlock { private long offset; // offset of the first byte of the block in the file private final DatanodeInfoWithStorage[] locs; /** Cached storage ID for each replica */ - private String[] storageIDs; + private final String[] storageIDs; /** Cached storage type for each replica, if reported. */ - private StorageType[] storageTypes; + private final StorageType[] storageTypes; // corrupt flag is true if all of the replicas of a block are corrupt. // else false. If block has few corrupt replicas, they are filtered and // their locations are not part of this object @@ -62,16 +62,8 @@ public class LocatedBlock { new DatanodeInfoWithStorage[0]; public LocatedBlock(ExtendedBlock b, DatanodeInfo[] locs) { - this(b, locs, -1, false); // startOffset is unknown - } - - public LocatedBlock(ExtendedBlock b, DatanodeInfo[] locs, long startOffset, - boolean corrupt) { - this(b, locs, null, null, startOffset, corrupt, EMPTY_LOCS); - } - - public LocatedBlock(ExtendedBlock b, DatanodeStorageInfo[] storages) { - this(b, storages, -1, false); // startOffset is unknown + // By default, startOffset is unknown(-1) and corrupt is false. + this(b, locs, null, null, -1, false, EMPTY_LOCS); } public LocatedBlock(ExtendedBlock b, DatanodeInfo[] locs, @@ -170,11 +162,11 @@ public long getBlockSize() { return b.getNumBytes(); } - void setStartOffset(long value) { + public void setStartOffset(long value) { this.offset = value; } - void setCorrupt(boolean corrupt) { + public void setCorrupt(boolean corrupt) { this.corrupt = corrupt; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java index fc739cf711e33..e35a43107444b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java @@ -119,7 +119,7 @@ public FileEncryptionInfo getFileEncryptionInfo() { public int findBlock(long offset) { // create fake block of size 0 as a key LocatedBlock key = new LocatedBlock( - new ExtendedBlock(), new DatanodeInfo[0], 0L, false); + new ExtendedBlock(), new DatanodeInfo[0]); key.setStartOffset(offset); key.getBlock().setNumBytes(1); Comparator comp = diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java index 98089bc6ef873..80e3e3478d3fb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java @@ -29,12 +29,12 @@ @InterfaceStability.Evolving public class RollingUpgradeInfo extends RollingUpgradeStatus { private final long startTime; - private final long finalizeTime; + private long finalizeTime; private boolean createdRollbackImages; public RollingUpgradeInfo(String blockPoolId, boolean createdRollbackImages, long startTime, long finalizeTime) { - super(blockPoolId); + super(blockPoolId, finalizeTime != 0); this.createdRollbackImages = createdRollbackImages; this.startTime = startTime; this.finalizeTime = finalizeTime; @@ -56,11 +56,23 @@ public boolean isStarted() { public long getStartTime() { return startTime; } - + + @Override public boolean isFinalized() { return finalizeTime != 0; } + /** + * Finalize the upgrade if not already finalized + * @param finalizeTime + */ + public void finalize(long finalizeTime) { + if (finalizeTime != 0) { + this.finalizeTime = finalizeTime; + createdRollbackImages = false; + } + } + public long getFinalizeTime() { return finalizeTime; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java index 9925920250b4e..1f969fbb0c12b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java @@ -27,15 +27,21 @@ @InterfaceStability.Evolving public class RollingUpgradeStatus { private final String blockPoolId; + private final boolean finalized; - public RollingUpgradeStatus(String blockPoolId) { + public RollingUpgradeStatus(String blockPoolId, boolean finalized) { this.blockPoolId = blockPoolId; + this.finalized = finalized; } public String getBlockPoolId() { return blockPoolId; } + public boolean isFinalized() { + return finalized; + } + @Override public int hashCode() { return blockPoolId.hashCode(); @@ -48,8 +54,9 @@ public boolean equals(Object obj) { } else if (obj == null || !(obj instanceof RollingUpgradeStatus)) { return false; } - final RollingUpgradeStatus that = (RollingUpgradeStatus)obj; - return this.blockPoolId.equals(that.blockPoolId); + final RollingUpgradeStatus that = (RollingUpgradeStatus) obj; + return this.blockPoolId.equals(that.blockPoolId) + && this.isFinalized() == that.isFinalized(); } @Override diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java index 4be42a8d35891..48e931d741a03 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/DataTransferProtocol.java @@ -138,10 +138,13 @@ public void transferBlock(final ExtendedBlock blk, * to use no slot id. * @param maxVersion Maximum version of the block data the client * can understand. + * @param supportsReceiptVerification True if the client supports + * receipt verification. */ public void requestShortCircuitFds(final ExtendedBlock blk, final Token blockToken, - SlotId slotId, int maxVersion) throws IOException; + SlotId slotId, int maxVersion, boolean supportsReceiptVerification) + throws IOException; /** * Release a pair of short-circuit FDs requested earlier. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java index 35e5bb84dc7b0..9bd4115b59ff9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java @@ -130,13 +130,16 @@ public PipelineAck(long seqno, int[] replies) { */ public PipelineAck(long seqno, int[] replies, long downstreamAckTimeNanos) { - ArrayList replyList = Lists.newArrayList(); + ArrayList statusList = Lists.newArrayList(); + ArrayList flagList = Lists.newArrayList(); for (int r : replies) { - replyList.add(r); + statusList.add(StatusFormat.getStatus(r)); + flagList.add(r); } proto = PipelineAckProto.newBuilder() .setSeqno(seqno) - .addAllReply(replyList) + .addAllReply(statusList) + .addAllFlag(flagList) .setDownstreamAckTimeNanos(downstreamAckTimeNanos) .build(); } @@ -158,11 +161,18 @@ public short getNumOfReplies() { } /** - * get the ith reply - * @return the the ith reply + * get the header flag of ith reply */ - public int getReply(int i) { - return proto.getReply(i); + public int getHeaderFlag(int i) { + if (proto.getFlagCount() > 0) { + return proto.getFlag(i); + } else { + return combineHeader(ECN.DISABLED, proto.getReply(i)); + } + } + + public int getFlag(int i) { + return proto.getFlag(i); } /** @@ -178,8 +188,8 @@ public long getDownstreamAckTimeNanos() { * @return true if all statuses are SUCCESS */ public boolean isSuccess() { - for (int reply : proto.getReplyList()) { - if (StatusFormat.getStatus(reply) != Status.SUCCESS) { + for (Status s : proto.getReplyList()) { + if (s != Status.SUCCESS) { return false; } } @@ -196,10 +206,9 @@ public Status getOOBStatus() { if (getSeqno() != UNKOWN_SEQNO) { return null; } - for (int reply : proto.getReplyList()) { + for (Status s : proto.getReplyList()) { // The following check is valid because protobuf guarantees to // preserve the ordering of enum elements. - Status s = StatusFormat.getStatus(reply); if (s.getNumber() >= OOB_START && s.getNumber() <= OOB_END) { return s; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java index 7994027c6ec83..31bdc5e2a52a4 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java @@ -186,7 +186,7 @@ private void opRequestShortCircuitFds(DataInputStream in) throws IOException { try { requestShortCircuitFds(PBHelper.convert(proto.getHeader().getBlock()), PBHelper.convert(proto.getHeader().getToken()), - slotId, proto.getMaxVersion()); + slotId, proto.getMaxVersion(), true); } finally { if (traceScope != null) traceScope.close(); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java index 7fea33efc59c5..df69125882bc6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Sender.java @@ -181,7 +181,8 @@ public void transferBlock(final ExtendedBlock blk, @Override public void requestShortCircuitFds(final ExtendedBlock blk, final Token blockToken, - SlotId slotId, int maxVersion) throws IOException { + SlotId slotId, int maxVersion, boolean supportsReceiptVerification) + throws IOException { OpRequestShortCircuitAccessProto.Builder builder = OpRequestShortCircuitAccessProto.newBuilder() .setHeader(DataTransferProtoUtil.buildBaseHeader( @@ -189,6 +190,7 @@ public void requestShortCircuitFds(final ExtendedBlock blk, if (slotId != null) { builder.setSlotId(PBHelper.convert(slotId)); } + builder.setSupportsReceiptVerification(supportsReceiptVerification); OpRequestShortCircuitAccessProto proto = builder.build(); send(out, Op.REQUEST_SHORT_CIRCUIT_FDS, proto); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java index ce8c3924e39a5..e26158b208df0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java @@ -277,10 +277,7 @@ public class ClientNamenodeProtocolServerSideTranslatorPB implements private static final RenewLeaseResponseProto VOID_RENEWLEASE_RESPONSE = RenewLeaseResponseProto.newBuilder().build(); - private static final SaveNamespaceResponseProto VOID_SAVENAMESPACE_RESPONSE = - SaveNamespaceResponseProto.newBuilder().build(); - - private static final RefreshNodesResponseProto VOID_REFRESHNODES_RESPONSE = + private static final RefreshNodesResponseProto VOID_REFRESHNODES_RESPONSE = RefreshNodesResponseProto.newBuilder().build(); private static final FinalizeUpgradeResponseProto VOID_FINALIZEUPGRADE_RESPONSE = @@ -748,14 +745,15 @@ public SetSafeModeResponseProto setSafeMode(RpcController controller, public SaveNamespaceResponseProto saveNamespace(RpcController controller, SaveNamespaceRequestProto req) throws ServiceException { try { - server.saveNamespace(); - return VOID_SAVENAMESPACE_RESPONSE; + final long timeWindow = req.hasTimeWindow() ? req.getTimeWindow() : 0; + final long txGap = req.hasTxGap() ? req.getTxGap() : 0; + boolean saved = server.saveNamespace(timeWindow, txGap); + return SaveNamespaceResponseProto.newBuilder().setSaved(saved).build(); } catch (IOException e) { throw new ServiceException(e); } - } - + @Override public RollEditsResponseProto rollEdits(RpcController controller, RollEditsRequestProto request) throws ServiceException { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java index e970293ede10a..4ec6f9ea05e94 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java @@ -670,9 +670,11 @@ public boolean setSafeMode(SafeModeAction action, boolean isChecked) throws IOEx } @Override - public void saveNamespace() throws AccessControlException, IOException { + public boolean saveNamespace(long timeWindow, long txGap) throws IOException { try { - rpcProxy.saveNamespace(null, VOID_SAVE_NAMESPACE_REQUEST); + SaveNamespaceRequestProto req = SaveNamespaceRequestProto.newBuilder() + .setTimeWindow(timeWindow).setTxGap(txGap).build(); + return rpcProxy.saveNamespace(null, req).getSaved(); } catch (ServiceException e) { throw ProtobufHelper.getRemoteException(e); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java index 192916f365d24..825e83586b323 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java @@ -26,6 +26,7 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlock; @@ -46,11 +47,13 @@ import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.StorageBlockReportProto; import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.StorageReceivedDeletedBlocksProto; import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos.VersionRequestProto; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.HeartbeatResponse; import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo; +import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.Capability; import org.apache.hadoop.hdfs.server.protocol.ReceivedDeletedBlockInfo; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; import org.apache.hadoop.hdfs.server.protocol.StorageReceivedDeletedBlocks; @@ -64,6 +67,7 @@ import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.UserGroupInformation; +import com.google.common.annotations.VisibleForTesting; import com.google.protobuf.RpcController; import com.google.protobuf.ServiceException; @@ -83,6 +87,11 @@ public class DatanodeProtocolClientSideTranslatorPB implements VersionRequestProto.newBuilder().build(); private final static RpcController NULL_CONTROLLER = null; + @VisibleForTesting + public DatanodeProtocolClientSideTranslatorPB(DatanodeProtocolPB rpcProxy) { + this.rpcProxy = rpcProxy; + } + public DatanodeProtocolClientSideTranslatorPB(InetSocketAddress nameNodeAddr, Configuration conf) throws IOException { RPC.setProtocolEngine(conf, DatanodeProtocolPB.class, @@ -161,20 +170,30 @@ public HeartbeatResponse sendHeartbeat(DatanodeRegistration registration, @Override public DatanodeCommand blockReport(DatanodeRegistration registration, - String poolId, StorageBlockReport[] reports) throws IOException { + String poolId, StorageBlockReport[] reports, BlockReportContext context) + throws IOException { BlockReportRequestProto.Builder builder = BlockReportRequestProto .newBuilder().setRegistration(PBHelper.convert(registration)) .setBlockPoolId(poolId); + boolean useBlocksBuffer = registration.getNamespaceInfo() + .isCapabilitySupported(Capability.STORAGE_BLOCK_REPORT_BUFFERS); + for (StorageBlockReport r : reports) { StorageBlockReportProto.Builder reportBuilder = StorageBlockReportProto .newBuilder().setStorage(PBHelper.convert(r.getStorage())); - long[] blocks = r.getBlocks(); - for (int i = 0; i < blocks.length; i++) { - reportBuilder.addBlocks(blocks[i]); + BlockListAsLongs blocks = r.getBlocks(); + if (useBlocksBuffer) { + reportBuilder.setNumberOfBlocks(blocks.getNumberOfBlocks()); + reportBuilder.addAllBlocksBuffers(blocks.getBlocksBuffers()); + } else { + for (long value : blocks.getBlockListAsLongs()) { + reportBuilder.addBlocks(value); + } } builder.addReports(reportBuilder.build()); } + builder.setContext(PBHelper.convert(context)); BlockReportResponseProto resp; try { resp = rpcProxy.blockReport(NULL_CONTROLLER, builder.build()); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java index 1a89090f13e3e..873eb6d1708e7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java @@ -21,6 +21,7 @@ import java.io.IOException; import java.util.List; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.protocol.RollingUpgradeStatus; @@ -58,6 +59,7 @@ import org.apache.hadoop.hdfs.server.protocol.StorageReport; import org.apache.hadoop.hdfs.server.protocol.VolumeFailureSummary; +import com.google.common.base.Preconditions; import com.google.protobuf.RpcController; import com.google.protobuf.ServiceException; @@ -145,17 +147,23 @@ public BlockReportResponseProto blockReport(RpcController controller, int index = 0; for (StorageBlockReportProto s : request.getReportsList()) { - List blockIds = s.getBlocksList(); - long[] blocks = new long[blockIds.size()]; - for (int i = 0; i < blockIds.size(); i++) { - blocks[i] = blockIds.get(i); + final BlockListAsLongs blocks; + if (s.hasNumberOfBlocks()) { // new style buffer based reports + int num = (int)s.getNumberOfBlocks(); + Preconditions.checkState(s.getBlocksCount() == 0, + "cannot send both blocks list and buffers"); + blocks = BlockListAsLongs.decodeBuffers(num, s.getBlocksBuffersList()); + } else { + blocks = BlockListAsLongs.decodeLongs(s.getBlocksList()); } report[index++] = new StorageBlockReport(PBHelper.convert(s.getStorage()), blocks); } try { cmd = impl.blockReport(PBHelper.convert(request.getRegistration()), - request.getBlockPoolId(), report); + request.getBlockPoolId(), report, + request.hasContext() ? + PBHelper.convert(request.getContext()) : null); } catch (IOException e) { throw new ServiceException(e); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java index ee1603ced30cb..1942ea9e98da7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java @@ -111,6 +111,7 @@ import org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos.SafeModeActionProto; import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.ShortCircuitShmIdProto; import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.ShortCircuitShmSlotProto; +import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos; import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.BalancerBandwidthCommandProto; import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.BlockCommandProto; import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.BlockIdCommandProto; @@ -123,6 +124,7 @@ import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.ReceivedDeletedBlockInfoProto; import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.RegisterCommandProto; import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.VolumeFailureSummaryProto; +import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.BlockReportContextProto; import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos; import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos.BlockKeyProto; import org.apache.hadoop.hdfs.protocol.proto.HdfsProtos.BlockProto; @@ -194,6 +196,7 @@ import org.apache.hadoop.hdfs.server.protocol.BlockIdCommand; import org.apache.hadoop.hdfs.server.protocol.BlockRecoveryCommand; import org.apache.hadoop.hdfs.server.protocol.BlockRecoveryCommand.RecoveringBlock; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations; import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations.BlockWithLocations; import org.apache.hadoop.hdfs.server.protocol.CheckpointCommand; @@ -573,7 +576,7 @@ public static NamespaceInfo convert(NamespaceInfoProto info) { StorageInfoProto storage = info.getStorageInfo(); return new NamespaceInfo(storage.getNamespceID(), storage.getClusterID(), info.getBlockPoolID(), storage.getCTime(), info.getBuildVersion(), - info.getSoftwareVersion()); + info.getSoftwareVersion(), info.getCapabilities()); } public static NamenodeCommand convert(NamenodeCommandProto cmd) { @@ -642,8 +645,8 @@ static public DatanodeInfo convert(DatanodeInfoProto di) { di.hasLocation() ? di.getLocation() : null , di.getCapacity(), di.getDfsUsed(), di.getRemaining(), di.getBlockPoolUsed(), di.getCacheCapacity(), di.getCacheUsed(), - di.getLastUpdate(), di.getXceiverCount(), - PBHelper.convert(di.getAdminState())); + di.getLastUpdate(), di.getLastUpdateMonotonic(), + di.getXceiverCount(), PBHelper.convert(di.getAdminState())); } static public DatanodeInfoProto convertDatanodeInfo(DatanodeInfo di) { @@ -704,6 +707,7 @@ public static DatanodeInfoProto convert(DatanodeInfo info) { .setCacheCapacity(info.getCacheCapacity()) .setCacheUsed(info.getCacheUsed()) .setLastUpdate(info.getLastUpdate()) + .setLastUpdateMonotonic(info.getLastUpdateMonotonic()) .setXceiverCount(info.getXceiverCount()) .setAdminState(PBHelper.convert(info.getAdminState())) .build(); @@ -1233,7 +1237,9 @@ public static NamespaceInfoProto convert(NamespaceInfo info) { .setBuildVersion(info.getBuildVersion()) .setUnused(0) .setStorageInfo(PBHelper.convert((StorageInfo)info)) - .setSoftwareVersion(info.getSoftwareVersion()).build(); + .setSoftwareVersion(info.getSoftwareVersion()) + .setCapabilities(info.getCapabilities()) + .build(); } // Located Block Arrays and Lists @@ -1680,11 +1686,13 @@ public static RollingUpgradeStatusProto convertRollingUpgradeStatus( RollingUpgradeStatus status) { return RollingUpgradeStatusProto.newBuilder() .setBlockPoolId(status.getBlockPoolId()) + .setFinalized(status.isFinalized()) .build(); } public static RollingUpgradeStatus convert(RollingUpgradeStatusProto proto) { - return new RollingUpgradeStatus(proto.getBlockPoolId()); + return new RollingUpgradeStatus(proto.getBlockPoolId(), + proto.getFinalized()); } public static RollingUpgradeInfoProto convert(RollingUpgradeInfo info) { @@ -1722,21 +1730,49 @@ public static CorruptFileBlocksProto convert(CorruptFileBlocks c) { public static ContentSummary convert(ContentSummaryProto cs) { if (cs == null) return null; - return new ContentSummary( - cs.getLength(), cs.getFileCount(), cs.getDirectoryCount(), cs.getQuota(), - cs.getSpaceConsumed(), cs.getSpaceQuota()); + ContentSummary.Builder builder = new ContentSummary.Builder(); + builder.length(cs.getLength()). + fileCount(cs.getFileCount()). + directoryCount(cs.getDirectoryCount()). + quota(cs.getQuota()). + spaceConsumed(cs.getSpaceConsumed()). + spaceQuota(cs.getSpaceQuota()); + if (cs.hasTypeQuotaInfos()) { + for (HdfsProtos.StorageTypeQuotaInfoProto info : + cs.getTypeQuotaInfos().getTypeQuotaInfoList()) { + StorageType type = PBHelper.convertStorageType(info.getType()); + builder.typeConsumed(type, info.getConsumed()); + builder.typeQuota(type, info.getQuota()); + } + } + return builder.build(); } public static ContentSummaryProto convert(ContentSummary cs) { if (cs == null) return null; - return ContentSummaryProto.newBuilder(). - setLength(cs.getLength()). + ContentSummaryProto.Builder builder = ContentSummaryProto.newBuilder(); + builder.setLength(cs.getLength()). setFileCount(cs.getFileCount()). setDirectoryCount(cs.getDirectoryCount()). setQuota(cs.getQuota()). setSpaceConsumed(cs.getSpaceConsumed()). - setSpaceQuota(cs.getSpaceQuota()). - build(); + setSpaceQuota(cs.getSpaceQuota()); + + if (cs.isTypeQuotaSet() || cs.isTypeConsumedAvailable()) { + HdfsProtos.StorageTypeQuotaInfosProto.Builder isb = + HdfsProtos.StorageTypeQuotaInfosProto.newBuilder(); + for (StorageType t: StorageType.getTypesSupportingQuota()) { + HdfsProtos.StorageTypeQuotaInfoProto info = + HdfsProtos.StorageTypeQuotaInfoProto.newBuilder(). + setType(convertStorageType(t)). + setConsumed(cs.getTypeConsumed(t)). + setQuota(cs.getTypeQuota(t)). + build(); + isb.addTypeQuotaInfo(info); + } + builder.setTypeQuotaInfos(isb); + } + return builder.build(); } public static NNHAStatusHeartbeat convert(NNHAStatusHeartbeatProto s) { @@ -3006,4 +3042,16 @@ public static boolean[] convertBooleanList( return targetPinnings; } + public static BlockReportContext convert(BlockReportContextProto proto) { + return new BlockReportContext(proto.getTotalRpcs(), + proto.getCurRpc(), proto.getId()); + } + + public static BlockReportContextProto convert(BlockReportContext context) { + return BlockReportContextProto.newBuilder(). + setTotalRpcs(context.getTotalRpcs()). + setCurRpc(context.getCurRpc()). + setId(context.getReportId()). + build(); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java index 90212a3858832..bc7e4489e0bc6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java @@ -672,7 +672,7 @@ static class Cli extends Configured implements Tool { */ @Override public int run(String[] args) { - final long startTime = Time.now(); + final long startTime = Time.monotonicNow(); final Configuration conf = getConf(); try { @@ -687,8 +687,10 @@ public int run(String[] args) { System.out.println(e + ". Exiting ..."); return ExitStatus.INTERRUPTED.getExitCode(); } finally { - System.out.format("%-24s ", DateFormat.getDateTimeInstance().format(new Date())); - System.out.println("Balancing took " + time2Str(Time.now()-startTime)); + System.out.format("%-24s ", + DateFormat.getDateTimeInstance().format(new Date())); + System.out.println("Balancing took " + + time2Str(Time.monotonicNow() - startTime)); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java index 15476118b9eec..e9baf8535bdac 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockCollection.java @@ -36,7 +36,7 @@ public interface BlockCollection { /** * Get content summary. */ - public ContentSummary computeContentSummary(); + public ContentSummary computeContentSummary(BlockStoragePolicySuite bsps); /** * @return the number of blocks diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java index 91b76ccb0e266..92153abb97686 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoContiguousUnderConstruction.java @@ -315,7 +315,8 @@ public void initializeBlockRecovery(long recoveryId) { continue; } final ReplicaUnderConstruction ruc = replicas.get(i); - final long lastUpdate = ruc.getExpectedStorageLocation().getDatanodeDescriptor().getLastUpdate(); + final long lastUpdate = ruc.getExpectedStorageLocation() + .getDatanodeDescriptor().getLastUpdateMonotonic(); if (lastUpdate > mostRecentLastUpdate) { primaryNodeIndex = i; primary = ruc; @@ -383,6 +384,7 @@ public void appendStringTo(StringBuilder sb) { private void appendUCParts(StringBuilder sb) { sb.append("{UCState=").append(blockUCState) + .append(", truncateBlock=" + truncateBlock) .append(", primaryNodeIndex=").append(primaryNodeIndex) .append(", replicas=["); if (replicas != null) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java index 58a8b94d30ede..d9aee62e7ef0b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java @@ -47,7 +47,7 @@ import org.apache.hadoop.hdfs.HAUtil; import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; -import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportIterator; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportReplica; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; @@ -69,6 +69,7 @@ import org.apache.hadoop.hdfs.server.namenode.Namesystem; import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics; import org.apache.hadoop.hdfs.server.protocol.BlockCommand; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations; import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations.BlockWithLocations; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; @@ -365,10 +366,10 @@ private static BlockTokenSecretManager createBlockTokenSecretManager( if (!isEnabled) { if (UserGroupInformation.isSecurityEnabled()) { - LOG.error("Security is enabled but block access tokens " + - "(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " + - "aren't enabled. This may cause issues " + - "when clients attempt to talk to a DataNode."); + LOG.error("Security is enabled but block access tokens " + + "(via " + DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY + ") " + + "aren't enabled. This may cause issues " + + "when clients attempt to talk to a DataNode."); } return null; } @@ -744,7 +745,7 @@ public LocatedBlock convertLastBlockToUnderConstruction( // always decrement total blocks -1); - final long fileLength = bc.computeContentSummary().getLength(); + final long fileLength = bc.computeContentSummary(getStoragePolicySuite()).getLength(); final long pos = fileLength - ucBlock.getNumBytes(); return createLocatedBlock(ucBlock, pos, AccessMode.WRITE); } @@ -1222,10 +1223,6 @@ private boolean invalidateBlock(BlockToMarkCorrupt b, DatanodeInfo dn // Check how many copies we have of the block NumberReplicas nr = countNodes(b.stored); if (nr.replicasOnStaleNodes() > 0) { - blockLog.info("BLOCK* invalidateBlocks: postponing " + - "invalidation of " + b + " on " + dn + " because " + - nr.replicasOnStaleNodes() + " replica(s) are located on nodes " + - "with potentially out-of-date block reports"); blockLog.info("BLOCK* invalidateBlocks: postponing " + "invalidation of {} on {} because {} replica(s) are located on " + "nodes with potentially out-of-date block reports", b, dn, @@ -1496,8 +1493,10 @@ int computeReplicationWorkForBlocks(List> blocksToReplicate) { } } } - blockLog.debug("BLOCK* neededReplications = {} pendingReplications = {}", - neededReplications.size(), pendingReplications.size()); + if (blockLog.isDebugEnabled()) { + blockLog.debug("BLOCK* neededReplications = {} pendingReplications = {}", + neededReplications.size(), pendingReplications.size()); + } return scheduledWork; } @@ -1638,7 +1637,8 @@ else if (excessBlocks != null && excessBlocks.contains(block)) { // If so, do not select the node as src node if ((nodesCorrupt != null) && nodesCorrupt.contains(node)) continue; - if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY + if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY + && !node.isDecommissionInProgress() && node.getNumberOfBlocksToBeReplicated() >= maxReplicationStreams) { continue; // already reached replication limit @@ -1653,13 +1653,12 @@ else if (excessBlocks != null && excessBlocks.contains(block)) { // never use already decommissioned nodes if(node.isDecommissioned()) continue; - // we prefer nodes that are in DECOMMISSION_INPROGRESS state - if(node.isDecommissionInProgress() || srcNode == null) { + + // We got this far, current node is a reasonable choice + if (srcNode == null) { srcNode = node; continue; } - if(srcNode.isDecommissionInProgress()) - continue; // switch to a different node randomly // this to prevent from deterministically selecting the same node even // if the node failed to replicate the block on previous iterations @@ -1772,9 +1771,10 @@ public String toString() { */ public boolean processReport(final DatanodeID nodeID, final DatanodeStorage storage, - final BlockListAsLongs newReport) throws IOException { + final BlockListAsLongs newReport, BlockReportContext context, + boolean lastStorageInRpc) throws IOException { namesystem.writeLock(); - final long startTime = Time.now(); //after acquiring write lock + final long startTime = Time.monotonicNow(); //after acquiring write lock final long endTime; DatanodeDescriptor node; Collection invalidatedBlocks = null; @@ -1811,8 +1811,31 @@ public boolean processReport(final DatanodeID nodeID, } storageInfo.receivedBlockReport(); + if (context != null) { + storageInfo.setLastBlockReportId(context.getReportId()); + if (lastStorageInRpc) { + int rpcsSeen = node.updateBlockReportContext(context); + if (rpcsSeen >= context.getTotalRpcs()) { + List zombies = node.removeZombieStorages(); + if (zombies.isEmpty()) { + LOG.debug("processReport 0x{}: no zombie storages found.", + Long.toHexString(context.getReportId())); + } else { + for (DatanodeStorageInfo zombie : zombies) { + removeZombieReplicas(context, zombie); + } + } + node.clearBlockReportContext(); + } else { + LOG.debug("processReport 0x{}: {} more RPCs remaining in this " + + "report.", Long.toHexString(context.getReportId()), + (context.getTotalRpcs() - rpcsSeen) + ); + } + } + } } finally { - endTime = Time.now(); + endTime = Time.monotonicNow(); namesystem.writeUnlock(); } @@ -1835,6 +1858,32 @@ public boolean processReport(final DatanodeID nodeID, return !node.hasStaleStorages(); } + private void removeZombieReplicas(BlockReportContext context, + DatanodeStorageInfo zombie) { + LOG.warn("processReport 0x{}: removing zombie storage {}, which no " + + "longer exists on the DataNode.", + Long.toHexString(context.getReportId()), zombie.getStorageID()); + assert(namesystem.hasWriteLock()); + Iterator iter = zombie.getBlockIterator(); + int prevBlocks = zombie.numBlocks(); + while (iter.hasNext()) { + BlockInfoContiguous block = iter.next(); + // We assume that a block can be on only one storage in a DataNode. + // That's why we pass in the DatanodeDescriptor rather than the + // DatanodeStorageInfo. + // TODO: remove this assumption in case we want to put a block on + // more than one storage on a datanode (and because it's a difficult + // assumption to really enforce) + removeStoredBlock(block, zombie.getDatanodeDescriptor()); + invalidateBlocks.remove(zombie.getDatanodeDescriptor(), block); + } + assert(zombie.numBlocks() == 0); + LOG.warn("processReport 0x{}: removed {} replicas from storage {}, " + + "which no longer exists on the DataNode.", + Long.toHexString(context.getReportId()), prevBlocks, + zombie.getStorageID()); + } + /** * Rescan the list of blocks which were previously postponed. */ @@ -1842,7 +1891,7 @@ void rescanPostponedMisreplicatedBlocks() { if (getPostponedMisreplicatedBlocksCount() == 0) { return; } - long startTimeRescanPostponedMisReplicatedBlocks = Time.now(); + long startTimeRescanPostponedMisReplicatedBlocks = Time.monotonicNow(); long startPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount(); namesystem.writeLock(); @@ -1902,7 +1951,7 @@ void rescanPostponedMisreplicatedBlocks() { long endPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount(); LOG.info("Rescan of postponedMisreplicatedBlocks completed in " + - (Time.now() - startTimeRescanPostponedMisReplicatedBlocks) + + (Time.monotonicNow() - startTimeRescanPostponedMisReplicatedBlocks) + " msecs. " + endPostponedMisReplicatedBlocksCount + " blocks are left. " + (startPostponedMisReplicatedBlocksCount - endPostponedMisReplicatedBlocksCount) + " blocks are removed."); @@ -1951,6 +2000,46 @@ private Collection processReport( return toInvalidate; } + /** + * Mark block replicas as corrupt except those on the storages in + * newStorages list. + */ + public void markBlockReplicasAsCorrupt(BlockInfoContiguous block, + long oldGenerationStamp, long oldNumBytes, + DatanodeStorageInfo[] newStorages) throws IOException { + assert namesystem.hasWriteLock(); + BlockToMarkCorrupt b = null; + if (block.getGenerationStamp() != oldGenerationStamp) { + b = new BlockToMarkCorrupt(block, oldGenerationStamp, + "genstamp does not match " + oldGenerationStamp + + " : " + block.getGenerationStamp(), Reason.GENSTAMP_MISMATCH); + } else if (block.getNumBytes() != oldNumBytes) { + b = new BlockToMarkCorrupt(block, + "length does not match " + oldNumBytes + + " : " + block.getNumBytes(), Reason.SIZE_MISMATCH); + } else { + return; + } + + for (DatanodeStorageInfo storage : getStorages(block)) { + boolean isCorrupt = true; + if (newStorages != null) { + for (DatanodeStorageInfo newStorage : newStorages) { + if (newStorage!= null && storage.equals(newStorage)) { + isCorrupt = false; + break; + } + } + } + if (isCorrupt) { + blockLog.info("BLOCK* markBlockReplicasAsCorrupt: mark block replica" + + " {} on {} as corrupt because the dn is not in the new committed " + + "storage list.", b, storage.getDatanodeDescriptor()); + markBlockAsCorrupt(b, storage, storage.getDatanodeDescriptor()); + } + } + } + /** * processFirstBlockReport is intended only for processing "initial" block * reports, the first block report received from a DN after it registers. @@ -1968,11 +2057,9 @@ private void processFirstBlockReport( if (report == null) return; assert (namesystem.hasWriteLock()); assert (storageInfo.numBlocks() == 0); - BlockReportIterator itBR = report.getBlockReportIterator(); - while(itBR.hasNext()) { - Block iblk = itBR.next(); - ReplicaState reportedState = itBR.getCurrentReplicaState(); + for (BlockReportReplica iblk : report) { + ReplicaState reportedState = iblk.getState(); if (shouldPostponeBlocksFromFuture && namesystem.isGenStampInFuture(iblk)) { @@ -2042,13 +2129,11 @@ private void reportDiff(DatanodeStorageInfo storageInfo, int curIndex; if (newReport == null) { - newReport = new BlockListAsLongs(); + newReport = BlockListAsLongs.EMPTY; } // scan the report and process newly reported blocks - BlockReportIterator itBR = newReport.getBlockReportIterator(); - while(itBR.hasNext()) { - Block iblk = itBR.next(); - ReplicaState iState = itBR.getCurrentReplicaState(); + for (BlockReportReplica iblk : newReport) { + ReplicaState iState = iblk.getState(); BlockInfoContiguous storedBlock = processReportedBlock(storageInfo, iblk, iState, toAdd, toInvalidate, toCorrupt, toUC); @@ -2441,9 +2526,6 @@ private Block addStoredBlock(final BlockInfoContiguous block, } } else if (result == AddBlockResult.REPLACED) { curReplicaDelta = 0; - blockLog.warn("BLOCK* addStoredBlock: " + "block " + storedBlock - + " moved to storageType " + storageInfo.getStorageType() - + " on node " + node); blockLog.warn("BLOCK* addStoredBlock: block {} moved to storageType " + "{} on node {}", storedBlock, storageInfo.getStorageType(), node); } else { @@ -2617,7 +2699,7 @@ private void stopReplicationInitializer() { private void processMisReplicatesAsync() throws InterruptedException { long nrInvalid = 0, nrOverReplicated = 0; long nrUnderReplicated = 0, nrPostponed = 0, nrUnderConstruction = 0; - long startTimeMisReplicatedScan = Time.now(); + long startTimeMisReplicatedScan = Time.monotonicNow(); Iterator blocksItr = blocksMap.getBlocks().iterator(); long totalBlocks = blocksMap.size(); replicationQueuesInitProgress = 0; @@ -2675,7 +2757,8 @@ private void processMisReplicatesAsync() throws InterruptedException { NameNode.stateChangeLog .info("STATE* Replication Queue initialization " + "scan for invalid, over- and under-replicated blocks " - + "completed in " + (Time.now() - startTimeMisReplicatedScan) + + "completed in " + + (Time.monotonicNow() - startTimeMisReplicatedScan) + " msec"); break; } @@ -3188,28 +3271,6 @@ int countLiveNodes(BlockInfoContiguous b) { } return live; } - - private void logBlockReplicationInfo(Block block, DatanodeDescriptor srcNode, - NumberReplicas num) { - int curReplicas = num.liveReplicas(); - int curExpectedReplicas = getReplication(block); - BlockCollection bc = blocksMap.getBlockCollection(block); - StringBuilder nodeList = new StringBuilder(); - for(DatanodeStorageInfo storage : blocksMap.getStorages(block)) { - final DatanodeDescriptor node = storage.getDatanodeDescriptor(); - nodeList.append(node); - nodeList.append(" "); - } - LOG.info("Block: " + block + ", Expected Replicas: " - + curExpectedReplicas + ", live replicas: " + curReplicas - + ", corrupt replicas: " + num.corruptReplicas() - + ", decommissioned replicas: " + num.decommissionedReplicas() - + ", excess replicas: " + num.excessReplicas() - + ", Is Open File: " + bc.isUnderConstruction() - + ", Datanodes having this block: " + nodeList + ", Current Datanode: " - + srcNode + ", Is current datanode decommissioning: " - + srcNode.isDecommissionInProgress()); - } /** * On stopping decommission, check if the node has excess replicas. @@ -3240,89 +3301,30 @@ void processOverReplicatedBlocksOnReCommission( } /** - * Return true if there are any blocks on this node that have not - * yet reached their replication factor. Otherwise returns false. + * Returns whether a node can be safely decommissioned based on its + * liveness. Dead nodes cannot always be safely decommissioned. */ - boolean isReplicationInProgress(DatanodeDescriptor srcNode) { - boolean status = false; - boolean firstReplicationLog = true; - int underReplicatedBlocks = 0; - int decommissionOnlyReplicas = 0; - int underReplicatedInOpenFiles = 0; - final Iterator it = srcNode.getBlockIterator(); - while(it.hasNext()) { - final Block block = it.next(); - BlockCollection bc = blocksMap.getBlockCollection(block); - - if (bc != null) { - NumberReplicas num = countNodes(block); - int curReplicas = num.liveReplicas(); - int curExpectedReplicas = getReplication(block); - - if (isNeededReplication(block, curExpectedReplicas, curReplicas)) { - if (curExpectedReplicas > curReplicas) { - if (bc.isUnderConstruction()) { - if (block.equals(bc.getLastBlock()) && curReplicas > minReplication) { - continue; - } - underReplicatedInOpenFiles++; - } - - // Log info about one block for this node which needs replication - if (!status) { - status = true; - if (firstReplicationLog) { - logBlockReplicationInfo(block, srcNode, num); - } - // Allowing decommission as long as default replication is met - if (curReplicas >= defaultReplication) { - status = false; - firstReplicationLog = false; - } - } - underReplicatedBlocks++; - if ((curReplicas == 0) && (num.decommissionedReplicas() > 0)) { - decommissionOnlyReplicas++; - } - } - if (!neededReplications.contains(block) && - pendingReplications.getNumReplicas(block) == 0 && - namesystem.isPopulatingReplQueues()) { - // - // These blocks have been reported from the datanode - // after the startDecommission method has been executed. These - // blocks were in flight when the decommissioning was started. - // Process these blocks only when active NN is out of safe mode. - // - neededReplications.add(block, - curReplicas, - num.decommissionedReplicas(), - curExpectedReplicas); - } - } - } + boolean isNodeHealthyForDecommission(DatanodeDescriptor node) { + if (node.isAlive) { + return true; } - if (!status && !srcNode.isAlive) { - updateState(); - if (pendingReplicationBlocksCount == 0 && - underReplicatedBlocksCount == 0) { - LOG.info("srcNode {} is dead and there are no under-replicated" + - " blocks or blocks pending replication. Marking as " + - "decommissioned."); - } else { - LOG.warn("srcNode " + srcNode + " is dead " + - "while decommission is in progress. Continuing to mark " + - "it as decommission in progress so when it rejoins the " + - "cluster it can continue the decommission process."); - status = true; - } + updateState(); + if (pendingReplicationBlocksCount == 0 && + underReplicatedBlocksCount == 0) { + LOG.info("Node {} is dead and there are no under-replicated" + + " blocks or blocks pending replication. Safe to decommission.", + node); + return true; } - srcNode.decommissioningStatus.set(underReplicatedBlocks, - decommissionOnlyReplicas, - underReplicatedInOpenFiles); - return status; + LOG.warn("Node {} is dead " + + "while decommission is in progress. Cannot be safely " + + "decommissioned since there is risk of reduced " + + "data durability or data loss. Either restart the failed node or" + + " force decommissioning by removing, calling refreshNodes, " + + "then re-adding to the excludes files.", node); + return false; } public int getActiveBlockCount() { @@ -3349,8 +3351,7 @@ public void removeBlock(Block block) { // file already removes them from the block map below. block.setNumBytes(BlockCommand.NO_ACK); addToInvalidates(block); - corruptReplicas.removeFromCorruptReplicasMap(block); - blocksMap.removeBlock(block); + removeBlockFromMap(block); // Remove the block from pendingReplications and neededReplications pendingReplications.remove(block); neededReplications.remove(block, UnderReplicatedBlocks.LEVEL); @@ -3493,7 +3494,7 @@ boolean blockHasEnoughRacks(Block b) { * A block needs replication if the number of replicas is less than expected * or if it does not have enough racks. */ - private boolean isNeededReplication(Block b, int expected, int current) { + boolean isNeededReplication(Block b, int expected, int current) { return current < expected || !blockHasEnoughRacks(b); } @@ -3526,11 +3527,30 @@ public int numCorruptReplicas(Block block) { } public void removeBlockFromMap(Block block) { + removeFromExcessReplicateMap(block); blocksMap.removeBlock(block); // If block is removed from blocksMap remove it from corruptReplicasMap corruptReplicas.removeFromCorruptReplicasMap(block); } + /** + * If a block is removed from blocksMap, remove it from excessReplicateMap. + */ + private void removeFromExcessReplicateMap(Block block) { + for (DatanodeStorageInfo info : blocksMap.getStorages(block)) { + String uuid = info.getDatanodeDescriptor().getDatanodeUuid(); + LightWeightLinkedSet excessReplicas = excessReplicateMap.get(uuid); + if (excessReplicas != null) { + if (excessReplicas.remove(block)) { + excessBlocksCount.decrementAndGet(); + if (excessReplicas.isEmpty()) { + excessReplicateMap.remove(uuid); + } + } + } + } + } + public int getCapacity() { return blocksMap.getCapacity(); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java index cb17596173e52..3262772613834 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdfs.server.blockmanagement; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.util.*; @@ -884,7 +884,7 @@ public DatanodeStorageInfo chooseReplicaToDelete(BlockCollection bc, Collection second, final List excessTypes) { long oldestHeartbeat = - now() - heartbeatInterval * tolerateHeartbeatMultiplier; + monotonicNow() - heartbeatInterval * tolerateHeartbeatMultiplier; DatanodeStorageInfo oldestHeartbeatStorage = null; long minSpace = Long.MAX_VALUE; DatanodeStorageInfo minSpaceStorage = null; @@ -898,8 +898,8 @@ public DatanodeStorageInfo chooseReplicaToDelete(BlockCollection bc, final DatanodeDescriptor node = storage.getDatanodeDescriptor(); long free = node.getRemaining(); - long lastHeartbeat = node.getLastUpdate(); - if(lastHeartbeat < oldestHeartbeat) { + long lastHeartbeat = node.getLastUpdateMonotonic(); + if (lastHeartbeat < oldestHeartbeat) { oldestHeartbeat = lastHeartbeat; oldestHeartbeatStorage = storage; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java index c0a17b1c9525f..4731ad44c3116 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdfs.server.blockmanagement; import java.util.ArrayList; +import java.util.BitSet; import java.util.Collection; import java.util.Collections; import java.util.HashMap; @@ -31,6 +32,7 @@ import com.google.common.annotations.VisibleForTesting; +import com.google.common.collect.ImmutableList; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience; @@ -40,6 +42,7 @@ import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.server.namenode.CachedBlock; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; import org.apache.hadoop.hdfs.server.protocol.StorageReport; import org.apache.hadoop.hdfs.server.protocol.VolumeFailureSummary; @@ -62,7 +65,25 @@ public class DatanodeDescriptor extends DatanodeInfo { // Stores status of decommissioning. // If node is not decommissioning, do not use this object for anything. public final DecommissioningStatus decommissioningStatus = new DecommissioningStatus(); - + + private long curBlockReportId = 0; + + private BitSet curBlockReportRpcsSeen = null; + + public int updateBlockReportContext(BlockReportContext context) { + if (curBlockReportId != context.getReportId()) { + curBlockReportId = context.getReportId(); + curBlockReportRpcsSeen = new BitSet(context.getTotalRpcs()); + } + curBlockReportRpcsSeen.set(context.getCurRpc()); + return curBlockReportRpcsSeen.cardinality(); + } + + public void clearBlockReportContext() { + curBlockReportId = 0; + curBlockReportRpcsSeen = null; + } + /** Block and targets pair */ @InterfaceAudience.Private @InterfaceStability.Evolving @@ -282,6 +303,34 @@ boolean hasStaleStorages() { } } + static final private List EMPTY_STORAGE_INFO_LIST = + ImmutableList.of(); + + List removeZombieStorages() { + List zombies = null; + synchronized (storageMap) { + Iterator> iter = + storageMap.entrySet().iterator(); + while (iter.hasNext()) { + Map.Entry entry = iter.next(); + DatanodeStorageInfo storageInfo = entry.getValue(); + if (storageInfo.getLastBlockReportId() != curBlockReportId) { + LOG.info(storageInfo.getStorageID() + " had lastBlockReportId 0x" + + Long.toHexString(storageInfo.getLastBlockReportId()) + + ", but curBlockReportId = 0x" + + Long.toHexString(curBlockReportId)); + iter.remove(); + if (zombies == null) { + zombies = new LinkedList(); + } + zombies.add(storageInfo); + } + storageInfo.setLastBlockReportId(0); + } + } + return zombies == null ? EMPTY_STORAGE_INFO_LIST : zombies; + } + /** * Remove block from the list of blocks belonging to the data-node. Remove * data-node from the block. @@ -398,14 +447,17 @@ public void updateHeartbeatState(StorageReport[] reports, long cacheCapacity, if (checkFailedStorages) { LOG.info("Number of failed storage changes from " + this.volumeFailures + " to " + volFailures); - failedStorageInfos = new HashSet( - storageMap.values()); + synchronized (storageMap) { + failedStorageInfos = + new HashSet(storageMap.values()); + } } setCacheCapacity(cacheCapacity); setCacheUsed(cacheUsed); setXceiverCount(xceiverCount); - setLastUpdate(Time.now()); + setLastUpdate(Time.now()); + setLastUpdateMonotonic(Time.monotonicNow()); this.volumeFailures = volFailures; this.volumeFailureSummary = volumeFailureSummary; for (StorageReport report : reports) { @@ -420,7 +472,7 @@ public void updateHeartbeatState(StorageReport[] reports, long cacheCapacity, totalBlockPoolUsed += report.getBlockPoolUsed(); totalDfsUsed += report.getDfsUsed(); } - rollBlocksScheduled(getLastUpdate()); + rollBlocksScheduled(getLastUpdateMonotonic()); // Update total metrics for the node. setCapacity(totalCapacity); @@ -430,8 +482,11 @@ public void updateHeartbeatState(StorageReport[] reports, long cacheCapacity, if (checkFailedStorages) { updateFailedStorage(failedStorageInfos); } - - if (storageMap.size() != reports.length) { + long storageMapSize; + synchronized (storageMap) { + storageMapSize = storageMap.size(); + } + if (storageMapSize != reports.length) { pruneStorageMap(reports); } } @@ -441,14 +496,14 @@ public void updateHeartbeatState(StorageReport[] reports, long cacheCapacity, * as long as they have associated block replicas. */ private void pruneStorageMap(final StorageReport[] reports) { - if (LOG.isDebugEnabled()) { - LOG.debug("Number of storages reported in heartbeat=" + reports.length + - "; Number of storages in storageMap=" + storageMap.size()); - } + synchronized (storageMap) { + if (LOG.isDebugEnabled()) { + LOG.debug("Number of storages reported in heartbeat=" + reports.length + + "; Number of storages in storageMap=" + storageMap.size()); + } - HashMap excessStorages; + HashMap excessStorages; - synchronized (storageMap) { // Init excessStorages with all known storages. excessStorages = new HashMap(storageMap); @@ -465,8 +520,8 @@ private void pruneStorageMap(final StorageReport[] reports) { LOG.info("Removed storage " + storageInfo + " from DataNode" + this); } else if (LOG.isDebugEnabled()) { // This can occur until all block reports are received. - LOG.debug("Deferring removal of stale storage " + storageInfo + - " with " + storageInfo.numBlocks() + " blocks"); + LOG.debug("Deferring removal of stale storage " + storageInfo + + " with " + storageInfo.numBlocks() + " blocks"); } } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java index 45c56a8b5fdd3..f68c4fd275499 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java @@ -17,6 +17,8 @@ */ package org.apache.hadoop.hdfs.server.blockmanagement; +import static org.apache.hadoop.util.Time.monotonicNow; + import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.net.InetAddresses; @@ -42,9 +44,7 @@ import org.apache.hadoop.ipc.Server; import org.apache.hadoop.net.*; import org.apache.hadoop.net.NetworkTopology.InvalidTopologyException; -import org.apache.hadoop.util.Daemon; import org.apache.hadoop.util.ReflectionUtils; -import org.apache.hadoop.util.Time; import java.io.IOException; import java.io.PrintWriter; @@ -53,8 +53,6 @@ import java.net.UnknownHostException; import java.util.*; -import static org.apache.hadoop.util.Time.now; - /** * Manage datanodes, include decommission and other activities. */ @@ -65,9 +63,9 @@ public class DatanodeManager { private final Namesystem namesystem; private final BlockManager blockManager; + private final DecommissionManager decomManager; private final HeartbeatManager heartbeatManager; private final FSClusterStats fsClusterStats; - private Daemon decommissionthread = null; /** * Stores the datanode -> block map. @@ -110,7 +108,7 @@ public class DatanodeManager { private final HostFileManager hostFileManager = new HostFileManager(); /** The period to wait for datanode heartbeat.*/ - private final long heartbeatExpireInterval; + private long heartbeatExpireInterval; /** Ask Datanode only up to this many blocks to delete. */ final int blockInvalidateLimit; @@ -184,6 +182,8 @@ public class DatanodeManager { networktopology = NetworkTopology.getInstance(conf); this.heartbeatManager = new HeartbeatManager(namesystem, blockManager, conf); + this.decomManager = new DecommissionManager(namesystem, blockManager, + heartbeatManager); this.fsClusterStats = newFSClusterStats(); this.defaultXferPort = NetUtils.createSocketAddr( @@ -307,25 +307,12 @@ private static long getStaleIntervalFromConf(Configuration conf, } void activate(final Configuration conf) { - final DecommissionManager dm = new DecommissionManager(namesystem, blockManager); - this.decommissionthread = new Daemon(dm.new Monitor( - conf.getInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, - DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_DEFAULT), - conf.getInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_NODES_PER_INTERVAL_KEY, - DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_NODES_PER_INTERVAL_DEFAULT))); - decommissionthread.start(); - + decomManager.activate(conf); heartbeatManager.activate(conf); } void close() { - if (decommissionthread != null) { - decommissionthread.interrupt(); - try { - decommissionthread.join(3000); - } catch (InterruptedException e) { - } - } + decomManager.close(); heartbeatManager.close(); } @@ -339,6 +326,20 @@ HeartbeatManager getHeartbeatManager() { return heartbeatManager; } + @VisibleForTesting + public DecommissionManager getDecomManager() { + return decomManager; + } + + HostFileManager getHostFileManager() { + return hostFileManager; + } + + @VisibleForTesting + public void setHeartbeatExpireInterval(long expiryMs) { + this.heartbeatExpireInterval = expiryMs; + } + @VisibleForTesting public FSClusterStats getFSClusterStats() { return fsClusterStats; @@ -581,8 +582,8 @@ void removeDeadDatanode(final DatanodeID nodeID) { /** Is the datanode dead? */ boolean isDatanodeDead(DatanodeDescriptor node) { - return (node.getLastUpdate() < - (Time.now() - heartbeatExpireInterval)); + return (node.getLastUpdateMonotonic() < + (monotonicNow() - heartbeatExpireInterval)); } /** Add a datanode. */ @@ -826,63 +827,14 @@ private void removeDecomNodeFromList(final List nodeList) { } /** - * Decommission the node if it is in exclude list. + * Decommission the node if it is in the host exclude list. + * + * @param nodeReg datanode */ - private void checkDecommissioning(DatanodeDescriptor nodeReg) { + void startDecommissioningIfExcluded(DatanodeDescriptor nodeReg) { // If the registered node is in exclude list, then decommission it - if (hostFileManager.isExcluded(nodeReg)) { - startDecommission(nodeReg); - } - } - - /** - * Change, if appropriate, the admin state of a datanode to - * decommission completed. Return true if decommission is complete. - */ - boolean checkDecommissionState(DatanodeDescriptor node) { - // Check to see if all blocks in this decommissioned - // node has reached their target replication factor. - if (node.isDecommissionInProgress() && node.checkBlockReportReceived()) { - if (!blockManager.isReplicationInProgress(node)) { - node.setDecommissioned(); - LOG.info("Decommission complete for " + node); - } - } - return node.isDecommissioned(); - } - - /** Start decommissioning the specified datanode. */ - @InterfaceAudience.Private - @VisibleForTesting - public void startDecommission(DatanodeDescriptor node) { - if (!node.isDecommissionInProgress()) { - if (!node.isAlive) { - LOG.info("Dead node " + node + " is decommissioned immediately."); - node.setDecommissioned(); - } else if (!node.isDecommissioned()) { - for (DatanodeStorageInfo storage : node.getStorageInfos()) { - LOG.info("Start Decommissioning " + node + " " + storage - + " with " + storage.numBlocks() + " blocks"); - } - heartbeatManager.startDecommission(node); - node.decommissioningStatus.setStartTime(now()); - - // all the blocks that reside on this node have to be replicated. - checkDecommissionState(node); - } - } - } - - /** Stop decommissioning the specified datanodes. */ - void stopDecommission(DatanodeDescriptor node) { - if (node.isDecommissionInProgress() || node.isDecommissioned()) { - LOG.info("Stop Decommissioning " + node); - heartbeatManager.stopDecommission(node); - // Over-replicated blocks will be detected and processed when - // the dead node comes back and send in its full block report. - if (node.isAlive) { - blockManager.processOverReplicatedBlocksOnReCommission(node); - } + if (getHostFileManager().isExcluded(nodeReg)) { + decomManager.startDecommission(nodeReg); } } @@ -993,7 +945,7 @@ nodes with its data cleared (or user can just remove the StorageID // also treat the registration message as a heartbeat heartbeatManager.register(nodeS); incrementVersionCount(nodeS.getSoftwareVersion()); - checkDecommissioning(nodeS); + startDecommissioningIfExcluded(nodeS); success = true; } finally { if (!success) { @@ -1029,7 +981,7 @@ nodes with its data cleared (or user can just remove the StorageID // because its is done when the descriptor is created heartbeatManager.addDatanode(nodeDescr); incrementVersionCount(nodeReg.getSoftwareVersion()); - checkDecommissioning(nodeDescr); + startDecommissioningIfExcluded(nodeDescr); success = true; } finally { if (!success) { @@ -1092,9 +1044,9 @@ private void refreshDatanodes() { node.setDisallowed(true); // case 2. } else { if (hostFileManager.isExcluded(node)) { - startDecommission(node); // case 3. + decomManager.startDecommission(node); // case 3. } else { - stopDecommission(node); // case 4. + decomManager.stopDecommission(node); // case 4. } } } @@ -1348,7 +1300,7 @@ public List getDatanodeListForReport( .getAddress().getHostAddress(), addr.getHostName(), "", addr.getPort() == 0 ? defaultXferPort : addr.getPort(), defaultInfoPort, defaultInfoSecurePort, defaultIpcPort)); - dn.setLastUpdate(0); // Consider this node dead for reporting + setDatanodeDead(dn); nodes.add(dn); } } @@ -1381,6 +1333,7 @@ private static boolean isNameResolved(InetAddress address) { private void setDatanodeDead(DatanodeDescriptor node) { node.setLastUpdate(0); + node.setLastUpdateMonotonic(0); } /** Handle heartbeat from datanodes. */ @@ -1486,7 +1439,7 @@ public DatanodeCommand[] handleHeartbeat(DatanodeRegistration nodeReg, blockPoolId, blks)); } boolean sendingCachingCommands = false; - long nowMs = Time.monotonicNow(); + long nowMs = monotonicNow(); if (shouldSendCachingCommands && ((nowMs - nodeinfo.getLastCachingDirectiveSentTimeMs()) >= timeBetweenResendingCachingDirectivesMs)) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java index c4612a325ba0c..be16a8731490d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java @@ -115,6 +115,9 @@ public void remove() { private volatile BlockInfoContiguous blockList = null; private int numBlocks = 0; + // The ID of the last full block report which updated this storage. + private long lastBlockReportId = 0; + /** The number of block reports received */ private int blockReportCount = 0; @@ -178,7 +181,15 @@ public void setUtilizationForTesting(long capacity, long dfsUsed, this.remaining = remaining; this.blockPoolUsed = blockPoolUsed; } - + + long getLastBlockReportId() { + return lastBlockReportId; + } + + void setLastBlockReportId(long lastBlockReportId) { + this.lastBlockReportId = lastBlockReportId; + } + State getState() { return this.state; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java index a234cf545fcc2..9355329637f65 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java @@ -17,88 +17,605 @@ */ package org.apache.hadoop.hdfs.server.blockmanagement; +import java.util.AbstractList; +import java.util.Iterator; +import java.util.LinkedList; +import java.util.List; import java.util.Map; +import java.util.Queue; +import java.util.TreeMap; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; +import com.google.common.util.concurrent.ThreadFactoryBuilder; import org.apache.hadoop.classification.InterfaceAudience; -import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.hdfs.protocol.Block; +import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.server.namenode.Namesystem; +import org.apache.hadoop.hdfs.util.CyclicIteration; +import org.apache.hadoop.util.ChunkedArrayList; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static com.google.common.base.Preconditions.checkArgument; +import static org.apache.hadoop.util.Time.monotonicNow; /** - * Manage node decommissioning. + * Manages datanode decommissioning. A background monitor thread + * periodically checks the status of datanodes that are in-progress of + * decommissioning. + *

+ * A datanode can be decommissioned in a few situations: + *

    + *
  • If a DN is dead, it is decommissioned immediately.
    • + *
  • If a DN is alive, it is decommissioned after all of its blocks + * are sufficiently replicated. Merely under-replicated blocks do not + * block decommissioning as long as they are above a replication + * threshold.
    • + *
+ * In the second case, the datanode transitions to a + * decommission-in-progress state and is tracked by the monitor thread. The + * monitor periodically scans through the list of insufficiently replicated + * blocks on these datanodes to + * determine if they can be decommissioned. The monitor also prunes this list + * as blocks become replicated, so monitor scans will become more efficient + * over time. + *

+ * Decommission-in-progress nodes that become dead do not progress to + * decommissioned until they become live again. This prevents potential + * durability loss for singly-replicated blocks (see HDFS-6791). + *

+ * This class depends on the FSNamesystem lock for synchronization. */ @InterfaceAudience.Private -@InterfaceStability.Evolving -class DecommissionManager { - static final Log LOG = LogFactory.getLog(DecommissionManager.class); +public class DecommissionManager { + private static final Logger LOG = LoggerFactory.getLogger(DecommissionManager + .class); private final Namesystem namesystem; - private final BlockManager blockmanager; + private final BlockManager blockManager; + private final HeartbeatManager hbManager; + private final ScheduledExecutorService executor; + + /** + * Map containing the decommission-in-progress datanodes that are being + * tracked so they can be be marked as decommissioned. + *

+ * This holds a set of references to the under-replicated blocks on the DN at + * the time the DN is added to the map, i.e. the blocks that are preventing + * the node from being marked as decommissioned. During a monitor tick, this + * list is pruned as blocks becomes replicated. + *

+ * Note also that the reference to the list of under-replicated blocks + * will be null on initial add + *

+ * However, this map can become out-of-date since it is not updated by block + * reports or other events. Before being finally marking as decommissioned, + * another check is done with the actual block map. + */ + private final TreeMap> + decomNodeBlocks; + + /** + * Tracking a node in decomNodeBlocks consumes additional memory. To limit + * the impact on NN memory consumption, we limit the number of nodes in + * decomNodeBlocks. Additional nodes wait in pendingNodes. + */ + private final Queue pendingNodes; + + private Monitor monitor = null; DecommissionManager(final Namesystem namesystem, - final BlockManager blockmanager) { + final BlockManager blockManager, final HeartbeatManager hbManager) { this.namesystem = namesystem; - this.blockmanager = blockmanager; + this.blockManager = blockManager; + this.hbManager = hbManager; + + executor = Executors.newScheduledThreadPool(1, + new ThreadFactoryBuilder().setNameFormat("DecommissionMonitor-%d") + .setDaemon(true).build()); + decomNodeBlocks = new TreeMap<>(); + pendingNodes = new LinkedList<>(); + } + + /** + * Start the decommission monitor thread. + * @param conf + */ + void activate(Configuration conf) { + final int intervalSecs = + conf.getInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_DEFAULT); + checkArgument(intervalSecs >= 0, "Cannot set a negative " + + "value for " + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY); + + // By default, the new configuration key overrides the deprecated one. + // No # node limit is set. + int blocksPerInterval = conf.getInt( + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY, + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_DEFAULT); + int nodesPerInterval = Integer.MAX_VALUE; + + // If the expected key isn't present and the deprecated one is, + // use the deprecated one into the new one. This overrides the + // default. + // + // Also print a deprecation warning. + final String deprecatedKey = + "dfs.namenode.decommission.nodes.per.interval"; + final String strNodes = conf.get(deprecatedKey); + if (strNodes != null) { + nodesPerInterval = Integer.parseInt(strNodes); + blocksPerInterval = Integer.MAX_VALUE; + LOG.warn("Using deprecated configuration key {} value of {}.", + deprecatedKey, nodesPerInterval); + LOG.warn("Please update your configuration to use {} instead.", + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY); + } + checkArgument(blocksPerInterval > 0, + "Must set a positive value for " + + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY); + + final int maxConcurrentTrackedNodes = conf.getInt( + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES, + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES_DEFAULT); + checkArgument(maxConcurrentTrackedNodes >= 0, "Cannot set a negative " + + "value for " + + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES); + + monitor = new Monitor(blocksPerInterval, + nodesPerInterval, maxConcurrentTrackedNodes); + executor.scheduleAtFixedRate(monitor, intervalSecs, intervalSecs, + TimeUnit.SECONDS); + + LOG.debug("Activating DecommissionManager with interval {} seconds, " + + "{} max blocks per interval, {} max nodes per interval, " + + "{} max concurrently tracked nodes.", intervalSecs, + blocksPerInterval, nodesPerInterval, maxConcurrentTrackedNodes); + } + + /** + * Stop the decommission monitor thread, waiting briefly for it to terminate. + */ + void close() { + executor.shutdownNow(); + try { + executor.awaitTermination(3000, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) {} + } + + /** + * Start decommissioning the specified datanode. + * @param node + */ + @VisibleForTesting + public void startDecommission(DatanodeDescriptor node) { + if (!node.isDecommissionInProgress()) { + if (!node.isAlive) { + LOG.info("Dead node {} is decommissioned immediately.", node); + node.setDecommissioned(); + } else if (!node.isDecommissioned()) { + for (DatanodeStorageInfo storage : node.getStorageInfos()) { + LOG.info("Starting decommission of {} {} with {} blocks", + node, storage, storage.numBlocks()); + } + // Update DN stats maintained by HeartbeatManager + hbManager.startDecommission(node); + node.decommissioningStatus.setStartTime(monotonicNow()); + pendingNodes.add(node); + } + } else { + LOG.trace("startDecommission: Node {} is already decommission in " + + "progress, nothing to do.", node); + } + } + + /** + * Stop decommissioning the specified datanode. + * @param node + */ + void stopDecommission(DatanodeDescriptor node) { + if (node.isDecommissionInProgress() || node.isDecommissioned()) { + LOG.info("Stopping decommissioning of node {}", node); + // Update DN stats maintained by HeartbeatManager + hbManager.stopDecommission(node); + // Over-replicated blocks will be detected and processed when + // the dead node comes back and send in its full block report. + if (node.isAlive) { + blockManager.processOverReplicatedBlocksOnReCommission(node); + } + // Remove from tracking in DecommissionManager + pendingNodes.remove(node); + decomNodeBlocks.remove(node); + } else { + LOG.trace("stopDecommission: Node {} is not decommission in progress " + + "or decommissioned, nothing to do.", node); + } + } + + private void setDecommissioned(DatanodeDescriptor dn) { + dn.setDecommissioned(); + LOG.info("Decommissioning complete for node {}", dn); } - /** Periodically check decommission status. */ - class Monitor implements Runnable { - /** recheckInterval is how often namenode checks - * if a node has finished decommission + /** + * Checks whether a block is sufficiently replicated for decommissioning. + * Full-strength replication is not always necessary, hence "sufficient". + * @return true if sufficient, else false. + */ + private boolean isSufficientlyReplicated(BlockInfoContiguous block, + BlockCollection bc, + NumberReplicas numberReplicas) { + final int numExpected = bc.getBlockReplication(); + final int numLive = numberReplicas.liveReplicas(); + if (!blockManager.isNeededReplication(block, numExpected, numLive)) { + // Block doesn't need replication. Skip. + LOG.trace("Block {} does not need replication.", block); + return true; + } + + // Block is under-replicated + LOG.trace("Block {} numExpected={}, numLive={}", block, numExpected, + numLive); + if (numExpected > numLive) { + if (bc.isUnderConstruction() && block.equals(bc.getLastBlock())) { + // Can decom a UC block as long as there will still be minReplicas + if (numLive >= blockManager.minReplication) { + LOG.trace("UC block {} sufficiently-replicated since numLive ({}) " + + ">= minR ({})", block, numLive, blockManager.minReplication); + return true; + } else { + LOG.trace("UC block {} insufficiently-replicated since numLive " + + "({}) < minR ({})", block, numLive, + blockManager.minReplication); + } + } else { + // Can decom a non-UC as long as the default replication is met + if (numLive >= blockManager.defaultReplication) { + return true; + } + } + } + return false; + } + + private static void logBlockReplicationInfo(Block block, BlockCollection bc, + DatanodeDescriptor srcNode, NumberReplicas num, + Iterable storages) { + int curReplicas = num.liveReplicas(); + int curExpectedReplicas = bc.getBlockReplication(); + StringBuilder nodeList = new StringBuilder(); + for (DatanodeStorageInfo storage : storages) { + final DatanodeDescriptor node = storage.getDatanodeDescriptor(); + nodeList.append(node); + nodeList.append(" "); + } + LOG.info("Block: " + block + ", Expected Replicas: " + + curExpectedReplicas + ", live replicas: " + curReplicas + + ", corrupt replicas: " + num.corruptReplicas() + + ", decommissioned replicas: " + num.decommissionedReplicas() + + ", excess replicas: " + num.excessReplicas() + + ", Is Open File: " + bc.isUnderConstruction() + + ", Datanodes having this block: " + nodeList + ", Current Datanode: " + + srcNode + ", Is current datanode decommissioning: " + + srcNode.isDecommissionInProgress()); + } + + @VisibleForTesting + public int getNumPendingNodes() { + return pendingNodes.size(); + } + + @VisibleForTesting + public int getNumTrackedNodes() { + return decomNodeBlocks.size(); + } + + @VisibleForTesting + public int getNumNodesChecked() { + return monitor.numNodesChecked; + } + + /** + * Checks to see if DNs have finished decommissioning. + *

+ * Since this is done while holding the namesystem lock, + * the amount of work per monitor tick is limited. + */ + private class Monitor implements Runnable { + /** + * The maximum number of blocks to check per tick. + */ + private final int numBlocksPerCheck; + /** + * The maximum number of nodes to check per tick. */ - private final long recheckInterval; - /** The number of decommission nodes to check for each interval */ private final int numNodesPerCheck; - /** firstkey can be initialized to anything. */ - private String firstkey = ""; + /** + * The maximum number of nodes to track in decomNodeBlocks. A value of 0 + * means no limit. + */ + private final int maxConcurrentTrackedNodes; + /** + * The number of blocks that have been checked on this tick. + */ + private int numBlocksChecked = 0; + /** + * The number of nodes that have been checked on this tick. Used for + * testing. + */ + private int numNodesChecked = 0; + /** + * The last datanode in decomNodeBlocks that we've processed + */ + private DatanodeDescriptor iterkey = new DatanodeDescriptor(new + DatanodeID("", "", "", 0, 0, 0, 0)); - Monitor(int recheckIntervalInSecond, int numNodesPerCheck) { - this.recheckInterval = recheckIntervalInSecond * 1000L; + Monitor(int numBlocksPerCheck, int numNodesPerCheck, int + maxConcurrentTrackedNodes) { + this.numBlocksPerCheck = numBlocksPerCheck; this.numNodesPerCheck = numNodesPerCheck; + this.maxConcurrentTrackedNodes = maxConcurrentTrackedNodes; + } + + private boolean exceededNumBlocksPerCheck() { + LOG.trace("Processed {} blocks so far this tick", numBlocksChecked); + return numBlocksChecked >= numBlocksPerCheck; + } + + @Deprecated + private boolean exceededNumNodesPerCheck() { + LOG.trace("Processed {} nodes so far this tick", numNodesChecked); + return numNodesChecked >= numNodesPerCheck; } - /** - * Check decommission status of numNodesPerCheck nodes - * for every recheckInterval milliseconds. - */ @Override public void run() { - for(; namesystem.isRunning(); ) { - namesystem.writeLock(); - try { - check(); - } finally { - namesystem.writeUnlock(); + if (!namesystem.isRunning()) { + LOG.info("Namesystem is not running, skipping decommissioning checks" + + "."); + return; + } + // Reset the checked count at beginning of each iteration + numBlocksChecked = 0; + numNodesChecked = 0; + // Check decom progress + namesystem.writeLock(); + try { + processPendingNodes(); + check(); + } finally { + namesystem.writeUnlock(); + } + if (numBlocksChecked + numNodesChecked > 0) { + LOG.info("Checked {} blocks and {} nodes this tick", numBlocksChecked, + numNodesChecked); + } + } + + /** + * Pop datanodes off the pending list and into decomNodeBlocks, + * subject to the maxConcurrentTrackedNodes limit. + */ + private void processPendingNodes() { + while (!pendingNodes.isEmpty() && + (maxConcurrentTrackedNodes == 0 || + decomNodeBlocks.size() < maxConcurrentTrackedNodes)) { + decomNodeBlocks.put(pendingNodes.poll(), null); + } + } + + private void check() { + final Iterator>> + it = new CyclicIteration<>(decomNodeBlocks, iterkey).iterator(); + final LinkedList toRemove = new LinkedList<>(); + + while (it.hasNext() + && !exceededNumBlocksPerCheck() + && !exceededNumNodesPerCheck()) { + numNodesChecked++; + final Map.Entry> + entry = it.next(); + final DatanodeDescriptor dn = entry.getKey(); + AbstractList blocks = entry.getValue(); + boolean fullScan = false; + if (blocks == null) { + // This is a newly added datanode, run through its list to schedule + // under-replicated blocks for replication and collect the blocks + // that are insufficiently replicated for further tracking + LOG.debug("Newly-added node {}, doing full scan to find " + + "insufficiently-replicated blocks.", dn); + blocks = handleInsufficientlyReplicated(dn); + decomNodeBlocks.put(dn, blocks); + fullScan = true; + } else { + // This is a known datanode, check if its # of insufficiently + // replicated blocks has dropped to zero and if it can be decommed + LOG.debug("Processing decommission-in-progress node {}", dn); + pruneSufficientlyReplicated(dn, blocks); } - - try { - Thread.sleep(recheckInterval); - } catch (InterruptedException ie) { - LOG.warn(this.getClass().getSimpleName() + " interrupted: " + ie); + if (blocks.size() == 0) { + if (!fullScan) { + // If we didn't just do a full scan, need to re-check with the + // full block map. + // + // We've replicated all the known insufficiently replicated + // blocks. Re-check with the full block map before finally + // marking the datanode as decommissioned + LOG.debug("Node {} has finished replicating current set of " + + "blocks, checking with the full block map.", dn); + blocks = handleInsufficientlyReplicated(dn); + decomNodeBlocks.put(dn, blocks); + } + // If the full scan is clean AND the node liveness is okay, + // we can finally mark as decommissioned. + final boolean isHealthy = + blockManager.isNodeHealthyForDecommission(dn); + if (blocks.size() == 0 && isHealthy) { + setDecommissioned(dn); + toRemove.add(dn); + LOG.debug("Node {} is sufficiently replicated and healthy, " + + "marked as decommissioned.", dn); + } else { + if (LOG.isDebugEnabled()) { + StringBuilder b = new StringBuilder("Node {} "); + if (isHealthy) { + b.append("is "); + } else { + b.append("isn't "); + } + b.append("healthy and still needs to replicate {} more blocks," + + " decommissioning is still in progress."); + LOG.debug(b.toString(), dn, blocks.size()); + } + } + } else { + LOG.debug("Node {} still has {} blocks to replicate " + + "before it is a candidate to finish decommissioning.", + dn, blocks.size()); } + iterkey = dn; + } + // Remove the datanodes that are decommissioned + for (DatanodeDescriptor dn : toRemove) { + Preconditions.checkState(dn.isDecommissioned(), + "Removing a node that is not yet decommissioned!"); + decomNodeBlocks.remove(dn); } } - - private void check() { - final DatanodeManager dm = blockmanager.getDatanodeManager(); - int count = 0; - for(Map.Entry entry - : dm.getDatanodeCyclicIteration(firstkey)) { - final DatanodeDescriptor d = entry.getValue(); - firstkey = entry.getKey(); - - if (d.isDecommissionInProgress()) { - try { - dm.checkDecommissionState(d); - } catch(Exception e) { - LOG.warn("entry=" + entry, e); + + /** + * Removes sufficiently replicated blocks from the block list of a + * datanode. + */ + private void pruneSufficientlyReplicated(final DatanodeDescriptor datanode, + AbstractList blocks) { + processBlocksForDecomInternal(datanode, blocks.iterator(), null, true); + } + + /** + * Returns a list of blocks on a datanode that are insufficiently + * replicated, i.e. are under-replicated enough to prevent decommission. + *

+ * As part of this, it also schedules replication work for + * any under-replicated blocks. + * + * @param datanode + * @return List of insufficiently replicated blocks + */ + private AbstractList handleInsufficientlyReplicated( + final DatanodeDescriptor datanode) { + AbstractList insufficient = new ChunkedArrayList<>(); + processBlocksForDecomInternal(datanode, datanode.getBlockIterator(), + insufficient, false); + return insufficient; + } + + /** + * Used while checking if decommission-in-progress datanodes can be marked + * as decommissioned. Combines shared logic of + * pruneSufficientlyReplicated and handleInsufficientlyReplicated. + * + * @param datanode Datanode + * @param it Iterator over the blocks on the + * datanode + * @param insufficientlyReplicated Return parameter. If it's not null, + * will contain the insufficiently + * replicated-blocks from the list. + * @param pruneSufficientlyReplicated whether to remove sufficiently + * replicated blocks from the iterator + * @return true if there are under-replicated blocks in the provided block + * iterator, else false. + */ + private void processBlocksForDecomInternal( + final DatanodeDescriptor datanode, + final Iterator it, + final List insufficientlyReplicated, + boolean pruneSufficientlyReplicated) { + boolean firstReplicationLog = true; + int underReplicatedBlocks = 0; + int decommissionOnlyReplicas = 0; + int underReplicatedInOpenFiles = 0; + while (it.hasNext()) { + numBlocksChecked++; + final BlockInfoContiguous block = it.next(); + // Remove the block from the list if it's no longer in the block map, + // e.g. the containing file has been deleted + if (blockManager.blocksMap.getStoredBlock(block) == null) { + LOG.trace("Removing unknown block {}", block); + it.remove(); + continue; + } + BlockCollection bc = blockManager.blocksMap.getBlockCollection(block); + if (bc == null) { + // Orphan block, will be invalidated eventually. Skip. + continue; + } + + final NumberReplicas num = blockManager.countNodes(block); + final int liveReplicas = num.liveReplicas(); + final int curReplicas = liveReplicas; + + // Schedule under-replicated blocks for replication if not already + // pending + if (blockManager.isNeededReplication(block, bc.getBlockReplication(), + liveReplicas)) { + if (!blockManager.neededReplications.contains(block) && + blockManager.pendingReplications.getNumReplicas(block) == 0 && + namesystem.isPopulatingReplQueues()) { + // Process these blocks only when active NN is out of safe mode. + blockManager.neededReplications.add(block, + curReplicas, + num.decommissionedReplicas(), + bc.getBlockReplication()); } - if (++count == numNodesPerCheck) { - return; + } + + // Even if the block is under-replicated, + // it doesn't block decommission if it's sufficiently replicated + if (isSufficientlyReplicated(block, bc, num)) { + if (pruneSufficientlyReplicated) { + it.remove(); } + continue; + } + + // We've found an insufficiently replicated block. + if (insufficientlyReplicated != null) { + insufficientlyReplicated.add(block); + } + // Log if this is our first time through + if (firstReplicationLog) { + logBlockReplicationInfo(block, bc, datanode, num, + blockManager.blocksMap.getStorages(block)); + firstReplicationLog = false; + } + // Update various counts + underReplicatedBlocks++; + if (bc.isUnderConstruction()) { + underReplicatedInOpenFiles++; + } + if ((curReplicas == 0) && (num.decommissionedReplicas() > 0)) { + decommissionOnlyReplicas++; } } + + datanode.decommissioningStatus.set(underReplicatedBlocks, + decommissionOnlyReplicas, + underReplicatedInOpenFiles); } } + + @VisibleForTesting + void runMonitor() throws ExecutionException, InterruptedException { + Future f = executor.submit(monitor); + f.get(); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java index d60a39b277def..d2905a29b7aa6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java @@ -353,7 +353,7 @@ private class Monitor implements Runnable { public void run() { while(namesystem.isRunning()) { try { - final long now = Time.now(); + final long now = Time.monotonicNow(); if (lastHeartbeatCheck + heartbeatRecheckInterval < now) { heartbeatCheck(); lastHeartbeatCheck = now; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java index 57c296262514a..796b878c92d41 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdfs.server.blockmanagement; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.PrintWriter; import java.sql.Time; @@ -177,7 +177,7 @@ static class PendingBlockInfo { private final List targets; PendingBlockInfo(DatanodeDescriptor[] targets) { - this.timeStamp = now(); + this.timeStamp = monotonicNow(); this.targets = targets == null ? new ArrayList() : new ArrayList(Arrays.asList(targets)); } @@ -187,7 +187,7 @@ long getTimeStamp() { } void setTimeStamp() { - timeStamp = now(); + timeStamp = monotonicNow(); } void incrementReplicas(DatanodeDescriptor... newTargets) { @@ -234,7 +234,7 @@ void pendingReplicationCheck() { synchronized (pendingReplications) { Iterator> iter = pendingReplications.entrySet().iterator(); - long now = now(); + long now = monotonicNow(); if(LOG.isDebugEnabled()) { LOG.debug("PendingReplicationMonitor checking Q"); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java index e6bd5b289332d..e6f099983d5a9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java @@ -672,7 +672,7 @@ public boolean isShared() { */ public void lock() throws IOException { if (isShared()) { - LOG.info("Locking is disabled"); + LOG.info("Locking is disabled for " + this.root); return; } FileLock newLock = tryLock(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java index 86c881689b43d..da9642adfab37 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java @@ -29,6 +29,7 @@ import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; +import org.apache.hadoop.hdfs.protocol.RollingUpgradeStatus; import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; import org.apache.hadoop.hdfs.server.protocol.*; import org.apache.hadoop.hdfs.server.protocol.ReceivedDeletedBlockInfo.BlockStatus; @@ -470,15 +471,19 @@ List getBPServiceActors() { /** * Signal the current rolling upgrade status as indicated by the NN. - * @param inProgress true if a rolling upgrade is in progress + * @param rollingUpgradeStatus rolling upgrade status */ - void signalRollingUpgrade(boolean inProgress) throws IOException { + void signalRollingUpgrade(RollingUpgradeStatus rollingUpgradeStatus) + throws IOException { + if (rollingUpgradeStatus == null) { + return; + } String bpid = getBlockPoolId(); - if (inProgress) { + if (!rollingUpgradeStatus.isFinalized()) { dn.getFSDataset().enableTrash(bpid); dn.getFSDataset().setRollingUpgradeMarker(bpid); } else { - dn.getFSDataset().restoreTrash(bpid); + dn.getFSDataset().clearTrash(bpid); dn.getFSDataset().clearRollingUpgradeMarker(bpid); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java index ff1ad786d1da4..df582f1846af1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdfs.server.datanode; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.EOFException; import java.io.IOException; @@ -44,6 +44,7 @@ import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; import org.apache.hadoop.hdfs.server.common.IncorrectVersionException; import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; @@ -57,7 +58,6 @@ import org.apache.hadoop.hdfs.server.protocol.VolumeFailureSummary; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.ipc.RemoteException; -import org.apache.hadoop.util.Time; import org.apache.hadoop.util.VersionInfo; import org.apache.hadoop.util.VersionUtil; @@ -82,12 +82,11 @@ class BPServiceActor implements Runnable { final BPOfferService bpos; - // lastBlockReport, lastDeletedReport and lastHeartbeat may be assigned/read + // lastBlockReport and lastHeartbeat may be assigned/read // by testing threads (through BPServiceActor#triggerXXX), while also // assigned/read by the actor thread. Thus they should be declared as volatile // to make sure the "happens-before" consistency. volatile long lastBlockReport = 0; - volatile long lastDeletedReport = 0; boolean resetBlockReportTime = true; @@ -228,7 +227,7 @@ private void connectToNNAndHandshake() throws IOException { bpos.verifyAndSetNamespaceInfo(nsInfo); // Second phase of the handshake with the NN. - register(); + register(nsInfo); } // This is useful to make sure NN gets Heartbeat before Blockreport @@ -249,7 +248,7 @@ void scheduleHeartbeat() { */ void scheduleBlockReport(long delay) { if (delay > 0) { // send BR after random delay - lastBlockReport = Time.now() + lastBlockReport = monotonicNow() - ( dnConf.blockReportInterval - DFSUtil.getRandom().nextInt((int)(delay))); } else { // send at next heartbeat lastBlockReport = lastHeartbeat - dnConf.blockReportInterval; @@ -291,12 +290,14 @@ private void reportReceivedDeletedBlocks() throws IOException { // Send incremental block reports to the Namenode outside the lock boolean success = false; + final long startTime = monotonicNow(); try { bpNamenode.blockReceivedAndDeleted(bpRegistration, bpos.getBlockPoolId(), reports.toArray(new StorageReceivedDeletedBlocks[reports.size()])); success = true; } finally { + dn.getMetrics().addIncrementalBlockReport(monotonicNow() - startTime); if (!success) { synchronized (pendingIncrementalBRperStorage) { for (StorageReceivedDeletedBlocks report : reports) { @@ -415,10 +416,10 @@ void triggerHeartbeatForTests() { @VisibleForTesting void triggerDeletionReportForTests() { synchronized (pendingIncrementalBRperStorage) { - lastDeletedReport = 0; + sendImmediateIBR = true; pendingIncrementalBRperStorage.notifyAll(); - while (lastDeletedReport == 0) { + while (sendImmediateIBR) { try { pendingIncrementalBRperStorage.wait(100); } catch (InterruptedException e) { @@ -433,6 +434,17 @@ boolean hasPendingIBR() { return sendImmediateIBR; } + private long prevBlockReportId = 0; + + private long generateUniqueBlockReportId() { + long id = System.nanoTime(); + if (id <= prevBlockReportId) { + id = prevBlockReportId + 1; + } + prevBlockReportId = id; + return id; + } + /** * Report the list blocks to the Namenode * @return DatanodeCommands returned by the NN. May be null. @@ -440,7 +452,7 @@ boolean hasPendingIBR() { */ List blockReport() throws IOException { // send block report if timer has expired. - final long startTime = now(); + final long startTime = monotonicNow(); if (startTime - lastBlockReport <= dnConf.blockReportInterval) { return null; } @@ -452,9 +464,8 @@ List blockReport() throws IOException { // or we will report an RBW replica after the BlockReport already reports // a FINALIZED one. reportReceivedDeletedBlocks(); - lastDeletedReport = startTime; - long brCreateStartTime = now(); + long brCreateStartTime = monotonicNow(); Map perVolumeBlockLists = dn.getFSDataset().getBlockReports(bpos.getBlockPoolId()); @@ -466,8 +477,7 @@ List blockReport() throws IOException { for(Map.Entry kvPair : perVolumeBlockLists.entrySet()) { BlockListAsLongs blockList = kvPair.getValue(); - reports[i++] = new StorageBlockReport( - kvPair.getKey(), blockList.getBlockListAsLongs()); + reports[i++] = new StorageBlockReport(kvPair.getKey(), blockList); totalBlockCount += blockList.getNumberOfBlocks(); } @@ -475,12 +485,14 @@ List blockReport() throws IOException { int numReportsSent = 0; int numRPCs = 0; boolean success = false; - long brSendStartTime = now(); + long brSendStartTime = monotonicNow(); + long reportId = generateUniqueBlockReportId(); try { if (totalBlockCount < dnConf.blockReportSplitThreshold) { // Below split threshold, send all reports in a single message. DatanodeCommand cmd = bpNamenode.blockReport( - bpRegistration, bpos.getBlockPoolId(), reports); + bpRegistration, bpos.getBlockPoolId(), reports, + new BlockReportContext(1, 0, reportId)); numRPCs = 1; numReportsSent = reports.length; if (cmd != null) { @@ -488,10 +500,11 @@ List blockReport() throws IOException { } } else { // Send one block report per message. - for (StorageBlockReport report : reports) { - StorageBlockReport singleReport[] = { report }; + for (int r = 0; r < reports.length; r++) { + StorageBlockReport singleReport[] = { reports[r] }; DatanodeCommand cmd = bpNamenode.blockReport( - bpRegistration, bpos.getBlockPoolId(), singleReport); + bpRegistration, bpos.getBlockPoolId(), singleReport, + new BlockReportContext(reports.length, r, reportId)); numReportsSent++; numRPCs++; if (cmd != null) { @@ -502,16 +515,17 @@ List blockReport() throws IOException { success = true; } finally { // Log the block report processing stats from Datanode perspective - long brSendCost = now() - brSendStartTime; + long brSendCost = monotonicNow() - brSendStartTime; long brCreateCost = brSendStartTime - brCreateStartTime; dn.getMetrics().addBlockReport(brSendCost); final int nCmds = cmds.size(); LOG.info((success ? "S" : "Uns") + - "uccessfully sent " + numReportsSent + - " of " + reports.length + - " blockreports for " + totalBlockCount + - " total blocks using " + numRPCs + - " RPCs. This took " + brCreateCost + + "uccessfully sent block report 0x" + + Long.toHexString(reportId) + ", containing " + reports.length + + " storage report(s), of which we sent " + numReportsSent + "." + + " The reports had " + totalBlockCount + + " total blocks and used " + numRPCs + + " RPC(s). This took " + brCreateCost + " msec to generate and " + brSendCost + " msecs for RPC and NN processing." + " Got back " + @@ -538,7 +552,7 @@ private void scheduleNextBlockReport(long previousReportStartTime) { * 1) normal like 9:20:18, next report should be at 10:20:14 * 2) unexpected like 11:35:43, next report should be at 12:20:14 */ - lastBlockReport += (now() - lastBlockReport) / + lastBlockReport += (monotonicNow() - lastBlockReport) / dnConf.blockReportInterval * dnConf.blockReportInterval; } } @@ -550,7 +564,7 @@ DatanodeCommand cacheReport() throws IOException { } // send cache report if timer has expired. DatanodeCommand cmd = null; - final long startTime = Time.monotonicNow(); + final long startTime = monotonicNow(); if (startTime - lastCacheReport > dnConf.cacheReportInterval) { if (LOG.isDebugEnabled()) { LOG.debug("Sending cacheReport from service actor: " + this); @@ -559,10 +573,10 @@ DatanodeCommand cacheReport() throws IOException { String bpid = bpos.getBlockPoolId(); List blockIds = dn.getFSDataset().getCacheReport(bpid); - long createTime = Time.monotonicNow(); + long createTime = monotonicNow(); cmd = bpNamenode.cacheReport(bpRegistration, bpid, blockIds); - long sendTime = Time.monotonicNow(); + long sendTime = monotonicNow(); long createCost = createTime - startTime; long sendCost = sendTime - createTime; dn.getMetrics().addCacheReport(sendCost); @@ -648,7 +662,7 @@ private void handleRollingUpgradeStatus(HeartbeatResponse resp) throws IOExcepti " in HeartbeatResponse. Expected " + bpos.getBlockPoolId()); } else { - bpos.signalRollingUpgrade(rollingUpgradeStatus != null); + bpos.signalRollingUpgrade(rollingUpgradeStatus); } } @@ -658,7 +672,6 @@ private void handleRollingUpgradeStatus(HeartbeatResponse resp) throws IOExcepti */ private void offerService() throws Exception { LOG.info("For namenode " + nnAddr + " using" - + " DELETEREPORT_INTERVAL of " + dnConf.deleteReportInterval + " msec " + " BLOCKREPORT_INTERVAL of " + dnConf.blockReportInterval + "msec" + " CACHEREPORT_INTERVAL of " + dnConf.cacheReportInterval + "msec" + " Initial delay: " + dnConf.initialBlockReportDelay + "msec" @@ -669,12 +682,14 @@ private void offerService() throws Exception { // while (shouldRun()) { try { - final long startTime = now(); + final long startTime = monotonicNow(); // // Every so often, send heartbeat or block-report // - if (startTime - lastHeartbeat >= dnConf.heartBeatInterval) { + boolean sendHeartbeat = + startTime - lastHeartbeat >= dnConf.heartBeatInterval; + if (sendHeartbeat) { // // All heartbeat messages include following info: // -- Datanode name @@ -686,7 +701,7 @@ private void offerService() throws Exception { if (!dn.areHeartbeatsDisabledForTests()) { HeartbeatResponse resp = sendHeartBeat(); assert resp != null; - dn.getMetrics().addHeartbeat(now() - startTime); + dn.getMetrics().addHeartbeat(monotonicNow() - startTime); // If the state of this NN has changed (eg STANDBY->ACTIVE) // then let the BPOfferService update itself. @@ -702,10 +717,10 @@ private void offerService() throws Exception { handleRollingUpgradeStatus(resp); } - long startProcessCommands = now(); + long startProcessCommands = monotonicNow(); if (!processCommand(resp.getCommands())) continue; - long endProcessCommands = now(); + long endProcessCommands = monotonicNow(); if (endProcessCommands - startProcessCommands > 2000) { LOG.info("Took " + (endProcessCommands - startProcessCommands) + "ms to process " + resp.getCommands().length @@ -713,10 +728,8 @@ private void offerService() throws Exception { } } } - if (sendImmediateIBR || - (startTime - lastDeletedReport > dnConf.deleteReportInterval)) { + if (sendImmediateIBR || sendHeartbeat) { reportReceivedDeletedBlocks(); - lastDeletedReport = startTime; } List cmds = blockReport(); @@ -730,7 +743,7 @@ private void offerService() throws Exception { // or work arrives, and then iterate again. // long waitTime = dnConf.heartBeatInterval - - (Time.now() - lastHeartbeat); + (monotonicNow() - lastHeartbeat); synchronized(pendingIncrementalBRperStorage) { if (waitTime > 0 && !sendImmediateIBR) { try { @@ -772,10 +785,11 @@ private void offerService() throws Exception { * * issued by the namenode to recognize registered datanodes. * + * @param nsInfo current NamespaceInfo * @see FSNamesystem#registerDatanode(DatanodeRegistration) * @throws IOException */ - void register() throws IOException { + void register(NamespaceInfo nsInfo) throws IOException { // The handshake() phase loaded the block pool storage // off disk - so update the bpRegistration object from that info bpRegistration = bpos.createRegistration(); @@ -786,6 +800,7 @@ void register() throws IOException { try { // Use returned registration from namenode with updated fields bpRegistration = bpNamenode.registerDatanode(bpRegistration); + bpRegistration.setNamespaceInfo(nsInfo); break; } catch(EOFException e) { // namenode might have just restarted LOG.info("Problem connecting to server: " + nnAddr + " :" @@ -913,9 +928,9 @@ void reRegister() throws IOException { if (shouldRun()) { // re-retrieve namespace info to make sure that, if the NN // was restarted, we still match its version (HDFS-2120) - retrieveNamespaceInfo(); + NamespaceInfo nsInfo = retrieveNamespaceInfo(); // and re-register - register(); + register(nsInfo); scheduleHeartbeat(); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolManager.java index 9f389952afea2..28a6cc7edbdae 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolManager.java @@ -20,11 +20,8 @@ import java.io.IOException; import java.net.InetSocketAddress; import java.security.PrivilegedExceptionAction; -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; -import java.util.Map; -import java.util.Set; +import java.util.*; +import java.util.concurrent.CopyOnWriteArrayList; import org.apache.commons.logging.Log; import org.apache.hadoop.classification.InterfaceAudience; @@ -53,7 +50,7 @@ class BlockPoolManager { private final Map bpByBlockPoolId = Maps.newHashMap(); private final List offerServices = - Lists.newArrayList(); + new CopyOnWriteArrayList<>(); private final DataNode dn; @@ -74,12 +71,14 @@ synchronized void addBlockPool(BPOfferService bpos) { } /** - * Returns the array of BPOfferService objects. + * Returns a list of BPOfferService objects. The underlying list + * implementation is a CopyOnWriteArrayList so it can be safely + * iterated while BPOfferServices are being added or removed. + * * Caution: The BPOfferService returned could be shutdown any time. */ - synchronized BPOfferService[] getAllNamenodeThreads() { - BPOfferService[] bposArray = new BPOfferService[offerServices.size()]; - return offerServices.toArray(bposArray); + synchronized List getAllNamenodeThreads() { + return Collections.unmodifiableList(offerServices); } synchronized BPOfferService get(String bpid) { @@ -110,15 +109,13 @@ synchronized void remove(BPOfferService t) { } } - void shutDownAll(BPOfferService[] bposArray) throws InterruptedException { - if (bposArray != null) { - for (BPOfferService bpos : bposArray) { - bpos.stop(); //interrupts the threads - } - //now join - for (BPOfferService bpos : bposArray) { - bpos.join(); - } + void shutDownAll(List bposList) throws InterruptedException { + for (BPOfferService bpos : bposList) { + bpos.stop(); //interrupts the threads + } + //now join + for (BPOfferService bpos : bposList) { + bpos.join(); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java index 4076a8b4c621d..d26a9a591b894 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java @@ -351,7 +351,8 @@ private void doTransition(DataNode datanode, StorageDirectory sd, sd.getPreviousDir() + " and " + getTrashRootDir(sd) + " should not " + " both be present."); doRollback(sd, nsInfo); // rollback if applicable - } else { + } else if (startOpt == StartupOption.ROLLBACK && + !sd.getPreviousDir().exists()) { // Restore all the files in the trash. The restored files are retained // during rolling upgrade rollback. They are deleted during rolling // upgrade downgrade. @@ -378,6 +379,12 @@ private void doTransition(DataNode datanode, StorageDirectory sd, && this.cTime == nsInfo.getCTime()) { return; // regular startup } + if (this.layoutVersion > HdfsConstants.DATANODE_LAYOUT_VERSION) { + int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd)); + LOG.info("Restored " + restored + " block files from trash " + + "before the layout upgrade. These blocks will be moved to " + + "the previous directory during the upgrade"); + } if (this.layoutVersion > HdfsConstants.DATANODE_LAYOUT_VERSION || this.cTime < nsInfo.getCTime()) { doUpgrade(datanode, sd, nsInfo); // upgrade @@ -730,16 +737,12 @@ String getRestoreDirectory(File blockFile) { /** * Delete all files and directories in the trash directories. */ - public void restoreTrash() { + public void clearTrash() { for (StorageDirectory sd : storageDirs) { File trashRoot = getTrashRootDir(sd); - try { - Preconditions.checkState(!(trashRoot.exists() && sd.getPreviousDir().exists())); - restoreBlockFilesFromTrash(trashRoot); - FileUtil.fullyDelete(getTrashRootDir(sd)); - } catch (IOException ioe) { - LOG.warn("Restoring trash failed for storage directory " + sd); - } + Preconditions.checkState(!(trashRoot.exists() && sd.getPreviousDir().exists())); + FileUtil.fullyDelete(trashRoot); + LOG.info("Cleared trash for storage directory " + sd); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java index 0a2b650f981e9..4e8ce94ab2e7c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java @@ -1372,7 +1372,7 @@ private void sendAckUpstreamUnprotected(PipelineAck ack, long seqno, replies = new int[ackLen + 1]; replies[0] = myHeader; for (int i = 0; i < ackLen; ++i) { - replies[i + 1] = ack.getReply(i); + replies[i + 1] = ack.getHeaderFlag(i); } // If the mirror has reported that it received a corrupt packet, // do self-destruct to mark myself bad, instead of making the diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java index f4cde11678adf..e76b93a5fd96a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java @@ -246,6 +246,13 @@ class BlockSender implements java.io.Closeable { if (replica.getGenerationStamp() < block.getGenerationStamp()) { throw new IOException("Replica gen stamp < block genstamp, block=" + block + ", replica=" + replica); + } else if (replica.getGenerationStamp() > block.getGenerationStamp()) { + if (DataNode.LOG.isDebugEnabled()) { + DataNode.LOG.debug("Bumping up the client provided" + + " block's genstamp to latest " + replica.getGenerationStamp() + + " for block " + block); + } + block.setGenerationStamp(replica.getGenerationStamp()); } if (replicaVisibleLength < 0) { throw new IOException("Replica is not readable, block=" diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java index 67cd1ce1aae5b..3406f29f4c4cb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java @@ -82,7 +82,6 @@ public class DNConf { final long heartBeatInterval; final long blockReportInterval; final long blockReportSplitThreshold; - final long deleteReportInterval; final long initialBlockReportDelay; final long cacheReportInterval; final long dfsclientSlowIoWarningThresholdMs; @@ -164,7 +163,6 @@ public DNConf(Configuration conf) { heartBeatInterval = conf.getLong(DFS_HEARTBEAT_INTERVAL_KEY, DFS_HEARTBEAT_INTERVAL_DEFAULT) * 1000L; - this.deleteReportInterval = 100 * heartBeatInterval; // do we need to sync block file contents to disk when blockfile is closed? this.syncOnClose = conf.getBoolean(DFS_DATANODE_SYNCONCLOSE_KEY, DFS_DATANODE_SYNCONCLOSE_DEFAULT); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java index 92ddb7bd9606e..071aba1feb7ea 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java @@ -41,8 +41,6 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_NETWORK_COUNTS_CACHE_MAX_SIZE_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_NETWORK_COUNTS_CACHE_MAX_SIZE_DEFAULT; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_PLUGINS_KEY; -import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_SCAN_PERIOD_HOURS_DEFAULT; -import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_SCAN_PERIOD_HOURS_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_STARTUP_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_MAX_NUM_BLOCKS_TO_LOG_DEFAULT; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_MAX_NUM_BLOCKS_TO_LOG_KEY; @@ -53,6 +51,7 @@ import java.io.DataInputStream; import java.io.DataOutputStream; import java.io.EOFException; +import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; @@ -73,6 +72,7 @@ import java.util.Collections; import java.util.EnumSet; import java.util.HashMap; +import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Map; @@ -157,6 +157,7 @@ import org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl; import org.apache.hadoop.hdfs.server.datanode.metrics.DataNodeMetrics; import org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer; import org.apache.hadoop.hdfs.server.protocol.BlockRecoveryCommand.RecoveringBlock; @@ -613,20 +614,16 @@ public IOException call() { errorMessageBuilder.append( String.format("FAILED to ADD: %s: %s%n", volume, e.toString())); + LOG.error("Failed to add volume: " + volume, e); } } } - if (!changedVolumes.deactivateLocations.isEmpty()) { - LOG.info("Deactivating volumes: " + - Joiner.on(",").join(changedVolumes.deactivateLocations)); - - data.removeVolumes(changedVolumes.deactivateLocations); - try { - storage.removeVolumes(changedVolumes.deactivateLocations); - } catch (IOException e) { - errorMessageBuilder.append(e.getMessage()); - } + try { + removeVolumes(changedVolumes.deactivateLocations); + } catch (IOException e) { + errorMessageBuilder.append(e.getMessage()); + LOG.error("Failed to remove volume: " + e.getMessage(), e); } if (errorMessageBuilder.length() > 0) { @@ -636,6 +633,83 @@ public IOException call() { conf.set(DFS_DATANODE_DATA_DIR_KEY, Joiner.on(",").join(effectiveVolumes)); dataDirs = getStorageLocations(conf); + + // Send a full block report to let NN acknowledge the volume changes. + triggerBlockReport(new BlockReportOptions.Factory() + .setIncremental(false).build()); + } + } + + /** + * Remove volumes from DataNode. + * See {@link removeVolumes(final Set, boolean)} for details. + * + * @param locations the StorageLocations of the volumes to be removed. + * @throws IOException + */ + private void removeVolumes(final Collection locations) + throws IOException { + if (locations.isEmpty()) { + return; + } + Set volumesToRemove = new HashSet<>(); + for (StorageLocation loc : locations) { + volumesToRemove.add(loc.getFile().getAbsoluteFile()); + } + removeVolumes(volumesToRemove, true); + } + + /** + * Remove volumes from DataNode. + * + * It does three things: + *

  • + *
      Remove volumes and block info from FsDataset.
    + *
      Remove volumes from DataStorage.
    + *
      Reset configuration DATA_DIR and {@link dataDirs} to represent + * active volumes.
    + *
    • + * @param absoluteVolumePaths the absolute path of volumes. + * @param clearFailure if true, clears the failure information related to the + * volumes. + * @throws IOException + */ + private synchronized void removeVolumes( + final Set absoluteVolumePaths, boolean clearFailure) + throws IOException { + for (File vol : absoluteVolumePaths) { + Preconditions.checkArgument(vol.isAbsolute()); + } + + if (absoluteVolumePaths.isEmpty()) { + return; + } + + LOG.info(String.format("Deactivating volumes (clear failure=%b): %s", + clearFailure, Joiner.on(",").join(absoluteVolumePaths))); + + IOException ioe = null; + // Remove volumes and block infos from FsDataset. + data.removeVolumes(absoluteVolumePaths, clearFailure); + + // Remove volumes from DataStorage. + try { + storage.removeVolumes(absoluteVolumePaths); + } catch (IOException e) { + ioe = e; + } + + // Set configuration and dataDirs to reflect volume changes. + for (Iterator it = dataDirs.iterator(); it.hasNext(); ) { + StorageLocation loc = it.next(); + if (absoluteVolumePaths.contains(loc.getFile().getAbsoluteFile())) { + it.remove(); + } + } + conf.set(DFS_DATANODE_DATA_DIR_KEY, Joiner.on(",").join(dataDirs)); + + if (ioe != null) { + throw ioe; } } @@ -1278,12 +1352,12 @@ void initBlockPool(BPOfferService bpos) throws IOException { blockScanner.enableBlockPoolId(bpos.getBlockPoolId()); } - BPOfferService[] getAllBpOs() { + List getAllBpOs() { return blockPoolManager.getAllNamenodeThreads(); } int getBpOsCount() { - return blockPoolManager.getAllNamenodeThreads().length; + return blockPoolManager.getAllNamenodeThreads().size(); } /** @@ -1580,11 +1654,8 @@ public void shutdown() { } } - // We need to make a copy of the original blockPoolManager#offerServices to - // make sure blockPoolManager#shutDownAll() can still access all the - // BPOfferServices, since after setting DataNode#shouldRun to false the - // offerServices may be modified. - BPOfferService[] bposArray = this.blockPoolManager == null ? null + List bposArray = (this.blockPoolManager == null) + ? new ArrayList() : this.blockPoolManager.getAllNamenodeThreads(); // If shutdown is not for restart, set shouldRun to false early. if (!shutdownForUpgrade) { @@ -1653,8 +1724,9 @@ public void shutdown() { // termination of receiver threads. if (!this.shutdownForUpgrade || (this.shutdownForUpgrade && (Time.monotonicNow() - timeNotified - > 2500))) { + > 1000))) { this.threadGroup.interrupt(); + break; } LOG.info("Waiting for threadgroup to exit, active threads is " + this.threadGroup.activeCount()); @@ -1665,8 +1737,8 @@ public void shutdown() { Thread.sleep(sleepMs); } catch (InterruptedException e) {} sleepMs = sleepMs * 3 / 2; // exponential backoff - if (sleepMs > 1000) { - sleepMs = 1000; + if (sleepMs > 200) { + sleepMs = 200; } } this.threadGroup = null; @@ -2263,8 +2335,7 @@ void join() { while (shouldRun) { try { blockPoolManager.joinAll(); - if (blockPoolManager.getAllNamenodeThreads() != null - && blockPoolManager.getAllNamenodeThreads().length == 0) { + if (blockPoolManager.getAllNamenodeThreads().size() == 0) { shouldRun = false; } // Terminate if shutdown is complete or 2 seconds after all BPs @@ -2424,6 +2495,10 @@ public BlockScanner getBlockScanner() { return blockScanner; } + @VisibleForTesting + DirectoryScanner getDirectoryScanner() { + return directoryScanner; + } public static void secureMain(String args[], SecureResources resources) { int errorCode = 0; @@ -3076,10 +3151,20 @@ public ShortCircuitRegistry getShortCircuitRegistry() { * Check the disk error */ private void checkDiskError() { - try { - data.checkDataDir(); - } catch (DiskErrorException de) { - handleDiskError(de.getMessage()); + Set unhealthyDataDirs = data.checkDataDir(); + if (unhealthyDataDirs != null && !unhealthyDataDirs.isEmpty()) { + try { + // Remove all unhealthy volumes from DataNode. + removeVolumes(unhealthyDataDirs, false); + } catch (IOException e) { + LOG.warn("Error occurred when removing unhealthy storage dirs: " + + e.getMessage(), e); + } + StringBuilder sb = new StringBuilder("DataNode failed volumes:"); + for (File dataDir : unhealthyDataDirs) { + sb.append(dataDir.getAbsolutePath() + ";"); + } + handleDiskError(sb.toString()); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java index 754df2c7353ce..77fcfedb96b81 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java @@ -83,7 +83,6 @@ public class DataStorage extends Storage { public final static String BLOCK_SUBDIR_PREFIX = "subdir"; - final static String BLOCK_FILE_PREFIX = "blk_"; final static String COPY_FILE_PREFIX = "dncp_"; final static String STORAGE_DIR_DETACHED = "detach"; public final static String STORAGE_DIR_RBW = "rbw"; @@ -169,11 +168,11 @@ public void enableTrash(String bpid) { } } - public void restoreTrash(String bpid) { + public void clearTrash(String bpid) { if (trashEnabledBpids.contains(bpid)) { - getBPStorage(bpid).restoreTrash(); + getBPStorage(bpid).clearTrash(); trashEnabledBpids.remove(bpid); - LOG.info("Restored trash for bpid " + bpid); + LOG.info("Cleared trash for bpid " + bpid); } } @@ -405,28 +404,23 @@ synchronized List addStorageLocations(DataNode datanode, } /** - * Remove volumes from DataStorage. All volumes are removed even when the + * Remove storage dirs from DataStorage. All storage dirs are removed even when the * IOException is thrown. * - * @param locations a collection of volumes. + * @param dirsToRemove a set of storage directories to be removed. * @throws IOException if I/O error when unlocking storage directory. */ - synchronized void removeVolumes(Collection locations) + synchronized void removeVolumes(final Set dirsToRemove) throws IOException { - if (locations.isEmpty()) { + if (dirsToRemove.isEmpty()) { return; } - Set dataDirs = new HashSet(); - for (StorageLocation sl : locations) { - dataDirs.add(sl.getFile()); - } - StringBuilder errorMsgBuilder = new StringBuilder(); for (Iterator it = this.storageDirs.iterator(); it.hasNext(); ) { StorageDirectory sd = it.next(); - if (dataDirs.contains(sd.getRoot())) { + if (dirsToRemove.contains(sd.getRoot())) { // Remove the block pool level storage first. for (Map.Entry entry : this.bpStorageMap.entrySet()) { @@ -1250,7 +1244,7 @@ static void linkBlocksHelper(File from, File to, int oldLV, HardLink hl, String[] blockNames = from.list(new java.io.FilenameFilter() { @Override public boolean accept(File dir, String name) { - return name.startsWith(BLOCK_FILE_PREFIX); + return name.startsWith(Block.BLOCK_FILE_PREFIX); } }); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java index e9547a84e2ef6..cf1b6bebeab13 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java @@ -22,8 +22,10 @@ import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status.ERROR_INVALID; import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status.ERROR_UNSUPPORTED; import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.Status.SUCCESS; +import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.ShortCircuitFdResponse.USE_RECEIPT_VERIFICATION; +import static org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.ShortCircuitFdResponse.DO_NOT_USE_RECEIPT_VERIFICATION; import static org.apache.hadoop.hdfs.server.datanode.DataNode.DN_CLIENTTRACE_FORMAT; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.BufferedInputStream; import java.io.BufferedOutputStream; @@ -245,7 +247,7 @@ public void run() { peer.setReadTimeout(dnConf.socketTimeout); } - opStartTime = now(); + opStartTime = monotonicNow(); processOp(op); ++opsProcessed; } while ((peer != null) && @@ -291,64 +293,83 @@ public void run() { @Override public void requestShortCircuitFds(final ExtendedBlock blk, final Token token, - SlotId slotId, int maxVersion) throws IOException { + SlotId slotId, int maxVersion, boolean supportsReceiptVerification) + throws IOException { updateCurrentThreadName("Passing file descriptors for block " + blk); BlockOpResponseProto.Builder bld = BlockOpResponseProto.newBuilder(); FileInputStream fis[] = null; + SlotId registeredSlotId = null; + boolean success = false; try { - if (peer.getDomainSocket() == null) { - throw new IOException("You cannot pass file descriptors over " + - "anything but a UNIX domain socket."); - } - if (slotId != null) { - boolean isCached = datanode.data. - isCached(blk.getBlockPoolId(), blk.getBlockId()); - datanode.shortCircuitRegistry.registerSlot( - ExtendedBlockId.fromExtendedBlock(blk), slotId, isCached); - } try { - fis = datanode.requestShortCircuitFdsForRead(blk, token, maxVersion); - } finally { - if ((fis == null) && (slotId != null)) { - datanode.shortCircuitRegistry.unregisterSlot(slotId); + if (peer.getDomainSocket() == null) { + throw new IOException("You cannot pass file descriptors over " + + "anything but a UNIX domain socket."); } + if (slotId != null) { + boolean isCached = datanode.data. + isCached(blk.getBlockPoolId(), blk.getBlockId()); + datanode.shortCircuitRegistry.registerSlot( + ExtendedBlockId.fromExtendedBlock(blk), slotId, isCached); + registeredSlotId = slotId; + } + fis = datanode.requestShortCircuitFdsForRead(blk, token, maxVersion); + Preconditions.checkState(fis != null); + bld.setStatus(SUCCESS); + bld.setShortCircuitAccessVersion(DataNode.CURRENT_BLOCK_FORMAT_VERSION); + } catch (ShortCircuitFdsVersionException e) { + bld.setStatus(ERROR_UNSUPPORTED); + bld.setShortCircuitAccessVersion(DataNode.CURRENT_BLOCK_FORMAT_VERSION); + bld.setMessage(e.getMessage()); + } catch (ShortCircuitFdsUnsupportedException e) { + bld.setStatus(ERROR_UNSUPPORTED); + bld.setMessage(e.getMessage()); + } catch (InvalidToken e) { + bld.setStatus(ERROR_ACCESS_TOKEN); + bld.setMessage(e.getMessage()); + } catch (IOException e) { + bld.setStatus(ERROR); + bld.setMessage(e.getMessage()); } - bld.setStatus(SUCCESS); - bld.setShortCircuitAccessVersion(DataNode.CURRENT_BLOCK_FORMAT_VERSION); - } catch (ShortCircuitFdsVersionException e) { - bld.setStatus(ERROR_UNSUPPORTED); - bld.setShortCircuitAccessVersion(DataNode.CURRENT_BLOCK_FORMAT_VERSION); - bld.setMessage(e.getMessage()); - } catch (ShortCircuitFdsUnsupportedException e) { - bld.setStatus(ERROR_UNSUPPORTED); - bld.setMessage(e.getMessage()); - } catch (InvalidToken e) { - bld.setStatus(ERROR_ACCESS_TOKEN); - bld.setMessage(e.getMessage()); - } catch (IOException e) { - bld.setStatus(ERROR); - bld.setMessage(e.getMessage()); - } - try { bld.build().writeDelimitedTo(socketOut); if (fis != null) { FileDescriptor fds[] = new FileDescriptor[fis.length]; for (int i = 0; i < fds.length; i++) { fds[i] = fis[i].getFD(); } - byte buf[] = new byte[] { (byte)0 }; - peer.getDomainSocket(). - sendFileDescriptors(fds, buf, 0, buf.length); + byte buf[] = new byte[1]; + if (supportsReceiptVerification) { + buf[0] = (byte)USE_RECEIPT_VERIFICATION.getNumber(); + } else { + buf[0] = (byte)DO_NOT_USE_RECEIPT_VERIFICATION.getNumber(); + } + DomainSocket sock = peer.getDomainSocket(); + sock.sendFileDescriptors(fds, buf, 0, buf.length); + if (supportsReceiptVerification) { + LOG.trace("Reading receipt verification byte for " + slotId); + int val = sock.getInputStream().read(); + if (val < 0) { + throw new EOFException(); + } + } else { + LOG.trace("Receipt verification is not enabled on the DataNode. " + + "Not verifying " + slotId); + } + success = true; } } finally { + if ((!success) && (registeredSlotId != null)) { + LOG.info("Unregistering " + registeredSlotId + " because the " + + "requestShortCircuitFdsForRead operation failed."); + datanode.shortCircuitRegistry.unregisterSlot(registeredSlotId); + } if (ClientTraceLog.isInfoEnabled()) { DatanodeRegistration dnR = datanode.getDNRegistrationForBP(blk .getBlockPoolId()); BlockSender.ClientTraceLog.info(String.format( "src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS," + " blockid: %s, srvID: %s, success: %b", - blk.getBlockId(), dnR.getDatanodeUuid(), (fis != null) - )); + blk.getBlockId(), dnR.getDatanodeUuid(), success)); } if (fis != null) { IOUtils.cleanup(LOG, fis); @@ -1181,7 +1202,7 @@ public void replaceBlock(final ExtendedBlock block, } private long elapsed() { - return now() - opStartTime; + return monotonicNow() - opStartTime; } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java index c7ee21e5ca6d7..61dfb14b9f4f9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java @@ -443,13 +443,14 @@ void scan() { int d = 0; // index for blockpoolReport int m = 0; // index for memReprot while (m < memReport.length && d < blockpoolReport.length) { - FinalizedReplica memBlock = memReport[Math.min(m, memReport.length - 1)]; - ScanInfo info = blockpoolReport[Math.min( - d, blockpoolReport.length - 1)]; + FinalizedReplica memBlock = memReport[m]; + ScanInfo info = blockpoolReport[d]; if (info.getBlockId() < memBlock.getBlockId()) { - // Block is missing in memory - statsRecord.missingMemoryBlocks++; - addDifference(diffRecord, statsRecord, info); + if (!dataset.isDeletingBlock(bpid, info.getBlockId())) { + // Block is missing in memory + statsRecord.missingMemoryBlocks++; + addDifference(diffRecord, statsRecord, info); + } d++; continue; } @@ -495,8 +496,11 @@ void scan() { current.getBlockId(), current.getVolume()); } while (d < blockpoolReport.length) { - statsRecord.missingMemoryBlocks++; - addDifference(diffRecord, statsRecord, blockpoolReport[d++]); + if (!dataset.isDeletingBlock(bpid, blockpoolReport[d].getBlockId())) { + statsRecord.missingMemoryBlocks++; + addDifference(diffRecord, statsRecord, blockpoolReport[d]); + } + d++; } LOG.info(statsRecord.toString()); } //end for @@ -633,7 +637,7 @@ private LinkedList compileReport(FsVolumeSpi vol, continue; } if (!Block.isBlockFilename(files[i])) { - if (isBlockMetaFile("blk_", files[i].getName())) { + if (isBlockMetaFile(Block.BLOCK_FILE_PREFIX, files[i].getName())) { long blockId = Block.getBlockId(files[i].getName()); verifyFileLocation(files[i].getParentFile(), bpFinalizedDir, blockId); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java index 965b40a0c4504..b32c0d167c5a2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ShortCircuitRegistry.java @@ -30,6 +30,7 @@ import java.util.Iterator; import java.util.Set; +import com.google.common.annotations.VisibleForTesting; import org.apache.commons.io.IOUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -83,7 +84,7 @@ public class ShortCircuitRegistry { private static final int SHM_LENGTH = 8192; - private static class RegisteredShm extends ShortCircuitShm + public static class RegisteredShm extends ShortCircuitShm implements DomainSocketWatcher.Handler { private final String clientName; private final ShortCircuitRegistry registry; @@ -176,7 +177,7 @@ public ShortCircuitRegistry(Configuration conf) throws IOException { if (dswLoadingFailure != null) { throw new IOException(dswLoadingFailure); } - watcher = new DomainSocketWatcher(interruptCheck); + watcher = new DomainSocketWatcher(interruptCheck, "datanode"); enabled = true; if (LOG.isDebugEnabled()) { LOG.debug("created new ShortCircuitRegistry with interruptCheck=" + @@ -383,4 +384,14 @@ public void shutdown() { } IOUtils.closeQuietly(watcher); } + + public static interface Visitor { + void accept(HashMap segments, + HashMultimap slots); + } + + @VisibleForTesting + public synchronized void visit(Visitor visitor) { + visitor.accept(segments, slots); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java index 10c83694f9e85..8a741de3a0ecc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java @@ -27,6 +27,7 @@ import java.util.Collection; import java.util.List; import java.util.Map; +import java.util.Set; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.conf.Configuration; @@ -113,9 +114,11 @@ public void addVolume( * If the FSDataset supports block scanning, this function removes * the volumes from the block scanner. * - * @param volumes The storage locations of the volumes to remove. + * @param volumes The paths of the volumes to be removed. + * @param clearFailure set true to clear the failure information about the + * volumes. */ - public void removeVolumes(Collection volumes); + public void removeVolumes(Set volumes, boolean clearFailure); /** @return a storage with the given storage ID */ public DatanodeStorage getStorage(final String storageUuid); @@ -388,9 +391,9 @@ public void checkBlock(ExtendedBlock b, long minLength, ReplicaState state) /** * Check if all the data directories are healthy - * @throws DiskErrorException + * @return A set of unhealthy data directories. */ - public void checkDataDir() throws DiskErrorException; + public Set checkDataDir(); /** * Shutdown the FSDataset @@ -487,9 +490,9 @@ public HdfsBlocksMetadata getHdfsBlocksMetadata(String bpid, public void enableTrash(String bpid); /** - * Restore trash + * Clear trash */ - public void restoreTrash(String bpid); + public void clearTrash(String bpid); /** * @return true when trash is enabled @@ -543,4 +546,9 @@ public ReplicaInfo moveBlockAcrossStorage(final ExtendedBlock block, * Check whether the block was pinned */ public boolean getPinning(ExtendedBlock block) throws IOException; + + /** + * Confirm whether the block is deleting + */ + public boolean isDeletingBlock(String bpid, long blockId); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index 5a69e1e4d666d..6daf03944ecd0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -23,12 +23,12 @@ import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.FileOutputStream; -import java.io.FileWriter; import java.io.IOException; import java.io.InputStream; import java.io.OutputStreamWriter; import java.io.RandomAccessFile; import java.io.Writer; +import java.util.Iterator; import java.util.Scanner; import org.apache.commons.io.FileUtils; @@ -39,6 +39,8 @@ import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.protocol.Block; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportReplica; import org.apache.hadoop.hdfs.protocol.HdfsConstants; import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader; import org.apache.hadoop.hdfs.server.datanode.DataStorage; @@ -55,6 +57,7 @@ import org.apache.hadoop.util.ShutdownHookManager; import org.apache.hadoop.util.Time; +import com.google.common.io.Files; /** * A block pool slice represents a portion of a block pool stored on a volume. * Taken together, all BlockPoolSlices sharing a block pool ID across a @@ -77,7 +80,9 @@ class BlockPoolSlice { private volatile boolean dfsUsedSaved = false; private static final int SHUTDOWN_HOOK_PRIORITY = 30; private final boolean deleteDuplicateReplicas; - + private static final String REPLICA_CACHE_FILE = "replicas"; + private final long replicaCacheExpiry = 5*60*1000; + // TODO:FEDERATION scalability issue - a thread per DU is needed private final DU dfsUsage; @@ -310,11 +315,14 @@ void getVolumeMap(ReplicaMap volumeMap, FsDatasetImpl.LOG.info( "Recovered " + numRecovered + " replicas from " + lazypersistDir); } - - // add finalized replicas - addToReplicasMap(volumeMap, finalizedDir, lazyWriteReplicaMap, true); - // add rbw replicas - addToReplicasMap(volumeMap, rbwDir, lazyWriteReplicaMap, false); + + boolean success = readReplicasFromCache(volumeMap, lazyWriteReplicaMap); + if (!success) { + // add finalized replicas + addToReplicasMap(volumeMap, finalizedDir, lazyWriteReplicaMap, true); + // add rbw replicas + addToReplicasMap(volumeMap, rbwDir, lazyWriteReplicaMap, false); + } } /** @@ -401,6 +409,75 @@ private int moveLazyPersistReplicasToFinalized(File source) FileUtil.fullyDelete(source); return numRecovered; } + + private void addReplicaToReplicasMap(Block block, ReplicaMap volumeMap, + final RamDiskReplicaTracker lazyWriteReplicaMap,boolean isFinalized) + throws IOException { + ReplicaInfo newReplica = null; + long blockId = block.getBlockId(); + long genStamp = block.getGenerationStamp(); + if (isFinalized) { + newReplica = new FinalizedReplica(blockId, + block.getNumBytes(), genStamp, volume, DatanodeUtil + .idToBlockDir(finalizedDir, blockId)); + } else { + File file = new File(rbwDir, block.getBlockName()); + boolean loadRwr = true; + File restartMeta = new File(file.getParent() + + File.pathSeparator + "." + file.getName() + ".restart"); + Scanner sc = null; + try { + sc = new Scanner(restartMeta, "UTF-8"); + // The restart meta file exists + if (sc.hasNextLong() && (sc.nextLong() > Time.now())) { + // It didn't expire. Load the replica as a RBW. + // We don't know the expected block length, so just use 0 + // and don't reserve any more space for writes. + newReplica = new ReplicaBeingWritten(blockId, + validateIntegrityAndSetLength(file, genStamp), + genStamp, volume, file.getParentFile(), null, 0); + loadRwr = false; + } + sc.close(); + if (!restartMeta.delete()) { + FsDatasetImpl.LOG.warn("Failed to delete restart meta file: " + + restartMeta.getPath()); + } + } catch (FileNotFoundException fnfe) { + // nothing to do hereFile dir = + } finally { + if (sc != null) { + sc.close(); + } + } + // Restart meta doesn't exist or expired. + if (loadRwr) { + newReplica = new ReplicaWaitingToBeRecovered(blockId, + validateIntegrityAndSetLength(file, genStamp), + genStamp, volume, file.getParentFile()); + } + } + + ReplicaInfo oldReplica = volumeMap.get(bpid, newReplica.getBlockId()); + if (oldReplica == null) { + volumeMap.add(bpid, newReplica); + } else { + // We have multiple replicas of the same block so decide which one + // to keep. + newReplica = resolveDuplicateReplicas(newReplica, oldReplica, volumeMap); + } + + // If we are retaining a replica on transient storage make sure + // it is in the lazyWriteReplicaMap so it can be persisted + // eventually. + if (newReplica.getVolume().isTransientStorage()) { + lazyWriteReplicaMap.addReplica(bpid, blockId, + (FsVolumeImpl) newReplica.getVolume()); + } else { + lazyWriteReplicaMap.discardReplica(bpid, blockId, false); + } + } + /** * Add replicas under the given directory to the volume map @@ -434,66 +511,9 @@ void addToReplicasMap(ReplicaMap volumeMap, File dir, long genStamp = FsDatasetUtil.getGenerationStampFromFile( files, file); long blockId = Block.filename2id(file.getName()); - ReplicaInfo newReplica = null; - if (isFinalized) { - newReplica = new FinalizedReplica(blockId, - file.length(), genStamp, volume, file.getParentFile()); - } else { - - boolean loadRwr = true; - File restartMeta = new File(file.getParent() + - File.pathSeparator + "." + file.getName() + ".restart"); - Scanner sc = null; - try { - sc = new Scanner(restartMeta, "UTF-8"); - // The restart meta file exists - if (sc.hasNextLong() && (sc.nextLong() > Time.now())) { - // It didn't expire. Load the replica as a RBW. - // We don't know the expected block length, so just use 0 - // and don't reserve any more space for writes. - newReplica = new ReplicaBeingWritten(blockId, - validateIntegrityAndSetLength(file, genStamp), - genStamp, volume, file.getParentFile(), null, 0); - loadRwr = false; - } - sc.close(); - if (!restartMeta.delete()) { - FsDatasetImpl.LOG.warn("Failed to delete restart meta file: " + - restartMeta.getPath()); - } - } catch (FileNotFoundException fnfe) { - // nothing to do hereFile dir = - } finally { - if (sc != null) { - sc.close(); - } - } - // Restart meta doesn't exist or expired. - if (loadRwr) { - newReplica = new ReplicaWaitingToBeRecovered(blockId, - validateIntegrityAndSetLength(file, genStamp), - genStamp, volume, file.getParentFile()); - } - } - - ReplicaInfo oldReplica = volumeMap.get(bpid, newReplica.getBlockId()); - if (oldReplica == null) { - volumeMap.add(bpid, newReplica); - } else { - // We have multiple replicas of the same block so decide which one - // to keep. - newReplica = resolveDuplicateReplicas(newReplica, oldReplica, volumeMap); - } - - // If we are retaining a replica on transient storage make sure - // it is in the lazyWriteReplicaMap so it can be persisted - // eventually. - if (newReplica.getVolume().isTransientStorage()) { - lazyWriteReplicaMap.addReplica(bpid, blockId, - (FsVolumeImpl) newReplica.getVolume()); - } else { - lazyWriteReplicaMap.discardReplica(bpid, blockId, false); - } + Block block = new Block(blockId, file.length(), genStamp); + addReplicaToReplicasMap(block, volumeMap, lazyWriteReplicaMap, + isFinalized); } } @@ -649,9 +669,121 @@ public String toString() { return currentDir.getAbsolutePath(); } - void shutdown() { + void shutdown(BlockListAsLongs blocksListToPersist) { + saveReplicas(blocksListToPersist); saveDfsUsed(); dfsUsedSaved = true; dfsUsage.shutdown(); } + + private boolean readReplicasFromCache(ReplicaMap volumeMap, + final RamDiskReplicaTracker lazyWriteReplicaMap) { + ReplicaMap tmpReplicaMap = new ReplicaMap(this); + File replicaFile = new File(currentDir, REPLICA_CACHE_FILE); + // Check whether the file exists or not. + if (!replicaFile.exists()) { + LOG.info("Replica Cache file: "+ replicaFile.getPath() + + " doesn't exist "); + return false; + } + long fileLastModifiedTime = replicaFile.lastModified(); + if (System.currentTimeMillis() > fileLastModifiedTime + replicaCacheExpiry) { + LOG.info("Replica Cache file: " + replicaFile.getPath() + + " has gone stale"); + // Just to make findbugs happy + if (!replicaFile.delete()) { + LOG.info("Replica Cache file: " + replicaFile.getPath() + + " cannot be deleted"); + } + return false; + } + FileInputStream inputStream = null; + try { + inputStream = new FileInputStream(replicaFile); + BlockListAsLongs blocksList = BlockListAsLongs.readFrom(inputStream); + Iterator iterator = blocksList.iterator(); + while (iterator.hasNext()) { + BlockReportReplica replica = iterator.next(); + switch (replica.getState()) { + case FINALIZED: + addReplicaToReplicasMap(replica, tmpReplicaMap, lazyWriteReplicaMap, true); + break; + case RUR: + case RBW: + case RWR: + addReplicaToReplicasMap(replica, tmpReplicaMap, lazyWriteReplicaMap, false); + break; + default: + break; + } + } + inputStream.close(); + // Now it is safe to add the replica into volumeMap + // In case of any exception during parsing this cache file, fall back + // to scan all the files on disk. + for (ReplicaInfo info: tmpReplicaMap.replicas(bpid)) { + volumeMap.add(bpid, info); + } + LOG.info("Successfully read replica from cache file : " + + replicaFile.getPath()); + return true; + } catch (Exception e) { + // Any exception we need to revert back to read from disk + // Log the error and return false + LOG.info("Exception occured while reading the replicas cache file: " + + replicaFile.getPath(), e ); + return false; + } + finally { + if (!replicaFile.delete()) { + LOG.info("Failed to delete replica cache file: " + + replicaFile.getPath()); + } + // close the inputStream + IOUtils.closeStream(inputStream); + } + } + + private void saveReplicas(BlockListAsLongs blocksListToPersist) { + if (blocksListToPersist == null || + blocksListToPersist.getNumberOfBlocks()== 0) { + return; + } + File tmpFile = new File(currentDir, REPLICA_CACHE_FILE + ".tmp"); + if (tmpFile.exists() && !tmpFile.delete()) { + LOG.warn("Failed to delete tmp replicas file in " + + tmpFile.getPath()); + return; + } + File replicaCacheFile = new File(currentDir, REPLICA_CACHE_FILE); + if (replicaCacheFile.exists() && !replicaCacheFile.delete()) { + LOG.warn("Failed to delete replicas file in " + + replicaCacheFile.getPath()); + return; + } + + FileOutputStream out = null; + try { + out = new FileOutputStream(tmpFile); + blocksListToPersist.writeTo(out); + out.close(); + // Renaming the tmp file to replicas + Files.move(tmpFile, replicaCacheFile); + } catch (Exception e) { + // If write failed, the volume might be bad. Since the cache file is + // not critical, log the error, delete both the files (tmp and cache) + // and continue. + LOG.warn("Failed to write replicas to cache ", e); + if (replicaCacheFile.exists() && !replicaCacheFile.delete()) { + LOG.warn("Failed to delete replicas file: " + + replicaCacheFile.getPath()); + } + } finally { + IOUtils.closeStream(out); + if (tmpFile.exists() && !tmpFile.delete()) { + LOG.warn("Failed to delete tmp file in " + + tmpFile.getPath()); + } + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java index 13e854f0b7c91..c1d3990e22bea 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java @@ -22,7 +22,10 @@ import java.io.FileDescriptor; import java.io.IOException; import java.util.HashMap; +import java.util.HashSet; import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.ThreadFactory; import java.util.concurrent.ThreadPoolExecutor; @@ -64,9 +67,14 @@ class FsDatasetAsyncDiskService { private static final long THREADS_KEEP_ALIVE_SECONDS = 60; private final DataNode datanode; + private final FsDatasetImpl fsdatasetImpl; private final ThreadGroup threadGroup; private Map executors = new HashMap(); + private Map> deletedBlockIds + = new HashMap>(); + private static final int MAX_DELETED_BLOCKS = 64; + private int numDeletedBlocks = 0; /** * Create a AsyncDiskServices with a set of volumes (specified by their @@ -75,8 +83,9 @@ class FsDatasetAsyncDiskService { * The AsyncDiskServices uses one ThreadPool per volume to do the async * disk operations. */ - FsDatasetAsyncDiskService(DataNode datanode) { + FsDatasetAsyncDiskService(DataNode datanode, FsDatasetImpl fsdatasetImpl) { this.datanode = datanode; + this.fsdatasetImpl = fsdatasetImpl; this.threadGroup = new ThreadGroup(getClass().getSimpleName()); } @@ -286,7 +295,27 @@ public void run() { LOG.info("Deleted " + block.getBlockPoolId() + " " + block.getLocalBlock() + " file " + blockFile); } + updateDeletedBlockId(block); IOUtils.cleanup(null, volumeRef); } } + + private synchronized void updateDeletedBlockId(ExtendedBlock block) { + Set blockIds = deletedBlockIds.get(block.getBlockPoolId()); + if (blockIds == null) { + blockIds = new HashSet(); + deletedBlockIds.put(block.getBlockPoolId(), blockIds); + } + blockIds.add(block.getBlockId()); + numDeletedBlocks++; + if (numDeletedBlocks == MAX_DELETED_BLOCKS) { + for (Entry> e : deletedBlockIds.entrySet()) { + String bpid = e.getKey(); + Set bs = e.getValue(); + fsdatasetImpl.removeDeletedBlocks(bpid, bs); + bs.clear(); + } + numDeletedBlocks = 0; + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java index cc6220aa80eb5..f15f6493b3065 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java @@ -46,6 +46,7 @@ import javax.management.ObjectName; import javax.management.StandardMBean; +import com.google.common.annotations.VisibleForTesting; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience; @@ -236,6 +237,7 @@ public LengthInputStream getMetaDataInputStream(ExtendedBlock b) private volatile boolean fsRunning; final ReplicaMap volumeMap; + final Map> deletingBlock; final RamDiskReplicaTracker ramDiskReplicaTracker; final RamDiskAsyncLazyPersistService asyncLazyPersistService; @@ -274,8 +276,10 @@ public LengthInputStream getMetaDataInputStream(ExtendedBlock b) this.validVolsRequired = volsConfigured - volFailuresTolerated; if (volFailuresTolerated < 0 || volFailuresTolerated >= volsConfigured) { - throw new DiskErrorException("Invalid volume failure " - + " config value: " + volFailuresTolerated); + throw new DiskErrorException("Invalid value configured for " + + "dfs.datanode.failed.volumes.tolerated - " + volFailuresTolerated + + ". Value configured is either less than 0 or >= " + + "to the number of configured volumes (" + volsConfigured + ")."); } if (volsFailed > volFailuresTolerated) { throw new DiskErrorException("Too many failed volumes - " @@ -297,8 +301,9 @@ public LengthInputStream getMetaDataInputStream(ExtendedBlock b) VolumeChoosingPolicy.class), conf); volumes = new FsVolumeList(volumeFailureInfos, datanode.getBlockScanner(), blockChooserImpl); - asyncDiskService = new FsDatasetAsyncDiskService(datanode); + asyncDiskService = new FsDatasetAsyncDiskService(datanode, this); asyncLazyPersistService = new RamDiskAsyncLazyPersistService(datanode); + deletingBlock = new HashMap>(); for (int idx = 0; idx < storage.getNumStorageDirs(); idx++) { addVolume(dataLocations, storage.getStorageDir(idx)); @@ -375,6 +380,12 @@ private void addVolume(Collection dataLocations, LOG.info("Added volume - " + dir + ", StorageType: " + storageType); } + @VisibleForTesting + public FsVolumeImpl createFsVolume(String storageUuid, File currentDir, + StorageType storageType) throws IOException { + return new FsVolumeImpl(this, storageUuid, currentDir, conf, storageType); + } + @Override public void addVolume(final StorageLocation location, final List nsInfos) @@ -394,8 +405,8 @@ public void addVolume(final StorageLocation location, final Storage.StorageDirectory sd = builder.getStorageDirectory(); StorageType storageType = location.getStorageType(); - final FsVolumeImpl fsVolume = new FsVolumeImpl( - this, sd.getStorageUuid(), sd.getCurrentDir(), this.conf, storageType); + final FsVolumeImpl fsVolume = + createFsVolume(sd.getStorageUuid(), sd.getCurrentDir(), storageType); final ReplicaMap tempVolumeMap = new ReplicaMap(fsVolume); ArrayList exceptions = Lists.newArrayList(); @@ -411,6 +422,11 @@ public void addVolume(final StorageLocation location, } } if (!exceptions.isEmpty()) { + try { + sd.unlock(); + } catch (IOException e) { + exceptions.add(e); + } throw MultipleIOException.createIOException(exceptions); } @@ -431,41 +447,42 @@ public void addVolume(final StorageLocation location, } /** - * Removes a collection of volumes from FsDataset. - * @param volumes the root directories of the volumes. + * Removes a set of volumes from FsDataset. + * @param volumesToRemove a set of absolute root path of each volume. + * @param clearFailure set true to clear failure information. * * DataNode should call this function before calling * {@link DataStorage#removeVolumes(java.util.Collection)}. */ @Override - public synchronized void removeVolumes(Collection volumes) { - Set volumeSet = new HashSet<>(); - for (StorageLocation sl : volumes) { - volumeSet.add(sl.getFile().getAbsolutePath()); + public synchronized void removeVolumes( + Set volumesToRemove, boolean clearFailure) { + // Make sure that all volumes are absolute path. + for (File vol : volumesToRemove) { + Preconditions.checkArgument(vol.isAbsolute(), + String.format("%s is not absolute path.", vol.getPath())); } for (int idx = 0; idx < dataStorage.getNumStorageDirs(); idx++) { Storage.StorageDirectory sd = dataStorage.getStorageDir(idx); - String volume = sd.getRoot().getAbsolutePath(); - if (volumeSet.contains(volume)) { - LOG.info("Removing " + volume + " from FsDataset."); + final File absRoot = sd.getRoot().getAbsoluteFile(); + if (volumesToRemove.contains(absRoot)) { + LOG.info("Removing " + absRoot + " from FsDataset."); // Disable the volume from the service. asyncDiskService.removeVolume(sd.getCurrentDir()); - this.volumes.removeVolume(sd.getRoot()); + volumes.removeVolume(absRoot, clearFailure); // Removed all replica information for the blocks on the volume. Unlike // updating the volumeMap in addVolume(), this operation does not scan // disks. for (String bpid : volumeMap.getBlockPoolList()) { - List blocks = new ArrayList(); for (Iterator it = volumeMap.replicas(bpid).iterator(); - it.hasNext(); ) { + it.hasNext(); ) { ReplicaInfo block = it.next(); - String absBasePath = - new File(block.getVolume().getBasePath()).getAbsolutePath(); - if (absBasePath.equals(volume)) { + final File absBasePath = + new File(block.getVolume().getBasePath()).getAbsoluteFile(); + if (absBasePath.equals(absRoot)) { invalidate(bpid, block); - blocks.add(block); it.remove(); } } @@ -1560,30 +1577,26 @@ public Map getBlockReports(String bpid) { Map blockReportsMap = new HashMap(); - Map> finalized = - new HashMap>(); - Map> uc = - new HashMap>(); + Map builders = + new HashMap(); List curVolumes = getVolumes(); for (FsVolumeSpi v : curVolumes) { - finalized.put(v.getStorageID(), new ArrayList()); - uc.put(v.getStorageID(), new ArrayList()); + builders.put(v.getStorageID(), BlockListAsLongs.builder()); } synchronized(this) { for (ReplicaInfo b : volumeMap.replicas(bpid)) { switch(b.getState()) { case FINALIZED: - finalized.get(b.getVolume().getStorageID()).add(b); - break; case RBW: case RWR: - uc.get(b.getVolume().getStorageID()).add(b); + builders.get(b.getVolume().getStorageID()).add(b); break; case RUR: ReplicaUnderRecovery rur = (ReplicaUnderRecovery)b; - uc.get(rur.getVolume().getStorageID()).add(rur.getOriginalReplica()); + builders.get(rur.getVolume().getStorageID()) + .add(rur.getOriginalReplica()); break; case TEMPORARY: break; @@ -1594,10 +1607,8 @@ public Map getBlockReports(String bpid) { } for (FsVolumeImpl v : curVolumes) { - ArrayList finalizedList = finalized.get(v.getStorageID()); - ArrayList ucList = uc.get(v.getStorageID()); blockReportsMap.put(v.toDatanodeStorage(), - new BlockListAsLongs(finalizedList, ucList)); + builders.get(v.getStorageID()).build()); } return blockReportsMap; @@ -1783,7 +1794,12 @@ public void invalidate(String bpid, Block invalidBlks[]) throws IOException { + ". Parent not found for file " + f); continue; } - volumeMap.remove(bpid, invalidBlks[i]); + ReplicaInfo removing = volumeMap.remove(bpid, invalidBlks[i]); + addDeletingBlock(bpid, removing.getBlockId()); + if (LOG.isDebugEnabled()) { + LOG.debug("Block file " + removing.getBlockFile().getName() + + " is to be deleted"); + } } if (v.isTransientStorage()) { @@ -1956,50 +1972,14 @@ File getFile(final String bpid, final long blockId, boolean touch) { /** * check if a data directory is healthy - * if some volumes failed - make sure to remove all the blocks that belong - * to these volumes - * @throws DiskErrorException + * + * if some volumes failed - the caller must emove all the blocks that belong + * to these failed volumes. + * @return the failed volumes. Returns null if no volume failed. */ @Override // FsDatasetSpi - public void checkDataDir() throws DiskErrorException { - long totalBlocks=0, removedBlocks=0; - List failedVols = volumes.checkDirs(); - - // If there no failed volumes return - if (failedVols == null) { - return; - } - - // Otherwise remove blocks for the failed volumes - long mlsec = Time.now(); - synchronized (this) { - for (FsVolumeImpl fv: failedVols) { - for (String bpid : fv.getBlockPoolList()) { - Iterator ib = volumeMap.replicas(bpid).iterator(); - while(ib.hasNext()) { - ReplicaInfo b = ib.next(); - totalBlocks++; - // check if the volume block belongs to still valid - if(b.getVolume() == fv) { - LOG.warn("Removing replica " + bpid + ":" + b.getBlockId() - + " on failed volume " + fv.getCurrentDir().getAbsolutePath()); - ib.remove(); - removedBlocks++; - } - } - } - } - } // end of sync - mlsec = Time.now() - mlsec; - LOG.warn("Removed " + removedBlocks + " out of " + totalBlocks + - "(took " + mlsec + " millisecs)"); - - // report the error - StringBuilder sb = new StringBuilder(); - for (FsVolumeImpl fv : failedVols) { - sb.append(fv.getCurrentDir().getAbsolutePath() + ";"); - } - throw new DiskErrorException("DataNode failed volumes:" + sb); + public Set checkDataDir() { + return volumes.checkDirs(); } @@ -2483,8 +2463,9 @@ public void addBlockPool(String bpid, Configuration conf) @Override public synchronized void shutdownBlockPool(String bpid) { LOG.info("Removing block pool " + bpid); + Map blocksPerVolume = getBlockReports(bpid); volumeMap.cleanUpBlockPool(bpid); - volumes.removeBlockPool(bpid); + volumes.removeBlockPool(bpid, blocksPerVolume); } /** @@ -2568,6 +2549,21 @@ public synchronized void deleteBlockPool(String bpid, boolean force) @Override // FsDatasetSpi public BlockLocalPathInfo getBlockLocalPathInfo(ExtendedBlock block) throws IOException { + synchronized(this) { + final Replica replica = volumeMap.get(block.getBlockPoolId(), + block.getBlockId()); + if (replica == null) { + throw new ReplicaNotFoundException(block); + } + if (replica.getGenerationStamp() < block.getGenerationStamp()) { + throw new IOException( + "Replica generation stamp < block generation stamp, block=" + + block + ", replica=" + replica); + } else if (replica.getGenerationStamp() > block.getGenerationStamp()) { + block.setGenerationStamp(replica.getGenerationStamp()); + } + } + File datafile = getBlockFile(block); File metafile = FsDatasetUtil.getMetaFile(datafile, block.getGenerationStamp()); BlockLocalPathInfo info = new BlockLocalPathInfo(block, @@ -2623,8 +2619,8 @@ public void enableTrash(String bpid) { } @Override - public void restoreTrash(String bpid) { - dataStorage.restoreTrash(bpid); + public void clearTrash(String bpid) { + dataStorage.clearTrash(bpid); } @Override @@ -2978,5 +2974,35 @@ public boolean getPinning(ExtendedBlock block) throws IOException { FileStatus fss = localFS.getFileStatus(new Path(f.getAbsolutePath())); return fss.getPermission().getStickyBit(); } + + @Override + public boolean isDeletingBlock(String bpid, long blockId) { + synchronized(deletingBlock) { + Set s = deletingBlock.get(bpid); + return s != null ? s.contains(blockId) : false; + } + } + + public void removeDeletedBlocks(String bpid, Set blockIds) { + synchronized (deletingBlock) { + Set s = deletingBlock.get(bpid); + if (s != null) { + for (Long id : blockIds) { + s.remove(id); + } + } + } + } + + private void addDeletingBlock(String bpid, Long blockId) { + synchronized(deletingBlock) { + Set s = deletingBlock.get(bpid); + if (s == null) { + s = new HashSet(); + deletingBlock.put(bpid, s); + } + s.add(blockId); + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java index 297a47d7b403a..4dbc7f17ee3fb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java @@ -50,6 +50,7 @@ import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.protocol.Block; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.server.datanode.DataStorage; import org.apache.hadoop.hdfs.server.datanode.DatanodeUtil; @@ -65,7 +66,6 @@ import org.apache.hadoop.util.Time; import org.codehaus.jackson.annotate.JsonProperty; import org.codehaus.jackson.map.ObjectMapper; -import org.codehaus.jackson.map.ObjectReader; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -289,7 +289,8 @@ void incDfsUsed(String bpid, long value) { } } - long getDfsUsed() throws IOException { + @VisibleForTesting + public long getDfsUsed() throws IOException { long dfsUsed = 0; synchronized(dataset) { for(BlockPoolSlice s : bpSlices.values()) { @@ -430,7 +431,8 @@ private enum BlockFileFilter implements FilenameFilter { @Override public boolean accept(File dir, String name) { - return !name.endsWith(".meta") && name.startsWith("blk_"); + return !name.endsWith(".meta") && + name.startsWith(Block.BLOCK_FILE_PREFIX); } } @@ -803,7 +805,7 @@ void shutdown() { } Set> set = bpSlices.entrySet(); for (Entry entry : set) { - entry.getValue().shutdown(); + entry.getValue().shutdown(null); } } @@ -813,10 +815,10 @@ void addBlockPool(String bpid, Configuration conf) throws IOException { bpSlices.put(bpid, bp); } - void shutdownBlockPool(String bpid) { + void shutdownBlockPool(String bpid, BlockListAsLongs blocksListsAsLongs) { BlockPoolSlice bp = bpSlices.get(bpid); if (bp != null) { - bp.shutdown(); + bp.shutdown(blocksListsAsLongs); } bpSlices.remove(bpid); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java index b38d41fb90f70..4fddfb9e644c6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java @@ -24,19 +24,23 @@ import java.util.Arrays; import java.util.Collection; import java.util.Collections; +import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.TreeMap; +import java.util.Set; import java.util.concurrent.atomic.AtomicReference; import com.google.common.collect.Lists; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.StorageType; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeReference; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi; import org.apache.hadoop.hdfs.server.datanode.fsdataset.VolumeChoosingPolicy; import org.apache.hadoop.hdfs.server.datanode.BlockScanner; +import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; import org.apache.hadoop.util.DiskChecker.DiskErrorException; import org.apache.hadoop.util.Time; @@ -218,16 +222,15 @@ public void run() { } /** - * Calls {@link FsVolumeImpl#checkDirs()} on each volume, removing any - * volumes from the active list that result in a DiskErrorException. + * Calls {@link FsVolumeImpl#checkDirs()} on each volume. * * Use checkDirsMutext to allow only one instance of checkDirs() call * - * @return list of all the removed volumes. + * @return list of all the failed volumes. */ - List checkDirs() { + Set checkDirs() { synchronized(checkDirsMutex) { - ArrayList removedVols = null; + Set failedVols = null; // Make a copy of volumes for performing modification final List volumeList = getVolumes(); @@ -238,12 +241,12 @@ List checkDirs() { fsv.checkDirs(); } catch (DiskErrorException e) { FsDatasetImpl.LOG.warn("Removing failed volume " + fsv + ": ", e); - if (removedVols == null) { - removedVols = new ArrayList<>(1); + if (failedVols == null) { + failedVols = new HashSet<>(1); } - removedVols.add(fsv); - removeVolume(fsv); + failedVols.add(new File(fsv.getBasePath()).getAbsoluteFile()); addVolumeFailureInfo(fsv); + removeVolume(fsv); } catch (ClosedChannelException e) { FsDatasetImpl.LOG.debug("Caught exception when obtaining " + "reference count on closed volume", e); @@ -252,12 +255,12 @@ List checkDirs() { } } - if (removedVols != null && removedVols.size() > 0) { - FsDatasetImpl.LOG.warn("Completed checkDirs. Removed " + removedVols.size() - + " volumes. Current volumes: " + this); + if (failedVols != null && failedVols.size() > 0) { + FsDatasetImpl.LOG.warn("Completed checkDirs. Found " + failedVols.size() + + " failure volumes."); } - return removedVols; + return failedVols; } } @@ -290,6 +293,9 @@ void addVolume(FsVolumeReference ref) { if (blockScanner != null) { blockScanner.addVolumeScanner(ref); } + // If the volume is used to replace a failed volume, it needs to reset the + // volume failure info for this volume. + removeVolumeFailureInfo(new File(ref.getVolume().getBasePath())); FsDatasetImpl.LOG.info("Added new volume: " + ref.getVolume().getStorageID()); } @@ -337,8 +343,9 @@ private void removeVolume(FsVolumeImpl target) { /** * Dynamically remove volume in the list. * @param volume the volume to be removed. + * @param clearFailure set true to remove failure info for this volume. */ - void removeVolume(File volume) { + void removeVolume(File volume, boolean clearFailure) { // Make a copy of volumes to remove one volume. final FsVolumeImpl[] curVolumes = volumes.get(); final List volumeList = Lists.newArrayList(curVolumes); @@ -352,7 +359,9 @@ void removeVolume(File volume) { removeVolume(fsVolume); } } - removeVolumeFailureInfo(volume); + if (clearFailure) { + removeVolumeFailureInfo(volume); + } } VolumeFailureInfo[] getVolumeFailureInfos() { @@ -366,7 +375,9 @@ void addVolumeFailureInfo(VolumeFailureInfo volumeFailureInfo) { } private void addVolumeFailureInfo(FsVolumeImpl vol) { - addVolumeFailureInfo(new VolumeFailureInfo(vol.getBasePath(), Time.now(), + addVolumeFailureInfo(new VolumeFailureInfo( + new File(vol.getBasePath()).getAbsolutePath(), + Time.now(), vol.getCapacity())); } @@ -419,9 +430,10 @@ public void run() { bpid + ": " + totalTimeTaken + "ms"); } - void removeBlockPool(String bpid) { + void removeBlockPool(String bpid, Map + blocksPerVolume) { for (FsVolumeImpl v : volumes.get()) { - v.shutdownBlockPool(bpid); + v.shutdownBlockPool(bpid, blocksPerVolume.get(v.toDatanodeStorage())); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java index 0fbc2ee10eb6a..2e8eb222f75d5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java @@ -107,6 +107,7 @@ public class DataNodeMetrics { @Metric MutableRate replaceBlockOp; @Metric MutableRate heartbeats; @Metric MutableRate blockReports; + @Metric MutableRate incrementalBlockReports; @Metric MutableRate cacheReports; @Metric MutableRate packetAckRoundTripTimeNanos; final MutableQuantiles[] packetAckRoundTripTimeNanosQuantiles; @@ -201,6 +202,10 @@ public void addBlockReport(long latency) { blockReports.add(latency); } + public void addIncrementalBlockReport(long latency) { + incrementalBlockReports.add(latency); + } + public void addCacheReport(long latency) { cacheReports.add(latency); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java index 57495046899cc..0ebf3dcc5224c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/ParameterParser.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdfs.server.datanode.web.webhdfs; import io.netty.handler.codec.http.QueryStringDecoder; +import org.apache.commons.io.Charsets; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.hdfs.HAUtil; @@ -27,6 +28,7 @@ import org.apache.hadoop.hdfs.web.resources.DelegationParam; import org.apache.hadoop.hdfs.web.resources.DoAsParam; import org.apache.hadoop.hdfs.web.resources.HttpOpParam; +import org.apache.hadoop.hdfs.web.resources.LengthParam; import org.apache.hadoop.hdfs.web.resources.NamenodeAddressParam; import org.apache.hadoop.hdfs.web.resources.OffsetParam; import org.apache.hadoop.hdfs.web.resources.OverwriteParam; @@ -38,6 +40,7 @@ import java.io.IOException; import java.net.URI; +import java.nio.charset.Charset; import java.util.List; import java.util.Map; @@ -50,7 +53,8 @@ class ParameterParser { private final Map> params; ParameterParser(QueryStringDecoder decoder, Configuration conf) { - this.path = QueryStringDecoder.decodeComponent(decoder.path().substring(WEBHDFS_PREFIX_LENGTH)); + this.path = decodeComponent(decoder.path().substring + (WEBHDFS_PREFIX_LENGTH), Charsets.UTF_8); this.params = decoder.parameters(); this.conf = conf; } @@ -62,7 +66,11 @@ String op() { } long offset() { - return new OffsetParam(param(OffsetParam.NAME)).getValue(); + return new OffsetParam(param(OffsetParam.NAME)).getOffset(); + } + + long length() { + return new LengthParam(param(LengthParam.NAME)).getLength(); } String namenodeId() { @@ -122,4 +130,78 @@ private String param(String key) { List p = params.get(key); return p == null ? null : p.get(0); } + + /** + * The following function behaves exactly the same as netty's + * QueryStringDecoder#decodeComponent except that it + * does not decode the '+' character as space. WebHDFS takes this scheme + * to maintain the backward-compatibility for pre-2.7 releases. + */ + private static String decodeComponent(final String s, final Charset charset) { + if (s == null) { + return ""; + } + final int size = s.length(); + boolean modified = false; + for (int i = 0; i < size; i++) { + final char c = s.charAt(i); + if (c == '%' || c == '+') { + modified = true; + break; + } + } + if (!modified) { + return s; + } + final byte[] buf = new byte[size]; + int pos = 0; // position in `buf'. + for (int i = 0; i < size; i++) { + char c = s.charAt(i); + if (c == '%') { + if (i == size - 1) { + throw new IllegalArgumentException("unterminated escape sequence at" + + " end of string: " + s); + } + c = s.charAt(++i); + if (c == '%') { + buf[pos++] = '%'; // "%%" -> "%" + break; + } + if (i == size - 1) { + throw new IllegalArgumentException("partial escape sequence at end " + + "of string: " + s); + } + c = decodeHexNibble(c); + final char c2 = decodeHexNibble(s.charAt(++i)); + if (c == Character.MAX_VALUE || c2 == Character.MAX_VALUE) { + throw new IllegalArgumentException( + "invalid escape sequence `%" + s.charAt(i - 1) + s.charAt( + i) + "' at index " + (i - 2) + " of: " + s); + } + c = (char) (c * 16 + c2); + // Fall through. + } + buf[pos++] = (byte) c; + } + return new String(buf, 0, pos, charset); + } + + /** + * Helper to decode half of a hexadecimal number from a string. + * @param c The ASCII character of the hexadecimal number to decode. + * Must be in the range {@code [0-9a-fA-F]}. + * @return The hexadecimal value represented in the ASCII character + * given, or {@link Character#MAX_VALUE} if the character is invalid. + */ + private static char decodeHexNibble(final char c) { + if ('0' <= c && c <= '9') { + return (char) (c - '0'); + } else if ('a' <= c && c <= 'f') { + return (char) (c - 'a' + 10); + } else if ('A' <= c && c <= 'F') { + return (char) (c - 'A' + 10); + } else { + return Character.MAX_VALUE; + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java index be1faecebddc3..4d705b0dd9fe0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java @@ -47,8 +47,10 @@ import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.util.LimitInputStream; import java.io.IOException; +import java.io.InputStream; import java.io.OutputStream; import java.net.URI; import java.net.URISyntaxException; @@ -188,6 +190,7 @@ private void onOpen(ChannelHandlerContext ctx) throws IOException { final String nnId = params.namenodeId(); final int bufferSize = params.bufferSize(); final long offset = params.offset(); + final long length = params.length(); DefaultHttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK); HttpHeaders headers = response.headers(); @@ -202,12 +205,20 @@ private void onOpen(ChannelHandlerContext ctx) throws IOException { dfsclient.open(path, bufferSize, true)); in.seek(offset); - if (in.getVisibleLength() >= offset) { - headers.set(CONTENT_LENGTH, in.getVisibleLength() - offset); + long contentLength = in.getVisibleLength() - offset; + if (length >= 0) { + contentLength = Math.min(contentLength, length); + } + final InputStream data; + if (contentLength >= 0) { + headers.set(CONTENT_LENGTH, contentLength); + data = new LimitInputStream(in, contentLength); + } else { + data = in; } ctx.write(response); - ctx.writeAndFlush(new ChunkedStream(in) { + ctx.writeAndFlush(new ChunkedStream(data) { @Override public void close() throws Exception { super.close(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java index 9327f4382ec7e..011a4599c5e2e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java @@ -19,7 +19,7 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_BACKUP_HTTP_ADDRESS_DEFAULT; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_BACKUP_HTTP_ADDRESS_KEY; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.File; import java.io.IOException; @@ -135,11 +135,11 @@ public void run() { long lastCheckpointTime = 0; if (!backupNode.shouldCheckpointAtStartup()) { - lastCheckpointTime = now(); + lastCheckpointTime = monotonicNow(); } while(shouldRun) { try { - long now = now(); + long now = monotonicNow(); boolean shouldCheckpoint = false; if(now >= lastCheckpointTime + periodMSec) { shouldCheckpoint = true; @@ -182,7 +182,7 @@ void doCheckpoint() throws IOException { BackupImage bnImage = getFSImage(); NNStorage bnStorage = bnImage.getStorage(); - long startTime = now(); + long startTime = monotonicNow(); bnImage.freezeNamespaceAtNextRoll(); NamenodeCommand cmd = @@ -276,7 +276,7 @@ void doCheckpoint() throws IOException { long imageSize = bnImage.getStorage().getFsImageName(txid).length(); LOG.info("Checkpoint completed in " - + (now() - startTime)/1000 + " seconds." + + (monotonicNow() - startTime)/1000 + " seconds." + " New Image Size: " + imageSize); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentCounts.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentCounts.java new file mode 100644 index 0000000000000..16f0771f1ad12 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentCounts.java @@ -0,0 +1,146 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.server.namenode; + +import org.apache.hadoop.fs.StorageType; +import org.apache.hadoop.hdfs.util.EnumCounters; + +/** + * The counter to be computed for content types such as file, directory and symlink, + * and the storage type usage such as SSD, DISK, ARCHIVE. + */ +public class ContentCounts { + private EnumCounters contents; + private EnumCounters types; + + public static class Builder { + private EnumCounters contents; + // storage spaces used by corresponding storage types + private EnumCounters types; + + public Builder() { + contents = new EnumCounters(Content.class); + types = new EnumCounters(StorageType.class); + } + + public Builder file(long file) { + contents.set(Content.FILE, file); + return this; + } + + public Builder directory(long directory) { + contents.set(Content.DIRECTORY, directory); + return this; + } + + public Builder symlink(long symlink) { + contents.set(Content.SYMLINK, symlink); + return this; + } + + public Builder length(long length) { + contents.set(Content.LENGTH, length); + return this; + } + + public Builder storagespace(long storagespace) { + contents.set(Content.DISKSPACE, storagespace); + return this; + } + + public Builder snapshot(long snapshot) { + contents.set(Content.SNAPSHOT, snapshot); + return this; + } + + public Builder snapshotable_directory(long snapshotable_directory) { + contents.set(Content.SNAPSHOTTABLE_DIRECTORY, snapshotable_directory); + return this; + } + + public ContentCounts build() { + return new ContentCounts(contents, types); + } + } + + private ContentCounts(EnumCounters contents, + EnumCounters types) { + this.contents = contents; + this.types = types; + } + + // Get the number of files. + public long getFileCount() { + return contents.get(Content.FILE); + } + + // Get the number of directories. + public long getDirectoryCount() { + return contents.get(Content.DIRECTORY); + } + + // Get the number of symlinks. + public long getSymlinkCount() { + return contents.get(Content.SYMLINK); + } + + // Get the total of file length in bytes. + public long getLength() { + return contents.get(Content.LENGTH); + } + + // Get the total of storage space usage in bytes including replication. + public long getStoragespace() { + return contents.get(Content.DISKSPACE); + } + + // Get the number of snapshots + public long getSnapshotCount() { + return contents.get(Content.SNAPSHOT); + } + + // Get the number of snapshottable directories. + public long getSnapshotableDirectoryCount() { + return contents.get(Content.SNAPSHOTTABLE_DIRECTORY); + } + + public long[] getTypeSpaces() { + return types.asArray(); + } + + public long getTypeSpace(StorageType t) { + return types.get(t); + } + + public void addContent(Content c, long val) { + contents.add(c, val); + } + + public void addContents(ContentCounts that) { + contents.add(that.contents); + types.add(that.types); + } + + public void addTypeSpace(StorageType t, long val) { + types.add(t, val); + } + + public void addTypeSpaces(EnumCounters that) { + this.types.add(that); + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java index 63fa8c1eb84e7..31f34b9133dfd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hdfs.server.namenode; +import com.google.common.base.Preconditions; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite; @@ -26,7 +27,8 @@ public class ContentSummaryComputationContext { private FSDirectory dir = null; private FSNamesystem fsn = null; - private Content.Counts counts = null; + private BlockStoragePolicySuite bsps = null; + private ContentCounts counts = null; private long nextCountLimit = 0; private long limitPerRun = 0; private long yieldCount = 0; @@ -46,12 +48,13 @@ public ContentSummaryComputationContext(FSDirectory dir, this.fsn = fsn; this.limitPerRun = limitPerRun; this.nextCountLimit = limitPerRun; - this.counts = Content.Counts.newInstance(); + this.counts = new ContentCounts.Builder().build(); } /** Constructor for blocking computation. */ - public ContentSummaryComputationContext() { + public ContentSummaryComputationContext(BlockStoragePolicySuite bsps) { this(null, null, 0); + this.bsps = bsps; } /** Return current yield count */ @@ -73,10 +76,10 @@ public boolean yield() { } // Have we reached the limit? - long currentCount = counts.get(Content.FILE) + - counts.get(Content.SYMLINK) + - counts.get(Content.DIRECTORY) + - counts.get(Content.SNAPSHOTTABLE_DIRECTORY); + long currentCount = counts.getFileCount() + + counts.getSymlinkCount() + + counts.getDirectoryCount() + + counts.getSnapshotableDirectoryCount(); if (currentCount <= nextCountLimit) { return false; } @@ -114,11 +117,15 @@ public boolean yield() { } /** Get the content counts */ - public Content.Counts getCounts() { + public ContentCounts getCounts() { return counts; } public BlockStoragePolicySuite getBlockStoragePolicySuite() { - return fsn.getBlockManager().getStoragePolicySuite(); + Preconditions.checkState((bsps != null || fsn != null), + "BlockStoragePolicySuite must be either initialized or available via" + + " FSNameSystem"); + return (bsps != null) ? bsps: + fsn.getBlockManager().getStoragePolicySuite(); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DefaultINodeAttributesProvider.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DefaultINodeAttributesProvider.java new file mode 100644 index 0000000000000..45aa1b5398abf --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DefaultINodeAttributesProvider.java @@ -0,0 +1,45 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.server.namenode; + +/** + * A default implementation of the INodeAttributesProvider + * + */ +public class DefaultINodeAttributesProvider extends INodeAttributeProvider { + + public static INodeAttributeProvider DEFAULT_PROVIDER = + new DefaultINodeAttributesProvider(); + + @Override + public void start() { + // NO-OP + } + + @Override + public void stop() { + // NO-OP + } + + @Override + public INodeAttributes getAttributes(String[] pathElements, + INodeAttributes inode) { + return inode; + } + +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java index 01eb22fe40130..31b45ad758efc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java @@ -126,12 +126,12 @@ QuotaCounts AddCurrentSpaceUsage(QuotaCounts counts) { ContentSummaryComputationContext computeContentSummary(final INodeDirectory dir, final ContentSummaryComputationContext summary) { - final long original = summary.getCounts().get(Content.DISKSPACE); + final long original = summary.getCounts().getStoragespace(); long oldYieldCount = summary.getYieldCount(); dir.computeDirectoryContentSummary(summary, Snapshot.CURRENT_STATE_ID); // Check only when the content has not changed in the middle. if (oldYieldCount == summary.getYieldCount()) { - checkStoragespace(dir, summary.getCounts().get(Content.DISKSPACE) - original); + checkStoragespace(dir, summary.getCounts().getStoragespace() - original); } return summary; } @@ -277,4 +277,4 @@ public String toString() { return "Quota[" + namespaceString() + ", " + storagespaceString() + ", " + typeSpaceString() + "]"; } -} \ No newline at end of file +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java index 5e6d9d8f686bf..b4ca2d6c0df4e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java @@ -20,7 +20,7 @@ import java.io.IOException; import java.io.Closeable; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceStability; @@ -109,9 +109,9 @@ public void flush() throws IOException { public void flush(boolean durable) throws IOException { numSync++; - long start = now(); + long start = monotonicNow(); flushAndSync(durable); - long end = now(); + long end = monotonicNow(); totalTimeSync += (end - start); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java index 5ccd3eafb36c7..31a6af7b7bcf2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirConcatOp.java @@ -34,6 +34,16 @@ import static org.apache.hadoop.util.Time.now; +/** + * Restrictions for a concat operation: + * 
    + * 1. the src file and the target file are in the same dir
+ * 2. all the source files are not in snapshot
+ * 3. any source file cannot be the same with the target file
+ * 4. source files cannot be under construction or empty
+ * 5. source file's preferred block size cannot be greater than the target file
+ *
    + */ class FSDirConcatOp { static HdfsFileStatus concat(FSDirectory fsd, String target, String[] srcs, @@ -123,14 +133,25 @@ private static INodeFile[] verifySrcFiles(FSDirectory fsd, String[] srcs, throw new SnapshotException("Concat: the source file " + src + " is referred by some other reference in some snapshot."); } + // source file cannot be the same with the target file if (srcINode == targetINode) { throw new HadoopIllegalArgumentException("concat: the src file " + src + " is the same with the target file " + targetIIP.getPath()); } + // source file cannot be under construction or empty if(srcINodeFile.isUnderConstruction() || srcINodeFile.numBlocks() == 0) { throw new HadoopIllegalArgumentException("concat: source file " + src + " is invalid or empty or underConstruction"); } + // source file's preferred block size cannot be greater than the target + // file + if (srcINodeFile.getPreferredBlockSize() > + targetINode.getPreferredBlockSize()) { + throw new HadoopIllegalArgumentException("concat: source file " + src + + " has preferred block size " + srcINodeFile.getPreferredBlockSize() + + " which is greater than the target file's preferred block size " + + targetINode.getPreferredBlockSize()); + } si.add(srcINodeFile); } @@ -143,9 +164,10 @@ private static INodeFile[] verifySrcFiles(FSDirectory fsd, String[] srcs, return si.toArray(new INodeFile[si.size()]); } - private static QuotaCounts computeQuotaDeltas(FSDirectory fsd, INodeFile target, INodeFile[] srcList) { + private static QuotaCounts computeQuotaDeltas(FSDirectory fsd, + INodeFile target, INodeFile[] srcList) { QuotaCounts deltas = new QuotaCounts.Builder().build(); - short targetRepl = target.getBlockReplication(); + final short targetRepl = target.getBlockReplication(); for (INodeFile src : srcList) { short srcRepl = src.getBlockReplication(); long fileSize = src.computeFileSize(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java index cb3da193e8f4e..43c2de3c471c8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java @@ -181,7 +181,7 @@ private static DirectoryListing getListing(FSDirectory fsd, INodesInPath iip, if (!targetNode.isDirectory()) { return new DirectoryListing( - new HdfsFileStatus[]{createFileStatus(fsd, + new HdfsFileStatus[]{createFileStatus(fsd, src, HdfsFileStatus.EMPTY_NAME, targetNode, needLocation, parentStoragePolicy, snapshot, isRawPath, iip)}, 0); } @@ -200,7 +200,7 @@ private static DirectoryListing getListing(FSDirectory fsd, INodesInPath iip, byte curPolicy = isSuperUser && !cur.isSymlink()? cur.getLocalStoragePolicyID(): BlockStoragePolicySuite.ID_UNSPECIFIED; - listing[i] = createFileStatus(fsd, cur.getLocalNameBytes(), cur, + listing[i] = createFileStatus(fsd, src, cur.getLocalNameBytes(), cur, needLocation, getStoragePolicyID(curPolicy, parentStoragePolicy), snapshot, isRawPath, iip); listingCnt++; @@ -253,7 +253,7 @@ private static DirectoryListing getSnapshotsListing( final HdfsFileStatus listing[] = new HdfsFileStatus[numOfListing]; for (int i = 0; i < numOfListing; i++) { Snapshot.Root sRoot = snapshots.get(i + skipSize).getRoot(); - listing[i] = createFileStatus(fsd, sRoot.getLocalNameBytes(), sRoot, + listing[i] = createFileStatus(fsd, src, sRoot.getLocalNameBytes(), sRoot, BlockStoragePolicySuite.ID_UNSPECIFIED, Snapshot.CURRENT_STATE_ID, false, INodesInPath.fromINode(sRoot)); } @@ -270,7 +270,7 @@ private static DirectoryListing getSnapshotsListing( * or null if file not found */ static HdfsFileStatus getFileInfo( - FSDirectory fsd, INodesInPath src, boolean isRawPath, + FSDirectory fsd, String path, INodesInPath src, boolean isRawPath, boolean includeStoragePolicy) throws IOException { fsd.readLock(); @@ -279,7 +279,7 @@ static HdfsFileStatus getFileInfo( byte policyId = includeStoragePolicy && i != null && !i.isSymlink() ? i.getStoragePolicyID() : BlockStoragePolicySuite.ID_UNSPECIFIED; return i == null ? null : createFileStatus( - fsd, HdfsFileStatus.EMPTY_NAME, i, policyId, + fsd, path, HdfsFileStatus.EMPTY_NAME, i, policyId, src.getPathSnapshotId(), isRawPath, src); } finally { fsd.readUnlock(); @@ -303,7 +303,7 @@ static HdfsFileStatus getFileInfo( fsd.readLock(); try { final INodesInPath iip = fsd.getINodesInPath(srcs, resolveLink); - return getFileInfo(fsd, iip, isRawPath, includeStoragePolicy); + return getFileInfo(fsd, src, iip, isRawPath, includeStoragePolicy); } finally { fsd.readUnlock(); } @@ -340,14 +340,15 @@ private static HdfsFileStatus getFileInfo4DotSnapshot( * @throws java.io.IOException if any error occurs */ static HdfsFileStatus createFileStatus( - FSDirectory fsd, byte[] path, INode node, boolean needLocation, - byte storagePolicy, int snapshot, boolean isRawPath, INodesInPath iip) + FSDirectory fsd, String fullPath, byte[] path, INode node, + boolean needLocation, byte storagePolicy, int snapshot, boolean isRawPath, + INodesInPath iip) throws IOException { if (needLocation) { - return createLocatedFileStatus(fsd, path, node, storagePolicy, + return createLocatedFileStatus(fsd, fullPath, path, node, storagePolicy, snapshot, isRawPath, iip); } else { - return createFileStatus(fsd, path, node, storagePolicy, snapshot, + return createFileStatus(fsd, fullPath, path, node, storagePolicy, snapshot, isRawPath, iip); } } @@ -356,8 +357,9 @@ static HdfsFileStatus createFileStatus( * Create FileStatus by file INode */ static HdfsFileStatus createFileStatus( - FSDirectory fsd, byte[] path, INode node, byte storagePolicy, - int snapshot, boolean isRawPath, INodesInPath iip) throws IOException { + FSDirectory fsd, String fullPath, byte[] path, INode node, + byte storagePolicy, int snapshot, boolean isRawPath, + INodesInPath iip) throws IOException { long size = 0; // length is zero for directories short replication = 0; long blocksize = 0; @@ -380,6 +382,8 @@ static HdfsFileStatus createFileStatus( int childrenNum = node.isDirectory() ? node.asDirectory().getChildrenNum(snapshot) : 0; + INodeAttributes nodeAttrs = + fsd.getAttributes(fullPath, path, node, snapshot); return new HdfsFileStatus( size, node.isDirectory(), @@ -387,9 +391,9 @@ static HdfsFileStatus createFileStatus( blocksize, node.getModificationTime(snapshot), node.getAccessTime(snapshot), - getPermissionForFileStatus(node, snapshot, isEncrypted), - node.getUserName(snapshot), - node.getGroupName(snapshot), + getPermissionForFileStatus(nodeAttrs, isEncrypted), + nodeAttrs.getUserName(), + nodeAttrs.getGroupName(), node.isSymlink() ? node.asSymlink().getSymlink() : null, path, node.getId(), @@ -402,8 +406,9 @@ static HdfsFileStatus createFileStatus( * Create FileStatus with location info by file INode */ private static HdfsLocatedFileStatus createLocatedFileStatus( - FSDirectory fsd, byte[] path, INode node, byte storagePolicy, - int snapshot, boolean isRawPath, INodesInPath iip) throws IOException { + FSDirectory fsd, String fullPath, byte[] path, INode node, + byte storagePolicy, int snapshot, boolean isRawPath, + INodesInPath iip) throws IOException { assert fsd.hasReadLock(); long size = 0; // length is zero for directories short replication = 0; @@ -437,12 +442,14 @@ private static HdfsLocatedFileStatus createLocatedFileStatus( int childrenNum = node.isDirectory() ? node.asDirectory().getChildrenNum(snapshot) : 0; + INodeAttributes nodeAttrs = + fsd.getAttributes(fullPath, path, node, snapshot); HdfsLocatedFileStatus status = new HdfsLocatedFileStatus(size, node.isDirectory(), replication, blocksize, node.getModificationTime(snapshot), node.getAccessTime(snapshot), - getPermissionForFileStatus(node, snapshot, isEncrypted), - node.getUserName(snapshot), node.getGroupName(snapshot), + getPermissionForFileStatus(nodeAttrs, isEncrypted), + nodeAttrs.getUserName(), nodeAttrs.getGroupName(), node.isSymlink() ? node.asSymlink().getSymlink() : null, path, node.getId(), loc, childrenNum, feInfo, storagePolicy); // Set caching information for the located blocks. @@ -467,9 +474,9 @@ private static HdfsLocatedFileStatus createLocatedFileStatus( * and encrypted bit on if it represents an encrypted file/dir. */ private static FsPermission getPermissionForFileStatus( - INode node, int snapshot, boolean isEncrypted) { - FsPermission perm = node.getFsPermission(snapshot); - boolean hasAcl = node.getAclFeature(snapshot) != null; + INodeAttributes node, boolean isEncrypted) { + FsPermission perm = node.getFsPermission(); + boolean hasAcl = node.getAclFeature() != null; if (hasAcl || isEncrypted) { perm = new FsPermissionExtension(perm, hasAcl, isEncrypted); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java index 45e63f2effe00..d5c9124166270 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java @@ -108,7 +108,8 @@ static List getXAttrs(FSDirectory fsd, final String srcArg, return filteredAll; } if (filteredAll == null || filteredAll.isEmpty()) { - return null; + throw new IOException( + "At least one of the attributes provided was not found."); } List toGet = Lists.newArrayListWithCapacity(xAttrs.size()); for (XAttr xAttr : xAttrs) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java index f6ab077fcd3ac..7eea343fdcb3a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java @@ -61,6 +61,7 @@ import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState; import org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo; +import org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature; import org.apache.hadoop.hdfs.util.ByteArray; import org.apache.hadoop.hdfs.util.EnumCounters; import org.apache.hadoop.security.AccessControlException; @@ -166,6 +167,12 @@ private static INodeDirectory createRoot(FSNamesystem namesystem) { private final FSEditLog editLog; + private INodeAttributeProvider attributeProvider; + + public void setINodeAttributeProvider(INodeAttributeProvider provider) { + attributeProvider = provider; + } + // utility methods to acquire and release read lock and write lock void readLock() { this.dirLock.readLock().lock(); @@ -677,7 +684,7 @@ void updateCount(INodesInPath iip, long nsDelta, long ssDelta, short oldRep, * @param checkQuota if true then check if quota is exceeded * @throws QuotaExceededException if the new count violates any quota limit */ - void updateCount(INodesInPath iip, int numOfINodes, + void updateCount(INodesInPath iip, int numOfINodes, QuotaCounts counts, boolean checkQuota) throws QuotaExceededException { assert hasWriteLock(); @@ -1050,7 +1057,7 @@ void unprotectedTruncate(String src, String clientName, String clientMachine, INodeFile file = iip.getLastINode().asFile(); BlocksMapUpdateInfo collectedBlocks = new BlocksMapUpdateInfo(); boolean onBlockBoundary = - unprotectedTruncate(iip, newLength, collectedBlocks, mtime); + unprotectedTruncate(iip, newLength, collectedBlocks, mtime, null); if(! onBlockBoundary) { BlockInfoContiguous oldBlock = file.getLastBlock(); @@ -1073,11 +1080,11 @@ void unprotectedTruncate(String src, String clientName, String clientMachine, boolean truncate(INodesInPath iip, long newLength, BlocksMapUpdateInfo collectedBlocks, - long mtime) + long mtime, QuotaCounts delta) throws IOException { writeLock(); try { - return unprotectedTruncate(iip, newLength, collectedBlocks, mtime); + return unprotectedTruncate(iip, newLength, collectedBlocks, mtime, delta); } finally { writeUnlock(); } @@ -1097,22 +1104,49 @@ boolean truncate(INodesInPath iip, long newLength, */ boolean unprotectedTruncate(INodesInPath iip, long newLength, BlocksMapUpdateInfo collectedBlocks, - long mtime) throws IOException { + long mtime, QuotaCounts delta) throws IOException { assert hasWriteLock(); INodeFile file = iip.getLastINode().asFile(); int latestSnapshot = iip.getLatestSnapshotId(); file.recordModification(latestSnapshot, true); - long oldDiskspaceNoRep = file.storagespaceConsumedNoReplication(); + + verifyQuotaForTruncate(iip, file, newLength, delta); + long remainingLength = file.collectBlocksBeyondMax(newLength, collectedBlocks); file.excludeSnapshotBlocks(latestSnapshot, collectedBlocks); file.setModificationTime(mtime); - updateCount(iip, 0, file.storagespaceConsumedNoReplication() - oldDiskspaceNoRep, - file.getBlockReplication(), true); // return whether on a block boundary return (remainingLength - newLength) == 0; } + private void verifyQuotaForTruncate(INodesInPath iip, INodeFile file, + long newLength, QuotaCounts delta) throws QuotaExceededException { + if (!getFSNamesystem().isImageLoaded() || shouldSkipQuotaChecks()) { + // Do not check quota if edit log is still being processed + return; + } + final long diff = file.computeQuotaDeltaForTruncate(newLength); + final short repl = file.getBlockReplication(); + delta.addStorageSpace(diff * repl); + final BlockStoragePolicy policy = getBlockStoragePolicySuite() + .getPolicy(file.getStoragePolicyID()); + List types = policy.chooseStorageTypes(repl); + for (StorageType t : types) { + if (t.supportTypeQuota()) { + delta.addTypeSpace(t, diff); + } + } + if (diff > 0) { + readLock(); + try { + verifyQuota(iip, iip.length() - 1, delta, null); + } finally { + readUnlock(); + } + } + } + /** * This method is always called with writeLock of FSDirectory held. */ @@ -1595,13 +1629,23 @@ INodesInPath getINodesInPath4Write(String src, boolean resolveLink) FSPermissionChecker getPermissionChecker() throws AccessControlException { try { - return new FSPermissionChecker(fsOwnerShortUserName, supergroup, + return getPermissionChecker(fsOwnerShortUserName, supergroup, NameNode.getRemoteUser()); - } catch (IOException ioe) { - throw new AccessControlException(ioe); + } catch (IOException e) { + throw new AccessControlException(e); } } + @VisibleForTesting + FSPermissionChecker getPermissionChecker(String fsOwner, String superGroup, + UserGroupInformation ugi) throws AccessControlException { + return new FSPermissionChecker( + fsOwner, superGroup, ugi, + attributeProvider == null ? + DefaultINodeAttributesProvider.DEFAULT_PROVIDER + : attributeProvider); + } + void checkOwner(FSPermissionChecker pc, INodesInPath iip) throws AccessControlException { checkPermission(pc, iip, true, null, null, null, null); @@ -1662,7 +1706,8 @@ void checkPermission(FSPermissionChecker pc, INodesInPath iip, HdfsFileStatus getAuditFileInfo(INodesInPath iip) throws IOException { return (namesystem.isAuditEnabled() && namesystem.isExternalInvocation()) - ? FSDirStatAndListingOp.getFileInfo(this, iip, false, false) : null; + ? FSDirStatAndListingOp.getFileInfo(this, iip.getPath(), iip, false, + false) : null; } /** @@ -1708,4 +1753,20 @@ void resetLastInodeId(long newValue) throws IOException { void resetLastInodeIdWithoutChecking(long newValue) { inodeId.setCurrentValue(newValue); } + + INodeAttributes getAttributes(String fullPath, byte[] path, + INode node, int snapshot) { + INodeAttributes nodeAttrs = node; + if (attributeProvider != null) { + nodeAttrs = node.getSnapshotINode(snapshot); + fullPath = fullPath + (fullPath.endsWith(Path.SEPARATOR) ? "" + : Path.SEPARATOR) + + DFSUtil.bytes2String(path); + nodeAttrs = attributeProvider.getAttributes(fullPath, nodeAttrs); + } else { + nodeAttrs = node.getSnapshotINode(snapshot); + } + return nodeAttrs; + } + } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java index 9d487e58c41e5..df9c58528f63d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java @@ -18,7 +18,7 @@ package org.apache.hadoop.hdfs.server.namenode; import static org.apache.hadoop.util.ExitUtil.terminate; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.IOException; import java.lang.reflect.Constructor; @@ -230,7 +230,7 @@ protected synchronized TransactionId initialValue() { this.conf = conf; this.storage = storage; metrics = NameNode.getNameNodeMetrics(); - lastPrintTime = now(); + lastPrintTime = monotonicNow(); // If this list is empty, an error will be thrown on first use // of the editlog, as no journals will exist @@ -486,14 +486,14 @@ private long beginTransaction() { // TransactionId id = myTransactionId.get(); id.txid = txid; - return now(); + return monotonicNow(); } private void endTransaction(long start) { assert Thread.holdsLock(this); // update statistics - long end = now(); + long end = monotonicNow(); numTransactions++; totalTimeTransactions += (end-start); if (metrics != null) // Metrics is non-null only when used inside name node @@ -640,7 +640,7 @@ public void logSync() { } // do the sync - long start = now(); + long start = monotonicNow(); try { if (logStream != null) { logStream.flush(); @@ -657,7 +657,7 @@ public void logSync() { terminate(1, msg); } } - long elapsed = now() - start; + long elapsed = monotonicNow() - start; if (metrics != null) { // Metrics non-null only when used inside name node metrics.addSync(elapsed); @@ -679,7 +679,7 @@ public void logSync() { // print statistics every 1 minute. // private void printStatistics(boolean force) { - long now = now(); + long now = monotonicNow(); if (lastPrintTime + 60000 > now && !force) { return; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java index 51c167a7be5e6..f50dc4d5781b1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java @@ -19,7 +19,7 @@ import static org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.TruncateOp; import static org.apache.hadoop.hdfs.server.namenode.FSImageFormat.renameReservedPathsOnUpgrade; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.FilterInputStream; import java.io.IOException; @@ -136,13 +136,13 @@ long loadFSEdits(EditLogInputStream edits, long expectedStartingTxId, prog.beginStep(Phase.LOADING_EDITS, step); fsNamesys.writeLock(); try { - long startTime = now(); + long startTime = monotonicNow(); FSImage.LOG.info("Start loading edits file " + edits.getName()); long numEdits = loadEditRecords(edits, false, expectedStartingTxId, startOpt, recovery); FSImage.LOG.info("Edits file " + edits.getName() + " of size " + edits.length() + " edits # " + numEdits - + " loaded in " + (now()-startTime)/1000 + " seconds"); + + " loaded in " + (monotonicNow()-startTime)/1000 + " seconds"); return numEdits; } finally { edits.close(); @@ -177,7 +177,7 @@ long loadEditRecords(EditLogInputStream in, boolean closeOnExit, Step step = createStartupProgressStep(in); prog.setTotal(Phase.LOADING_EDITS, step, numTxns); Counter counter = prog.getCounter(Phase.LOADING_EDITS, step); - long lastLogTime = now(); + long lastLogTime = monotonicNow(); long lastInodeId = fsNamesys.dir.getLastInodeId(); try { @@ -257,7 +257,7 @@ long loadEditRecords(EditLogInputStream in, boolean closeOnExit, } // log progress if (op.hasTransactionId()) { - long now = now(); + long now = monotonicNow(); if (now - lastLogTime > REPLAY_TRANSACTION_LOG_INTERVAL) { long deltaTxId = lastAppliedTxId - expectedStartingTxId + 1; int percent = Math.round((float) deltaTxId / numTxns * 100); @@ -378,7 +378,7 @@ private long applyEditLogOp(FSEditLogOp op, FSDirectory fsDir, // add the op into retry cache if necessary if (toAddRetryCache) { HdfsFileStatus stat = FSDirStatAndListingOp.createFileStatus( - fsNamesys.dir, HdfsFileStatus.EMPTY_NAME, newFile, + fsNamesys.dir, path, HdfsFileStatus.EMPTY_NAME, newFile, BlockStoragePolicySuite.ID_UNSPECIFIED, Snapshot.CURRENT_STATE_ID, false, iip); fsNamesys.addCacheEntryWithPayload(addCloseOp.rpcClientId, @@ -397,7 +397,7 @@ private long applyEditLogOp(FSEditLogOp op, FSDirectory fsDir, // add the op into retry cache if necessary if (toAddRetryCache) { HdfsFileStatus stat = FSDirStatAndListingOp.createFileStatus( - fsNamesys.dir, + fsNamesys.dir, path, HdfsFileStatus.EMPTY_NAME, newFile, BlockStoragePolicySuite.ID_UNSPECIFIED, Snapshot.CURRENT_STATE_ID, false, iip); @@ -471,7 +471,7 @@ private long applyEditLogOp(FSEditLogOp op, FSDirectory fsDir, // add the op into retry cache if necessary if (toAddRetryCache) { HdfsFileStatus stat = FSDirStatAndListingOp.createFileStatus( - fsNamesys.dir, HdfsFileStatus.EMPTY_NAME, file, + fsNamesys.dir, path, HdfsFileStatus.EMPTY_NAME, file, BlockStoragePolicySuite.ID_UNSPECIFIED, Snapshot.CURRENT_STATE_ID, false, iip); fsNamesys.addCacheEntryWithPayload(appendOp.rpcClientId, diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java index 44c41d07b4457..7454850c16d36 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java @@ -406,7 +406,7 @@ void doUpgrade(FSNamesystem target) throws IOException { for (Iterator it = storage.dirIterator(false); it.hasNext();) { StorageDirectory sd = it.next(); try { - NNUpgradeUtil.doPreUpgrade(sd); + NNUpgradeUtil.doPreUpgrade(conf, sd); } catch (Exception e) { LOG.error("Failed to move aside pre-upgrade storage " + "in image directory " + sd.getRoot(), e); @@ -793,7 +793,7 @@ private boolean needsResaveBasedOnStaleCheckpoint( DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_PERIOD_KEY, DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_PERIOD_DEFAULT); final long checkpointTxnCount = conf.getLong( - DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_KEY, + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_KEY, DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_DEFAULT); long checkpointAge = Time.now() - imageFile.lastModified(); @@ -883,7 +883,7 @@ private static void updateCountForQuotaRecursively(BlockStoragePolicySuite bsps, final long namespace = counts.getNameSpace() - parentNamespace; final long nsQuota = q.getNameSpace(); if (Quota.isViolated(nsQuota, namespace)) { - LOG.error("BUG: Namespace quota violation in image for " + LOG.warn("Namespace quota violation in image for " + dir.getFullPathName() + " quota = " + nsQuota + " < consumed = " + namespace); } @@ -891,7 +891,7 @@ private static void updateCountForQuotaRecursively(BlockStoragePolicySuite bsps, final long ssConsumed = counts.getStorageSpace() - parentStoragespace; final long ssQuota = q.getStorageSpace(); if (Quota.isViolated(ssQuota, ssConsumed)) { - LOG.error("BUG: Storagespace quota violation in image for " + LOG.warn("Storagespace quota violation in image for " + dir.getFullPathName() + " quota = " + ssQuota + " < consumed = " + ssConsumed); } @@ -903,7 +903,7 @@ private static void updateCountForQuotaRecursively(BlockStoragePolicySuite bsps, parentTypeSpaces.get(t); final long typeQuota = q.getTypeSpaces().get(t); if (Quota.isViolated(typeQuota, typeSpace)) { - LOG.error("BUG: Storage type quota violation in image for " + LOG.warn("Storage type quota violation in image for " + dir.getFullPathName() + " type = " + t.toString() + " quota = " + typeQuota + " < consumed " + typeSpace); @@ -1062,11 +1062,35 @@ public synchronized void updateStorageVersion() throws IOException { } /** + * @param timeWindow a checkpoint is done if the latest checkpoint + * was done more than this number of seconds ago. + * @param txGap a checkpoint is done also if the gap between the latest tx id + * and the latest checkpoint is greater than this number. + * @return true if a checkpoint has been made * @see #saveNamespace(FSNamesystem, NameNodeFile, Canceler) */ - public synchronized void saveNamespace(FSNamesystem source) - throws IOException { + public synchronized boolean saveNamespace(long timeWindow, long txGap, + FSNamesystem source) throws IOException { + if (timeWindow > 0 || txGap > 0) { + final FSImageStorageInspector inspector = storage.readAndInspectDirs( + EnumSet.of(NameNodeFile.IMAGE, NameNodeFile.IMAGE_ROLLBACK), + StartupOption.REGULAR); + FSImageFile image = inspector.getLatestImages().get(0); + File imageFile = image.getFile(); + + final long checkpointTxId = image.getCheckpointTxId(); + final long checkpointAge = Time.now() - imageFile.lastModified(); + if (checkpointAge <= timeWindow * 1000 && + checkpointTxId >= this.getLastAppliedOrWrittenTxId() - txGap) { + return false; + } + } saveNamespace(source, NameNodeFile.IMAGE, null); + return true; + } + + public void saveNamespace(FSNamesystem source) throws IOException { + saveNamespace(0, 0, source); } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java index 956a2193cfec4..cce991f3f09fa 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdfs.server.namenode; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.DataInput; import java.io.DataInputStream; @@ -309,7 +309,7 @@ public void load(File curFile) throws IOException { StartupProgress prog = NameNode.getStartupProgress(); Step step = new Step(StepType.INODES); prog.beginStep(Phase.LOADING_FSIMAGE, step); - long startTime = now(); + long startTime = monotonicNow(); // // Load in bits @@ -441,8 +441,9 @@ public void load(File curFile) throws IOException { imgDigest = new MD5Hash(digester.digest()); loaded = true; - LOG.info("Image file " + curFile + " of size " + curFile.length() + - " bytes loaded in " + (now() - startTime)/1000 + " seconds."); + LOG.info("Image file " + curFile + " of size " + curFile.length() + + " bytes loaded in " + (monotonicNow() - startTime) / 1000 + + " seconds."); } /** Update the root node's attributes */ @@ -1240,7 +1241,7 @@ void save(File newFile, FSImageCompression compression) throws IOException { prog.beginStep(Phase.SAVING_CHECKPOINT, step); prog.setTotal(Phase.SAVING_CHECKPOINT, step, numINodes); Counter counter = prog.getCounter(Phase.SAVING_CHECKPOINT, step); - long startTime = now(); + long startTime = monotonicNow(); // // Write out data // @@ -1308,8 +1309,9 @@ void save(File newFile, FSImageCompression compression) throws IOException { // set md5 of the saved image savedDigest = new MD5Hash(digester.digest()); - LOG.info("Image file " + newFile + " of size " + newFile.length() + - " bytes saved in " + (now() - startTime)/1000 + " seconds."); + LOG.info("Image file " + newFile + " of size " + newFile.length() + + " bytes saved in " + (monotonicNow() - startTime) / 1000 + + " seconds."); } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java index d2b48f37454d7..59e1900bac5cd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java @@ -62,6 +62,7 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_ENABLE_RETRY_CACHE_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_LAZY_PERSIST_FILE_SCRUB_INTERVAL_SEC; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_LAZY_PERSIST_FILE_SCRUB_INTERVAL_SEC_DEFAULT; +import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_INODE_ATTRIBUTES_PROVIDER_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_MAX_OBJECTS_DEFAULT; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_MAX_OBJECTS_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY; @@ -88,6 +89,7 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_REPLICATION_KEY; import static org.apache.hadoop.hdfs.server.common.HdfsServerConstants.SECURITY_XATTR_UNREADABLE_BY_SUPERUSER; import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.BufferedWriter; import java.io.ByteArrayInputStream; @@ -276,8 +278,8 @@ import org.apache.hadoop.util.ChunkedArrayList; import org.apache.hadoop.util.Daemon; import org.apache.hadoop.util.DataChecksum; +import org.apache.hadoop.util.ReflectionUtils; import org.apache.hadoop.util.StringUtils; -import org.apache.hadoop.util.Time; import org.apache.hadoop.util.VersionInfo; import org.apache.log4j.Appender; import org.apache.log4j.AsyncAppender; @@ -536,6 +538,8 @@ private void logAuditEvent(boolean succeeded, private final TopConf topConf; private TopMetrics topMetrics; + private INodeAttributeProvider inodeAttributeProvider; + /** * Notify that loading of this FSDirectory is complete, and * it is imageLoaded for use @@ -683,7 +687,7 @@ static FSNamesystem loadFromDisk(Configuration conf) throws IOException { namesystem.setSafeMode(SafeModeAction.SAFEMODE_ENTER); } - long loadStart = now(); + long loadStart = monotonicNow(); try { namesystem.loadFSImage(startOpt); } catch (IOException ioe) { @@ -691,7 +695,7 @@ static FSNamesystem loadFromDisk(Configuration conf) throws IOException { fsImage.close(); throw ioe; } - long timeTakenToLoadFSImage = now() - loadStart; + long timeTakenToLoadFSImage = monotonicNow() - loadStart; LOG.info("Finished loading FSImage in " + timeTakenToLoadFSImage + " msecs"); NameNodeMetrics nnMetrics = NameNode.getNameNodeMetrics(); if (nnMetrics != null) { @@ -841,6 +845,13 @@ static FSNamesystem loadFromDisk(Configuration conf) throws IOException { this.isDefaultAuditLogger = auditLoggers.size() == 1 && auditLoggers.get(0) instanceof DefaultAuditLogger; this.retryCache = ignoreRetryCache ? null : initRetryCache(conf); + Class klass = conf.getClass( + DFS_NAMENODE_INODE_ATTRIBUTES_PROVIDER_KEY, + null, INodeAttributeProvider.class); + if (klass != null) { + inodeAttributeProvider = ReflectionUtils.newInstance(klass, conf); + LOG.info("Using INode attribute provider: " + klass.getName()); + } } catch(IOException e) { LOG.error(getClass().getSimpleName() + " initialization failed.", e); close(); @@ -1067,6 +1078,10 @@ void startCommonServices(Configuration conf, HAContext haContext) throws IOExcep registerMXBean(); DefaultMetricsSystem.instance().register(this); + if (inodeAttributeProvider != null) { + inodeAttributeProvider.start(); + dir.setINodeAttributeProvider(inodeAttributeProvider); + } snapshotManager.registerMXBean(); } @@ -1075,6 +1090,10 @@ void startCommonServices(Configuration conf, HAContext haContext) throws IOExcep */ void stopCommonServices() { writeLock(); + if (inodeAttributeProvider != null) { + dir.setINodeAttributeProvider(null); + inodeAttributeProvider.stop(); + } try { if (blockManager != null) blockManager.close(); } finally { @@ -1966,6 +1985,21 @@ boolean truncateInternal(String src, long newLength, throw new UnsupportedOperationException( "Cannot truncate lazy persist file " + src); } + + // Check if the file is already being truncated with the same length + final BlockInfoContiguous last = file.getLastBlock(); + if (last != null && last.getBlockUCState() == BlockUCState.UNDER_RECOVERY) { + final Block truncateBlock + = ((BlockInfoContiguousUnderConstruction)last).getTruncateBlock(); + if (truncateBlock != null) { + final long truncateLength = file.computeFileSize(false, false) + + truncateBlock.getNumBytes(); + if (newLength == truncateLength) { + return false; + } + } + } + // Opening an existing file for truncate. May need lease recovery. recoverLeaseInternal(RecoverLeaseOp.TRUNCATE_FILE, iip, src, clientName, clientMachine, false); @@ -1980,16 +2014,26 @@ boolean truncateInternal(String src, long newLength, ", truncate size: " + newLength + "."); } // Perform INodeFile truncation. - boolean onBlockBoundary = dir.truncate(iip, newLength, - toRemoveBlocks, mtime); + final QuotaCounts delta = new QuotaCounts.Builder().build(); + boolean onBlockBoundary = dir.truncate(iip, newLength, toRemoveBlocks, + mtime, delta); Block truncateBlock = null; - if(! onBlockBoundary) { + if(!onBlockBoundary) { // Open file for write, but don't log into edits long lastBlockDelta = file.computeFileSize() - newLength; assert lastBlockDelta > 0 : "delta is 0 only if on block bounday"; truncateBlock = prepareFileForTruncate(iip, clientName, clientMachine, lastBlockDelta, null); } + + // update the quota: use the preferred block size for UC block + dir.writeLock(); + try { + dir.updateCountNoQuotaCheck(iip, iip.length() - 1, delta); + } finally { + dir.writeUnlock(); + } + getEditLog().logTruncate(src, clientName, clientMachine, newLength, mtime, truncateBlock); return onBlockBoundary; @@ -2058,13 +2102,10 @@ Block prepareFileForTruncate(INodesInPath iip, + truncatedBlockUC.getTruncateBlock().getNumBytes() + " block=" + truncatedBlockUC); } - if(shouldRecoverNow) + if (shouldRecoverNow) { truncatedBlockUC.initializeBlockRecovery(newBlock.getGenerationStamp()); + } - // update the quota: use the preferred block size for UC block - final long diff = - file.getPreferredBlockSize() - truncatedBlockUC.getNumBytes(); - dir.updateSpaceConsumed(iip, 0, diff, file.getBlockReplication()); return newBlock; } @@ -2663,6 +2704,8 @@ LocatedBlock prepareFileForAppend(String src, INodesInPath iip, String leaseHolder, String clientMachine, boolean newBlock, boolean writeToEditLog, boolean logRetryCache) throws IOException { final INodeFile file = iip.getLastINode().asFile(); + final QuotaCounts delta = verifyQuotaForUCBlock(file, iip); + file.recordModification(iip.getLatestSnapshotId()); file.toUnderConstruction(leaseHolder, clientMachine); @@ -2672,10 +2715,15 @@ LocatedBlock prepareFileForAppend(String src, INodesInPath iip, LocatedBlock ret = null; if (!newBlock) { ret = blockManager.convertLastBlockToUnderConstruction(file, 0); - if (ret != null) { - // update the quota: use the preferred block size for UC block - final long diff = file.getPreferredBlockSize() - ret.getBlockSize(); - dir.updateSpaceConsumed(iip, 0, diff, file.getBlockReplication()); + if (ret != null && delta != null) { + Preconditions.checkState(delta.getStorageSpace() >= 0, + "appending to a block with size larger than the preferred block size"); + dir.writeLock(); + try { + dir.updateCountNoQuotaCheck(iip, iip.length() - 1, delta); + } finally { + dir.writeUnlock(); + } } } else { BlockInfoContiguous lastBlock = file.getLastBlock(); @@ -2691,6 +2739,52 @@ LocatedBlock prepareFileForAppend(String src, INodesInPath iip, return ret; } + /** + * Verify quota when using the preferred block size for UC block. This is + * usually used by append and truncate + * @throws QuotaExceededException when violating the storage quota + * @return expected quota usage update. null means no change or no need to + * update quota usage later + */ + private QuotaCounts verifyQuotaForUCBlock(INodeFile file, INodesInPath iip) + throws QuotaExceededException { + if (!isImageLoaded() || dir.shouldSkipQuotaChecks()) { + // Do not check quota if editlog is still being processed + return null; + } + if (file.getLastBlock() != null) { + final QuotaCounts delta = computeQuotaDeltaForUCBlock(file); + dir.readLock(); + try { + FSDirectory.verifyQuota(iip, iip.length() - 1, delta, null); + return delta; + } finally { + dir.readUnlock(); + } + } + return null; + } + + /** Compute quota change for converting a complete block to a UC block */ + private QuotaCounts computeQuotaDeltaForUCBlock(INodeFile file) { + final QuotaCounts delta = new QuotaCounts.Builder().build(); + final BlockInfoContiguous lastBlock = file.getLastBlock(); + if (lastBlock != null) { + final long diff = file.getPreferredBlockSize() - lastBlock.getNumBytes(); + final short repl = file.getBlockReplication(); + delta.addStorageSpace(diff * repl); + final BlockStoragePolicy policy = dir.getBlockStoragePolicySuite() + .getPolicy(file.getStoragePolicyID()); + List types = policy.chooseStorageTypes(repl); + for (StorageType t : types) { + if (t.supportTypeQuota()) { + delta.addTypeSpace(t, diff); + } + } + } + return delta; + } + /** * Recover lease; * Immediately revoke the lease of the current lease holder and start lease @@ -3091,7 +3185,7 @@ FileState analyzeFileState(String src, // doesn't match up with what we think is the last block. There are // four possibilities: // 1) This is the first block allocation of an append() pipeline - // which started appending exactly at a block boundary. + // which started appending exactly at or exceeding the block boundary. // In this case, the client isn't passed the previous block, // so it makes the allocateBlock() call with previous=null. // We can distinguish this since the last block of the file @@ -3116,7 +3210,7 @@ FileState analyzeFileState(String src, BlockInfoContiguous penultimateBlock = pendingFile.getPenultimateBlock(); if (previous == null && lastBlockInFile != null && - lastBlockInFile.getNumBytes() == pendingFile.getPreferredBlockSize() && + lastBlockInFile.getNumBytes() >= pendingFile.getPreferredBlockSize() && lastBlockInFile.isComplete()) { // Case 1 if (NameNode.stateChangeLog.isDebugEnabled()) { @@ -3224,7 +3318,7 @@ LocatedBlock getAdditionalDatanode(String src, long fileId, final DatanodeStorageInfo[] targets = blockManager.chooseTarget4AdditionalDatanode( src, numAdditionalNodes, clientnode, chosen, excludes, preferredblocksize, storagePolicyID); - final LocatedBlock lb = new LocatedBlock(blk, targets); + final LocatedBlock lb = new LocatedBlock(blk, targets, -1, false); blockManager.setBlockToken(lb, AccessMode.COPY); return lb; } @@ -4153,6 +4247,8 @@ void commitBlockSynchronization(ExtendedBlock oldBlock, throw new IOException("Block (=" + oldBlock + ") not found"); } } + final long oldGenerationStamp = storedBlock.getGenerationStamp(); + final long oldNumBytes = storedBlock.getNumBytes(); // // The implementation of delete operation (see @deleteInternal method) // first removes the file paths from namespace, and delays the removal @@ -4213,8 +4309,6 @@ void commitBlockSynchronization(ExtendedBlock oldBlock, } // find the DatanodeDescriptor objects - // There should be no locations in the blockManager till now because the - // file is underConstruction ArrayList trimmedTargets = new ArrayList(newtargets.length); ArrayList trimmedStorages = @@ -4258,6 +4352,10 @@ void commitBlockSynchronization(ExtendedBlock oldBlock, iFile.setLastBlock(truncatedBlock, trimmedStorageInfos); } else { iFile.setLastBlock(storedBlock, trimmedStorageInfos); + if (closeFile) { + blockManager.markBlockReplicasAsCorrupt(storedBlock, + oldGenerationStamp, oldNumBytes, trimmedStorageInfos); + } } } @@ -4686,7 +4784,7 @@ public int getExpiredHeartbeats() { @Metric({"TransactionsSinceLastCheckpoint", "Number of transactions since last checkpoint"}) public long getTransactionsSinceLastCheckpoint() { - return getEditLog().getLastWrittenTxId() - + return getFSImage().getLastAppliedOrWrittenTxId() - getFSImage().getStorage().getMostRecentCheckpointTxId(); } @@ -4849,14 +4947,13 @@ DatanodeStorageReport[] getDatanodeStorageReport(final DatanodeReportType type * Save namespace image. * This will save current namespace into fsimage file and empty edits file. * Requires superuser privilege and safe mode. - * - * @throws AccessControlException if superuser privilege is violated. - * @throws IOException if */ - void saveNamespace() throws AccessControlException, IOException { + boolean saveNamespace(final long timeWindow, final long txGap) + throws IOException { checkOperation(OperationCategory.UNCHECKED); checkSuperuserPrivilege(); + boolean saved = false; cpLock(); // Block if a checkpointing is in progress on standby. readLock(); try { @@ -4866,12 +4963,15 @@ void saveNamespace() throws AccessControlException, IOException { throw new IOException("Safe mode should be turned ON " + "in order to create namespace image."); } - getFSImage().saveNamespace(this); + saved = getFSImage().saveNamespace(timeWindow, txGap, this); } finally { readUnlock(); cpUnlock(); } - LOG.info("New namespace image has been created"); + if (saved) { + LOG.info("New namespace image has been created"); + } + return saved; } /** @@ -4992,6 +5092,7 @@ public class SafeModeInfo { *
    >0 safe mode is on, but we are in extension period */ private long reached = -1; + private long reachedTimestamp = -1; /** Total number of blocks. */ int blockTotal; /** Number of safe blocks. */ @@ -5092,6 +5193,7 @@ private synchronized boolean isOn() { */ private void enter() { this.reached = 0; + this.reachedTimestamp = 0; } /** @@ -5115,6 +5217,7 @@ private synchronized void leave() { NameNode.stateChangeLog.info("STATE* Safe mode is OFF"); } reached = -1; + reachedTimestamp = -1; safeMode = null; final NetworkTopology nt = blockManager.getDatanodeManager().getNetworkTopology(); NameNode.stateChangeLog.info("STATE* Network topology has " @@ -5153,7 +5256,7 @@ private synchronized boolean canLeave() { return false; } - if (now() - reached < extension) { + if (monotonicNow() - reached < extension) { reportStatus("STATE* Safe mode ON, in safe mode extension.", false); return false; } @@ -5209,7 +5312,8 @@ private void checkMode() { return; } // start monitor - reached = now(); + reached = monotonicNow(); + reachedTimestamp = now(); if (smmthread == null) { smmthread = new Daemon(new SafeModeMonitor()); smmthread.start(); @@ -5334,7 +5438,7 @@ String getTurnOffTip() { msg += String.format( "The reported blocks %d needs additional %d" + " blocks to reach the threshold %.4f of total blocks %d.%n", - blockSafe, (blockThreshold - blockSafe) + 1, threshold, blockTotal); + blockSafe, (blockThreshold - blockSafe), threshold, blockTotal); thresholdsMet = false; } else { msg += String.format("The reported blocks %d has reached the threshold" @@ -5356,8 +5460,8 @@ String getTurnOffTip() { if (!thresholdsMet) { msg += "once the thresholds have been reached."; - } else if (reached + extension - now() > 0) { - msg += ("in " + (reached + extension - now()) / 1000 + " seconds."); + } else if (reached + extension - monotonicNow() > 0) { + msg += ("in " + (reached + extension - monotonicNow()) / 1000 + " seconds."); } else { msg += "soon."; } @@ -5383,7 +5487,7 @@ public String toString() { + ". Target blocks = " + blockThreshold + " for threshold = %" + threshold + ". Minimal replication = " + safeReplication + "."; if (reached > 0) - resText += " Threshold was reached " + new Date(reached) + "."; + resText += " Threshold was reached " + new Date(reachedTimestamp) + "."; return resText; } @@ -5862,7 +5966,7 @@ public String getHAState() { @Metric public long getMillisSinceLastLoadedEdits() { if (isInStandbyState() && editLogTailer != null) { - return now() - editLogTailer.getLastLoadTimestamp(); + return monotonicNow() - editLogTailer.getLastLoadTimeMs(); } else { return 0; } @@ -6904,7 +7008,7 @@ public String getDecomNodes() { } private long getLastContact(DatanodeDescriptor alivenode) { - return (Time.now() - alivenode.getLastUpdate())/1000; + return (monotonicNow() - alivenode.getLastUpdateMonotonic())/1000; } private long getDfsUsed(DatanodeDescriptor alivenode) { @@ -7464,7 +7568,7 @@ public RollingUpgradeInfo.Bean getRollingUpgradeStatus() { /** Is rolling upgrade in progress? */ public boolean isRollingUpgrade() { - return rollingUpgradeInfo != null; + return rollingUpgradeInfo != null && !rollingUpgradeInfo.isFinalized(); } void checkRollingUpgrade(String action) throws RollingUpgradeException { @@ -7475,20 +7579,19 @@ void checkRollingUpgrade(String action) throws RollingUpgradeException { } } - void finalizeRollingUpgrade() throws IOException { + RollingUpgradeInfo finalizeRollingUpgrade() throws IOException { checkSuperuserPrivilege(); checkOperation(OperationCategory.WRITE); writeLock(); - final RollingUpgradeInfo returnInfo; try { checkOperation(OperationCategory.WRITE); if (!isRollingUpgrade()) { - return; + return null; } checkNameNodeSafeMode("Failed to finalize rolling upgrade"); - returnInfo = finalizeRollingUpgradeInternal(now()); - getEditLog().logFinalizeRollingUpgrade(returnInfo.getFinalizeTime()); + finalizeRollingUpgradeInternal(now()); + getEditLog().logFinalizeRollingUpgrade(rollingUpgradeInfo.getFinalizeTime()); if (haEnabled) { // roll the edit log to make sure the standby NameNode can tail getFSImage().rollEditLog(); @@ -7508,14 +7611,12 @@ void finalizeRollingUpgrade() throws IOException { if (auditLog.isInfoEnabled() && isExternalInvocation()) { logAuditEvent(true, "finalizeRollingUpgrade", null, null, null); } - return; + return rollingUpgradeInfo; } - RollingUpgradeInfo finalizeRollingUpgradeInternal(long finalizeTime) - throws RollingUpgradeException { - final long startTime = rollingUpgradeInfo.getStartTime(); - rollingUpgradeInfo = null; - return new RollingUpgradeInfo(blockPoolId, false, startTime, finalizeTime); + void finalizeRollingUpgradeInternal(long finalizeTime) { + // Set the finalize time + rollingUpgradeInfo.finalize(finalizeTime); } long addCacheDirective(CacheDirectiveInfo directive, @@ -7853,7 +7954,7 @@ void createEncryptionZone(final String src, final String keyName, throw new IOException("Key " + keyName + " doesn't exist."); } // If the provider supports pool for EDEKs, this will fill in the pool - generateEncryptedDataEncryptionKey(keyName); + provider.warmUpEncryptedKeys(keyName); createEncryptionZoneInt(src, metadata.getCipher(), keyName, logRetryCache); } catch (AccessControlException e) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java index 050848492d4d0..e6570f544e622 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java @@ -20,7 +20,6 @@ import java.util.Arrays; import java.util.Collections; import java.util.HashSet; -import java.util.List; import java.util.Set; import java.util.Stack; @@ -30,6 +29,8 @@ import org.apache.hadoop.fs.permission.AclEntryType; import org.apache.hadoop.fs.permission.FsAction; import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.hdfs.DFSUtil; +import org.apache.hadoop.hdfs.server.namenode.INodeAttributeProvider.AccessControlEnforcer; import org.apache.hadoop.hdfs.util.ReadOnlyList; import org.apache.hadoop.security.AccessControlException; import org.apache.hadoop.security.UserGroupInformation; @@ -41,25 +42,25 @@ * * Some of the helper methods are gaurded by {@link FSNamesystem#readLock()}. */ -class FSPermissionChecker { +class FSPermissionChecker implements AccessControlEnforcer { static final Log LOG = LogFactory.getLog(UserGroupInformation.class); /** @return a string for throwing {@link AccessControlException} */ - private String toAccessControlString(INode inode, int snapshotId, + private String toAccessControlString(INodeAttributes inodeAttrib, String path, FsAction access, FsPermission mode) { - return toAccessControlString(inode, snapshotId, access, mode, false); + return toAccessControlString(inodeAttrib, path, access, mode, false); } /** @return a string for throwing {@link AccessControlException} */ - private String toAccessControlString(INode inode, int snapshotId, FsAction access, - FsPermission mode, boolean deniedFromAcl) { + private String toAccessControlString(INodeAttributes inodeAttrib, + String path, FsAction access, FsPermission mode, boolean deniedFromAcl) { StringBuilder sb = new StringBuilder("Permission denied: ") - .append("user=").append(user).append(", ") + .append("user=").append(getUser()).append(", ") .append("access=").append(access).append(", ") - .append("inode=\"").append(inode.getFullPathName()).append("\":") - .append(inode.getUserName(snapshotId)).append(':') - .append(inode.getGroupName(snapshotId)).append(':') - .append(inode.isDirectory() ? 'd' : '-') + .append("inode=\"").append(path).append("\":") + .append(inodeAttrib.getUserName()).append(':') + .append(inodeAttrib.getGroupName()).append(':') + .append(inodeAttrib.isDirectory() ? 'd' : '-') .append(mode); if (deniedFromAcl) { sb.append("+"); @@ -67,42 +68,59 @@ private String toAccessControlString(INode inode, int snapshotId, FsAction acces return sb.toString(); } + private final String fsOwner; + private final String supergroup; + private final UserGroupInformation callerUgi; + private final String user; - /** A set with group namess. Not synchronized since it is unmodifiable */ private final Set groups; private final boolean isSuper; + private final INodeAttributeProvider attributeProvider; + FSPermissionChecker(String fsOwner, String supergroup, - UserGroupInformation callerUgi) { - HashSet s = new HashSet(Arrays.asList(callerUgi.getGroupNames())); + UserGroupInformation callerUgi, + INodeAttributeProvider attributeProvider) { + this.fsOwner = fsOwner; + this.supergroup = supergroup; + this.callerUgi = callerUgi; + HashSet s = + new HashSet(Arrays.asList(callerUgi.getGroupNames())); groups = Collections.unmodifiableSet(s); user = callerUgi.getShortUserName(); isSuper = user.equals(fsOwner) || groups.contains(supergroup); + this.attributeProvider = attributeProvider; } - /** - * Check if the callers group contains the required values. - * @param group group to check - */ - public boolean containsGroup(String group) {return groups.contains(group);} + public boolean containsGroup(String group) { + return groups.contains(group); + } public String getUser() { return user; } - + + public Set getGroups() { + return groups; + } + public boolean isSuperUser() { return isSuper; } - + + public INodeAttributeProvider getAttributesProvider() { + return attributeProvider; + } + /** * Verify if the caller has the required permission. This will result into * an exception if the caller is not allowed to access the resource. */ public void checkSuperuserPrivilege() throws AccessControlException { - if (!isSuper) { + if (!isSuperUser()) { throw new AccessControlException("Access denied for user " - + user + ". Superuser privilege is required"); + + getUser() + ". Superuser privilege is required"); } } @@ -154,64 +172,98 @@ void checkPermission(INodesInPath inodesInPath, boolean doCheckOwner, // check if (parentAccess != null) && file exists, then check sb // If resolveLink, the check is performed on the link target. final int snapshotId = inodesInPath.getPathSnapshotId(); - final int length = inodesInPath.length(); - final INode last = length > 0 ? inodesInPath.getLastINode() : null; - final INode parent = length > 1 ? inodesInPath.getINode(-2) : null; + final INode[] inodes = inodesInPath.getINodesArray(); + final INodeAttributes[] inodeAttrs = new INodeAttributes[inodes.length]; + final byte[][] pathByNameArr = new byte[inodes.length][]; + for (int i = 0; i < inodes.length && inodes[i] != null; i++) { + if (inodes[i] != null) { + pathByNameArr[i] = inodes[i].getLocalNameBytes(); + inodeAttrs[i] = getINodeAttrs(pathByNameArr, i, inodes[i], snapshotId); + } + } + + String path = inodesInPath.getPath(); + int ancestorIndex = inodes.length - 2; + + AccessControlEnforcer enforcer = + getAttributesProvider().getExternalAccessControlEnforcer(this); + enforcer.checkPermission(fsOwner, supergroup, callerUgi, inodeAttrs, inodes, + pathByNameArr, snapshotId, path, ancestorIndex, doCheckOwner, + ancestorAccess, parentAccess, access, subAccess, ignoreEmptyDir); + } - checkTraverse(inodesInPath, snapshotId); + @Override + public void checkPermission(String fsOwner, String supergroup, + UserGroupInformation callerUgi, INodeAttributes[] inodeAttrs, + INode[] inodes, byte[][] pathByNameArr, int snapshotId, String path, + int ancestorIndex, boolean doCheckOwner, FsAction ancestorAccess, + FsAction parentAccess, FsAction access, FsAction subAccess, + boolean ignoreEmptyDir) + throws AccessControlException { + for(; ancestorIndex >= 0 && inodes[ancestorIndex] == null; + ancestorIndex--); + checkTraverse(inodeAttrs, path, ancestorIndex); + final INodeAttributes last = inodeAttrs[inodeAttrs.length - 1]; if (parentAccess != null && parentAccess.implies(FsAction.WRITE) - && length > 1 && last != null) { - checkStickyBit(parent, last, snapshotId); + && inodeAttrs.length > 1 && last != null) { + checkStickyBit(inodeAttrs[inodeAttrs.length - 2], last); } - if (ancestorAccess != null && length > 1) { - List inodes = inodesInPath.getReadOnlyINodes(); - INode ancestor = null; - for (int i = inodes.size() - 2; i >= 0 && (ancestor = inodes.get(i)) == - null; i--); - check(ancestor, snapshotId, ancestorAccess); + if (ancestorAccess != null && inodeAttrs.length > 1) { + check(inodeAttrs, path, ancestorIndex, ancestorAccess); } - if (parentAccess != null && length > 1 && parent != null) { - check(parent, snapshotId, parentAccess); + if (parentAccess != null && inodeAttrs.length > 1) { + check(inodeAttrs, path, inodeAttrs.length - 2, parentAccess); } if (access != null) { - check(last, snapshotId, access); + check(last, path, access); } if (subAccess != null) { - checkSubAccess(last, snapshotId, subAccess, ignoreEmptyDir); + INode rawLast = inodes[inodeAttrs.length - 1]; + checkSubAccess(pathByNameArr, inodeAttrs.length - 1, rawLast, + snapshotId, subAccess, ignoreEmptyDir); } if (doCheckOwner) { - checkOwner(last, snapshotId); + checkOwner(last); } } + private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx, + INode inode, int snapshotId) { + INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId); + if (getAttributesProvider() != null) { + String[] elements = new String[pathIdx + 1]; + for (int i = 0; i < elements.length; i++) { + elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); + } + inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs); + } + return inodeAttrs; + } + /** Guarded by {@link FSNamesystem#readLock()} */ - private void checkOwner(INode inode, int snapshotId + private void checkOwner(INodeAttributes inode ) throws AccessControlException { - if (inode != null && user.equals(inode.getUserName(snapshotId))) { + if (getUser().equals(inode.getUserName())) { return; } throw new AccessControlException( "Permission denied. user=" - + user + " is not the owner of inode=" + inode); + + getUser() + " is not the owner of inode=" + inode); } /** Guarded by {@link FSNamesystem#readLock()} */ - private void checkTraverse(INodesInPath iip, int snapshotId) - throws AccessControlException { - List inodes = iip.getReadOnlyINodes(); - for (int i = 0; i < inodes.size() - 1; i++) { - INode inode = inodes.get(i); - if (inode == null) { - break; - } - check(inode, snapshotId, FsAction.EXECUTE); + private void checkTraverse(INodeAttributes[] inodes, String path, int last + ) throws AccessControlException { + for(int j = 0; j <= last; j++) { + check(inodes[j], path, FsAction.EXECUTE); } } /** Guarded by {@link FSNamesystem#readLock()} */ - private void checkSubAccess(INode inode, int snapshotId, FsAction access, - boolean ignoreEmptyDir) throws AccessControlException { + private void checkSubAccess(byte[][] pathByNameArr, int pathIdx, INode inode, + int snapshotId, FsAction access, boolean ignoreEmptyDir) + throws AccessControlException { if (inode == null || !inode.isDirectory()) { return; } @@ -221,7 +273,9 @@ private void checkSubAccess(INode inode, int snapshotId, FsAction access, INodeDirectory d = directories.pop(); ReadOnlyList cList = d.getChildrenList(snapshotId); if (!(cList.isEmpty() && ignoreEmptyDir)) { - check(d, snapshotId, access); + //TODO have to figure this out with inodeattribute provider + check(getINodeAttrs(pathByNameArr, pathIdx, d, snapshotId), + inode.getFullPathName(), access); } for(INode child : cList) { @@ -233,37 +287,37 @@ private void checkSubAccess(INode inode, int snapshotId, FsAction access, } /** Guarded by {@link FSNamesystem#readLock()} */ - private void check(INode inode, int snapshotId, FsAction access) - throws AccessControlException { + private void check(INodeAttributes[] inodes, String path, int i, FsAction access + ) throws AccessControlException { + check(i >= 0 ? inodes[i] : null, path, access); + } + + private void check(INodeAttributes inode, String path, FsAction access + ) throws AccessControlException { if (inode == null) { return; } - FsPermission mode = inode.getFsPermission(snapshotId); - AclFeature aclFeature = inode.getAclFeature(snapshotId); + final FsPermission mode = inode.getFsPermission(); + final AclFeature aclFeature = inode.getAclFeature(); if (aclFeature != null) { // It's possible that the inode has a default ACL but no access ACL. int firstEntry = aclFeature.getEntryAt(0); if (AclEntryStatusFormat.getScope(firstEntry) == AclEntryScope.ACCESS) { - checkAccessAcl(inode, snapshotId, access, mode, aclFeature); + checkAccessAcl(inode, path, access, mode, aclFeature); return; } } - checkFsPermission(inode, snapshotId, access, mode); - } - - private void checkFsPermission(INode inode, int snapshotId, FsAction access, - FsPermission mode) throws AccessControlException { - if (user.equals(inode.getUserName(snapshotId))) { //user class + if (getUser().equals(inode.getUserName())) { //user class if (mode.getUserAction().implies(access)) { return; } } - else if (groups.contains(inode.getGroupName(snapshotId))) { //group class + else if (getGroups().contains(inode.getGroupName())) { //group class if (mode.getGroupAction().implies(access)) { return; } } else { //other class if (mode.getOtherAction().implies(access)) { return; } } throw new AccessControlException( - toAccessControlString(inode, snapshotId, access, mode)); + toAccessControlString(inode, path, access, mode)); } /** @@ -282,20 +336,20 @@ else if (groups.contains(inode.getGroupName(snapshotId))) { //group class * - The other entry must not have a name. * - Default entries may be present, but they are ignored during enforcement. * - * @param inode INode accessed inode + * @param inode INodeAttributes accessed inode * @param snapshotId int snapshot ID * @param access FsAction requested permission * @param mode FsPermission mode from inode * @param aclFeature AclFeature of inode * @throws AccessControlException if the ACL denies permission */ - private void checkAccessAcl(INode inode, int snapshotId, FsAction access, - FsPermission mode, AclFeature aclFeature) + private void checkAccessAcl(INodeAttributes inode, String path, + FsAction access, FsPermission mode, AclFeature aclFeature) throws AccessControlException { boolean foundMatch = false; // Use owner entry from permission bits if user is owner. - if (user.equals(inode.getUserName(snapshotId))) { + if (getUser().equals(inode.getUserName())) { if (mode.getUserAction().implies(access)) { return; } @@ -314,7 +368,7 @@ private void checkAccessAcl(INode inode, int snapshotId, FsAction access, if (type == AclEntryType.USER) { // Use named user entry with mask from permission bits applied if user // matches name. - if (user.equals(name)) { + if (getUser().equals(name)) { FsAction masked = AclEntryStatusFormat.getPermission(entry).and( mode.getGroupAction()); if (masked.implies(access)) { @@ -328,8 +382,8 @@ private void checkAccessAcl(INode inode, int snapshotId, FsAction access, // applied if user is a member and entry grants access. If user is a // member of multiple groups that have entries that grant access, then // it doesn't matter which is chosen, so exit early after first match. - String group = name == null ? inode.getGroupName(snapshotId) : name; - if (groups.contains(group)) { + String group = name == null ? inode.getGroupName() : name; + if (getGroups().contains(group)) { FsAction masked = AclEntryStatusFormat.getPermission(entry).and( mode.getGroupAction()); if (masked.implies(access)) { @@ -347,28 +401,28 @@ private void checkAccessAcl(INode inode, int snapshotId, FsAction access, } throw new AccessControlException( - toAccessControlString(inode, snapshotId, access, mode, true)); + toAccessControlString(inode, path, access, mode)); } /** Guarded by {@link FSNamesystem#readLock()} */ - private void checkStickyBit(INode parent, INode inode, int snapshotId + private void checkStickyBit(INodeAttributes parent, INodeAttributes inode ) throws AccessControlException { - if(!parent.getFsPermission(snapshotId).getStickyBit()) { + if (!parent.getFsPermission().getStickyBit()) { return; } // If this user is the directory owner, return - if(parent.getUserName(snapshotId).equals(user)) { + if (parent.getUserName().equals(getUser())) { return; } // if this user is the file owner, return - if(inode.getUserName(snapshotId).equals(user)) { + if (inode.getUserName().equals(getUser())) { return; } throw new AccessControlException("Permission denied by sticky bit setting:" + - " user=" + user + ", inode=" + inode); + " user=" + getUser() + ", inode=" + inode); } /** @@ -384,11 +438,11 @@ public void checkPermission(CachePool pool, FsAction access) if (isSuperUser()) { return; } - if (user.equals(pool.getOwnerName()) + if (getUser().equals(pool.getOwnerName()) && mode.getUserAction().implies(access)) { return; } - if (groups.contains(pool.getGroupName()) + if (getGroups().contains(pool.getGroupName()) && mode.getGroupAction().implies(access)) { return; } @@ -396,7 +450,7 @@ public void checkPermission(CachePool pool, FsAction access) return; } throw new AccessControlException("Permission denied while accessing pool " - + pool.getPoolName() + ": user " + user + " does not have " + + pool.getPoolName() + ": user " + getUser() + " does not have " + access.toString() + " permissions."); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java index 921803c96c4b2..12733fd0d3df6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java @@ -581,7 +581,7 @@ public String toString() { public void doPreUpgrade() throws IOException { LOG.info("Starting upgrade of edits directory " + sd.getRoot()); try { - NNUpgradeUtil.doPreUpgrade(sd); + NNUpgradeUtil.doPreUpgrade(conf, sd); } catch (IOException ioe) { LOG.error("Failed to move aside pre-upgrade storage " + "in image directory " + sd.getRoot(), ioe); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java index 8c4e466e2d91f..586cce4b1aa9c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java @@ -432,9 +432,9 @@ public abstract void destroyAndCollectBlocks( BlocksMapUpdateInfo collectedBlocks, List removedINodes); /** Compute {@link ContentSummary}. Blocking call */ - public final ContentSummary computeContentSummary() { + public final ContentSummary computeContentSummary(BlockStoragePolicySuite bsps) { return computeAndConvertContentSummary( - new ContentSummaryComputationContext()); + new ContentSummaryComputationContext(bsps)); } /** @@ -442,17 +442,22 @@ public final ContentSummary computeContentSummary() { */ public final ContentSummary computeAndConvertContentSummary( ContentSummaryComputationContext summary) { - Content.Counts counts = computeContentSummary(summary).getCounts(); + ContentCounts counts = computeContentSummary(summary).getCounts(); final QuotaCounts q = getQuotaCounts(); - return new ContentSummary(counts.get(Content.LENGTH), - counts.get(Content.FILE) + counts.get(Content.SYMLINK), - counts.get(Content.DIRECTORY), q.getNameSpace(), - counts.get(Content.DISKSPACE), q.getStorageSpace()); - // TODO: storage type quota reporting HDFS-7701. + return new ContentSummary.Builder(). + length(counts.getLength()). + fileCount(counts.getFileCount() + counts.getSymlinkCount()). + directoryCount(counts.getDirectoryCount()). + quota(q.getNameSpace()). + spaceConsumed(counts.getStoragespace()). + spaceQuota(q.getStorageSpace()). + typeConsumed(counts.getTypeSpaces()). + typeQuota(q.getTypeSpaces().asArray()). + build(); } /** - * Count subtree content summary with a {@link Content.Counts}. + * Count subtree content summary with a {@link ContentCounts}. * * @param summary the context object holding counts for the subtree. * @return The same objects as summary. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java new file mode 100644 index 0000000000000..b12e147116dce --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.server.namenode; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; +import java.util.Set; + +import com.google.common.annotations.VisibleForTesting; + +import org.apache.commons.lang.StringUtils; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsAction; +import org.apache.hadoop.security.AccessControlException; +import org.apache.hadoop.security.UserGroupInformation; + +@InterfaceAudience.Public +@InterfaceStability.Unstable +public abstract class INodeAttributeProvider { + + /** + * The AccessControlEnforcer allows implementations to override the + * default File System permission checking logic enforced on a file system + * object + */ + public interface AccessControlEnforcer { + + /** + * Checks permission on a file system object. Has to throw an Exception + * if the filesystem object is not accessessible by the calling Ugi. + * @param fsOwner Filesystem owner (The Namenode user) + * @param supergroup super user geoup + * @param callerUgi UserGroupInformation of the caller + * @param inodeAttrs Array of INode attributes for each path element in the + * the path + * @param inodes Array of INodes for each path element in the path + * @param pathByNameArr Array of byte arrays of the LocalName + * @param snapshotId the snapshotId of the requested path + * @param path Path String + * @param ancestorIndex Index of ancestor + * @param doCheckOwner perform ownership check + * @param ancestorAccess The access required by the ancestor of the path. + * @param parentAccess The access required by the parent of the path. + * @param access The access required by the path. + * @param subAccess If path is a directory, It is the access required of + * the path and all the sub-directories. If path is not a + * directory, there should ideally be no effect. + * @param ignoreEmptyDir Ignore permission checking for empty directory? + * @throws AccessControlException + */ + public abstract void checkPermission(String fsOwner, String supergroup, + UserGroupInformation callerUgi, INodeAttributes[] inodeAttrs, + INode[] inodes, byte[][] pathByNameArr, int snapshotId, String path, + int ancestorIndex, boolean doCheckOwner, FsAction ancestorAccess, + FsAction parentAccess, FsAction access, FsAction subAccess, + boolean ignoreEmptyDir) + throws AccessControlException; + + } + /** + * Initialize the provider. This method is called at NameNode startup + * time. + */ + public abstract void start(); + + /** + * Shutdown the provider. This method is called at NameNode shutdown time. + */ + public abstract void stop(); + + @VisibleForTesting + String[] getPathElements(String path) { + path = path.trim(); + if (path.charAt(0) != Path.SEPARATOR_CHAR) { + throw new IllegalArgumentException("It must be an absolute path: " + + path); + } + int numOfElements = StringUtils.countMatches(path, Path.SEPARATOR); + if (path.length() > 1 && path.endsWith(Path.SEPARATOR)) { + numOfElements--; + } + String[] pathElements = new String[numOfElements]; + int elementIdx = 0; + int idx = 0; + int found = path.indexOf(Path.SEPARATOR_CHAR, idx); + while (found > -1) { + if (found > idx) { + pathElements[elementIdx++] = path.substring(idx, found); + } + idx = found + 1; + found = path.indexOf(Path.SEPARATOR_CHAR, idx); + } + if (idx < path.length()) { + pathElements[elementIdx] = path.substring(idx); + } + return pathElements; + } + + public INodeAttributes getAttributes(String fullPath, INodeAttributes inode) { + return getAttributes(getPathElements(fullPath), inode); + } + + public abstract INodeAttributes getAttributes(String[] pathElements, + INodeAttributes inode); + + /** + * Can be over-ridden by implementations to provide a custom Access Control + * Enforcer that can provide an alternate implementation of the + * default permission checking logic. + * @param defaultEnforcer The Default AccessControlEnforcer + * @return The AccessControlEnforcer to use + */ + public AccessControlEnforcer getExternalAccessControlEnforcer( + AccessControlEnforcer defaultEnforcer) { + return defaultEnforcer; + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java index 0f76b682ebff6..7b780c2177a16 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributes.java @@ -28,6 +28,9 @@ */ @InterfaceAudience.Private public interface INodeAttributes { + + public boolean isDirectory(); + /** * @return null if the local name is null; * otherwise, return the local name byte array. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java index ebb8ae4552cec..dadb8c70652a0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java @@ -664,7 +664,7 @@ protected ContentSummaryComputationContext computeDirectoryContentSummary( } // Increment the directory count for this directory. - summary.getCounts().add(Content.DIRECTORY, 1); + summary.getCounts().addContent(Content.DIRECTORY, 1); // Relinquish and reacquire locks if necessary. summary.yield(); return summary; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java index 956deaecd4179..240aa15ee5fb5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectoryAttributes.java @@ -52,6 +52,10 @@ public QuotaCounts getQuotaCounts() { storageSpace(-1).typeSpaces(-1).build(); } + public boolean isDirectory() { + return true; + } + @Override public boolean metadataEquals(INodeDirectoryAttributes other) { return other != null diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java index 887a25911ee73..a6f07f99bc7a1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java @@ -24,6 +24,7 @@ import java.io.IOException; import java.io.PrintWriter; import java.util.Arrays; +import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Set; @@ -598,22 +599,36 @@ public final QuotaCounts computeQuotaUsage( @Override public final ContentSummaryComputationContext computeContentSummary( final ContentSummaryComputationContext summary) { - final Content.Counts counts = summary.getCounts(); + final ContentCounts counts = summary.getCounts(); FileWithSnapshotFeature sf = getFileWithSnapshotFeature(); + long fileLen = 0; if (sf == null) { - counts.add(Content.LENGTH, computeFileSize()); - counts.add(Content.FILE, 1); + fileLen = computeFileSize(); + counts.addContent(Content.FILE, 1); } else { final FileDiffList diffs = sf.getDiffs(); final int n = diffs.asList().size(); - counts.add(Content.FILE, n); + counts.addContent(Content.FILE, n); if (n > 0 && sf.isCurrentFileDeleted()) { - counts.add(Content.LENGTH, diffs.getLast().getFileSize()); + fileLen = diffs.getLast().getFileSize(); } else { - counts.add(Content.LENGTH, computeFileSize()); + fileLen = computeFileSize(); + } + } + counts.addContent(Content.LENGTH, fileLen); + counts.addContent(Content.DISKSPACE, storagespaceConsumed()); + + if (getStoragePolicyID() != BlockStoragePolicySuite.ID_UNSPECIFIED){ + BlockStoragePolicy bsp = summary.getBlockStoragePolicySuite(). + getPolicy(getStoragePolicyID()); + List storageTypes = bsp.chooseStorageTypes(getFileReplication()); + for (StorageType t : storageTypes) { + if (!t.supportTypeQuota()) { + continue; + } + counts.addTypeSpace(t, fileLen); } } - counts.add(Content.DISKSPACE, storagespaceConsumed()); return summary; } @@ -804,6 +819,43 @@ public long collectBlocksBeyondMax(final long max, return size; } + /** + * compute the quota usage change for a truncate op + * @param newLength the length for truncation + * @return the quota usage delta (not considering replication factor) + */ + long computeQuotaDeltaForTruncate(final long newLength) { + final BlockInfoContiguous[] blocks = getBlocks(); + if (blocks == null || blocks.length == 0) { + return 0; + } + + int n = 0; + long size = 0; + for (; n < blocks.length && newLength > size; n++) { + size += blocks[n].getNumBytes(); + } + final boolean onBoundary = size == newLength; + + long truncateSize = 0; + for (int i = (onBoundary ? n : n - 1); i < blocks.length; i++) { + truncateSize += blocks[i].getNumBytes(); + } + + FileWithSnapshotFeature sf = getFileWithSnapshotFeature(); + if (sf != null) { + FileDiff diff = sf.getDiffs().getLast(); + BlockInfoContiguous[] sblocks = diff != null ? diff.getBlocks() : null; + if (sblocks != null) { + for (int i = (onBoundary ? n : n-1); i < blocks.length + && i < sblocks.length && blocks[i].equals(sblocks[i]); i++) { + truncateSize -= blocks[i].getNumBytes(); + } + } + } + return onBoundary ? -truncateSize : (getPreferredBlockSize() - truncateSize); + } + void truncateBlocksTo(int n) { final BlockInfoContiguous[] newBlocks; if (n == 0) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java index 0f85bab5feb2f..204c8ac6a8b67 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java @@ -59,6 +59,11 @@ public SnapshotCopy(INodeFile file) { this.header = file.getHeaderLong(); } + @Override + public boolean isDirectory() { + return false; + } + @Override public short getFileReplication() { return HeaderFormat.getReplication(header); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java index 911279a5fbb48..eee50a52919b0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java @@ -23,6 +23,7 @@ import java.util.Comparator; import java.util.List; +import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.fs.permission.PermissionStatus; import org.apache.hadoop.hdfs.protocol.QuotaExceededException; @@ -512,7 +513,8 @@ public final ContentSummaryComputationContext computeContentSummary( //only count storagespace for WithName final QuotaCounts q = new QuotaCounts.Builder().build(); computeQuotaUsage(summary.getBlockStoragePolicySuite(), q, false, lastSnapshotId); - summary.getCounts().add(Content.DISKSPACE, q.getStorageSpace()); + summary.getCounts().addContent(Content.DISKSPACE, q.getStorageSpace()); + summary.getCounts().addTypeSpaces(q.getTypeSpaces()); return summary; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java index fe75687dffb7f..120d0dcd76213 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeSymlink.java @@ -102,7 +102,7 @@ public QuotaCounts computeQuotaUsage( @Override public ContentSummaryComputationContext computeContentSummary( final ContentSummaryComputationContext summary) { - summary.getCounts().add(Content.SYMLINK, 1); + summary.getCounts().addContent(Content.SYMLINK, 1); return summary; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java index 389b62b2cbf60..f1892c5e87e1d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java @@ -376,6 +376,12 @@ public List getReadOnlyINodes() { return Collections.unmodifiableList(Arrays.asList(inodes)); } + public INode[] getINodesArray() { + INode[] retArr = new INode[inodes.length]; + System.arraycopy(inodes, 0, retArr, 0, inodes.length); + return retArr; + } + /** * @param length number of ancestral INodes in the returned INodesInPath * instance diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java index 0dafaae7fc7d9..9ce8ebc650ed3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdfs.server.namenode; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import java.io.IOException; import java.util.ArrayList; @@ -256,17 +256,17 @@ private Lease(String holder) { } /** Only LeaseManager object can renew a lease */ private void renew() { - this.lastUpdate = now(); + this.lastUpdate = monotonicNow(); } /** @return true if the Hard Limit Timer has expired */ public boolean expiredHardLimit() { - return now() - lastUpdate > hardLimit; + return monotonicNow() - lastUpdate > hardLimit; } /** @return true if the Soft Limit Timer has expired */ public boolean expiredSoftLimit() { - return now() - lastUpdate > softLimit; + return monotonicNow() - lastUpdate > softLimit; } /** Does this lease contain any path? */ diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java index 546480d11b78a..8e105bd6fd5bb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java @@ -18,15 +18,19 @@ package org.apache.hadoop.hdfs.server.namenode; import java.io.File; +import java.io.FilenameFilter; import java.io.IOException; +import java.util.List; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.common.Storage.StorageDirectory; import org.apache.hadoop.hdfs.server.common.StorageInfo; import com.google.common.base.Preconditions; +import org.apache.hadoop.io.IOUtils; abstract class NNUpgradeUtil { @@ -99,15 +103,17 @@ static void doFinalize(StorageDirectory sd) throws IOException { * a call to any JM's or local storage dir's doPreUpgrade method fails, then * doUpgrade will not be called for any JM. The existing current dir is * renamed to previous.tmp, and then a new, empty current dir is created. - * + * + * @param conf configuration for creating {@link EditLogFileOutputStream} * @param sd the storage directory to perform the pre-upgrade procedure. * @throws IOException in the event of error */ - static void doPreUpgrade(StorageDirectory sd) throws IOException { + static void doPreUpgrade(Configuration conf, StorageDirectory sd) + throws IOException { LOG.info("Starting upgrade of storage directory " + sd.getRoot()); File curDir = sd.getCurrentDir(); File prevDir = sd.getPreviousDir(); - File tmpDir = sd.getPreviousTmp(); + final File tmpDir = sd.getPreviousTmp(); Preconditions.checkState(curDir.exists(), "Current directory must exist for preupgrade."); @@ -123,6 +129,35 @@ static void doPreUpgrade(StorageDirectory sd) throws IOException { if (!curDir.mkdir()) { throw new IOException("Cannot create directory " + curDir); } + + List fileNameList = IOUtils.listDirectory(tmpDir, new FilenameFilter() { + @Override + public boolean accept(File dir, String name) { + return dir.equals(tmpDir) + && name.startsWith(NNStorage.NameNodeFile.EDITS.getName()); + } + }); + + for (String s : fileNameList) { + File prevFile = new File(tmpDir, s); + Preconditions.checkState(prevFile.canRead(), + "Edits log file " + s + " is not readable."); + File newFile = new File(curDir, prevFile.getName()); + Preconditions.checkState(newFile.createNewFile(), + "Cannot create new edits log file in " + curDir); + EditLogFileInputStream in = new EditLogFileInputStream(prevFile); + EditLogFileOutputStream out = + new EditLogFileOutputStream(conf, newFile, 512*1024); + FSEditLogOp logOp = in.nextValidOp(); + while (logOp != null) { + out.write(logOp); + logOp = in.nextOp(); + } + out.setReadyToFlush(); + out.flushAndSync(true); + out.close(); + in.close(); + } } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java index 662c0e9340361..a671d21dd4dd6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java @@ -67,30 +67,28 @@ public class NameNodeHttpServer { } private void initWebHdfs(Configuration conf) throws IOException { - if (WebHdfsFileSystem.isEnabled(conf, HttpServer2.LOG)) { - // set user pattern based on configuration file - UserParam.setUserPattern(conf.get( - DFSConfigKeys.DFS_WEBHDFS_USER_PATTERN_KEY, - DFSConfigKeys.DFS_WEBHDFS_USER_PATTERN_DEFAULT)); - - // add authentication filter for webhdfs - final String className = conf.get( - DFSConfigKeys.DFS_WEBHDFS_AUTHENTICATION_FILTER_KEY, - DFSConfigKeys.DFS_WEBHDFS_AUTHENTICATION_FILTER_DEFAULT); - final String name = className; - - final String pathSpec = WebHdfsFileSystem.PATH_PREFIX + "/*"; - Map params = getAuthFilterParams(conf); - HttpServer2.defineFilter(httpServer.getWebAppContext(), name, className, - params, new String[] { pathSpec }); - HttpServer2.LOG.info("Added filter '" + name + "' (class=" + className - + ")"); - - // add webhdfs packages - httpServer.addJerseyResourcePackage(NamenodeWebHdfsMethods.class - .getPackage().getName() + ";" + Param.class.getPackage().getName(), - pathSpec); - } + // set user pattern based on configuration file + UserParam.setUserPattern(conf.get( + DFSConfigKeys.DFS_WEBHDFS_USER_PATTERN_KEY, + DFSConfigKeys.DFS_WEBHDFS_USER_PATTERN_DEFAULT)); + + // add authentication filter for webhdfs + final String className = conf.get( + DFSConfigKeys.DFS_WEBHDFS_AUTHENTICATION_FILTER_KEY, + DFSConfigKeys.DFS_WEBHDFS_AUTHENTICATION_FILTER_DEFAULT); + final String name = className; + + final String pathSpec = WebHdfsFileSystem.PATH_PREFIX + "/*"; + Map params = getAuthFilterParams(conf); + HttpServer2.defineFilter(httpServer.getWebAppContext(), name, className, + params, new String[] { pathSpec }); + HttpServer2.LOG.info("Added filter '" + name + "' (class=" + className + + ")"); + + // add webhdfs packages + httpServer.addJerseyResourcePackage(NamenodeWebHdfsMethods.class + .getPackage().getName() + ";" + Param.class.getPackage().getName(), + pathSpec); } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java index 9ccdb40c93ef9..7ab8b8661a63e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java @@ -121,6 +121,7 @@ import org.apache.hadoop.hdfs.server.namenode.NameNode.OperationCategory; import org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics; import org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol; @@ -1069,19 +1070,20 @@ public boolean restoreFailedStorage(String arg) throws IOException { } @Override // ClientProtocol - public void saveNamespace() throws IOException { + public boolean saveNamespace(long timeWindow, long txGap) throws IOException { checkNNStartup(); CacheEntry cacheEntry = RetryCache.waitForCompletion(retryCache); if (cacheEntry != null && cacheEntry.isSuccess()) { - return; // Return previous response + return true; // Return previous response } boolean success = false; try { - namesystem.saveNamespace(); + namesystem.saveNamespace(timeWindow, txGap); success = true; } finally { RetryCache.setState(cacheEntry, success); } + return true; } @Override // ClientProtocol @@ -1145,8 +1147,7 @@ public RollingUpgradeInfo rollingUpgrade(RollingUpgradeAction action) throws IOE case PREPARE: return namesystem.startRollingUpgrade(); case FINALIZE: - namesystem.finalizeRollingUpgrade(); - return null; + return namesystem.finalizeRollingUpgrade(); default: throw new UnsupportedActionException(action + " is not yet supported."); } @@ -1293,7 +1294,8 @@ public HeartbeatResponse sendHeartbeat(DatanodeRegistration nodeReg, @Override // DatanodeProtocol public DatanodeCommand blockReport(DatanodeRegistration nodeReg, - String poolId, StorageBlockReport[] reports) throws IOException { + String poolId, StorageBlockReport[] reports, + BlockReportContext context) throws IOException { checkNNStartup(); verifyRequest(nodeReg); if(blockStateChangeLog.isDebugEnabled()) { @@ -1302,14 +1304,15 @@ public DatanodeCommand blockReport(DatanodeRegistration nodeReg, } final BlockManager bm = namesystem.getBlockManager(); boolean noStaleStorages = false; - for(StorageBlockReport r : reports) { - final BlockListAsLongs blocks = new BlockListAsLongs(r.getBlocks()); + for (int r = 0; r < reports.length; r++) { + final BlockListAsLongs blocks = reports[r].getBlocks(); // // BlockManager.processReport accumulates information of prior calls // for the same node and storage, so the value returned by the last // call of this loop is the final updated value for noStaleStorage. // - noStaleStorages = bm.processReport(nodeReg, r.getStorage(), blocks); + noStaleStorages = bm.processReport(nodeReg, reports[r].getStorage(), + blocks, context, (r == reports.length - 1)); metrics.incrStorageBlockReportOps(); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java index 36b4461be5962..669f68a83bdb7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java @@ -289,7 +289,7 @@ public void blockIdCK(String blockId) { * Check files on DFS, starting from the indicated path. */ public void fsck() { - final long startTime = Time.now(); + final long startTime = Time.monotonicNow(); try { if(blockIds != null) { @@ -301,7 +301,7 @@ public void fsck() { out.println(sb.toString()); sb.append(" for blockIds: \n"); for (String blk: blocks) { - if(blk == null || !blk.contains("blk_")) { + if(blk == null || !blk.contains(Block.BLOCK_FILE_PREFIX)) { out.println("Incorrect blockId format: " + blk); continue; } @@ -357,7 +357,7 @@ public void fsck() { } out.println("FSCK ended at " + new Date() + " in " - + (Time.now() - startTime + " milliseconds")); + + (Time.monotonicNow() - startTime + " milliseconds")); // If there were internal errors during the fsck operation, we want to // return FAILURE_STATUS, even if those errors were not immediately @@ -383,7 +383,7 @@ public void fsck() { String errMsg = "Fsck on path '" + path + "' " + FAILURE_STATUS; LOG.warn(errMsg, e); out.println("FSCK ended at " + new Date() + " in " - + (Time.now() - startTime + " milliseconds")); + + (Time.monotonicNow() - startTime + " milliseconds")); out.println(e.getMessage()); out.print("\n\n" + errMsg); } finally { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java index 3d726454b0ca3..1897d8db7cedf 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java @@ -42,9 +42,11 @@ import org.apache.hadoop.hdfs.server.namenode.NameNode; import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol; import org.apache.hadoop.ipc.RPC; +import org.apache.hadoop.ipc.RemoteException; +import org.apache.hadoop.ipc.StandbyException; import org.apache.hadoop.security.SecurityUtil; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import static org.apache.hadoop.util.ExitUtil.terminate; import com.google.common.annotations.VisibleForTesting; @@ -84,7 +86,7 @@ public class EditLogTailer { * The last time we successfully loaded a non-zero number of edits from the * shared directory. */ - private long lastLoadTimestamp; + private long lastLoadTimeMs; /** * How often the Standby should roll edit logs. Since the Standby only reads @@ -105,7 +107,7 @@ public EditLogTailer(FSNamesystem namesystem, Configuration conf) { this.namesystem = namesystem; this.editLog = namesystem.getEditLog(); - lastLoadTimestamp = now(); + lastLoadTimeMs = monotonicNow(); logRollPeriodMs = conf.getInt(DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_KEY, DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_DEFAULT) * 1000; @@ -241,7 +243,7 @@ void doTailEdits() throws IOException, InterruptedException { } if (editsLoaded > 0) { - lastLoadTimestamp = now(); + lastLoadTimeMs = monotonicNow(); } lastLoadedTxnId = image.getLastAppliedTxId(); } finally { @@ -250,10 +252,10 @@ void doTailEdits() throws IOException, InterruptedException { } /** - * @return timestamp (in msec) of when we last loaded a non-zero number of edits. + * @return time in msec of when we last loaded a non-zero number of edits. */ - public long getLastLoadTimestamp() { - return lastLoadTimestamp; + public long getLastLoadTimeMs() { + return lastLoadTimeMs; } /** @@ -261,7 +263,7 @@ public long getLastLoadTimestamp() { */ private boolean tooLongSinceLastLoad() { return logRollPeriodMs >= 0 && - (now() - lastLoadTimestamp) > logRollPeriodMs ; + (monotonicNow() - lastLoadTimeMs) > logRollPeriodMs ; } /** @@ -273,6 +275,15 @@ private void triggerActiveLogRoll() { getActiveNodeProxy().rollEditLog(); lastRollTriggerTxId = lastLoadedTxnId; } catch (IOException ioe) { + if (ioe instanceof RemoteException) { + ioe = ((RemoteException)ioe).unwrapRemoteException(); + if (ioe instanceof StandbyException) { + LOG.info("Skipping log roll. Remote node is not in Active state: " + + ioe.getMessage().split("\n")[0]); + return; + } + } + LOG.warn("Unable to trigger a roll of the active NN", ioe); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java index 5168f0b2b90d6..fa1bf94f56d1c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java @@ -237,8 +237,8 @@ public ContentSummaryComputationContext computeContentSummary( final INodeDirectory snapshotRoot, final ContentSummaryComputationContext summary) { snapshotRoot.computeContentSummary(summary); - summary.getCounts().add(Content.SNAPSHOT, snapshotsByNames.size()); - summary.getCounts().add(Content.SNAPSHOTTABLE_DIRECTORY, 1); + summary.getCounts().addContent(Content.SNAPSHOT, snapshotsByNames.size()); + summary.getCounts().addContent(Content.SNAPSHOTTABLE_DIRECTORY, 1); return summary; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java index 07ff744f5e15c..d55332f544ad5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java @@ -31,6 +31,7 @@ import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite; import org.apache.hadoop.hdfs.server.namenode.AclStorage; import org.apache.hadoop.hdfs.server.namenode.Content; +import org.apache.hadoop.hdfs.server.namenode.ContentCounts; import org.apache.hadoop.hdfs.server.namenode.ContentSummaryComputationContext; import org.apache.hadoop.hdfs.server.namenode.FSImageSerialization; import org.apache.hadoop.hdfs.server.namenode.INode; @@ -650,19 +651,19 @@ public QuotaCounts computeQuotaUsage4CurrentDirectory( } public void computeContentSummary4Snapshot(final BlockStoragePolicySuite bsps, - final Content.Counts counts) { + final ContentCounts counts) { // Create a new blank summary context for blocking processing of subtree. ContentSummaryComputationContext summary = - new ContentSummaryComputationContext(); + new ContentSummaryComputationContext(bsps); for(DirectoryDiff d : diffs) { for(INode deleted : d.getChildrenDiff().getList(ListType.DELETED)) { deleted.computeContentSummary(summary); } } // Add the counts from deleted trees. - counts.add(summary.getCounts()); + counts.addContents(summary.getCounts()); // Add the deleted directory count. - counts.add(Content.DIRECTORY, diffs.asList().size()); + counts.addContent(Content.DIRECTORY, diffs.asList().size()); } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java index 0c94554738a49..5c9e1219a5930 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileDiffList.java @@ -20,8 +20,11 @@ import java.util.Collections; import java.util.List; +import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous; +import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguousUnderConstruction; import org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite; +import org.apache.hadoop.hdfs.server.common.HdfsServerConstants; import org.apache.hadoop.hdfs.server.namenode.INode; import org.apache.hadoop.hdfs.server.namenode.INode.BlocksMapUpdateInfo; import org.apache.hadoop.hdfs.server.namenode.INodeFile; @@ -125,9 +128,19 @@ void combineAndCollectSnapshotBlocks(BlockStoragePolicySuite bsps, INodeFile fil continue; break; } - // Collect the remaining blocks of the file - while(i < removedBlocks.length) { - collectedBlocks.addDeleteBlock(removedBlocks[i++]); + // Check if last block is part of truncate recovery + BlockInfoContiguous lastBlock = file.getLastBlock(); + Block dontRemoveBlock = null; + if(lastBlock != null && lastBlock.getBlockUCState().equals( + HdfsServerConstants.BlockUCState.UNDER_RECOVERY)) { + dontRemoveBlock = ((BlockInfoContiguousUnderConstruction) lastBlock) + .getTruncateBlock(); + } + // Collect the remaining blocks of the file, ignoring truncate block + for(;i < removedBlocks.length; i++) { + if(dontRemoveBlock == null || !removedBlocks[i].equals(dontRemoveBlock)) { + collectedBlocks.addDeleteBlock(removedBlocks[i]); + } } } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/window/RollingWindowManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/window/RollingWindowManager.java index 00e708766e0d1..4759cc8ca5d3f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/window/RollingWindowManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/window/RollingWindowManager.java @@ -245,7 +245,7 @@ private TopN getTopUsersForMetric(long time, String metricName, metricName, userName, windowSum); topN.offer(new NameValuePair(userName, windowSum)); } - LOG.info("topN size for command {} is: {}", metricName, topN.size()); + LOG.debug("topN users size for command {} is: {}", metricName, topN.size()); return topN; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java index ced3296f48637..3adc85c5a21bf 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java @@ -61,7 +61,7 @@ public static class RecoveringBlock extends LocatedBlock { * Create RecoveringBlock. */ public RecoveringBlock(ExtendedBlock b, DatanodeInfo[] locs, long newGS) { - super(b, locs, -1, false); // startOffset is unknown + super(b, locs); // startOffset is unknown this.newGenerationStamp = newGS; this.recoveryBlock = null; } @@ -71,7 +71,7 @@ public RecoveringBlock(ExtendedBlock b, DatanodeInfo[] locs, long newGS) { */ public RecoveringBlock(ExtendedBlock b, DatanodeInfo[] locs, Block recoveryBlock) { - super(b, locs, -1, false); // startOffset is unknown + super(b, locs); // startOffset is unknown this.newGenerationStamp = recoveryBlock.getGenerationStamp(); this.recoveryBlock = recoveryBlock; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockReportContext.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockReportContext.java new file mode 100644 index 0000000000000..a084a813cf6f1 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockReportContext.java @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdfs.server.protocol; + +/** + * The context of the block report. + * + * This is a set of fields that the Datanode sends to provide context about a + * block report RPC. The context includes a unique 64-bit ID which + * identifies the block report as a whole. It also includes the total number + * of RPCs which this block report is split into, and the index into that + * total for the current RPC. + */ +public class BlockReportContext { + private final int totalRpcs; + private final int curRpc; + private final long reportId; + + public BlockReportContext(int totalRpcs, int curRpc, long reportId) { + this.totalRpcs = totalRpcs; + this.curRpc = curRpc; + this.reportId = reportId; + } + + public int getTotalRpcs() { + return totalRpcs; + } + + public int getCurRpc() { + return curRpc; + } + + public long getReportId() { + return reportId; + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java index 047de56961878..a3b6004644b9b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java @@ -23,7 +23,6 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.hdfs.DFSConfigKeys; -import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlock; @@ -128,20 +127,23 @@ public HeartbeatResponse sendHeartbeat(DatanodeRegistration registration, * Each finalized block is represented as 3 longs. Each under- * construction replica is represented as 4 longs. * This is done instead of Block[] to reduce memory used by block reports. - * + * @param reports report of blocks per storage + * @param context Context information for this block report. + * * @return - the next command for DN to process. * @throws IOException */ @Idempotent public DatanodeCommand blockReport(DatanodeRegistration registration, - String poolId, StorageBlockReport[] reports) throws IOException; + String poolId, StorageBlockReport[] reports, + BlockReportContext context) throws IOException; /** * Communicates the complete list of locally cached blocks to the NameNode. * * This method is similar to - * {@link #blockReport(DatanodeRegistration, String, StorageBlockReport[])}, + * {@link #blockReport(DatanodeRegistration, String, StorageBlockReport[], BlockReportContext)}, * which is used to communicated blocks stored on disk. * * @param The datanode registration. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java index aaa18c666a721..711973838bc06 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java @@ -25,6 +25,8 @@ import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.common.StorageInfo; +import com.google.common.annotations.VisibleForTesting; + /** * DatanodeRegistration class contains all information the name-node needs * to identify and verify a data-node when it contacts the name-node. @@ -38,6 +40,15 @@ public class DatanodeRegistration extends DatanodeID private final StorageInfo storageInfo; private ExportedBlockKeys exportedKeys; private final String softwareVersion; + private NamespaceInfo nsInfo; + + @VisibleForTesting + public DatanodeRegistration(String uuid, DatanodeRegistration dnr) { + this(new DatanodeID(uuid, dnr), + dnr.getStorageInfo(), + dnr.getExportedKeys(), + dnr.getSoftwareVersion()); + } public DatanodeRegistration(DatanodeID dn, StorageInfo info, ExportedBlockKeys keys, String softwareVersion) { @@ -67,6 +78,14 @@ public String getSoftwareVersion() { public int getVersion() { return storageInfo.getLayoutVersion(); } + + public void setNamespaceInfo(NamespaceInfo nsInfo) { + this.nsInfo = nsInfo; + } + + public NamespaceInfo getNamespaceInfo() { + return nsInfo; + } @Override // NodeRegistration public String getRegistrationID() { @@ -81,7 +100,7 @@ public String getAddress() { @Override public String toString() { return getClass().getSimpleName() - + "(" + getIpAddr() + + "(" + super.toString() + ", datanodeUuid=" + getDatanodeUuid() + ", infoPort=" + getInfoPort() + ", infoSecurePort=" + getInfoSecurePort() diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java index 07337434e23ea..a7439a0e17477 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamespaceInfo.java @@ -29,6 +29,9 @@ import org.apache.hadoop.hdfs.server.namenode.NNStorage; import org.apache.hadoop.util.VersionInfo; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; + /** * NamespaceInfo is returned by the name-node in reply * to a data-node handshake. @@ -40,19 +43,52 @@ public class NamespaceInfo extends StorageInfo { final String buildVersion; String blockPoolID = ""; // id of the block pool String softwareVersion; + long capabilities; + + // only authoritative on the server-side to determine advertisement to + // clients. enum will update the supported values + private static long CAPABILITIES_SUPPORTED = 0; + + public enum Capability { + UNKNOWN(false), + STORAGE_BLOCK_REPORT_BUFFERS(true); // use optimized ByteString buffers + private final long mask; + Capability(boolean isSupported) { + int bits = ordinal() - 1; + mask = (bits < 0) ? 0 : (1L << bits); + if (isSupported) { + CAPABILITIES_SUPPORTED |= mask; + } + } + public long getMask() { + return mask; + } + } + // defaults to enabled capabilites since this ctor is for server public NamespaceInfo() { super(NodeType.NAME_NODE); buildVersion = null; + capabilities = CAPABILITIES_SUPPORTED; } + // defaults to enabled capabilites since this ctor is for server public NamespaceInfo(int nsID, String clusterID, String bpID, long cT, String buildVersion, String softwareVersion) { + this(nsID, clusterID, bpID, cT, buildVersion, softwareVersion, + CAPABILITIES_SUPPORTED); + } + + // for use by server and/or client + public NamespaceInfo(int nsID, String clusterID, String bpID, + long cT, String buildVersion, String softwareVersion, + long capabilities) { super(HdfsConstants.NAMENODE_LAYOUT_VERSION, nsID, clusterID, cT, NodeType.NAME_NODE); blockPoolID = bpID; this.buildVersion = buildVersion; this.softwareVersion = softwareVersion; + this.capabilities = capabilities; } public NamespaceInfo(int nsID, String clusterID, String bpID, @@ -61,6 +97,22 @@ public NamespaceInfo(int nsID, String clusterID, String bpID, VersionInfo.getVersion()); } + public long getCapabilities() { + return capabilities; + } + + @VisibleForTesting + public void setCapabilities(long capabilities) { + this.capabilities = capabilities; + } + + public boolean isCapabilitySupported(Capability capability) { + Preconditions.checkArgument(capability != Capability.UNKNOWN, + "cannot test for unknown capability"); + long mask = capability.getMask(); + return (capabilities & mask) == mask; + } + public String getBuildVersion() { return buildVersion; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageBlockReport.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageBlockReport.java index 152169339d803..4ef5ebcbba9a3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageBlockReport.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/StorageBlockReport.java @@ -18,14 +18,16 @@ package org.apache.hadoop.hdfs.server.protocol; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; + /** * Block report for a Datanode storage */ public class StorageBlockReport { private final DatanodeStorage storage; - private final long[] blocks; + private final BlockListAsLongs blocks; - public StorageBlockReport(DatanodeStorage storage, long[] blocks) { + public StorageBlockReport(DatanodeStorage storage, BlockListAsLongs blocks) { this.storage = storage; this.blocks = blocks; } @@ -34,7 +36,7 @@ public DatanodeStorage getStorage() { return storage; } - public long[] getBlocks() { + public BlockListAsLongs getBlocks() { return blocks; } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java index 6dbaf84d2692b..9092bc531f178 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java @@ -412,7 +412,8 @@ final void shutdown(DfsClientShm shm) { private final DomainSocketWatcher domainSocketWatcher; DfsClientShmManager(int interruptCheckPeriodMs) throws IOException { - this.domainSocketWatcher = new DomainSocketWatcher(interruptCheckPeriodMs); + this.domainSocketWatcher = new DomainSocketWatcher(interruptCheckPeriodMs, + "client"); } public Slot allocSlot(DatanodeInfo datanode, DomainPeer peer, diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java index abbd46b7de37b..b8dcbbf056c42 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java @@ -18,7 +18,6 @@ package org.apache.hadoop.hdfs.tools; import java.io.File; -import java.io.FileNotFoundException; import java.io.IOException; import java.io.PrintStream; import java.net.InetSocketAddress; @@ -341,7 +340,7 @@ static boolean matches(String cmd) { private static void printMessage(RollingUpgradeInfo info, PrintStream out) { if (info != null && info.isStarted()) { - if (!info.createdRollbackImages()) { + if (!info.createdRollbackImages() && !info.isFinalized()) { out.println( "Preparing for upgrade. Data is being saved for rollback." + "\nRun \"dfsadmin -rollingUpgrade query\" to check the status" @@ -393,7 +392,7 @@ static int run(DistributedFileSystem dfs, String[] argv, int idx) throws IOExcep private static final String commonUsageSummary = "\t[-report [-live] [-dead] [-decommissioning]]\n" + "\t[-safemode ]\n" + - "\t[-saveNamespace]\n" + + "\t[-saveNamespace [-beforeShutdown]]\n" + "\t[-rollEdits]\n" + "\t[-restoreFailedStorage true|false|check]\n" + "\t[-refreshNodes]\n" + @@ -694,34 +693,57 @@ public void disallowSnapshot(String[] argv) throws IOException { /** * Command to ask the namenode to save the namespace. * Usage: hdfs dfsadmin -saveNamespace - * @exception IOException - * @see org.apache.hadoop.hdfs.protocol.ClientProtocol#saveNamespace() + * @see ClientProtocol#saveNamespace(long, long) */ - public int saveNamespace() throws IOException { - int exitCode = -1; + public int saveNamespace(String[] argv) throws IOException { + final DistributedFileSystem dfs = getDFS(); + final Configuration dfsConf = dfs.getConf(); + + long timeWindow = 0; + long txGap = 0; + if (argv.length > 1 && "-beforeShutdown".equals(argv[1])) { + final long checkpointPeriod = dfsConf.getLong( + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_PERIOD_KEY, + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_PERIOD_DEFAULT); + final long checkpointTxnCount = dfsConf.getLong( + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_KEY, + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_DEFAULT); + final int toleratePeriodNum = dfsConf.getInt( + DFSConfigKeys.DFS_NAMENODE_MISSING_CHECKPOINT_PERIODS_BEFORE_SHUTDOWN_KEY, + DFSConfigKeys.DFS_NAMENODE_MISSING_CHECKPOINT_PERIODS_BEFORE_SHUTDONW_DEFAULT); + timeWindow = checkpointPeriod * toleratePeriodNum; + txGap = checkpointTxnCount * toleratePeriodNum; + System.out.println("Do checkpoint if necessary before stopping " + + "namenode. The time window is " + timeWindow + " seconds, and the " + + "transaction gap is " + txGap); + } - DistributedFileSystem dfs = getDFS(); - Configuration dfsConf = dfs.getConf(); URI dfsUri = dfs.getUri(); boolean isHaEnabled = HAUtil.isLogicalUri(dfsConf, dfsUri); - if (isHaEnabled) { String nsId = dfsUri.getHost(); List> proxies = HAUtil.getProxiesForAllNameNodesInNameservice(dfsConf, nsId, ClientProtocol.class); for (ProxyAndInfo proxy : proxies) { - proxy.getProxy().saveNamespace(); - System.out.println("Save namespace successful for " + - proxy.getAddress()); + boolean saved = proxy.getProxy().saveNamespace(timeWindow, txGap); + if (saved) { + System.out.println("Save namespace successful for " + + proxy.getAddress()); + } else { + System.out.println("No extra checkpoint has been made for " + + proxy.getAddress()); + } } } else { - dfs.saveNamespace(); - System.out.println("Save namespace successful"); + boolean saved = dfs.saveNamespace(timeWindow, txGap); + if (saved) { + System.out.println("Save namespace successful"); + } else { + System.out.println("No extra checkpoint has been made"); + } } - exitCode = 0; - - return exitCode; + return 0; } public int rollEdits() throws IOException { @@ -902,9 +924,14 @@ private void printHelp(String cmd) { "\t\tcondition. Safe mode can also be entered manually, but then\n" + "\t\tit can only be turned off manually as well.\n"; - String saveNamespace = "-saveNamespace:\t" + - "Save current namespace into storage directories and reset edits log.\n" + - "\t\tRequires safe mode.\n"; + String saveNamespace = "-saveNamespace [-beforeShutdown]:\t" + + "Save current namespace into storage directories and reset edits \n" + + "\t\t log. Requires safe mode.\n" + + "\t\tIf the \"beforeShutdown\" option is given, the NameNode does a \n" + + "\t\tcheckpoint if and only if there is no checkpoint done during \n" + + "\t\ta time window (a configurable number of checkpoint periods).\n" + + "\t\tThis is usually used before shutting down the NameNode to \n" + + "\t\tprevent potential fsimage/editlog corruption.\n"; String rollEdits = "-rollEdits:\t" + "Rolls the edit log.\n"; @@ -1546,10 +1573,9 @@ private static void printUsage(String cmd) { + " [-disallowSnapshot ]"); } else if ("-saveNamespace".equals(cmd)) { System.err.println("Usage: hdfs dfsadmin" - + " [-saveNamespace]"); + + " [-saveNamespace [-beforeShutdown]]"); } else if ("-rollEdits".equals(cmd)) { - System.err.println("Usage: hdfs dfsadmin" - + " [-rollEdits]"); + System.err.println("Usage: hdfs dfsadmin [-rollEdits]"); } else if ("-restoreFailedStorage".equals(cmd)) { System.err.println("Usage: hdfs dfsadmin" + " [-restoreFailedStorage true|false|check ]"); @@ -1668,7 +1694,7 @@ public int run(String[] argv) throws Exception { return exitCode; } } else if ("-saveNamespace".equals(cmd)) { - if (argv.length != 1) { + if (argv.length != 1 && argv.length != 2) { printUsage(cmd); return exitCode; } @@ -1788,7 +1814,7 @@ public int run(String[] argv) throws Exception { } else if ("-disallowSnapshot".equalsIgnoreCase(cmd)) { disallowSnapshot(argv); } else if ("-saveNamespace".equals(cmd)) { - exitCode = saveNamespace(); + exitCode = saveNamespace(argv); } else if ("-rollEdits".equals(cmd)) { exitCode = rollEdits(); } else if ("-restoreFailedStorage".equals(cmd)) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java index 85f77f1b7d106..4e256a2174743 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java @@ -167,6 +167,8 @@ protected String getScopeInsideParentNode() { public static void main(String args[]) throws Exception { + StringUtils.startupShutdownMessage(DFSZKFailoverController.class, + args, LOG); if (DFSUtil.parseHelpArgument(args, ZKFailoverController.USAGE, System.out, true)) { System.exit(0); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java index 659036691e3e7..4169af1d34753 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java @@ -65,7 +65,7 @@ public class OfflineImageViewerPB { + " -step defines the granularity of the distribution. (2MB by default)\n" + " * Web: Run a viewer to expose read-only WebHDFS API.\n" + " -addr specifies the address to listen. (localhost:5978 by default)\n" - + " * Delimited: Generate a text file with all of the elements common\n" + + " * Delimited (experimental): Generate a text file with all of the elements common\n" + " to both inodes and inodes-under-construction, separated by a\n" + " delimiter. The default delimiter is \\t, though this may be\n" + " changed via the -delimiter argument.\n" @@ -80,10 +80,10 @@ public class OfflineImageViewerPB { + "-p,--processor Select which type of processor to apply\n" + " against image file. (XML|FileDistribution|Web|Delimited)\n" + " (Web by default)\n" - + "-delimiter Delimiting string to use with Delimited processor\n" + + "-delimiter Delimiting string to use with Delimited processor. \n" + "-t,--temp Use temporary dir to cache intermediate result to generate\n" + " Delimited outputs. If not set, Delimited processor constructs\n" - + " the namespace in memory before outputting text." + + " the namespace in memory before outputting text.\n" + "-h,--help Display usage information and exit\n"; /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/EnumCounters.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/EnumCounters.java index 18f4bd6969147..86ba341e7f8e1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/EnumCounters.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/EnumCounters.java @@ -21,6 +21,7 @@ import java.util.HashMap; import com.google.common.base.Preconditions; +import org.apache.commons.lang.ArrayUtils; /** * Counters for an enum type. @@ -64,6 +65,11 @@ public final long get(final E e) { return counters[e.ordinal()]; } + /** @return the values of counter as a shadow copy of array*/ + public long[] asArray() { + return ArrayUtils.clone(counters); + } + /** Negate all counters. */ public final void negation() { for(int i = 0; i < counters.length; i++) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java index aa6100cc318e9..d53bc318cbafb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java @@ -21,7 +21,6 @@ import org.apache.hadoop.fs.permission.AclEntry; import org.apache.hadoop.fs.permission.AclStatus; import org.apache.hadoop.fs.permission.FsPermission; -import org.apache.hadoop.hdfs.protocol.BlockStoragePolicy; import org.apache.hadoop.hdfs.DFSUtil; import org.apache.hadoop.hdfs.XAttrHelper; import org.apache.hadoop.hdfs.protocol.*; @@ -35,7 +34,8 @@ import org.apache.hadoop.security.token.TokenIdentifier; import org.apache.hadoop.util.DataChecksum; import org.apache.hadoop.util.StringUtils; -import org.mortbay.util.ajax.JSON; +import org.codehaus.jackson.map.ObjectMapper; +import org.codehaus.jackson.map.ObjectReader; import com.google.common.collect.Lists; import com.google.common.collect.Maps; @@ -95,59 +95,6 @@ private static Token toBlockToken( return (Token)toToken(m); } - /** Convert a Token[] to a JSON array. */ - private static Object[] toJsonArray(final Token[] array - ) throws IOException { - if (array == null) { - return null; - } else if (array.length == 0) { - return EMPTY_OBJECT_ARRAY; - } else { - final Object[] a = new Object[array.length]; - for(int i = 0; i < array.length; i++) { - a[i] = toJsonMap(array[i]); - } - return a; - } - } - - /** Convert a token object to a JSON string. */ - public static String toJsonString(final Token[] tokens - ) throws IOException { - if (tokens == null) { - return null; - } - - final Map m = new TreeMap(); - m.put(Token.class.getSimpleName(), toJsonArray(tokens)); - return toJsonString(Token.class.getSimpleName() + "s", m); - } - - /** Convert an Object[] to a List>. */ - private static List> toTokenList(final Object[] objects) throws IOException { - if (objects == null) { - return null; - } else if (objects.length == 0) { - return Collections.emptyList(); - } else { - final List> list = new ArrayList>(objects.length); - for(int i = 0; i < objects.length; i++) { - list.add(toToken((Map)objects[i])); - } - return list; - } - } - - /** Convert a JSON map to a List>. */ - public static List> toTokenList(final Map json) throws IOException { - if (json == null) { - return null; - } - - final Map m = (Map)json.get(Token.class.getSimpleName() + "s"); - return toTokenList((Object[])m.get(Token.class.getSimpleName())); - } - /** Convert an exception object to a Json string. */ public static String toJsonString(final Exception e) { final Map m = new TreeMap(); @@ -173,7 +120,12 @@ private static String toJsonString(final Class clazz, final Object value) { public static String toJsonString(final String key, final Object value) { final Map m = new TreeMap(); m.put(key, value); - return JSON.toString(m); + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(m); + } catch (IOException ignored) { + } + return null; } /** Convert a FsPermission object to a string. */ @@ -233,7 +185,13 @@ public static String toJsonString(final HdfsFileStatus status, m.put("fileId", status.getFileId()); m.put("childrenNum", status.getChildrenNum()); m.put("storagePolicy", status.getStoragePolicy()); - return includeType ? toJsonString(FileStatus.class, m): JSON.toString(m); + ObjectMapper mapper = new ObjectMapper(); + try { + return includeType ? + toJsonString(FileStatus.class, m) : mapper.writeValueAsString(m); + } catch (IOException ignored) { + } + return null; } /** Convert a Json map to a HdfsFileStatus object. */ @@ -249,25 +207,23 @@ public static HdfsFileStatus toFileStatus(final Map json, boolean includes final byte[] symlink = type != PathType.SYMLINK? null : DFSUtil.string2Bytes((String)m.get("symlink")); - final long len = (Long) m.get("length"); + final long len = ((Number) m.get("length")).longValue(); final String owner = (String) m.get("owner"); final String group = (String) m.get("group"); final FsPermission permission = toFsPermission((String) m.get("permission"), (Boolean)m.get("aclBit"), (Boolean)m.get("encBit")); - final long aTime = (Long) m.get("accessTime"); - final long mTime = (Long) m.get("modificationTime"); - final long blockSize = (Long) m.get("blockSize"); + final long aTime = ((Number) m.get("accessTime")).longValue(); + final long mTime = ((Number) m.get("modificationTime")).longValue(); + final long blockSize = ((Number) m.get("blockSize")).longValue(); final boolean isLazyPersist = m.containsKey("lazyPersist") ? (Boolean) m.get("lazyPersist") : false; - final short replication = (short) (long) (Long) m.get("replication"); - final long fileId = m.containsKey("fileId") ? (Long) m.get("fileId") - : INodeId.GRANDFATHER_INODE_ID; - Long childrenNumLong = (Long) m.get("childrenNum"); - final int childrenNum = (childrenNumLong == null) ? -1 - : childrenNumLong.intValue(); + final short replication = ((Number) m.get("replication")).shortValue(); + final long fileId = m.containsKey("fileId") ? + ((Number) m.get("fileId")).longValue() : INodeId.GRANDFATHER_INODE_ID; + final int childrenNum = getInt(m, "childrenNum", -1); final byte storagePolicy = m.containsKey("storagePolicy") ? - (byte) (long) (Long) m.get("storagePolicy") : - BlockStoragePolicySuite.ID_UNSPECIFIED; + (byte) ((Number) m.get("storagePolicy")).longValue() : + BlockStoragePolicySuite.ID_UNSPECIFIED; return new HdfsFileStatus(len, type == PathType.DIRECTORY, replication, blockSize, mTime, aTime, permission, owner, group, symlink, DFSUtil.string2Bytes(localName), fileId, childrenNum, null, @@ -295,9 +251,10 @@ private static ExtendedBlock toExtendedBlock(final Map m) { } final String blockPoolId = (String)m.get("blockPoolId"); - final long blockId = (Long)m.get("blockId"); - final long numBytes = (Long)m.get("numBytes"); - final long generationStamp = (Long)m.get("generationStamp"); + final long blockId = ((Number) m.get("blockId")).longValue(); + final long numBytes = ((Number) m.get("numBytes")).longValue(); + final long generationStamp = + ((Number) m.get("generationStamp")).longValue(); return new ExtendedBlock(blockPoolId, blockId, numBytes, generationStamp); } @@ -327,6 +284,7 @@ static Map toJsonMap(final DatanodeInfo datanodeinfo) { m.put("cacheCapacity", datanodeinfo.getCacheCapacity()); m.put("cacheUsed", datanodeinfo.getCacheUsed()); m.put("lastUpdate", datanodeinfo.getLastUpdate()); + m.put("lastUpdateMonotonic", datanodeinfo.getLastUpdateMonotonic()); m.put("xceiverCount", datanodeinfo.getXceiverCount()); m.put("networkLocation", datanodeinfo.getNetworkLocation()); m.put("adminState", datanodeinfo.getAdminState().name()); @@ -338,7 +296,7 @@ private static int getInt(Map m, String key, final int defaultValue) { if (value == null) { return defaultValue; } - return (int) (long) (Long) value; + return ((Number) value).intValue(); } private static long getLong(Map m, String key, final long defaultValue) { @@ -346,7 +304,7 @@ private static long getLong(Map m, String key, final long defaultValue) { if (value == null) { return defaultValue; } - return (Long) value; + return ((Number) value).longValue(); } private static String getString(Map m, String key, @@ -358,6 +316,15 @@ private static String getString(Map m, String key, return (String) value; } + static List getList(Map m, String key) { + Object list = m.get(key); + if (list instanceof List) { + return (List) list; + } else { + return null; + } + } + /** Convert a Json map to an DatanodeInfo object. */ static DatanodeInfo toDatanodeInfo(final Map m) throws IOException { @@ -402,9 +369,9 @@ static DatanodeInfo toDatanodeInfo(final Map m) (String)m.get("hostName"), (String)m.get("storageID"), xferPort, - (int)(long)(Long)m.get("infoPort"), + ((Number) m.get("infoPort")).intValue(), getInt(m, "infoSecurePort", 0), - (int)(long)(Long)m.get("ipcPort"), + ((Number) m.get("ipcPort")).intValue(), getLong(m, "capacity", 0l), getLong(m, "dfsUsed", 0l), @@ -413,6 +380,7 @@ static DatanodeInfo toDatanodeInfo(final Map m) getLong(m, "cacheCapacity", 0l), getLong(m, "cacheUsed", 0l), getLong(m, "lastUpdate", 0l), + getLong(m, "lastUpdateMonotonic", 0l), getInt(m, "xceiverCount", 0), getString(m, "networkLocation", ""), AdminStates.valueOf(getString(m, "adminState", "NORMAL"))); @@ -434,16 +402,17 @@ private static Object[] toJsonArray(final DatanodeInfo[] array) { } /** Convert an Object[] to a DatanodeInfo[]. */ - private static DatanodeInfo[] toDatanodeInfoArray(final Object[] objects) + private static DatanodeInfo[] toDatanodeInfoArray(final List objects) throws IOException { if (objects == null) { return null; - } else if (objects.length == 0) { + } else if (objects.isEmpty()) { return EMPTY_DATANODE_INFO_ARRAY; } else { - final DatanodeInfo[] array = new DatanodeInfo[objects.length]; - for(int i = 0; i < array.length; i++) { - array[i] = toDatanodeInfo((Map) objects[i]); + final DatanodeInfo[] array = new DatanodeInfo[objects.size()]; + int i = 0; + for (Object object : objects) { + array[i++] = toDatanodeInfo((Map) object); } return array; } @@ -474,11 +443,11 @@ private static LocatedBlock toLocatedBlock(final Map m) throws IOException final ExtendedBlock b = toExtendedBlock((Map)m.get("block")); final DatanodeInfo[] locations = toDatanodeInfoArray( - (Object[])m.get("locations")); - final long startOffset = (Long)m.get("startOffset"); + getList(m, "locations")); + final long startOffset = ((Number) m.get("startOffset")).longValue(); final boolean isCorrupt = (Boolean)m.get("isCorrupt"); final DatanodeInfo[] cachedLocations = toDatanodeInfoArray( - (Object[])m.get("cachedLocations")); + getList(m, "cachedLocations")); final LocatedBlock locatedblock = new LocatedBlock(b, locations, null, null, startOffset, isCorrupt, cachedLocations); @@ -502,17 +471,17 @@ private static Object[] toJsonArray(final List array } } - /** Convert an Object[] to a List of LocatedBlock. */ - private static List toLocatedBlockList(final Object[] objects - ) throws IOException { + /** Convert an List of Object to a List of LocatedBlock. */ + private static List toLocatedBlockList( + final List objects) throws IOException { if (objects == null) { return null; - } else if (objects.length == 0) { + } else if (objects.isEmpty()) { return Collections.emptyList(); } else { - final List list = new ArrayList(objects.length); - for(int i = 0; i < objects.length; i++) { - list.add(toLocatedBlock((Map)objects[i])); + final List list = new ArrayList<>(objects.size()); + for (Object object : objects) { + list.add(toLocatedBlock((Map) object)); } return list; } @@ -543,10 +512,10 @@ public static LocatedBlocks toLocatedBlocks(final Map json } final Map m = (Map)json.get(LocatedBlocks.class.getSimpleName()); - final long fileLength = (Long)m.get("fileLength"); + final long fileLength = ((Number) m.get("fileLength")).longValue(); final boolean isUnderConstruction = (Boolean)m.get("isUnderConstruction"); final List locatedBlocks = toLocatedBlockList( - (Object[])m.get("locatedBlocks")); + getList(m, "locatedBlocks")); final LocatedBlock lastLocatedBlock = toLocatedBlock( (Map)m.get("lastLocatedBlock")); final boolean isLastBlockComplete = (Boolean)m.get("isLastBlockComplete"); @@ -577,15 +546,16 @@ public static ContentSummary toContentSummary(final Map json) { } final Map m = (Map)json.get(ContentSummary.class.getSimpleName()); - final long length = (Long)m.get("length"); - final long fileCount = (Long)m.get("fileCount"); - final long directoryCount = (Long)m.get("directoryCount"); - final long quota = (Long)m.get("quota"); - final long spaceConsumed = (Long)m.get("spaceConsumed"); - final long spaceQuota = (Long)m.get("spaceQuota"); + final long length = ((Number) m.get("length")).longValue(); + final long fileCount = ((Number) m.get("fileCount")).longValue(); + final long directoryCount = ((Number) m.get("directoryCount")).longValue(); + final long quota = ((Number) m.get("quota")).longValue(); + final long spaceConsumed = ((Number) m.get("spaceConsumed")).longValue(); + final long spaceQuota = ((Number) m.get("spaceQuota")).longValue(); - return new ContentSummary(length, fileCount, directoryCount, - quota, spaceConsumed, spaceQuota); + return new ContentSummary.Builder().length(length).fileCount(fileCount). + directoryCount(directoryCount).quota(quota).spaceConsumed(spaceConsumed). + spaceQuota(spaceQuota).build(); } /** Convert a MD5MD5CRC32FileChecksum to a Json string. */ @@ -610,7 +580,7 @@ public static MD5MD5CRC32FileChecksum toMD5MD5CRC32FileChecksum( final Map m = (Map)json.get(FileChecksum.class.getSimpleName()); final String algorithm = (String)m.get("algorithm"); - final int length = (int)(long)(Long)m.get("length"); + final int length = ((Number) m.get("length")).intValue(); final byte[] bytes = StringUtils.hexStringToByte((String)m.get("bytes")); final DataInputStream in = new DataInputStream(new ByteArrayInputStream(bytes)); @@ -654,7 +624,13 @@ public static String toJsonString(final AclStatus status) { m.put("owner", status.getOwner()); m.put("group", status.getGroup()); m.put("stickyBit", status.isStickyBit()); - m.put("entries", status.getEntries()); + + final List stringEntries = new ArrayList<>(); + for (AclEntry entry : status.getEntries()) { + stringEntries.add(entry.toString()); + } + m.put("entries", stringEntries); + FsPermission perm = status.getPermission(); if (perm != null) { m.put("permission", toString(perm)); @@ -668,7 +644,13 @@ public static String toJsonString(final AclStatus status) { final Map> finalMap = new TreeMap>(); finalMap.put(AclStatus.class.getSimpleName(), m); - return JSON.toString(finalMap); + + ObjectMapper mapper = new ObjectMapper(); + try { + return mapper.writeValueAsString(finalMap); + } catch (IOException ignored) { + } + return null; } /** Convert a Json map to a AclStatus object. */ @@ -689,11 +671,11 @@ public static AclStatus toAclStatus(final Map json) { (Boolean) m.get("aclBit"), (Boolean) m.get("encBit")); aclStatusBuilder.setPermission(permission); } - final Object[] entries = (Object[]) m.get("entries"); + final List entries = (List) m.get("entries"); List aclEntryList = new ArrayList(); - for (int i = 0; i < entries.length; i++) { - AclEntry aclEntry = AclEntry.parseAclEntry((String) entries[i], true); + for (Object entry : entries) { + AclEntry aclEntry = AclEntry.parseAclEntry((String) entry, true); aclEntryList.add(aclEntry); } aclStatusBuilder.addEntries(aclEntryList); @@ -732,7 +714,8 @@ public static String toJsonString(final List xAttrs, final XAttrCodec encoding) throws IOException { final Map finalMap = new TreeMap(); finalMap.put("XAttrs", toJsonArray(xAttrs, encoding)); - return JSON.toString(finalMap); + ObjectMapper mapper = new ObjectMapper(); + return mapper.writeValueAsString(finalMap); } public static String toJsonString(final List xAttrs) @@ -741,10 +724,11 @@ public static String toJsonString(final List xAttrs) for (XAttr xAttr : xAttrs) { names.add(XAttrHelper.getPrefixName(xAttr)); } - String ret = JSON.toString(names); + ObjectMapper mapper = new ObjectMapper(); + String ret = mapper.writeValueAsString(names); final Map finalMap = new TreeMap(); finalMap.put("XAttrNames", ret); - return JSON.toString(finalMap); + return mapper.writeValueAsString(finalMap); } public static byte[] getXAttr(final Map json, final String name) @@ -760,14 +744,13 @@ public static byte[] getXAttr(final Map json, final String name) return null; } - + public static Map toXAttrs(final Map json) throws IOException { if (json == null) { return null; } - - return toXAttrMap((Object[])json.get("XAttrs")); + return toXAttrMap(getList(json, "XAttrs")); } public static List toXAttrNames(final Map json) @@ -777,26 +760,27 @@ public static List toXAttrNames(final Map json) } final String namesInJson = (String) json.get("XAttrNames"); - final Object[] xattrs = (Object[]) JSON.parse(namesInJson); + ObjectReader reader = new ObjectMapper().reader(List.class); + final List xattrs = reader.readValue(namesInJson); final List names = Lists.newArrayListWithCapacity(json.keySet().size()); - for (int i = 0; i < xattrs.length; i++) { - names.add((String) (xattrs[i])); + for (Object xattr : xattrs) { + names.add((String) xattr); } return names; } - private static Map toXAttrMap(final Object[] objects) + private static Map toXAttrMap(final List objects) throws IOException { if (objects == null) { return null; - } else if (objects.length == 0) { + } else if (objects.isEmpty()) { return Maps.newHashMap(); } else { final Map xAttrs = Maps.newHashMap(); - for(int i = 0; i < objects.length; i++) { - Map m = (Map) objects[i]; + for (Object object : objects) { + Map m = (Map) object; String name = (String) m.get("name"); String value = (String) m.get("value"); xAttrs.put(name, decodeXAttrValue(value)); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java index a907404bed73b..12adb05ab7ffa 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java @@ -22,7 +22,6 @@ import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; -import java.io.InputStreamReader; import java.net.HttpURLConnection; import java.net.InetSocketAddress; import java.net.MalformedURLException; @@ -81,10 +80,9 @@ import org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSelector; import org.apache.hadoop.util.Progressable; import org.apache.hadoop.util.StringUtils; -import org.mortbay.util.ajax.JSON; +import org.codehaus.jackson.map.ObjectMapper; import com.google.common.annotations.VisibleForTesting; -import com.google.common.base.Charsets; import com.google.common.base.Preconditions; import com.google.common.collect.Lists; @@ -213,13 +211,6 @@ public URI getCanonicalUri() { return super.getCanonicalUri(); } - /** Is WebHDFS enabled in conf? */ - public static boolean isEnabled(final Configuration conf, final Log log) { - final boolean b = conf.getBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, - DFSConfigKeys.DFS_WEBHDFS_ENABLED_DEFAULT); - return b; - } - TokenSelector tokenSelector = new AbstractDelegationTokenSelector(getTokenKind()){}; @@ -323,7 +314,8 @@ private Path makeAbsolute(Path f) { + "\" (parsed=\"" + parsed + "\")"); } } - return (Map)JSON.parse(new InputStreamReader(in, Charsets.UTF_8)); + ObjectMapper mapper = new ObjectMapper(); + return mapper.reader(Map.class).readValue(in); } finally { in.close(); } @@ -1291,13 +1283,15 @@ public FileStatus[] listStatus(final Path f) throws IOException { @Override FileStatus[] decodeResponse(Map json) { final Map rootmap = (Map)json.get(FileStatus.class.getSimpleName() + "es"); - final Object[] array = (Object[])rootmap.get(FileStatus.class.getSimpleName()); + final List array = JsonUtil.getList( + rootmap, FileStatus.class.getSimpleName()); //convert FileStatus - final FileStatus[] statuses = new FileStatus[array.length]; - for (int i = 0; i < array.length; i++) { - final Map m = (Map)array[i]; - statuses[i] = makeQualified(JsonUtil.toFileStatus(m, false), f); + final FileStatus[] statuses = new FileStatus[array.size()]; + int i = 0; + for (Object object : array) { + final Map m = (Map) object; + statuses[i++] = makeQualified(JsonUtil.toFileStatus(m, false), f); } return statuses; } @@ -1348,7 +1342,7 @@ public synchronized long renewDelegationToken(final Token token new TokenArgumentParam(token.encodeToUrlString())) { @Override Long decodeResponse(Map json) throws IOException { - return (Long) json.get("long"); + return ((Number) json.get("long")).longValue(); } }.run(); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LengthParam.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LengthParam.java index 6c59ee5143252..5a609eeec0a53 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LengthParam.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LengthParam.java @@ -46,4 +46,9 @@ public LengthParam(final String str) { public String getName() { return NAME; } + + public long getLength() { + Long v = getValue(); + return v == null ? -1 : v; + } } \ No newline at end of file diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/OffsetParam.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/OffsetParam.java index 6973787847df4..6d88703da0abc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/OffsetParam.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/OffsetParam.java @@ -46,4 +46,9 @@ public OffsetParam(final String str) { public String getName() { return NAME; } + + public Long getOffset() { + Long offset = getValue(); + return (offset == null) ? Long.valueOf(0) : offset; + } } \ No newline at end of file diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c b/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c index 34acccce70596..504d47e85e48b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c @@ -1037,6 +1037,43 @@ hdfsFile hdfsOpenFile(hdfsFS fs, const char *path, int flags, return file; } +int hdfsTruncateFile(hdfsFS fs, const char* path, tOffset newlength) +{ + jobject jFS = (jobject)fs; + jthrowable jthr; + jvalue jVal; + jobject jPath = NULL; + + JNIEnv *env = getJNIEnv(); + + if (!env) { + errno = EINTERNAL; + return -1; + } + + /* Create an object of org.apache.hadoop.fs.Path */ + jthr = constructNewObjectOfPath(env, path, &jPath); + if (jthr) { + errno = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, + "hdfsTruncateFile(%s): constructNewObjectOfPath", path); + return -1; + } + + jthr = invokeMethod(env, &jVal, INSTANCE, jFS, HADOOP_FS, + "truncate", JMETHOD2(JPARAM(HADOOP_PATH), "J", "Z"), + jPath, newlength); + destroyLocalReference(env, jPath); + if (jthr) { + errno = printExceptionAndFree(env, jthr, PRINT_EXC_ALL, + "hdfsTruncateFile(%s): FileSystem#truncate", path); + return -1; + } + if (jVal.z == JNI_TRUE) { + return 1; + } + return 0; +} + int hdfsUnbufferFile(hdfsFile file) { int ret; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h b/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h index 3406d6b481e48..5b7bc1e3867ca 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h @@ -24,6 +24,30 @@ #include /* for uint64_t, etc. */ #include /* for time_t */ +/* + * Support export of DLL symbols during libhdfs build, and import of DLL symbols + * during client application build. A client application may optionally define + * symbol LIBHDFS_DLL_IMPORT in its build. This is not strictly required, but + * the compiler can produce more efficient code with it. + */ +#ifdef WIN32 + #ifdef LIBHDFS_DLL_EXPORT + #define LIBHDFS_EXTERNAL __declspec(dllexport) + #elif LIBHDFS_DLL_IMPORT + #define LIBHDFS_EXTERNAL __declspec(dllimport) + #else + #define LIBHDFS_EXTERNAL + #endif +#else + #ifdef LIBHDFS_DLL_EXPORT + #define LIBHDFS_EXTERNAL __attribute__((visibility("default"))) + #elif LIBHDFS_DLL_IMPORT + #define LIBHDFS_EXTERNAL __attribute__((visibility("default"))) + #else + #define LIBHDFS_EXTERNAL + #endif +#endif + #ifndef O_RDONLY #define O_RDONLY 1 #endif @@ -77,6 +101,7 @@ extern "C" { * @param file The HDFS file * @return 1 if the file is open for read; 0 otherwise */ + LIBHDFS_EXTERNAL int hdfsFileIsOpenForRead(hdfsFile file); /** @@ -85,6 +110,7 @@ extern "C" { * @param file The HDFS file * @return 1 if the file is open for write; 0 otherwise */ + LIBHDFS_EXTERNAL int hdfsFileIsOpenForWrite(hdfsFile file); struct hdfsReadStatistics { @@ -107,6 +133,7 @@ extern "C" { * ENOTSUP. webhdfs, LocalFilesystem, and so forth may * not support read statistics. */ + LIBHDFS_EXTERNAL int hdfsFileGetReadStatistics(hdfsFile file, struct hdfsReadStatistics **stats); @@ -115,6 +142,7 @@ extern "C" { * * @return the number of remote bytes read. */ + LIBHDFS_EXTERNAL int64_t hdfsReadStatisticsGetRemoteBytesRead( const struct hdfsReadStatistics *stats); @@ -129,6 +157,7 @@ extern "C" { * statistics. * Errno will also be set to this code on failure. */ + LIBHDFS_EXTERNAL int hdfsFileClearReadStatistics(hdfsFile file); /** @@ -136,6 +165,7 @@ extern "C" { * * @param stats The HDFS read statistics to free. */ + LIBHDFS_EXTERNAL void hdfsFileFreeReadStatistics(struct hdfsReadStatistics *stats); /** @@ -147,6 +177,7 @@ extern "C" { * @return Returns a handle to the filesystem or NULL on error. * @deprecated Use hdfsBuilderConnect instead. */ + LIBHDFS_EXTERNAL hdfsFS hdfsConnectAsUser(const char* nn, tPort port, const char *user); /** @@ -157,6 +188,7 @@ extern "C" { * @return Returns a handle to the filesystem or NULL on error. * @deprecated Use hdfsBuilderConnect instead. */ + LIBHDFS_EXTERNAL hdfsFS hdfsConnect(const char* nn, tPort port); /** @@ -170,6 +202,7 @@ extern "C" { * @return Returns a handle to the filesystem or NULL on error. * @deprecated Use hdfsBuilderConnect instead. */ + LIBHDFS_EXTERNAL hdfsFS hdfsConnectAsUserNewInstance(const char* nn, tPort port, const char *user ); /** @@ -182,6 +215,7 @@ extern "C" { * @return Returns a handle to the filesystem or NULL on error. * @deprecated Use hdfsBuilderConnect instead. */ + LIBHDFS_EXTERNAL hdfsFS hdfsConnectNewInstance(const char* nn, tPort port); /** @@ -196,6 +230,7 @@ extern "C" { * @param bld The HDFS builder * @return Returns a handle to the filesystem, or NULL on error. */ + LIBHDFS_EXTERNAL hdfsFS hdfsBuilderConnect(struct hdfsBuilder *bld); /** @@ -203,6 +238,7 @@ extern "C" { * * @return The HDFS builder, or NULL on error. */ + LIBHDFS_EXTERNAL struct hdfsBuilder *hdfsNewBuilder(void); /** @@ -211,6 +247,7 @@ extern "C" { * * @param bld The HDFS builder */ + LIBHDFS_EXTERNAL void hdfsBuilderSetForceNewInstance(struct hdfsBuilder *bld); /** @@ -234,6 +271,7 @@ extern "C" { * hdfsBuilderSetNameNodePort. However, you must not pass the * port in two different ways. */ + LIBHDFS_EXTERNAL void hdfsBuilderSetNameNode(struct hdfsBuilder *bld, const char *nn); /** @@ -242,6 +280,7 @@ extern "C" { * @param bld The HDFS builder * @param port The port. */ + LIBHDFS_EXTERNAL void hdfsBuilderSetNameNodePort(struct hdfsBuilder *bld, tPort port); /** @@ -250,6 +289,7 @@ extern "C" { * @param bld The HDFS builder * @param userName The user name. The string will be shallow-copied. */ + LIBHDFS_EXTERNAL void hdfsBuilderSetUserName(struct hdfsBuilder *bld, const char *userName); /** @@ -260,6 +300,7 @@ extern "C" { * @param kerbTicketCachePath The Kerberos ticket cache path. The string * will be shallow-copied. */ + LIBHDFS_EXTERNAL void hdfsBuilderSetKerbTicketCachePath(struct hdfsBuilder *bld, const char *kerbTicketCachePath); @@ -271,6 +312,7 @@ extern "C" { * * @param bld The HDFS builder */ + LIBHDFS_EXTERNAL void hdfsFreeBuilder(struct hdfsBuilder *bld); /** @@ -284,6 +326,7 @@ extern "C" { * * @return 0 on success; nonzero error code otherwise. */ + LIBHDFS_EXTERNAL int hdfsBuilderConfSetStr(struct hdfsBuilder *bld, const char *key, const char *val); @@ -298,6 +341,7 @@ extern "C" { * @return 0 on success; nonzero error code otherwise. * Failure to find the key is not an error. */ + LIBHDFS_EXTERNAL int hdfsConfGetStr(const char *key, char **val); /** @@ -310,6 +354,7 @@ extern "C" { * @return 0 on success; nonzero error code otherwise. * Failure to find the key is not an error. */ + LIBHDFS_EXTERNAL int hdfsConfGetInt(const char *key, int32_t *val); /** @@ -317,6 +362,7 @@ extern "C" { * * @param val A configuration string obtained from hdfsConfGetStr */ + LIBHDFS_EXTERNAL void hdfsConfStrFree(char *val); /** @@ -327,6 +373,7 @@ extern "C" { * Even if there is an error, the resources associated with the * hdfsFS will be freed. */ + LIBHDFS_EXTERNAL int hdfsDisconnect(hdfsFS fs); @@ -344,9 +391,25 @@ extern "C" { * default configured values. * @return Returns the handle to the open file or NULL on error. */ + LIBHDFS_EXTERNAL hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags, int bufferSize, short replication, tSize blocksize); + /** + * hdfsTruncateFile - Truncate a hdfs file to given lenght. + * @param fs The configured filesystem handle. + * @param path The full path to the file. + * @param newlength The size the file is to be truncated to + * @return 1 if the file has been truncated to the desired newlength + * and is immediately available to be reused for write operations + * such as append. + * 0 if a background process of adjusting the length of the last + * block has been started, and clients should wait for it to + * complete before proceeding with further file updates. + * -1 on error. + */ + int hdfsTruncateFile(hdfsFS fs, const char* path, tOffset newlength); + /** * hdfsUnbufferFile - Reduce the buffering done on a file. * @@ -355,6 +418,7 @@ extern "C" { * ENOTSUP if the file does not support unbuffering * Errno will also be set to this value. */ + LIBHDFS_EXTERNAL int hdfsUnbufferFile(hdfsFile file); /** @@ -367,6 +431,7 @@ extern "C" { * be freed at the end of this call, even if there was an I/O * error. */ + LIBHDFS_EXTERNAL int hdfsCloseFile(hdfsFS fs, hdfsFile file); @@ -376,6 +441,7 @@ extern "C" { * @param path The path to look for * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsExists(hdfsFS fs, const char *path); @@ -387,6 +453,7 @@ extern "C" { * @param desiredPos Offset into the file to seek into. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsSeek(hdfsFS fs, hdfsFile file, tOffset desiredPos); @@ -396,6 +463,7 @@ extern "C" { * @param file The file handle. * @return Current offset, -1 on error. */ + LIBHDFS_EXTERNAL tOffset hdfsTell(hdfsFS fs, hdfsFile file); @@ -413,6 +481,7 @@ extern "C" { * and set errno to EINTR if data is temporarily unavailable, * but we are not yet at the end of the file. */ + LIBHDFS_EXTERNAL tSize hdfsRead(hdfsFS fs, hdfsFile file, void* buffer, tSize length); /** @@ -424,6 +493,7 @@ extern "C" { * @param length The length of the buffer. * @return See hdfsRead */ + LIBHDFS_EXTERNAL tSize hdfsPread(hdfsFS fs, hdfsFile file, tOffset position, void* buffer, tSize length); @@ -436,6 +506,7 @@ extern "C" { * @param length The no. of bytes to write. * @return Returns the number of bytes written, -1 on error. */ + LIBHDFS_EXTERNAL tSize hdfsWrite(hdfsFS fs, hdfsFile file, const void* buffer, tSize length); @@ -446,6 +517,7 @@ extern "C" { * @param file The file handle. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsFlush(hdfsFS fs, hdfsFile file); @@ -456,6 +528,7 @@ extern "C" { * @param file file handle * @return 0 on success, -1 on error and sets errno */ + LIBHDFS_EXTERNAL int hdfsHFlush(hdfsFS fs, hdfsFile file); @@ -467,6 +540,7 @@ extern "C" { * @param file file handle * @return 0 on success, -1 on error and sets errno */ + LIBHDFS_EXTERNAL int hdfsHSync(hdfsFS fs, hdfsFile file); @@ -477,6 +551,7 @@ extern "C" { * @param file The file handle. * @return Returns available bytes; -1 on error. */ + LIBHDFS_EXTERNAL int hdfsAvailable(hdfsFS fs, hdfsFile file); @@ -488,6 +563,7 @@ extern "C" { * @param dst The path of destination file. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsCopy(hdfsFS srcFS, const char* src, hdfsFS dstFS, const char* dst); @@ -499,6 +575,7 @@ extern "C" { * @param dst The path of destination file. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsMove(hdfsFS srcFS, const char* src, hdfsFS dstFS, const char* dst); @@ -511,6 +588,7 @@ extern "C" { * case of a file the recursive argument is irrelevant. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsDelete(hdfsFS fs, const char* path, int recursive); /** @@ -520,6 +598,7 @@ extern "C" { * @param newPath The path of the destination file. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsRename(hdfsFS fs, const char* oldPath, const char* newPath); @@ -531,6 +610,7 @@ extern "C" { * @param bufferSize The length of user-buffer. * @return Returns buffer, NULL on error. */ + LIBHDFS_EXTERNAL char* hdfsGetWorkingDirectory(hdfsFS fs, char *buffer, size_t bufferSize); @@ -541,6 +621,7 @@ extern "C" { * @param path The path of the new 'cwd'. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsSetWorkingDirectory(hdfsFS fs, const char* path); @@ -551,6 +632,7 @@ extern "C" { * @param path The path of the directory. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsCreateDirectory(hdfsFS fs, const char* path); @@ -561,6 +643,7 @@ extern "C" { * @param path The path of the file. * @return Returns 0 on success, -1 on error. */ + LIBHDFS_EXTERNAL int hdfsSetReplication(hdfsFS fs, const char* path, int16_t replication); @@ -590,6 +673,7 @@ extern "C" { * @return Returns a dynamically-allocated array of hdfsFileInfo * objects; NULL on error. */ + LIBHDFS_EXTERNAL hdfsFileInfo *hdfsListDirectory(hdfsFS fs, const char* path, int *numEntries); @@ -603,6 +687,7 @@ extern "C" { * @return Returns a dynamically-allocated hdfsFileInfo object; * NULL on error. */ + LIBHDFS_EXTERNAL hdfsFileInfo *hdfsGetPathInfo(hdfsFS fs, const char* path); @@ -612,6 +697,7 @@ extern "C" { * objects. * @param numEntries The size of the array. */ + LIBHDFS_EXTERNAL void hdfsFreeFileInfo(hdfsFileInfo *hdfsFileInfo, int numEntries); /** @@ -620,6 +706,7 @@ extern "C" { * @return -1 if there was an error (errno will be set), 0 if the file is * not encrypted, 1 if the file is encrypted. */ + LIBHDFS_EXTERNAL int hdfsFileIsEncrypted(hdfsFileInfo *hdfsFileInfo); @@ -635,6 +722,7 @@ extern "C" { * @return Returns a dynamically-allocated 2-d array of blocks-hosts; * NULL on error. */ + LIBHDFS_EXTERNAL char*** hdfsGetHosts(hdfsFS fs, const char* path, tOffset start, tOffset length); @@ -645,6 +733,7 @@ extern "C" { * objects. * @param numEntries The size of the array. */ + LIBHDFS_EXTERNAL void hdfsFreeHosts(char ***blockHosts); @@ -656,6 +745,7 @@ extern "C" { * * @return Returns the default blocksize, or -1 on error. */ + LIBHDFS_EXTERNAL tOffset hdfsGetDefaultBlockSize(hdfsFS fs); @@ -669,6 +759,7 @@ extern "C" { * * @return Returns the default blocksize, or -1 on error. */ + LIBHDFS_EXTERNAL tOffset hdfsGetDefaultBlockSizeAtPath(hdfsFS fs, const char *path); @@ -677,6 +768,7 @@ extern "C" { * @param fs The configured filesystem handle. * @return Returns the raw-capacity; -1 on error. */ + LIBHDFS_EXTERNAL tOffset hdfsGetCapacity(hdfsFS fs); @@ -685,6 +777,7 @@ extern "C" { * @param fs The configured filesystem handle. * @return Returns the total-size; -1 on error. */ + LIBHDFS_EXTERNAL tOffset hdfsGetUsed(hdfsFS fs); /** @@ -696,6 +789,7 @@ extern "C" { * @param group Group string. Set to NULL for 'no change' * @return 0 on success else -1 */ + LIBHDFS_EXTERNAL int hdfsChown(hdfsFS fs, const char* path, const char *owner, const char *group); @@ -706,7 +800,8 @@ extern "C" { * @param mode the bitmask to set it to * @return 0 on success else -1 */ - int hdfsChmod(hdfsFS fs, const char* path, short mode); + LIBHDFS_EXTERNAL + int hdfsChmod(hdfsFS fs, const char* path, short mode); /** * hdfsUtime @@ -716,6 +811,7 @@ extern "C" { * @param atime new access time or -1 for no change * @return 0 on success else -1 */ + LIBHDFS_EXTERNAL int hdfsUtime(hdfsFS fs, const char* path, tTime mtime, tTime atime); /** @@ -728,6 +824,7 @@ extern "C" { * not be allocated. If NULL is returned, errno will * contain the error number. */ + LIBHDFS_EXTERNAL struct hadoopRzOptions *hadoopRzOptionsAlloc(void); /** @@ -739,6 +836,7 @@ extern "C" { * * @return 0 on success; -1 plus errno on failure. */ + LIBHDFS_EXTERNAL int hadoopRzOptionsSetSkipChecksum( struct hadoopRzOptions *opts, int skip); @@ -756,6 +854,7 @@ extern "C" { * instantiated; * -1 plus errno otherwise. */ + LIBHDFS_EXTERNAL int hadoopRzOptionsSetByteBufferPool( struct hadoopRzOptions *opts, const char *className); @@ -765,6 +864,7 @@ extern "C" { * @param opts The options structure to free. * Any associated ByteBufferPool will also be freed. */ + LIBHDFS_EXTERNAL void hadoopRzOptionsFree(struct hadoopRzOptions *opts); /** @@ -790,6 +890,7 @@ extern "C" { * zero-copy read, and there was no ByteBufferPool * supplied. */ + LIBHDFS_EXTERNAL struct hadoopRzBuffer* hadoopReadZero(hdfsFile file, struct hadoopRzOptions *opts, int32_t maxLength); @@ -799,6 +900,7 @@ extern "C" { * @param buffer a buffer returned from readZero. * @return the length of the buffer. */ + LIBHDFS_EXTERNAL int32_t hadoopRzBufferLength(const struct hadoopRzBuffer *buffer); /** @@ -811,6 +913,7 @@ extern "C" { * @return a pointer to the start of the buffer. This will be * NULL when end-of-file has been reached. */ + LIBHDFS_EXTERNAL const void *hadoopRzBufferGet(const struct hadoopRzBuffer *buffer); /** @@ -820,12 +923,14 @@ extern "C" { * the same stream you called hadoopReadZero on. * @param buffer The buffer to release. */ + LIBHDFS_EXTERNAL void hadoopRzBufferFree(hdfsFile file, struct hadoopRzBuffer *buffer); #ifdef __cplusplus } #endif +#undef LIBHDFS_EXTERNAL #endif /*LIBHDFS_HDFS_H*/ /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c b/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c index b37ebcce0b6c8..ab6abdae404f9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c @@ -128,22 +128,6 @@ struct NativeMiniDfsCluster* nmdCreate(struct NativeMiniDfsConf *conf) "nmdCreate: new Configuration"); goto error; } - if (conf->webhdfsEnabled) { - jthr = newJavaStr(env, DFS_WEBHDFS_ENABLED_KEY, &jconfStr); - if (jthr) { - printExceptionAndFree(env, jthr, PRINT_EXC_ALL, - "nmdCreate: new String"); - goto error; - } - jthr = invokeMethod(env, NULL, INSTANCE, cobj, HADOOP_CONF, - "setBoolean", "(Ljava/lang/String;Z)V", - jconfStr, conf->webhdfsEnabled); - if (jthr) { - printExceptionAndFree(env, jthr, PRINT_EXC_ALL, - "nmdCreate: Configuration::setBoolean"); - goto error; - } - } if (jthr) { printExceptionAndFree(env, jthr, PRINT_EXC_ALL, "nmdCreate: Configuration::setBoolean"); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto index 82709a6d8b308..b44c556bbaa68 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto @@ -350,10 +350,13 @@ message SetSafeModeResponseProto { required bool result = 1; } -message SaveNamespaceRequestProto { // no parameters +message SaveNamespaceRequestProto { + optional uint64 timeWindow = 1 [default = 0]; + optional uint64 txGap = 2 [default = 0]; } message SaveNamespaceResponseProto { // void response + optional bool saved = 1 [default = true]; } message RollEditsRequestProto { // no parameters diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto index 348f34606e21b..3083dc90f6d9c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto @@ -224,11 +224,25 @@ message HeartbeatResponseProto { * second long represents length * third long represents gen stamp * fourth long (if under construction) represents replica state + * context - An optional field containing information about the context + * of this block report. */ message BlockReportRequestProto { required DatanodeRegistrationProto registration = 1; required string blockPoolId = 2; repeated StorageBlockReportProto reports = 3; + optional BlockReportContextProto context = 4; +} + +message BlockReportContextProto { + // The total number of RPCs this block report is broken into. + required int32 totalRpcs = 1; + + // The index of the current RPC (zero-based) + required int32 curRpc = 2; + + // The unique 64-bit ID of this block report + required int64 id = 3; } /** @@ -237,6 +251,8 @@ message BlockReportRequestProto { message StorageBlockReportProto { required DatanodeStorageProto storage = 1; // Storage repeated uint64 blocks = 2 [packed=true]; + optional uint64 numberOfBlocks = 3; + repeated bytes blocksBuffers = 4; } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto index d72bb5e150e66..5071d15a8dfa8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto @@ -179,6 +179,12 @@ message OpRequestShortCircuitAccessProto { * The shared memory slot to use, if we are using one. */ optional ShortCircuitShmSlotProto slotId = 3; + + /** + * True if the client supports verifying that the file descriptor has been + * sent successfully. + */ + optional bool supportsReceiptVerification = 4 [default = false]; } message ReleaseShortCircuitAccessRequestProto { @@ -230,10 +236,16 @@ enum Status { IN_PROGRESS = 12; } +enum ShortCircuitFdResponse { + DO_NOT_USE_RECEIPT_VERIFICATION = 0; + USE_RECEIPT_VERIFICATION = 1; +} + message PipelineAckProto { required sint64 seqno = 1; - repeated uint32 reply = 2; + repeated Status reply = 2; optional uint64 downstreamAckTimeNanos = 3 [default = 0]; + repeated uint32 flag = 4 [packed=true]; } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto index 97906b164028e..86fb46297d974 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto @@ -97,6 +97,7 @@ message DatanodeInfoProto { optional AdminState adminState = 10 [default = NORMAL]; optional uint64 cacheCapacity = 11 [default = 0]; optional uint64 cacheUsed = 12 [default = 0]; + optional uint64 lastUpdateMonotonic = 13 [default = 0]; } /** @@ -133,6 +134,20 @@ message ContentSummaryProto { required uint64 quota = 4; required uint64 spaceConsumed = 5; required uint64 spaceQuota = 6; + optional StorageTypeQuotaInfosProto typeQuotaInfos = 7; +} + +/** + * Storage type quota and usage information of a file or directory + */ +message StorageTypeQuotaInfosProto { + repeated StorageTypeQuotaInfoProto typeQuotaInfo = 1; +} + +message StorageTypeQuotaInfoProto { + required StorageTypeProto type = 1; + required uint64 quota = 2; + required uint64 consumed = 3; } /** @@ -517,6 +532,7 @@ message NamespaceInfoProto { required string blockPoolID = 3; // block pool used by the namespace required StorageInfoProto storageInfo = 4;// Node information required string softwareVersion = 5; // Software version number (e.g. 2.0.0) + optional uint64 capabilities = 6 [default = 0]; // feature flags } /** @@ -591,4 +607,5 @@ message SnapshotInfoProto { */ message RollingUpgradeStatusProto { required string blockPoolId = 1; + optional bool finalized = 2 [default = false]; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml index 7eacfc58c868a..092d5aa6de5bc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml @@ -736,10 +736,25 @@ - dfs.namenode.decommission.nodes.per.interval - 5 - The number of nodes namenode checks if decommission is complete - in each dfs.namenode.decommission.interval. + dfs.namenode.decommission.blocks.per.interval + 500000 + The approximate number of blocks to process per + decommission interval, as defined in dfs.namenode.decommission.interval. + + + + + dfs.namenode.decommission.max.concurrent.tracked.nodes + 100 + + The maximum number of decommission-in-progress datanodes nodes that will be + tracked at one time by the namenode. Tracking a decommission-in-progress + datanode consumes additional NN memory proportional to the number of blocks + on the datnode. Having a conservative limit reduces the potential impact + of decomissioning a large number of nodes at once. + + A value of 0 means no limit will be enforced. + @@ -2299,4 +2314,11 @@ Whether pin blocks on favored DataNode. + + dfs.client.block.write.locateFollowingBlock.initial.delay.ms + 400 + The initial delay (unit is ms) for locateFollowingBlock, + the delay time will increase exponentially(double) for each retry. + + diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs b/hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs.sh similarity index 100% rename from hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs rename to hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs.sh diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html index 391ca79ce676e..928431cefa8f8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html @@ -79,6 +79,9 @@ {#RollingUpgradeStatus} + {@if cond="{finalizeTime} > 0"} +

    Rolling upgrade finalized at {#helper_date_tostring value="{finalizeTime}"/}.

    + {:else}

    Rolling upgrade started at {#helper_date_tostring value="{startTime}"/}.
    {#createdRollbackImages} Rollback image has been created. Proceed to upgrade daemons. @@ -86,6 +89,7 @@ Rollback image has not been created. {/createdRollbackImages}

    + {/if} {/RollingUpgradeStatus} {@if cond="{DistinctVersionCount} > 1"} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.html b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.html index 7b340441b5fb4..cd6623cc97d39 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.html +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.html @@ -87,13 +87,56 @@

    Browse Directory

    + + +
    -
    -
    -
    +
    +
    +
    + + + + +
    +
    +
    +
    +
    +
    +
    diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.js b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.js index 0a53dcd65a86d..5572880d184bb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.js +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.js @@ -78,6 +78,12 @@ return data.RemoteException !== undefined ? data.RemoteException.message : ""; } + function encode_path(abs_path) { + abs_path = encodeURIComponent(abs_path); + var re = /%2F/g; + return abs_path.replace(re, '/'); + } + function view_file_details(path, abs_path) { function show_block_info(blocks) { var menus = $('#file-info-blockinfo-list'); @@ -102,12 +108,6 @@ menus.change(); } - function encode_path(abs_path) { - abs_path = encodeURIComponent(abs_path); - var re = /%2F/g; - return abs_path.replace(re, '/'); - } - abs_path = encode_path(abs_path); var url = '/webhdfs/v1' + abs_path + '?op=GET_BLOCK_LOCATIONS'; $.get(url).done(function(data) { @@ -149,7 +149,7 @@ return chunk.write('' + new Date(Number(value)).toLocaleString()); } }; - var url = '/webhdfs/v1' + dir + '?op=LISTSTATUS'; + var url = '/webhdfs/v1' + encode_path(dir) + '?op=LISTSTATUS'; $.get(url, function(data) { var d = get_response(data, "FileStatuses"); if (d === null) { @@ -193,5 +193,27 @@ } } + $('#btn-create-directory').on('show.bs.modal', function(event) { + var modal = $(this) + $('#new_directory_pwd').html(current_directory); + }); + + $('#btn-create-directory-send').click(function () { + $(this).prop('disabled', true); + $(this).button('complete'); + + var url = '/webhdfs/v1' + encode_path(append_path(current_directory, + $('#new_directory').val())) + '?op=MKDIRS'; + + $.ajax(url, { type: 'PUT' } + ).done(function(data) { + browse_directory(current_directory); + }).error(network_error_handler(url) + ).complete(function() { + $('#btn-create-directory').modal('hide'); + $('#btn-create-directory-send').button('reset'); + }); + }) + init(); })(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md index 191b5bc7ca11f..bdb051ba875cc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md @@ -307,8 +307,8 @@ Usage: [-refreshNodes] [-setQuota ...] [-clrQuota ...] - [-setSpaceQuota ...] - [-clrSpaceQuota ...] + [-setSpaceQuota [-storageType ] ...] + [-clrSpaceQuota [-storageType ] ...] [-setStoragePolicy ] [-getStoragePolicy ] [-finalizeUpgrade] @@ -342,8 +342,8 @@ Usage: | `-refreshNodes` | Re-read the hosts and exclude files to update the set of Datanodes that are allowed to connect to the Namenode and those that should be decommissioned or recommissioned. | | `-setQuota` \ \...\ | See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html#Administrative_Commands) for the detail. | | `-clrQuota` \...\ | See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html#Administrative_Commands) for the detail. | -| `-setSpaceQuota` \ \...\ | See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html#Administrative_Commands) for the detail. | -| `-clrSpaceQuota` \...\ | See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html#Administrative_Commands) for the detail. | +| `-setSpaceQuota` \ `[-storageType ]` \...\ | See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html#Administrative_Commands) for the detail. | +| `-clrSpaceQuota` `[-storageType ]` \...\ | See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html#Administrative_Commands) for the detail. | | `-setStoragePolicy` \ \ | Set a storage policy to a file or a directory. | | `-getStoragePolicy` \ | Get the storage policy of a file or a directory. | | `-finalizeUpgrade` | Finalize upgrade of HDFS. Datanodes delete their previous version working directories, followed by Namenode doing the same. This completes the upgrade process. | diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md index 87a9fcd1cefcf..5a8e3662c4012 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md @@ -224,9 +224,9 @@ Space Reclamation ### File Deletes and Undeletes -When a file is deleted by a user or an application, it is not immediately removed from HDFS. Instead, HDFS first renames it to a file in the `/trash` directory. The file can be restored quickly as long as it remains in `/trash`. A file remains in `/trash` for a configurable amount of time. After the expiry of its life in `/trash`, the NameNode deletes the file from the HDFS namespace. The deletion of a file causes the blocks associated with the file to be freed. Note that there could be an appreciable time delay between the time a file is deleted by a user and the time of the corresponding increase in free space in HDFS. +When a file is deleted by a user or an application, it is not immediately removed from HDFS. Instead, HDFS first renames it to a file in the trash directory(`/user//.Trash`). The file can be restored quickly as long as it remains in trash. A file remains in trash for a configurable amount of time. After the expiry of its life in trash, the NameNode deletes the file from the HDFS namespace. The deletion of a file causes the blocks associated with the file to be freed. Note that there could be an appreciable time delay between the time a file is deleted by a user and the time of the corresponding increase in free space in HDFS. -A user can Undelete a file after deleting it as long as it remains in the `/trash` directory. If a user wants to undelete a file that he/she has deleted, he/she can navigate the `/trash` directory and retrieve the file. The `/trash` directory contains only the latest copy of the file that was deleted. The `/trash` directory is just like any other directory with one special feature: HDFS applies specified policies to automatically delete files from this directory. Current default trash interval is set to 0 (Deletes file without storing in trash). This value is configurable parameter stored as `fs.trash.interval` stored in core-site.xml. +A user can Undelete a file after deleting it as long as it remains in the trash directory. If a user wants to undelete a file that he/she has deleted, he/she can navigate the trash directory and retrieve the file. The trash directory contains only the latest copy of the file that was deleted. The trash directory is just like any other directory with one special feature: HDFS applies specified policies to automatically delete files from this directory. Current default trash interval is set to 0 (Deletes file without storing in trash). This value is configurable parameter stored as `fs.trash.interval` stored in core-site.xml. ### Decrease Replication Factor diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md index 864eee8d70755..e3952cdef48cb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md @@ -55,6 +55,11 @@ The Offline Image Viewer provides several output processors: and numFiles is the number of files form the image which size falls in this segment. +4. Delimited (experimental): Generate a text file with all of the elements + common to both inodes and inodes-under-construction, separated by a + delimiter. The default delimiter is \t, though this may be changed via + the -delimiter argument. + Usage ----- @@ -137,6 +142,8 @@ Options | `-addr` *address* | Specify the address(host:port) to listen. (localhost:5978 by default). This option is used with Web processor. | | `-maxSize` *size* | Specify the range [0, maxSize] of file sizes to be analyzed in bytes (128GB by default). This option is used with FileDistribution processor. | | `-step` *size* | Specify the granularity of the distribution in bytes (2MB by default). This option is used with FileDistribution processor. | +| `-delimiter` *arg* | Delimiting string to use with Delimited processor. | +| `-t`\|`--temp` *temporary dir* | Use temporary dir to cache intermediate result to generate Delimited outputs. If not set, Delimited processor constructs the namespace in memory before outputting text. | | `-h`\|`--help` | Display the tool usage and help information and exit. | Analyzing Results diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md index cea491f12e742..a49d1688b2b0f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsNfsGateway.md @@ -80,14 +80,33 @@ The above are the only required configuration for the NFS gateway in non-secure The rest of the NFS gateway configurations are optional for both secure and non-secure mode. -The AIX NFS client has a [few known issues](https://issues.apache.org/jira/browse/HDFS-6549) that prevent it from working correctly by default with the HDFS NFS Gateway. If you want to be able to access the HDFS NFS Gateway from AIX, you should set the following configuration setting to enable work-arounds for these issues: +* The AIX NFS client has a [few known issues](https://issues.apache.org/jira/browse/HDFS-6549) + that prevent it from working correctly by default with the HDFS NFS Gateway. If you want to + be able to access the HDFS NFS Gateway from AIX, you should set the following configuration + setting to enable work-arounds for these issues: - - nfs.aix.compatibility.mode.enabled - true - + + nfs.aix.compatibility.mode.enabled + true + + + Note that regular, non-AIX clients should NOT enable AIX compatibility mode. The work-arounds + implemented by AIX compatibility mode effectively disable safeguards to ensure that listing + of directory contents via NFS returns consistent results, and that all data sent to the NFS + server can be assured to have been committed. -Note that regular, non-AIX clients should NOT enable AIX compatibility mode. The work-arounds implemented by AIX compatibility mode effectively disable safeguards to ensure that listing of directory contents via NFS returns consistent results, and that all data sent to the NFS server can be assured to have been committed. +* HDFS super-user is the user with the same identity as NameNode process itself and + the super-user can do anything in that permissions checks never fail for the super-user. + If the following property is configured, the superuser on NFS client can access any file + on HDFS. By default, the super user is not configured in the gateway. + Note that, even the the superuser is configured, "nfs.exports.allowed.hosts" still takes effect. + For example, the superuser will not have write access to HDFS files through the gateway if + the NFS client host is not allowed to have write access in "nfs.exports.allowed.hosts". + + + nfs.superuser + the_name_of_hdfs_superuser + It's strongly recommended for the users to update a few configuration properties based on their use cases. All the following configuration properties can be added or updated in hdfs-site.xml. @@ -107,7 +126,8 @@ It's strongly recommended for the users to update a few configuration properties * Users are expected to update the file dump directory. NFS client often - reorders writes. Sequential writes can arrive at the NFS gateway at random + reorders writes, especially when the export is not mounted with "sync" option. + Sequential writes can arrive at the NFS gateway at random order. This directory is used to temporarily save out-of-order writes before writing to HDFS. For each file, the out-of-order writes are dumped after they are accumulated to exceed certain threshold (e.g., 1MB) in memory. @@ -125,16 +145,43 @@ It's strongly recommended for the users to update a few configuration properties * By default, the export can be mounted by any client. To better control the access, users can update the following property. The value string contains machine name and access privilege, separated by whitespace - characters. The machine name format can be a single host, a Java regular expression, or an IPv4 address. The access + characters. The machine name format can be a single host, a "*", a Java regular expression, or an IPv4 address. The access privilege uses rw or ro to specify read/write or read-only access of the machines to exports. If the access privilege is not provided, the default is read-only. Entries are separated by ";". - For example: "192.168.0.0/22 rw ; host.\*\\.example\\.com ; host1.test.org ro;". Only the NFS gateway needs to restart after - this property is updated. + For example: "192.168.0.0/22 rw ; \\\\w\*\\\\.example\\\\.com ; host1.test.org ro;". Only the NFS gateway needs to restart after + this property is updated. Note that, here Java regular expression is differnt with the regrulation expression used in + Linux NFS export table, such as, using "\\\\w\*\\\\.example\\\\.com" instead of "\*.example.com", "192\\\\.168\\\\.0\\\\.(11|22)" + instead of "192.168.0.[11|22]" and so on. nfs.exports.allowed.hosts * rw +* HDFS super-user is the user with the same identity as NameNode process itself and + the super-user can do anything in that permissions checks never fail for the super-user. + If the following property is configured, the superuser on NFS client can access any file + on HDFS. By default, the super user is not configured in the gateway. + Note that, even the the superuser is configured, "nfs.exports.allowed.hosts" still takes effect. + For example, the superuser will not have write access to HDFS files through the gateway if + the NFS client host is not allowed to have write access in "nfs.exports.allowed.hosts". + + + nfs.superuser + the_name_of_hdfs_superuser + + +* Metrics. Like other HDFS daemons, the gateway exposes runtime metrics. It is available at `http://gateway-ip:50079/jmx` as a JSON document. + The NFS handler related metrics is exposed under the name "Nfs3Metrics". The latency histograms can be enabled by adding the following + property to hdfs-site.xml file. + + + nfs.metrics.percentiles.intervals + 100 + Enable the latency histograms for read, write and + commit requests. The time unit is 100 seconds in this example. + + + * JVM and log settings. You can export JVM settings (e.g., heap size and GC log) in HADOOP\_NFS3\_OPTS. More NFS related settings can be found in hadoop-env.sh. To get NFS debug trace, you can edit the log4j.property file @@ -220,13 +267,16 @@ Verify validity of NFS related services Mount the export "/" -------------------- -Currently NFS v3 only uses TCP as the transportation protocol. NLM is not supported so mount option "nolock" is needed. It's recommended to use hard mount. This is because, even after the client sends all data to NFS gateway, it may take NFS gateway some extra time to transfer data to HDFS when writes were reorderd by NFS client Kernel. +Currently NFS v3 only uses TCP as the transportation protocol. NLM is not supported so mount option "nolock" is needed. +Mount option "sync" is strongly recommended since it can minimize or avoid reordered writes, which results in more predictable throughput. + Not specifying the sync option may cause unreliable behavior when uploading large files. + It's recommended to use hard mount. This is because, even after the client sends all data to NFS gateway, it may take NFS gateway some extra time to transfer data to HDFS when writes were reorderd by NFS client Kernel. If soft mount has to be used, the user should give it a relatively long timeout (at least no less than the default timeout on the host) . The users can mount the HDFS namespace as shown below: - [root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl $server:/ $mount_point + [root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl,sync $server:/ $mount_point Then the users can access HDFS as part of the local file system except that, hard link and random write are not supported yet. To optimize the performance of large file I/O, one can increase the NFS transfer size(rsize and wsize) during mount. By default, NFS gateway supports 1MB as the maximum transfer size. For larger data transfer size, one needs to update "nfs.rtmax" and "nfs.rtmax" in hdfs-site.xml. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md index a1bcd78ebcf92..7c15bb11ed6ca 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsQuotaAdminGuide.md @@ -19,6 +19,7 @@ HDFS Quotas Guide * [Overview](#Overview) * [Name Quotas](#Name_Quotas) * [Space Quotas](#Space_Quotas) + * [Storage Type Quotas](#Storage_Type_Quotas) * [Administrative Commands](#Administrative_Commands) * [Reporting Command](#Reporting_Command) @@ -41,6 +42,17 @@ The space quota is a hard limit on the number of bytes used by files in the tree Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in violation of a quota (perhaps the fsimage was surreptitiously modified), a warning is printed for each of such violations. Setting or removing a quota creates a journal entry. +Storage Type Quotas +------------------ + +The storage type quota is a hard limit on the usage of specific storage type (SSD, DISK, ARCHIVE) by files in the tree rooted at the directory. It works similar to storage space quota in many aspects but offers fine-grain control over the cluster storage space usage. To set storage type quota on a directory, storage policies must be configured on the directory in order to allow files to be stored in different storage types according to the storage policy. See the [HDFS Storage Policy Documentation](./ArchivalStorage.html) for more information. + +The storage type quota can be combined with the space quotas and name quotas to efficiently manage the cluster storage usage. For example, + +1. For directories with storage policy configured, administrator should set storage type quotas for resource constraint storage types such as SSD and leave quotas for other storage types and overall space quota with either less restrictive values or default unlimited. HDFS will deduct quotas from both target storage type based on storage policy and the overall space quota. +2. For directories without storage policy configured, administrator should not configure storage type quota. Storage type quota can be configured even though the specific storage type is unavailable (or available but not configured properly with storage type information). However, overall space quota is recommended in this case as the storage type information is either unavailable or inaccurate for storage type quota enforcement. +3. Storage type quota on DISK are of limited use except when DISK is not the dominant storage medium. (e.g. cluster with predominantly ARCHIVE storage). + Administrative Commands ----------------------- @@ -77,17 +89,40 @@ Quotas are managed by a set of commands available only to the administrator. directory, with faults reported if the directory does not exist or it is a file. It is not a fault if the directory has no quota. + +* `hdfs dfsadmin -setSpaceQuota -storageType ...` + + Set the storage type quota to be N bytes of storage type specified for each directory. + This is a hard limit on total storage type usage for all the files under the directory tree. + The storage type quota usage reflects the intended usage based on storage policy. For example, + one GB of data with replication of 3 and ALL_SSD storage policy consumes 3GB of SSD quota. N + can also be specified with a binary prefix for convenience, for e.g. 50g for 50 + gigabytes and 2t for 2 terabytes etc. Best effort for each + directory, with faults reported if N is neither zero nor a positive + integer, the directory does not exist or it is a file, or the + directory would immediately exceed the new quota. + +* `hdfs dfsadmin -clrSpaceQuota -storageType ...` + + Remove storage type quota specified for each directory. Best effort + for each directory, with faults reported if the directory does not exist or + it is a file. It is not a fault if the directory has no storage type quota on + for storage type specified. + Reporting Command ----------------- An an extension to the count command of the HDFS shell reports quota values and the current count of names and bytes in use. -* `hadoop fs -count -q [-h] [-v] ...` +* `hadoop fs -count -q [-h] [-v] [-t [comma-separated list of storagetypes]] ...` With the -q option, also report the name quota value set for each directory, the available name quota remaining, the space quota value set, and the available space quota remaining. If the directory does not have a quota set, the reported values are `none` and `inf`. The -h option shows sizes in human readable format. - The -v option displays a header line. - + The -v option displays a header line. The -t option displays the per + storage type quota set and the available quota remaining for each directory. + If specific storage types are given after -t option, only quota and remaining + quota of the types specified will be displayed. Otherwise, quota and + remaining quota of all storage types that support quota will be displayed. \ No newline at end of file diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md index 37fa4be0099c3..ffd8532eb6ec2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md @@ -264,7 +264,7 @@ For command usage, see [balancer](./HDFSCommands.html#balancer). Rack Awareness -------------- -Typically large Hadoop clusters are arranged in racks and network traffic between different nodes with in the same rack is much more desirable than network traffic across the racks. In addition NameNode tries to place replicas of block on multiple racks for improved fault tolerance. Hadoop lets the cluster administrators decide which rack a node belongs to through configuration variable `net.topology.script.file.name`. When this script is configured, each node runs the script to determine its rack id. A default installation assumes all the nodes belong to the same rack. This feature and configuration is further described in PDF attached to [HADOOP-692](https://issues.apache.org/jira/browse/HADOOP-692). +A HDFS cluster can recognize the topology of racks where each nodes are put. It is important to configure this topology in order to optimize the data capacity and usage. For more detail, please check the [rack awareness](../hadoop-common/RackAwareness.html) in common document. Safemode -------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml b/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml index f2f3ebe9e1cb5..2ad28e110a4cd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml @@ -47,7 +47,18 @@ These two capabilities make it feasible to upgrade HDFS without incurring HDFS downtime. In order to upgrade a HDFS cluster without downtime, the cluster must be setup with HA.

    - +

    + If there is any new feature which is enabled in new software release, may not work with old software release after upgrade. + In such cases upgrade should be done by following steps. +

    +
      +
    1. Disable new feature.
      2. +
    2. Upgrade the cluster.
      3. +
    3. Enable the new feature.
      4. +
    +

    + Note that rolling upgrade is supported only from Hadoop-2.4.0 onwards. +

    In a HA cluster, there are two or more NameNodes (NNs), many DataNodes (DNs), diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java index 0c3abec90c13b..2ff7050515706 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfs.java @@ -85,7 +85,6 @@ protected IOException unwrapException(IOException e) { @BeforeClass public static void beforeClassSetup() throws Exception { Configuration conf = new HdfsConfiguration(); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); conf.set(FsPermission.UMASK_LABEL, "000"); conf.setInt(DFSConfigKeys.DFS_NAMENODE_MAX_COMPONENT_LENGTH_KEY, 0); cluster = new MiniDFSCluster.Builder(conf).build(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java index 7e7ff39e8913f..5fc78d1bf3a7e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdfs; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Charsets; import com.google.common.base.Joiner; import com.google.common.base.Preconditions; @@ -89,6 +90,7 @@ import org.apache.hadoop.security.token.Token; import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.Time; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.VersionInfo; import org.apache.log4j.Level; @@ -903,7 +905,7 @@ public static byte[] loadFile(String filename) throws IOException { public static BlockOpResponseProto transferRbw(final ExtendedBlock b, final DFSClient dfsClient, final DatanodeInfo... datanodes) throws IOException { assertEquals(2, datanodes.length); - final Socket s = DFSOutputStream.createSocketForPipeline(datanodes[0], + final Socket s = DataStreamer.createSocketForPipeline(datanodes[0], datanodes.length, dfsClient); final long writeTimeout = dfsClient.getDatanodeWriteTimeout(datanodes.length); final DataOutputStream out = new DataOutputStream(new BufferedOutputStream( @@ -1019,7 +1021,7 @@ public static DatanodeInfo getLocalDatanodeInfo(String ipAddr, DFSConfigKeys.DFS_DATANODE_HTTP_DEFAULT_PORT, DFSConfigKeys.DFS_DATANODE_HTTPS_DEFAULT_PORT, DFSConfigKeys.DFS_DATANODE_IPC_DEFAULT_PORT, - 1l, 2l, 3l, 4l, 0l, 0l, 5, 6, "local", adminState); + 1l, 2l, 3l, 4l, 0l, 0l, 0l, 5, 6, "local", adminState); } public static DatanodeDescriptor getDatanodeDescriptor(String ipAddr, @@ -1571,9 +1573,11 @@ public static DatanodeDescriptor getExpectedPrimaryNode(NameNode nn, // the one to be in charge of the synchronization / recovery protocol. final DatanodeStorageInfo[] storages = ucBlock.getExpectedStorageLocations(); DatanodeStorageInfo expectedPrimary = storages[0]; - long mostRecentLastUpdate = expectedPrimary.getDatanodeDescriptor().getLastUpdate(); + long mostRecentLastUpdate = expectedPrimary.getDatanodeDescriptor() + .getLastUpdateMonotonic(); for (int i = 1; i < storages.length; i++) { - final long lastUpdate = storages[i].getDatanodeDescriptor().getLastUpdate(); + final long lastUpdate = storages[i].getDatanodeDescriptor() + .getLastUpdateMonotonic(); if (lastUpdate > mostRecentLastUpdate) { expectedPrimary = storages[i]; mostRecentLastUpdate = lastUpdate; @@ -1710,4 +1714,21 @@ public static void setNameNodeLogLevel(Level level) { GenericTestUtils.setLogLevel(NameNode.stateChangeLog, level); GenericTestUtils.setLogLevel(NameNode.blockStateChangeLog, level); } + + /** + * Set the datanode dead + */ + public static void setDatanodeDead(DatanodeInfo dn) { + dn.setLastUpdate(0); + dn.setLastUpdateMonotonic(0); + } + + /** + * Update lastUpdate and lastUpdateMonotonic with some offset. + */ + public static void resetLastUpdatesWithOffset(DatanodeInfo dn, long offset) { + dn.setLastUpdate(Time.now() + offset); + dn.setLastUpdateMonotonic(Time.monotonicNow() + offset); + } + } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java index 2c1d07e71603d..d92f49e2fd054 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java @@ -60,6 +60,7 @@ import java.util.List; import java.util.Map; import java.util.Random; +import java.util.Set; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; @@ -77,9 +78,12 @@ import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.protocol.ClientProtocol; +import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType; +import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil; +import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.StartupOption; import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.common.Util; @@ -115,6 +119,7 @@ import com.google.common.base.Joiner; import com.google.common.base.Preconditions; import com.google.common.collect.Lists; +import com.google.common.collect.Sets; /** * This class creates a single-process DFS cluster for junit testing. @@ -520,7 +525,8 @@ public void setDnArgs(String ... args) { private boolean federation; private boolean checkExitOnShutdown = true; protected final int storagesPerDatanode; - + private Set fileSystems = Sets.newHashSet(); + /** * A unique instance identifier for the cluster. This * is used to disambiguate HA filesystems in the case where @@ -1343,7 +1349,6 @@ public synchronized void startDataNodes(Configuration conf, int numDataNodes, } int curDatanodesNum = dataNodes.size(); - final int curDatanodesNumSaved = curDatanodesNum; // for mincluster's the default initialDelay for BRs is 0 if (conf.get(DFS_BLOCKREPORT_INITIAL_DELAY_KEY) == null) { conf.setLong(DFS_BLOCKREPORT_INITIAL_DELAY_KEY, 0); @@ -1703,6 +1708,13 @@ public void shutdown() { * Shutdown all the nodes in the cluster. */ public void shutdown(boolean deleteDfsDir) { + shutdown(deleteDfsDir, true); + } + + /** + * Shutdown all the nodes in the cluster. + */ + public void shutdown(boolean deleteDfsDir, boolean closeFileSystem) { LOG.info("Shutting down the Mini HDFS Cluster"); if (checkExitOnShutdown) { if (ExitUtil.terminateCalled()) { @@ -1712,6 +1724,16 @@ public void shutdown(boolean deleteDfsDir) { throw new AssertionError("Test resulted in an unexpected exit"); } } + if (closeFileSystem) { + for (FileSystem fs : fileSystems) { + try { + fs.close(); + } catch (IOException ioe) { + LOG.warn("Exception while closing file system", ioe); + } + } + fileSystems.clear(); + } shutdownDataNodes(); for (NameNodeInfo nnInfo : nameNodes) { if (nnInfo == null) continue; @@ -2022,7 +2044,23 @@ public boolean restartDataNode(int i) throws IOException { */ public synchronized boolean restartDataNode(int i, boolean keepPort) throws IOException { - DataNodeProperties dnprop = stopDataNode(i); + return restartDataNode(i, keepPort, false); + } + + /** + * Restart a particular DataNode. + * @param idn index of the DataNode + * @param keepPort true if should restart on the same port + * @param expireOnNN true if NameNode should expire the DataNode heartbeat + * @return + * @throws IOException + */ + public synchronized boolean restartDataNode( + int idn, boolean keepPort, boolean expireOnNN) throws IOException { + DataNodeProperties dnprop = stopDataNode(idn); + if(expireOnNN) { + setDataNodeDead(dnprop.datanode.getDatanodeId()); + } if (dnprop == null) { return false; } else { @@ -2030,6 +2068,24 @@ public synchronized boolean restartDataNode(int i, boolean keepPort) } } + /** + * Expire a DataNode heartbeat on the NameNode + * @param dnId + * @throws IOException + */ + public void setDataNodeDead(DatanodeID dnId) throws IOException { + DatanodeDescriptor dnd = + NameNodeAdapter.getDatanode(getNamesystem(), dnId); + DFSTestUtil.setDatanodeDead(dnd); + BlockManagerTestUtil.checkHeartbeat(getNamesystem().getBlockManager()); + } + + public void setDataNodesDead() throws IOException { + for (DataNodeProperties dnp : dataNodes) { + setDataNodeDead(dnp.datanode.getDatanodeId()); + } + } + /* * Restart all datanodes, on the same ports if keepPort is true */ @@ -2108,8 +2164,10 @@ public DistributedFileSystem getFileSystem() throws IOException { * Get a client handle to the DFS cluster for the namenode at given index. */ public DistributedFileSystem getFileSystem(int nnIndex) throws IOException { - return (DistributedFileSystem)FileSystem.get(getURI(nnIndex), - nameNodes[nnIndex].conf); + DistributedFileSystem dfs = (DistributedFileSystem) FileSystem.get( + getURI(nnIndex), nameNodes[nnIndex].conf); + fileSystems.add(dfs); + return dfs; } /** @@ -2117,7 +2175,9 @@ public DistributedFileSystem getFileSystem(int nnIndex) throws IOException { * This simulating different threads working on different FileSystem instances. */ public FileSystem getNewFileSystemInstance(int nnIndex) throws IOException { - return FileSystem.newInstance(getURI(nnIndex), nameNodes[nnIndex].conf); + FileSystem dfs = FileSystem.newInstance(getURI(nnIndex), nameNodes[nnIndex].conf); + fileSystems.add(dfs); + return dfs; } /** @@ -2255,8 +2315,8 @@ private synchronized boolean shouldWait(DatanodeInfo[] dnInfo, // make sure all datanodes have sent first heartbeat to namenode, // using (capacity == 0) as proxy. for (DatanodeInfo dn : dnInfo) { - if (dn.getCapacity() == 0) { - LOG.info("dn.getCapacity() == 0"); + if (dn.getCapacity() == 0 || dn.getLastUpdate() <= 0) { + LOG.info("No heartbeat from DataNode: " + dn.toString()); return true; } } @@ -2544,8 +2604,8 @@ public static List getAllBlockMetadataFiles(File storageDir) { return null; } for (File f : files) { - if (f.getName().startsWith("blk_") && f.getName().endsWith( - Block.METADATA_EXTENSION)) { + if (f.getName().startsWith(Block.BLOCK_FILE_PREFIX) && + f.getName().endsWith(Block.METADATA_EXTENSION)) { results.add(f); } else if (f.isDirectory()) { List subdirResults = getAllBlockMetadataFiles(f); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAppendSnapshotTruncate.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAppendSnapshotTruncate.java new file mode 100644 index 0000000000000..e80e14f56557c --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAppendSnapshotTruncate.java @@ -0,0 +1,497 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdfs; + +import java.io.File; +import java.io.FileFilter; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; +import java.util.Random; +import java.util.concurrent.Callable; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.server.namenode.NameNode; +import org.apache.hadoop.hdfs.server.namenode.TestFileTruncate; +import org.apache.hadoop.test.GenericTestUtils; +import org.apache.log4j.Level; +import org.junit.AfterClass; +import org.junit.Assert; +import org.junit.BeforeClass; +import org.junit.Test; + +import com.google.common.base.Preconditions; + +/** + * Test randomly mixing append, snapshot and truncate operations. + * Use local file system to simulate the each operation and verify + * the correctness. + */ +public class TestAppendSnapshotTruncate { + static { + GenericTestUtils.setLogLevel(NameNode.stateChangeLog, Level.ALL); + } + private static final Log LOG = LogFactory.getLog(TestAppendSnapshotTruncate.class); + private static final int BLOCK_SIZE = 1024; + private static final int DATANODE_NUM = 3; + private static final short REPLICATION = 3; + private static final int FILE_WORKER_NUM = 3; + private static final long TEST_TIME_SECOND = 10; + private static final long TEST_TIMEOUT_SECOND = TEST_TIME_SECOND + 60; + + static final int SHORT_HEARTBEAT = 1; + static final String[] EMPTY_STRINGS = {}; + + static Configuration conf; + static MiniDFSCluster cluster; + static DistributedFileSystem dfs; + + @BeforeClass + public static void startUp() throws IOException { + conf = new HdfsConfiguration(); + conf.setLong(DFSConfigKeys.DFS_NAMENODE_MIN_BLOCK_SIZE_KEY, BLOCK_SIZE); + conf.setInt(DFSConfigKeys.DFS_BYTES_PER_CHECKSUM_KEY, BLOCK_SIZE); + conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, SHORT_HEARTBEAT); + conf.setLong( + DFSConfigKeys.DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_KEY, 1); + cluster = new MiniDFSCluster.Builder(conf) + .format(true) + .numDataNodes(DATANODE_NUM) + .nameNodePort(NameNode.DEFAULT_PORT) + .waitSafeMode(true) + .build(); + dfs = cluster.getFileSystem(); + } + + @AfterClass + public static void tearDown() throws IOException { + if(dfs != null) { + dfs.close(); + } + if(cluster != null) { + cluster.shutdown(); + } + } + + + /** Test randomly mixing append, snapshot and truncate operations. */ + @Test(timeout=TEST_TIMEOUT_SECOND*1000) + public void testAST() throws Exception { + final String dirPathString = "/dir"; + final Path dir = new Path(dirPathString); + dfs.mkdirs(dir); + dfs.allowSnapshot(dir); + + final File localDir = new File( + System.getProperty("test.build.data", "target/test/data") + + dirPathString); + if (localDir.exists()) { + FileUtil.fullyDelete(localDir); + } + localDir.mkdirs(); + + final DirWorker w = new DirWorker(dir, localDir, FILE_WORKER_NUM); + w.startAllFiles(); + w.start(); + Worker.sleep(TEST_TIME_SECOND * 1000); + w.stop(); + w.stopAllFiles(); + w.checkEverything(); + } + + static final FileFilter FILE_ONLY = new FileFilter() { + @Override + public boolean accept(File f) { + return f.isFile(); + } + }; + + static class DirWorker extends Worker { + final Path dir; + final File localDir; + + final FileWorker[] files; + + private Map snapshotPaths = new HashMap(); + private AtomicInteger snapshotCount = new AtomicInteger(); + + DirWorker(Path dir, File localDir, int nFiles) throws IOException { + super(dir.getName()); + this.dir = dir; + this.localDir = localDir; + + this.files = new FileWorker[nFiles]; + for(int i = 0; i < files.length; i++) { + files[i] = new FileWorker(dir, localDir, String.format("file%02d", i)); + } + } + + static String getSnapshotName(int n) { + return String.format("s%02d", n); + } + + String createSnapshot(String snapshot) throws IOException { + final StringBuilder b = new StringBuilder("createSnapshot: ") + .append(snapshot).append(" for ").append(dir); + + { + //copy all local files to a sub dir to simulate snapshot. + final File subDir = new File(localDir, snapshot); + Assert.assertFalse(subDir.exists()); + subDir.mkdir(); + + for(File f : localDir.listFiles(FILE_ONLY)) { + FileUtils.copyFile(f, new File(subDir, f.getName())); + } + } + + final Path p = dfs.createSnapshot(dir, snapshot); + snapshotPaths.put(snapshot, p); + return b.toString(); + } + + String checkSnapshot(String snapshot) throws IOException { + final StringBuilder b = new StringBuilder("checkSnapshot: ") + .append(snapshot); + + final File subDir = new File(localDir, snapshot); + Assert.assertTrue(subDir.exists()); + + final File[] localFiles = subDir.listFiles(FILE_ONLY); + final Path p = snapshotPaths.get(snapshot); + final FileStatus[] statuses = dfs.listStatus(p); + Assert.assertEquals(localFiles.length, statuses.length); + b.append(p).append(" vs ").append(subDir).append(", ") + .append(statuses.length).append(" entries"); + + Arrays.sort(localFiles); + Arrays.sort(statuses); + for(int i = 0; i < statuses.length; i++) { + FileWorker.checkFullFile(statuses[i].getPath(), localFiles[i]); + } + return b.toString(); + } + + String deleteSnapshot(String snapshot) throws IOException { + final StringBuilder b = new StringBuilder("deleteSnapshot: ") + .append(snapshot).append(" from ").append(dir); + FileUtil.fullyDelete(new File(localDir, snapshot)); + dfs.deleteSnapshot(dir, snapshot); + snapshotPaths.remove(snapshot); + return b.toString(); + } + + + @Override + public String call() throws Exception { + final Random r = DFSUtil.getRandom(); + final int op = r.nextInt(6); + if (op <= 1) { + pauseAllFiles(); + try { + final String snapshot = getSnapshotName(snapshotCount.getAndIncrement()); + return createSnapshot(snapshot); + } finally { + startAllFiles(); + } + } else if (op <= 3) { + final String[] keys = snapshotPaths.keySet().toArray(EMPTY_STRINGS); + if (keys.length == 0) { + return "NO-OP"; + } + final String snapshot = keys[r.nextInt(keys.length)]; + final String s = checkSnapshot(snapshot); + + if (op == 2) { + return deleteSnapshot(snapshot); + } + return s; + } else { + return "NO-OP"; + } + } + + void pauseAllFiles() { + for(FileWorker f : files) { + f.pause(); + } + + for(int i = 0; i < files.length; ) { + sleep(100); + for(; i < files.length && files[i].isPaused(); i++); + } + } + + void startAllFiles() { + for(FileWorker f : files) { + f.start(); + } + } + + void stopAllFiles() throws InterruptedException { + for(FileWorker f : files) { + f.stop(); + } + } + + void checkEverything() throws IOException { + LOG.info("checkEverything"); + for(FileWorker f : files) { + f.checkFullFile(); + f.checkErrorState(); + } + for(String snapshot : snapshotPaths.keySet()) { + checkSnapshot(snapshot); + } + checkErrorState(); + } + } + + static class FileWorker extends Worker { + final Path file; + final File localFile; + + FileWorker(Path dir, File localDir, String filename) throws IOException { + super(filename); + this.file = new Path(dir, filename); + this.localFile = new File(localDir, filename); + + localFile.createNewFile(); + dfs.create(file, false, 4096, REPLICATION, BLOCK_SIZE).close(); + } + + @Override + public String call() throws IOException { + final Random r = DFSUtil.getRandom(); + final int op = r.nextInt(9); + if (op == 0) { + return checkFullFile(); + } else { + final int nBlocks = r.nextInt(4) + 1; + final int lastBlockSize = r.nextInt(BLOCK_SIZE) + 1; + final int nBytes = nBlocks*BLOCK_SIZE + lastBlockSize; + + if (op <= 4) { + return append(nBytes); + } else if (op <= 6) { + return truncateArbitrarily(nBytes); + } else { + return truncateToBlockBoundary(nBlocks); + } + } + } + + String append(int n) throws IOException { + final StringBuilder b = new StringBuilder("append ") + .append(n).append(" bytes to ").append(file.getName()); + + final byte[] bytes = new byte[n]; + DFSUtil.getRandom().nextBytes(bytes); + + { // write to local file + final FileOutputStream out = new FileOutputStream(localFile, true); + out.write(bytes, 0, bytes.length); + out.close(); + } + + { + final FSDataOutputStream out = dfs.append(file); + out.write(bytes, 0, bytes.length); + out.close(); + } + return b.toString(); + } + + String truncateArbitrarily(int nBytes) throws IOException { + Preconditions.checkArgument(nBytes > 0); + final int length = checkLength(); + final StringBuilder b = new StringBuilder("truncateArbitrarily: ") + .append(nBytes).append(" bytes from ").append(file.getName()) + .append(", length=" + length); + + truncate(length > nBytes? length - nBytes: 0, b); + return b.toString(); + } + + String truncateToBlockBoundary(int nBlocks) throws IOException { + Preconditions.checkArgument(nBlocks > 0); + final int length = checkLength(); + final StringBuilder b = new StringBuilder("truncateToBlockBoundary: ") + .append(nBlocks).append(" blocks from ").append(file.getName()) + .append(", length=" + length); + final int n = (nBlocks - 1)*BLOCK_SIZE + (length%BLOCK_SIZE); + Preconditions.checkState(truncate(length > n? length - n: 0, b), b); + return b.toString(); + } + + private boolean truncate(long newLength, StringBuilder b) throws IOException { + final RandomAccessFile raf = new RandomAccessFile(localFile, "rw"); + raf.setLength(newLength); + raf.close(); + + final boolean isReady = dfs.truncate(file, newLength); + b.append(", newLength=").append(newLength) + .append(", isReady=").append(isReady); + if (!isReady) { + TestFileTruncate.checkBlockRecovery(file, dfs, 100, 300L); + } + return isReady; + } + + int checkLength() throws IOException { + return checkLength(file, localFile); + } + + static int checkLength(Path file, File localFile) throws IOException { + final long length = dfs.getFileStatus(file).getLen(); + Assert.assertEquals(localFile.length(), length); + Assert.assertTrue(length <= Integer.MAX_VALUE); + return (int)length; + } + + String checkFullFile() throws IOException { + return checkFullFile(file, localFile); + } + + static String checkFullFile(Path file, File localFile) throws IOException { + final StringBuilder b = new StringBuilder("checkFullFile: ") + .append(file.getName()).append(" vs ").append(localFile); + final byte[] bytes = new byte[checkLength(file, localFile)]; + b.append(", length=").append(bytes.length); + + final FileInputStream in = new FileInputStream(localFile); + for(int n = 0; n < bytes.length; ) { + n += in.read(bytes, n, bytes.length - n); + } + in.close(); + + AppendTestUtil.checkFullFile(dfs, file, bytes.length, bytes, + "File content mismatch: " + b, false); + return b.toString(); + } + } + + static abstract class Worker implements Callable { + enum State { + IDLE(false), RUNNING(false), STOPPED(true), ERROR(true); + + final boolean isTerminated; + + State(boolean isTerminated) { + this.isTerminated = isTerminated; + } + }; + + final String name; + final AtomicReference state = new AtomicReference(State.IDLE); + final AtomicBoolean isCalling = new AtomicBoolean(); + final AtomicReference thread = new AtomicReference(); + + private Throwable thrown = null; + + Worker(String name) { + this.name = name; + } + + State checkErrorState() { + final State s = state.get(); + if (s == State.ERROR) { + throw new IllegalStateException(name + " has " + s, thrown); + } + return s; + } + + void setErrorState(Throwable t) { + checkErrorState(); + + LOG.error("Worker " + name + " failed.", t); + state.set(State.ERROR); + thrown = t; + } + + void start() { + Preconditions.checkState(state.compareAndSet(State.IDLE, State.RUNNING)); + + if (thread.get() == null) { + final Thread t = new Thread(null, new Runnable() { + @Override + public void run() { + final Random r = DFSUtil.getRandom(); + for(State s; !(s = checkErrorState()).isTerminated;) { + if (s == State.RUNNING) { + isCalling.set(true); + try { + LOG.info(call()); + } catch(Throwable t) { + setErrorState(t); + return; + } + isCalling.set(false); + } + sleep(r.nextInt(100) + 50); + } + } + }, name); + Preconditions.checkState(thread.compareAndSet(null, t)); + t.start(); + } + } + + boolean isPaused() { + final State s = checkErrorState(); + if (s == State.STOPPED) { + throw new IllegalStateException(name + " is " + s); + } + return s == State.IDLE && !isCalling.get(); + } + + void pause() { + Preconditions.checkState(state.compareAndSet(State.RUNNING, State.IDLE)); + } + + void stop() throws InterruptedException { + checkErrorState(); + + state.set(State.STOPPED); + thread.get().join(); + } + + static void sleep(final long sleepTimeMs) { + try { + Thread.sleep(sleepTimeMs); + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + } + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java index cb50539462080..1c4134f1f0aee 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java @@ -30,11 +30,16 @@ import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo; +import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; +import org.apache.hadoop.hdfs.protocol.LocatedBlock; +import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.net.unix.DomainSocket; import org.apache.hadoop.net.unix.TemporarySocketDirectory; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; import org.junit.Assert; import org.junit.Assume; import org.junit.BeforeClass; @@ -153,4 +158,62 @@ public void testBothOldAndNewShortCircuitConfigured() throws Exception { Arrays.equals(orig, buf); cluster.shutdown(); } + + @Test(timeout=20000) + public void testBlockReaderLocalLegacyWithAppend() throws Exception { + final short REPL_FACTOR = 1; + final HdfsConfiguration conf = getConfiguration(null); + conf.setBoolean(DFSConfigKeys.DFS_CLIENT_USE_LEGACY_BLOCKREADERLOCAL, true); + + final MiniDFSCluster cluster = + new MiniDFSCluster.Builder(conf).numDataNodes(1).build(); + cluster.waitActive(); + + final DistributedFileSystem dfs = cluster.getFileSystem(); + final Path path = new Path("/testBlockReaderLocalLegacy"); + DFSTestUtil.createFile(dfs, path, 10, REPL_FACTOR, 0); + DFSTestUtil.waitReplication(dfs, path, REPL_FACTOR); + + final ClientDatanodeProtocol proxy; + final Token token; + final ExtendedBlock originalBlock; + final long originalGS; + { + final LocatedBlock lb = cluster.getNameNode().getRpcServer() + .getBlockLocations(path.toString(), 0, 1).get(0); + proxy = DFSUtil.createClientDatanodeProtocolProxy( + lb.getLocations()[0], conf, 60000, false); + token = lb.getBlockToken(); + + // get block and generation stamp + final ExtendedBlock blk = new ExtendedBlock(lb.getBlock()); + originalBlock = new ExtendedBlock(blk); + originalGS = originalBlock.getGenerationStamp(); + + // test getBlockLocalPathInfo + final BlockLocalPathInfo info = proxy.getBlockLocalPathInfo(blk, token); + Assert.assertEquals(originalGS, info.getBlock().getGenerationStamp()); + } + + { // append one byte + FSDataOutputStream out = dfs.append(path); + out.write(1); + out.close(); + } + + { + // get new generation stamp + final LocatedBlock lb = cluster.getNameNode().getRpcServer() + .getBlockLocations(path.toString(), 0, 1).get(0); + final long newGS = lb.getBlock().getGenerationStamp(); + Assert.assertTrue(newGS > originalGS); + + // getBlockLocalPathInfo using the original block. + Assert.assertEquals(originalGS, originalBlock.getGenerationStamp()); + final BlockLocalPathInfo info = proxy.getBlockLocalPathInfo( + originalBlock, token); + Assert.assertEquals(newGS, info.getBlock().getGenerationStamp()); + } + cluster.shutdown(); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java index 969cdd5223bb3..f0c4c4228288e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java @@ -35,6 +35,7 @@ import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DFSClientFaultInjector; +import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.io.IOUtils; import org.junit.Before; @@ -176,7 +177,7 @@ private void thistest(Configuration conf, DFSTestUtil util) throws Exception { assertTrue("Blocks do not exist in data-dir", (blocks != null) && (blocks.length > 0)); int num = 0; for (int idx = 0; idx < blocks.length; idx++) { - if (blocks[idx].getName().startsWith("blk_") && + if (blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX) && blocks[idx].getName().endsWith(".meta")) { num++; if (num % 3 == 0) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java index 9c9111c7a5b74..391277417a44c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java @@ -489,9 +489,8 @@ private LocatedBlocks makeBadBlockList(LocatedBlocks goodBlockList) { goodLocatedBlock.getBlock(), new DatanodeInfo[] { DFSTestUtil.getDatanodeInfo("1.2.3.4", "bogus", 1234) - }, - goodLocatedBlock.getStartOffset(), - false); + }); + badLocatedBlock.setStartOffset(goodLocatedBlock.getStartOffset()); List badBlocks = new ArrayList(); @@ -1131,4 +1130,26 @@ static void parseMultipleLinearRandomRetry(String expected, String s) { assertEquals("MultipleLinearRandomRetry" + expected, r.toString()); } } + + @Test + public void testDFSClientConfigurationLocateFollowingBlockInitialDelay() + throws Exception { + // test if DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_KEY + // is not configured, verify DFSClient uses the default value 400. + Configuration dfsConf = new HdfsConfiguration(); + MiniDFSCluster cluster = new MiniDFSCluster.Builder(dfsConf).build(); + cluster.waitActive(); + NamenodeProtocols nn = cluster.getNameNodeRpc(); + DFSClient client = new DFSClient(null, nn, dfsConf, null); + assertEquals(client.getConf(). + getBlockWriteLocateFollowingInitialDelayMs(), 400); + + // change DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_KEY, + // verify DFSClient uses the configured value 1000. + dfsConf.setInt(DFSConfigKeys. + DFS_CLIENT_BLOCK_WRITE_LOCATEFOLLOWINGBLOCK_INITIAL_DELAY_KEY, 1000); + client = new DFSClient(null, nn, dfsConf, null); + assertEquals(client.getConf(). + getBlockWriteLocateFollowingInitialDelayMs(), 1000); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java index 7269e3910d218..b47e7f1510549 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java @@ -51,8 +51,11 @@ public void testCloseTwice() throws IOException { DFSOutputStream dos = (DFSOutputStream) Whitebox.getInternalState(os, "wrappedStream"); @SuppressWarnings("unchecked") + DataStreamer streamer = (DataStreamer) Whitebox + .getInternalState(dos, "streamer"); + @SuppressWarnings("unchecked") AtomicReference ex = (AtomicReference) Whitebox - .getInternalState(dos, "lastException"); + .getInternalState(streamer, "lastException"); Assert.assertEquals(null, ex.get()); dos.close(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java new file mode 100755 index 0000000000000..daee6083ebbfe --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSPacket.java @@ -0,0 +1,93 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs; + +import java.util.Random; +import org.apache.hadoop.hdfs.protocol.datatransfer.PacketHeader; +import org.apache.hadoop.io.DataOutputBuffer; +import org.junit.Assert; +import org.junit.Test; + +public class TestDFSPacket { + private static final int chunkSize = 512; + private static final int checksumSize = 4; + private static final int maxChunksPerPacket = 4; + + @Test + public void testPacket() throws Exception { + Random r = new Random(12345L); + byte[] data = new byte[chunkSize]; + r.nextBytes(data); + byte[] checksum = new byte[checksumSize]; + r.nextBytes(checksum); + + DataOutputBuffer os = new DataOutputBuffer(data.length * 2); + + byte[] packetBuf = new byte[data.length * 2]; + DFSPacket p = new DFSPacket(packetBuf, maxChunksPerPacket, + 0, 0, checksumSize, false); + p.setSyncBlock(true); + p.writeData(data, 0, data.length); + p.writeChecksum(checksum, 0, checksum.length); + p.writeTo(os); + + //we have set syncBlock to true, so the header has the maximum length + int headerLen = PacketHeader.PKT_MAX_HEADER_LEN; + byte[] readBuf = os.getData(); + + assertArrayRegionsEqual(readBuf, headerLen, checksum, 0, checksum.length); + assertArrayRegionsEqual(readBuf, headerLen + checksum.length, data, 0, data.length); + + } + + public static void assertArrayRegionsEqual(byte []buf1, int off1, byte []buf2, + int off2, int len) { + for (int i = 0; i < len; i++) { + if (buf1[off1 + i] != buf2[off2 + i]) { + Assert.fail("arrays differ at byte " + i + ". " + + "The first array has " + (int) buf1[off1 + i] + + ", but the second array has " + (int) buf2[off2 + i]); + } + } + } + + @Test + public void testAddParentsGetParents() throws Exception { + DFSPacket p = new DFSPacket(null, maxChunksPerPacket, + 0, 0, checksumSize, false); + long parents[] = p.getTraceParents(); + Assert.assertEquals(0, parents.length); + p.addTraceParent(123); + p.addTraceParent(123); + parents = p.getTraceParents(); + Assert.assertEquals(1, parents.length); + Assert.assertEquals(123, parents[0]); + parents = p.getTraceParents(); // test calling 'get' again. + Assert.assertEquals(1, parents.length); + Assert.assertEquals(123, parents[0]); + p.addTraceParent(1); + p.addTraceParent(456); + p.addTraceParent(789); + parents = p.getTraceParents(); + Assert.assertEquals(4, parents.length); + Assert.assertEquals(1, parents[0]); + Assert.assertEquals(123, parents[1]); + Assert.assertEquals(456, parents[2]); + Assert.assertEquals(789, parents[3]); + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java index bb00144ca47e5..f0a094ed12cc7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java @@ -28,7 +28,10 @@ import static org.junit.Assert.fail; import java.io.File; +import java.io.FilenameFilter; import java.io.IOException; +import java.util.LinkedList; +import java.util.List; import java.util.regex.Pattern; import org.apache.commons.logging.Log; @@ -42,7 +45,9 @@ import org.apache.hadoop.hdfs.server.common.InconsistentFSStateException; import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.common.StorageInfo; +import org.apache.hadoop.hdfs.server.namenode.NNStorage; import org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite; +import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.util.StringUtils; import org.junit.BeforeClass; @@ -450,7 +455,50 @@ public void test203LayoutVersion() { assertTrue(Storage.is203LayoutVersion(lv)); } } - + + @Test + public void testPreserveEditLogs() throws Exception { + conf = new HdfsConfiguration(); + conf = UpgradeUtilities.initializeStorageStateConf(1, conf); + String[] nameNodeDirs = conf.getStrings(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY); + conf.setBoolean(DFSConfigKeys.DFS_DATANODE_DUPLICATE_REPLICA_DELETION, false); + + log("Normal NameNode upgrade", 1); + File[] created = + UpgradeUtilities.createNameNodeStorageDirs(nameNodeDirs, "current"); + List beforeUpgrade = new LinkedList<>(); + for (final File createdDir : created) { + List fileNameList = + IOUtils.listDirectory(createdDir, EditLogsFilter.INSTANCE); + beforeUpgrade.addAll(fileNameList); + } + + cluster = createCluster(); + + List afterUpgrade = new LinkedList<>(); + for (final File createdDir : created) { + List fileNameList = + IOUtils.listDirectory(createdDir, EditLogsFilter.INSTANCE); + afterUpgrade.addAll(fileNameList); + } + + for (String s : beforeUpgrade) { + assertTrue(afterUpgrade.contains(s)); + } + + cluster.shutdown(); + UpgradeUtilities.createEmptyDirs(nameNodeDirs); + } + + private static enum EditLogsFilter implements FilenameFilter { + INSTANCE; + + @Override + public boolean accept(File dir, String name) { + return name.startsWith(NNStorage.NameNodeFile.EDITS.getName()); + } + } + public static void main(String[] args) throws Exception { TestDFSUpgrade t = new TestDFSUpgrade(); TestDFSUpgrade.initialize(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java index 046265f5520f8..7de121f4193a1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java @@ -99,11 +99,15 @@ public void testLocatedBlocks2Locations() { // ok ExtendedBlock b1 = new ExtendedBlock("bpid", 1, 1, 1); - LocatedBlock l1 = new LocatedBlock(b1, ds, 0, false); + LocatedBlock l1 = new LocatedBlock(b1, ds); + l1.setStartOffset(0); + l1.setCorrupt(false); // corrupt ExtendedBlock b2 = new ExtendedBlock("bpid", 2, 1, 1); - LocatedBlock l2 = new LocatedBlock(b2, ds, 0, true); + LocatedBlock l2 = new LocatedBlock(b2, ds); + l2.setStartOffset(0); + l2.setCorrupt(true); List ls = Arrays.asList(l1, l2); LocatedBlocks lbs = new LocatedBlocks(10, false, ls, l2, true, null); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java index 040a91cd523b2..bf011f79c129a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java @@ -51,6 +51,7 @@ import org.apache.hadoop.hdfs.protocol.datatransfer.PacketHeader; import org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck; import org.apache.hadoop.hdfs.protocol.datatransfer.Sender; +import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos; import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.BlockOpResponseProto; import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.BlockOpResponseProto.Builder; import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.ReadOpChecksumInfoProto; @@ -518,6 +519,35 @@ public void testPacketHeader() throws IOException { assertFalse(hdr.sanityCheck(100)); } + @Test + public void TestPipeLineAckCompatibility() throws IOException { + DataTransferProtos.PipelineAckProto proto = DataTransferProtos + .PipelineAckProto.newBuilder() + .setSeqno(0) + .addReply(Status.CHECKSUM_OK) + .build(); + + DataTransferProtos.PipelineAckProto newProto = DataTransferProtos + .PipelineAckProto.newBuilder().mergeFrom(proto) + .addFlag(PipelineAck.combineHeader(PipelineAck.ECN.SUPPORTED, + Status.CHECKSUM_OK)) + .build(); + + ByteArrayOutputStream oldAckBytes = new ByteArrayOutputStream(); + proto.writeDelimitedTo(oldAckBytes); + PipelineAck oldAck = new PipelineAck(); + oldAck.readFields(new ByteArrayInputStream(oldAckBytes.toByteArray())); + assertEquals(PipelineAck.combineHeader(PipelineAck.ECN.DISABLED, Status + .CHECKSUM_OK), oldAck.getHeaderFlag(0)); + + PipelineAck newAck = new PipelineAck(); + ByteArrayOutputStream newAckBytes = new ByteArrayOutputStream(); + newProto.writeDelimitedTo(newAckBytes); + newAck.readFields(new ByteArrayInputStream(newAckBytes.toByteArray())); + assertEquals(PipelineAck.combineHeader(PipelineAck.ECN.SUPPORTED, Status + .CHECKSUM_OK), newAck.getHeaderFlag(0)); + } + void writeBlock(String poolId, long blockId, DataChecksum checksum) throws IOException { writeBlock(new ExtendedBlock(poolId, blockId), BlockConstructionStage.PIPELINE_SETUP_CREATE, 0L, checksum); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java index 35c0d8c70100d..081e40f6278c7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdfs; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -26,39 +27,56 @@ import java.util.Arrays; import java.util.Collection; import java.util.Iterator; +import java.util.List; import java.util.Random; +import java.util.concurrent.ExecutionException; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; +import com.google.common.base.Supplier; +import com.google.common.collect.Lists; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.BlockLocation; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.client.HdfsDataInputStream; +import org.apache.hadoop.hdfs.client.HdfsDataOutputStream; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.DatanodeInfo.AdminStates; +import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlocks; +import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous; +import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager; +import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil; +import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor; import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager; +import org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager; import org.apache.hadoop.hdfs.server.datanode.DataNode; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; import org.apache.hadoop.hdfs.server.namenode.ha.HATestUtil; import org.apache.hadoop.hdfs.server.namenode.NameNode; import org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter; +import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.test.PathUtils; +import org.apache.log4j.Level; import org.junit.After; import org.junit.Assert; import org.junit.Before; +import org.junit.Ignore; import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * This class tests the decommissioning of nodes. */ public class TestDecommission { - public static final Log LOG = LogFactory.getLog(TestDecommission.class); + public static final Logger LOG = LoggerFactory.getLogger(TestDecommission + .class); static final long seed = 0xDEADBEEFL; static final int blockSize = 8192; static final int fileSize = 16384; @@ -90,6 +108,7 @@ public void setup() throws IOException { conf.set(DFSConfigKeys.DFS_HOSTS_EXCLUDE, excludeFile.toUri().getPath()); conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 2000); conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, HEARTBEAT_INTERVAL); + conf.setInt(DFSConfigKeys.DFS_NAMENODE_REPLICATION_INTERVAL_KEY, 1); conf.setInt(DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_KEY, BLOCKREPORT_INTERVAL_MSEC); conf.setInt(DFSConfigKeys.DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_KEY, 4); conf.setInt(DFSConfigKeys.DFS_NAMENODE_REPLICATION_INTERVAL_KEY, NAMENODE_REPLICATION_INTERVAL); @@ -106,7 +125,7 @@ public void teardown() throws IOException { } } - private void writeConfigFile(Path name, ArrayList nodes) + private void writeConfigFile(Path name, List nodes) throws IOException { // delete if it already exists if (localFileSys.exists(name)) { @@ -150,7 +169,7 @@ private void writeFile(FileSystem fileSys, Path name, int repl) * @param downnode - if null, there is no decommissioned node for this file. * @return - null if no failure found, else an error message string. */ - private String checkFile(FileSystem fileSys, Path name, int repl, + private static String checkFile(FileSystem fileSys, Path name, int repl, String downnode, int numDatanodes) throws IOException { boolean isNodeDown = (downnode != null); // need a raw stream @@ -262,7 +281,7 @@ private DatanodeInfo decommissionNode(int nnIndex, /* Ask a specific NN to stop decommission of the datanode and wait for each * to reach the NORMAL state. */ - private void recomissionNode(int nnIndex, DatanodeInfo decommissionedNode) throws IOException { + private void recommissionNode(int nnIndex, DatanodeInfo decommissionedNode) throws IOException { LOG.info("Recommissioning node: " + decommissionedNode); writeConfigFile(excludeFile, null); refreshNodes(cluster.getNamesystem(nnIndex), conf); @@ -280,7 +299,7 @@ private void waitNodeState(DatanodeInfo node, LOG.info("Waiting for node " + node + " to change state to " + state + " current state: " + node.getAdminState()); try { - Thread.sleep(HEARTBEAT_INTERVAL * 1000); + Thread.sleep(HEARTBEAT_INTERVAL * 500); } catch (InterruptedException e) { // nothing } @@ -322,28 +341,27 @@ static void refreshNodes(final FSNamesystem ns, final Configuration conf } private void verifyStats(NameNode namenode, FSNamesystem fsn, - DatanodeInfo node, boolean decommissioning) + DatanodeInfo info, DataNode node, boolean decommissioning) throws InterruptedException, IOException { - // Do the stats check over 10 iterations + // Do the stats check over 10 heartbeats for (int i = 0; i < 10; i++) { long[] newStats = namenode.getRpcServer().getStats(); // For decommissioning nodes, ensure capacity of the DN is no longer // counted. Only used space of the DN is counted in cluster capacity - assertEquals(newStats[0], decommissioning ? node.getDfsUsed() : - node.getCapacity()); + assertEquals(newStats[0], + decommissioning ? info.getDfsUsed() : info.getCapacity()); // Ensure cluster used capacity is counted for both normal and // decommissioning nodes - assertEquals(newStats[1], node.getDfsUsed()); + assertEquals(newStats[1], info.getDfsUsed()); // For decommissioning nodes, remaining space from the DN is not counted - assertEquals(newStats[2], decommissioning ? 0 : node.getRemaining()); + assertEquals(newStats[2], decommissioning ? 0 : info.getRemaining()); // Ensure transceiver count is same as that DN - assertEquals(fsn.getTotalLoad(), node.getXceiverCount()); - - Thread.sleep(HEARTBEAT_INTERVAL * 1000); // Sleep heart beat interval + assertEquals(fsn.getTotalLoad(), info.getXceiverCount()); + DataNodeTestUtils.triggerHeartbeat(node); } } @@ -407,14 +425,6 @@ public void testDecommission2() throws IOException { cluster.shutdown(); } - /** - * Tests recommission for non federated cluster - */ - @Test(timeout=360000) - public void testRecommission() throws IOException { - testRecommission(1, 6); - } - /** * Test decommission for federeated cluster */ @@ -501,12 +511,12 @@ public void testDecommissionOnStandby() throws Exception { // 1. the last DN would have been chosen as excess replica, given its // heartbeat is considered old. // Please refer to BlockPlacementPolicyDefault#chooseReplicaToDelete - // 2. After recomissionNode finishes, SBN has 3 live replicas ( 0, 1, 2 ) + // 2. After recommissionNode finishes, SBN has 3 live replicas ( 0, 1, 2 ) // and one excess replica ( 3 ) // After the fix, - // After recomissionNode finishes, SBN has 4 live replicas ( 0, 1, 2, 3 ) + // After recommissionNode finishes, SBN has 4 live replicas ( 0, 1, 2, 3 ) Thread.sleep(slowHeartbeatDNwaitTime); - recomissionNode(1, decomNodeFromSBN); + recommissionNode(1, decomNodeFromSBN); // Step 3.b, ask ANN to recommission the first DN. // To verify the fix, the test makes sure the excess replica picked by ANN @@ -525,7 +535,7 @@ public void testDecommissionOnStandby() throws Exception { cluster.restartDataNode(nextToLastDNprop); cluster.waitActive(); Thread.sleep(slowHeartbeatDNwaitTime); - recomissionNode(0, decommissionedNodeFromANN); + recommissionNode(0, decommissionedNodeFromANN); // Step 3.c, make sure the DN has deleted the block and report to NNs cluster.triggerHeartbeats(); @@ -607,69 +617,88 @@ private void testDecommission(int numNamenodes, int numDatanodes) cluster.shutdown(); } + /** + * Test that over-replicated blocks are deleted on recommission. + */ + @Test(timeout=120000) + public void testRecommission() throws Exception { + final int numDatanodes = 6; + try { + LOG.info("Starting test testRecommission"); - private void testRecommission(int numNamenodes, int numDatanodes) - throws IOException { - LOG.info("Starting test testRecommission"); + startCluster(1, numDatanodes, conf); - startCluster(numNamenodes, numDatanodes, conf); - - ArrayList> namenodeDecomList = - new ArrayList>(numNamenodes); - for(int i = 0; i < numNamenodes; i++) { - namenodeDecomList.add(i, new ArrayList(numDatanodes)); - } - Path file1 = new Path("testDecommission.dat"); - int replicas = numDatanodes - 1; - - for (int i = 0; i < numNamenodes; i++) { - ArrayList decommissionedNodes = namenodeDecomList.get(i); - FileSystem fileSys = cluster.getFileSystem(i); + final Path file1 = new Path("testDecommission.dat"); + final int replicas = numDatanodes - 1; + + ArrayList decommissionedNodes = Lists.newArrayList(); + final FileSystem fileSys = cluster.getFileSystem(); + + // Write a file to n-1 datanodes writeFile(fileSys, file1, replicas); - - // Decommission one node. Verify that node is decommissioned. - DatanodeInfo decomNode = decommissionNode(i, null, decommissionedNodes, - AdminStates.DECOMMISSIONED); + + // Decommission one of the datanodes with a replica + BlockLocation loc = fileSys.getFileBlockLocations(file1, 0, 1)[0]; + assertEquals("Unexpected number of replicas from getFileBlockLocations", + replicas, loc.getHosts().length); + final String toDecomHost = loc.getNames()[0]; + String toDecomUuid = null; + for (DataNode d : cluster.getDataNodes()) { + if (d.getDatanodeId().getXferAddr().equals(toDecomHost)) { + toDecomUuid = d.getDatanodeId().getDatanodeUuid(); + break; + } + } + assertNotNull("Could not find a dn with the block!", toDecomUuid); + final DatanodeInfo decomNode = + decommissionNode(0, toDecomUuid, decommissionedNodes, + AdminStates.DECOMMISSIONED); decommissionedNodes.add(decomNode); - + final BlockManager blockManager = + cluster.getNamesystem().getBlockManager(); + final DatanodeManager datanodeManager = + blockManager.getDatanodeManager(); + BlockManagerTestUtil.recheckDecommissionState(datanodeManager); + // Ensure decommissioned datanode is not automatically shutdown - DFSClient client = getDfsClient(cluster.getNameNode(i), conf); - assertEquals("All datanodes must be alive", numDatanodes, + DFSClient client = getDfsClient(cluster.getNameNode(), conf); + assertEquals("All datanodes must be alive", numDatanodes, client.datanodeReport(DatanodeReportType.LIVE).length); - int tries =0; + // wait for the block to be replicated - while (tries++ < 20) { - try { - Thread.sleep(1000); - if (checkFile(fileSys, file1, replicas, decomNode.getXferAddr(), - numDatanodes) == null) { - break; - } - } catch (InterruptedException ie) { - } - } - assertTrue("Checked if block was replicated after decommission, tried " - + tries + " times.", tries < 20); - - // stop decommission and check if the new replicas are removed - recomissionNode(0, decomNode); - // wait for the block to be deleted - tries = 0; - while (tries++ < 20) { - try { - Thread.sleep(1000); - if (checkFile(fileSys, file1, replicas, null, numDatanodes) == null) { - break; + final ExtendedBlock b = DFSTestUtil.getFirstBlock(fileSys, file1); + final String uuid = toDecomUuid; + GenericTestUtils.waitFor(new Supplier() { + @Override + public Boolean get() { + BlockInfoContiguous info = + blockManager.getStoredBlock(b.getLocalBlock()); + int count = 0; + StringBuilder sb = new StringBuilder("Replica locations: "); + for (int i = 0; i < info.numNodes(); i++) { + DatanodeDescriptor dn = info.getDatanode(i); + sb.append(dn + ", "); + if (!dn.getDatanodeUuid().equals(uuid)) { + count++; + } } - } catch (InterruptedException ie) { + LOG.info(sb.toString()); + LOG.info("Count: " + count); + return count == replicas; } - } + }, 500, 30000); + + // redecommission and wait for over-replication to be fixed + recommissionNode(0, decomNode); + BlockManagerTestUtil.recheckDecommissionState(datanodeManager); + DFSTestUtil.waitForReplication(cluster, b, 1, replicas, 0); + cleanupFile(fileSys, file1); - assertTrue("Checked if node was recommissioned " + tries + " times.", - tries < 20); - LOG.info("tried: " + tries + " times before recommissioned"); + } finally { + if (cluster != null) { + cluster.shutdown(); + } } - cluster.shutdown(); } /** @@ -703,20 +732,35 @@ public void testClusterStats(int numNameNodes) throws IOException, FSNamesystem fsn = cluster.getNamesystem(i); NameNode namenode = cluster.getNameNode(i); - DatanodeInfo downnode = decommissionNode(i, null, null, + + DatanodeInfo decomInfo = decommissionNode(i, null, null, AdminStates.DECOMMISSION_INPROGRESS); + DataNode decomNode = getDataNode(decomInfo); // Check namenode stats for multiple datanode heartbeats - verifyStats(namenode, fsn, downnode, true); + verifyStats(namenode, fsn, decomInfo, decomNode, true); // Stop decommissioning and verify stats writeConfigFile(excludeFile, null); refreshNodes(fsn, conf); - DatanodeInfo ret = NameNodeAdapter.getDatanode(fsn, downnode); - waitNodeState(ret, AdminStates.NORMAL); - verifyStats(namenode, fsn, ret, false); + DatanodeInfo retInfo = NameNodeAdapter.getDatanode(fsn, decomInfo); + DataNode retNode = getDataNode(decomInfo); + waitNodeState(retInfo, AdminStates.NORMAL); + verifyStats(namenode, fsn, retInfo, retNode, false); } } - + + private DataNode getDataNode(DatanodeInfo decomInfo) { + DataNode decomNode = null; + for (DataNode dn: cluster.getDataNodes()) { + if (decomInfo.equals(dn.getDatanodeId())) { + decomNode = dn; + break; + } + } + assertNotNull("Could not find decomNode in cluster!", decomNode); + return decomNode; + } + /** * Test host/include file functionality. Only datanodes * in the include file are allowed to connect to the namenode in a non @@ -902,9 +946,9 @@ public void testDecommissionWithNamenodeRestart()throws IOException, Interrupted * It is not recommended to use a registration name which is not also a * valid DNS hostname for the DataNode. See HDFS-5237 for background. */ + @Ignore @Test(timeout=360000) - public void testIncludeByRegistrationName() throws IOException, - InterruptedException { + public void testIncludeByRegistrationName() throws Exception { Configuration hdfsConf = new Configuration(conf); // Any IPv4 address starting with 127 functions as a "loopback" address // which is connected to the current host. So by choosing 127.0.0.100 @@ -927,15 +971,22 @@ public void testIncludeByRegistrationName() throws IOException, refreshNodes(cluster.getNamesystem(0), hdfsConf); // Wait for the DN to be marked dead. - DFSClient client = getDfsClient(cluster.getNameNode(0), hdfsConf); - while (true) { - DatanodeInfo info[] = client.datanodeReport(DatanodeReportType.DEAD); - if (info.length == 1) { - break; + LOG.info("Waiting for DN to be marked as dead."); + final DFSClient client = getDfsClient(cluster.getNameNode(0), hdfsConf); + GenericTestUtils.waitFor(new Supplier() { + @Override + public Boolean get() { + BlockManagerTestUtil + .checkHeartbeat(cluster.getNamesystem().getBlockManager()); + try { + DatanodeInfo info[] = client.datanodeReport(DatanodeReportType.DEAD); + return info.length == 1; + } catch (IOException e) { + LOG.warn("Failed to check dead DNs", e); + return false; + } } - LOG.info("Waiting for datanode to be marked dead"); - Thread.sleep(HEARTBEAT_INTERVAL * 1000); - } + }, 500, 5000); // Use a non-empty include file with our registration name. // It should work. @@ -945,18 +996,169 @@ public void testIncludeByRegistrationName() throws IOException, writeConfigFile(hostsFile, nodes); refreshNodes(cluster.getNamesystem(0), hdfsConf); cluster.restartDataNode(0); + cluster.triggerHeartbeats(); // Wait for the DN to come back. - while (true) { - DatanodeInfo info[] = client.datanodeReport(DatanodeReportType.LIVE); - if (info.length == 1) { - Assert.assertFalse(info[0].isDecommissioned()); - Assert.assertFalse(info[0].isDecommissionInProgress()); - assertEquals(registrationName, info[0].getHostName()); - break; + LOG.info("Waiting for DN to come back."); + GenericTestUtils.waitFor(new Supplier() { + @Override + public Boolean get() { + BlockManagerTestUtil + .checkHeartbeat(cluster.getNamesystem().getBlockManager()); + try { + DatanodeInfo info[] = client.datanodeReport(DatanodeReportType.LIVE); + if (info.length == 1) { + Assert.assertFalse(info[0].isDecommissioned()); + Assert.assertFalse(info[0].isDecommissionInProgress()); + assertEquals(registrationName, info[0].getHostName()); + return true; + } + } catch (IOException e) { + LOG.warn("Failed to check dead DNs", e); + } + return false; } - LOG.info("Waiting for datanode to come back"); - Thread.sleep(HEARTBEAT_INTERVAL * 1000); + }, 500, 5000); + } + + @Test(timeout=120000) + public void testBlocksPerInterval() throws Exception { + Configuration newConf = new Configuration(conf); + org.apache.log4j.Logger.getLogger(DecommissionManager.class) + .setLevel(Level.TRACE); + // Turn the blocks per interval way down + newConf.setInt( + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_BLOCKS_PER_INTERVAL_KEY, + 3); + // Disable the normal monitor runs + newConf.setInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, + Integer.MAX_VALUE); + startCluster(1, 3, newConf); + final FileSystem fs = cluster.getFileSystem(); + final DatanodeManager datanodeManager = + cluster.getNamesystem().getBlockManager().getDatanodeManager(); + final DecommissionManager decomManager = datanodeManager.getDecomManager(); + + // Write a 3 block file, so each node has one block. Should scan 3 nodes. + DFSTestUtil.createFile(fs, new Path("/file1"), 64, (short) 3, 0xBAD1DEA); + doDecomCheck(datanodeManager, decomManager, 3); + // Write another file, should only scan two + DFSTestUtil.createFile(fs, new Path("/file2"), 64, (short)3, 0xBAD1DEA); + doDecomCheck(datanodeManager, decomManager, 2); + // One more file, should only scan 1 + DFSTestUtil.createFile(fs, new Path("/file3"), 64, (short)3, 0xBAD1DEA); + doDecomCheck(datanodeManager, decomManager, 1); + // blocks on each DN now exceeds limit, still scan at least one node + DFSTestUtil.createFile(fs, new Path("/file4"), 64, (short)3, 0xBAD1DEA); + doDecomCheck(datanodeManager, decomManager, 1); + } + + @Deprecated + @Test(timeout=120000) + public void testNodesPerInterval() throws Exception { + Configuration newConf = new Configuration(conf); + org.apache.log4j.Logger.getLogger(DecommissionManager.class) + .setLevel(Level.TRACE); + // Set the deprecated configuration key which limits the # of nodes per + // interval + newConf.setInt("dfs.namenode.decommission.nodes.per.interval", 1); + // Disable the normal monitor runs + newConf.setInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, + Integer.MAX_VALUE); + startCluster(1, 3, newConf); + final FileSystem fs = cluster.getFileSystem(); + final DatanodeManager datanodeManager = + cluster.getNamesystem().getBlockManager().getDatanodeManager(); + final DecommissionManager decomManager = datanodeManager.getDecomManager(); + + // Write a 3 block file, so each node has one block. Should scan 1 node + // each time. + DFSTestUtil.createFile(fs, new Path("/file1"), 64, (short) 3, 0xBAD1DEA); + for (int i=0; i<3; i++) { + doDecomCheck(datanodeManager, decomManager, 1); } } + + private void doDecomCheck(DatanodeManager datanodeManager, + DecommissionManager decomManager, int expectedNumCheckedNodes) + throws IOException, ExecutionException, InterruptedException { + // Decom all nodes + ArrayList decommissionedNodes = Lists.newArrayList(); + for (DataNode d: cluster.getDataNodes()) { + DatanodeInfo dn = decommissionNode(0, d.getDatanodeUuid(), + decommissionedNodes, + AdminStates.DECOMMISSION_INPROGRESS); + decommissionedNodes.add(dn); + } + // Run decom scan and check + BlockManagerTestUtil.recheckDecommissionState(datanodeManager); + assertEquals("Unexpected # of nodes checked", expectedNumCheckedNodes, + decomManager.getNumNodesChecked()); + // Recommission all nodes + for (DatanodeInfo dn : decommissionedNodes) { + recommissionNode(0, dn); + } + } + + @Test(timeout=120000) + public void testPendingNodes() throws Exception { + Configuration newConf = new Configuration(conf); + org.apache.log4j.Logger.getLogger(DecommissionManager.class) + .setLevel(Level.TRACE); + // Only allow one node to be decom'd at a time + newConf.setInt( + DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MAX_CONCURRENT_TRACKED_NODES, + 1); + // Disable the normal monitor runs + newConf.setInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, + Integer.MAX_VALUE); + startCluster(1, 3, newConf); + final FileSystem fs = cluster.getFileSystem(); + final DatanodeManager datanodeManager = + cluster.getNamesystem().getBlockManager().getDatanodeManager(); + final DecommissionManager decomManager = datanodeManager.getDecomManager(); + + // Keep a file open to prevent decom from progressing + HdfsDataOutputStream open1 = + (HdfsDataOutputStream) fs.create(new Path("/openFile1"), (short)3); + // Flush and trigger block reports so the block definitely shows up on NN + open1.write(123); + open1.hflush(); + for (DataNode d: cluster.getDataNodes()) { + DataNodeTestUtils.triggerBlockReport(d); + } + // Decom two nodes, so one is still alive + ArrayList decommissionedNodes = Lists.newArrayList(); + for (int i=0; i<2; i++) { + final DataNode d = cluster.getDataNodes().get(i); + DatanodeInfo dn = decommissionNode(0, d.getDatanodeUuid(), + decommissionedNodes, + AdminStates.DECOMMISSION_INPROGRESS); + decommissionedNodes.add(dn); + } + + for (int i=2; i>=0; i--) { + assertTrackedAndPending(decomManager, 0, i); + BlockManagerTestUtil.recheckDecommissionState(datanodeManager); + } + + // Close file, try to decom the last node, should get stuck in tracked + open1.close(); + final DataNode d = cluster.getDataNodes().get(2); + DatanodeInfo dn = decommissionNode(0, d.getDatanodeUuid(), + decommissionedNodes, + AdminStates.DECOMMISSION_INPROGRESS); + decommissionedNodes.add(dn); + BlockManagerTestUtil.recheckDecommissionState(datanodeManager); + + assertTrackedAndPending(decomManager, 1, 0); + } + + private void assertTrackedAndPending(DecommissionManager decomManager, + int tracked, int pending) { + assertEquals("Unexpected number of tracked nodes", tracked, + decomManager.getNumTrackedNodes()); + assertEquals("Unexpected number of pending nodes", pending, + decomManager.getNumPendingNodes()); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java index da81d2fdd7fad..5be492ff4ce61 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java @@ -503,7 +503,6 @@ public void testFileChecksum() throws Exception { RAN.setSeed(seed); final Configuration conf = getTestConfiguration(); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).build(); final FileSystem hdfs = cluster.getFileSystem(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java index 0d4435779aaf3..6f6100302ae7b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java @@ -70,7 +70,7 @@ public void testFetchImage() throws Exception { cluster.getNameNodeRpc() .setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); cluster.getNameNodeRpc() .setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java index 7d3946ac78abc..8001bfb407861 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java @@ -77,7 +77,7 @@ public void testFileCorruption() throws Exception { File[] blocks = data_dir.listFiles(); assertTrue("Blocks do not exist in data-dir", (blocks != null) && (blocks.length > 0)); for (int idx = 0; idx < blocks.length; idx++) { - if (!blocks[idx].getName().startsWith("blk_")) { + if (!blocks[idx].getName().startsWith(Block.BLOCK_FILE_PREFIX)) { continue; } System.out.println("Deliberately removing file "+blocks[idx].getName()); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java index a0129da1f4a4a..fd916a942ec77 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java @@ -43,6 +43,8 @@ import java.io.FileNotFoundException; import java.io.FileReader; import java.io.IOException; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; import java.net.InetSocketAddress; import java.net.URI; import java.net.UnknownHostException; @@ -603,7 +605,8 @@ public void testFileCreationError3() throws IOException { * Test that file leases are persisted across namenode restarts. */ @Test - public void testFileCreationNamenodeRestart() throws IOException { + public void testFileCreationNamenodeRestart() + throws IOException, NoSuchFieldException, IllegalAccessException { Configuration conf = new HdfsConfiguration(); final int MAX_IDLE_TIME = 2000; // 2s conf.setInt("ipc.client.connection.maxidletime", MAX_IDLE_TIME); @@ -675,7 +678,7 @@ public void testFileCreationNamenodeRestart() throws IOException { // restart cluster with the same namenode port as before. // This ensures that leases are persisted in fsimage. - cluster.shutdown(); + cluster.shutdown(false, false); try { Thread.sleep(2*MAX_IDLE_TIME); } catch (InterruptedException e) { @@ -687,7 +690,7 @@ public void testFileCreationNamenodeRestart() throws IOException { // restart cluster yet again. This triggers the code to read in // persistent leases from fsimage. - cluster.shutdown(); + cluster.shutdown(false, false); try { Thread.sleep(5000); } catch (InterruptedException e) { @@ -702,11 +705,18 @@ public void testFileCreationNamenodeRestart() throws IOException { // new blocks for files that were renamed. DFSOutputStream dfstream = (DFSOutputStream) (stm.getWrappedStream()); - dfstream.setTestFilename(file1.toString()); + + Field f = DFSOutputStream.class.getDeclaredField("src"); + Field modifiersField = Field.class.getDeclaredField("modifiers"); + modifiersField.setAccessible(true); + modifiersField.setInt(f, f.getModifiers() & ~Modifier.FINAL); + f.setAccessible(true); + + f.set(dfstream, file1.toString()); dfstream = (DFSOutputStream) (stm3.getWrappedStream()); - dfstream.setTestFilename(file3new.toString()); + f.set(dfstream, file3new.toString()); dfstream = (DFSOutputStream) (stm4.getWrappedStream()); - dfstream.setTestFilename(file4new.toString()); + f.set(dfstream, file4new.toString()); // write 1 byte to file. This should succeed because the // namenode should have persisted leases. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java index cc898527882c1..fcc8e3560b40f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java @@ -132,7 +132,8 @@ public void testReadSelectNonStaleDatanode() throws Exception { staleNodeInfo = cluster.getNameNode().getNamesystem().getBlockManager() .getDatanodeManager() .getDatanode(staleNode.getDatanodeId()); - staleNodeInfo.setLastUpdate(Time.now() - staleInterval - 1); + DFSTestUtil.resetLastUpdatesWithOffset(staleNodeInfo, + -(staleInterval + 1)); LocatedBlocks blocksAfterStale = client.getNamenode().getBlockLocations( fileName.toString(), 0, blockSize); @@ -143,8 +144,7 @@ public void testReadSelectNonStaleDatanode() throws Exception { // restart the staleNode's heartbeat DataNodeTestUtils.setHeartbeatsDisabledForTests(staleNode, false); // reset the first node as non-stale, so as to avoid two stale nodes - staleNodeInfo.setLastUpdate(Time.now()); - + DFSTestUtil.resetLastUpdatesWithOffset(staleNodeInfo, 0); LocatedBlock lastBlock = client.getLocatedBlocks(fileName.toString(), 0, Long.MAX_VALUE).getLastLocatedBlock(); nodes = lastBlock.getLocations(); @@ -153,10 +153,10 @@ public void testReadSelectNonStaleDatanode() throws Exception { staleNode = this.stopDataNodeHeartbeat(cluster, nodes[0].getHostName()); assertNotNull(staleNode); // set the node as stale - cluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager() - .getDatanode(staleNode.getDatanodeId()) - .setLastUpdate(Time.now() - staleInterval - 1); + DatanodeDescriptor dnDesc = cluster.getNameNode().getNamesystem() + .getBlockManager().getDatanodeManager() + .getDatanode(staleNode.getDatanodeId()); + DFSTestUtil.resetLastUpdatesWithOffset(dnDesc, -(staleInterval + 1)); LocatedBlock lastBlockAfterStale = client.getLocatedBlocks( fileName.toString(), 0, Long.MAX_VALUE).getLastLocatedBlock(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInjectionForSimulatedStorage.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInjectionForSimulatedStorage.java index 8f2873ef06131..e49cf5838cbaa 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInjectionForSimulatedStorage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestInjectionForSimulatedStorage.java @@ -59,7 +59,7 @@ private void waitForBlockReplication(String filename, ClientProtocol namenode, int expected, long maxWaitSec) throws IOException { - long start = Time.now(); + long start = Time.monotonicNow(); //wait for all the blocks to be replicated; LOG.info("Checking for block replication for " + filename); @@ -84,7 +84,7 @@ private void waitForBlockReplication(String filename, actual + "."); if (maxWaitSec > 0 && - (Time.now() - start) > (maxWaitSec * 1000)) { + (Time.monotonicNow() - start) > (maxWaitSec * 1000)) { throw new IOException("Timedout while waiting for all blocks to " + " be replicated for " + filename); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java index 5d93db4e47a15..1cf7add170ed9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java @@ -101,7 +101,7 @@ public void testLeaseAbort() throws Exception { // call renewLease() manually. // make it look like the soft limit has been exceeded. LeaseRenewer originalRenewer = dfs.getLeaseRenewer(); - dfs.lastLeaseRenewal = Time.now() + dfs.lastLeaseRenewal = Time.monotonicNow() - HdfsConstants.LEASE_SOFTLIMIT_PERIOD - 1000; try { dfs.renewLease(); @@ -117,7 +117,7 @@ public void testLeaseAbort() throws Exception { } // make it look like the hard limit has been exceeded. - dfs.lastLeaseRenewal = Time.now() + dfs.lastLeaseRenewal = Time.monotonicNow() - HdfsConstants.LEASE_HARDLIMIT_PERIOD - 1000; dfs.renewLease(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java index 8ca4576a9eaba..11cbcad5596b6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java @@ -111,8 +111,8 @@ public Boolean answer(InvocationOnMock invocation) throws Throwable { renewer.put(fileId, mockStream, MOCK_DFSCLIENT); // Wait for lease to get renewed - long failTime = Time.now() + 5000; - while (Time.now() < failTime && + long failTime = Time.monotonicNow() + 5000; + while (Time.monotonicNow() < failTime && leaseRenewalCount.get() == 0) { Thread.sleep(50); } @@ -193,11 +193,11 @@ public void testThreadName() throws Exception { // Pretend to close the file renewer.closeFile(fileId, MOCK_DFSCLIENT); - renewer.setEmptyTime(Time.now()); + renewer.setEmptyTime(Time.monotonicNow()); // Should stop the renewer running within a few seconds - long failTime = Time.now() + 5000; - while (renewer.isRunning() && Time.now() < failTime) { + long failTime = Time.monotonicNow() + 5000; + while (renewer.isRunning() && Time.monotonicNow() < failTime) { Thread.sleep(50); } Assert.assertFalse(renewer.isRunning()); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java index 0163d956ac1cf..23e2a7a2f95dd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestParallelReadUtil.java @@ -333,7 +333,7 @@ boolean runParallelRead(int nFiles, int nWorkerEach, ReadWorkerHelper helper) th } // Start the workers and wait - long starttime = Time.now(); + long starttime = Time.monotonicNow(); for (ReadWorker worker : workers) { worker.start(); } @@ -343,7 +343,7 @@ boolean runParallelRead(int nFiles, int nWorkerEach, ReadWorkerHelper helper) th worker.join(); } catch (InterruptedException ignored) { } } - long endtime = Time.now(); + long endtime = Time.monotonicNow(); // Cleanup for (TestFileInfo testInfo : testInfoArr) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java index d108d59233fcc..163378c691649 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java @@ -851,8 +851,7 @@ public void testBlockAllocationAdjustsUsageConservatively() Configuration conf = new HdfsConfiguration(); final int BLOCK_SIZE = 6 * 1024; conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); - MiniDFSCluster cluster = + MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build(); cluster.waitActive(); FileSystem fs = cluster.getFileSystem(); @@ -913,7 +912,6 @@ public void testMultipleFilesSmallerThanOneBlock() throws Exception { Configuration conf = new HdfsConfiguration(); final int BLOCK_SIZE = 6 * 1024; conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); // Make it relinquish locks. When run serially, the result should // be identical. conf.setInt(DFSConfigKeys.DFS_CONTENT_SUMMARY_LIMIT_KEY, 2); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java index d116f82d5090c..b702da0b06d71 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java @@ -44,6 +44,7 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlocks; +import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset; import org.apache.hadoop.util.Time; import org.junit.Test; @@ -271,7 +272,7 @@ private void waitForBlockReplication(String filename, ClientProtocol namenode, int expected, long maxWaitSec) throws IOException { - long start = Time.now(); + long start = Time.monotonicNow(); //wait for all the blocks to be replicated; LOG.info("Checking for block replication for " + filename); @@ -297,7 +298,7 @@ private void waitForBlockReplication(String filename, } if (maxWaitSec > 0 && - (Time.now() - start) > (maxWaitSec * 1000)) { + (Time.monotonicNow() - start) > (maxWaitSec * 1000)) { throw new IOException("Timedout while waiting for all blocks to " + " be replicated for " + filename); } @@ -511,7 +512,7 @@ public void testReplicationWhenBlockCorruption() throws Exception { String blockFile = null; File[] listFiles = participatedNodeDirs.listFiles(); for (File file : listFiles) { - if (file.getName().startsWith("blk_") + if (file.getName().startsWith(Block.BLOCK_FILE_PREFIX) && !file.getName().endsWith("meta")) { blockFile = file.getName(); for (File file1 : nonParticipatedNodeDirs) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java index 974604976e65d..8baebd8a625a2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java @@ -244,7 +244,7 @@ public void testRollingUpgradeWithQJM() throws Exception { //finalize rolling upgrade final RollingUpgradeInfo finalize = dfs2.rollingUpgrade( RollingUpgradeAction.FINALIZE); - Assert.assertNull(finalize); + Assert.assertTrue(finalize.isFinalized()); LOG.info("RESTART cluster 2 with regular startup option"); cluster2.getNameNodeInfos()[0].setStartOpt(StartupOption.REGULAR); @@ -405,7 +405,7 @@ public void testFinalize() throws Exception { Assert.assertTrue(fsimage.hasRollbackFSImage()); info = dfs.rollingUpgrade(RollingUpgradeAction.FINALIZE); - Assert.assertNull(info); + Assert.assertTrue(info.isFinalized()); Assert.assertTrue(dfs.exists(foo)); // Once finalized, there should be no more fsimage for rollbacks. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java index 2d5bef2b6abf3..80fe9eead5491 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java @@ -198,7 +198,7 @@ public void testInitializeReplQueuesEarly() throws Exception { String status = nn.getNamesystem().getSafemode(); assertEquals("Safe mode is ON. The reported blocks 0 needs additional " + - "15 blocks to reach the threshold 0.9990 of total blocks 15." + NEWLINE + + "14 blocks to reach the threshold 0.9990 of total blocks 15." + NEWLINE + "The number of live datanodes 0 has reached the minimum number 0. " + "Safe mode will be turned off automatically once the thresholds " + "have been reached.", status); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java index dac26a08b9890..e9891bfe5950e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/UpgradeUtilities.java @@ -144,7 +144,7 @@ public static void initialize() throws Exception { // save image namenode.setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - namenode.saveNamespace(); + namenode.saveNamespace(0, 0); namenode.setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); // write more files @@ -304,10 +304,11 @@ public static void checksumContentsHelper(NodeType nodeType, File dir, continue; } - // skip VERSION and dfsUsed file for DataNodes + // skip VERSION and dfsUsed and replicas file for DataNodes if (nodeType == DATA_NODE && (list[i].getName().equals("VERSION") || - list[i].getName().equals("dfsUsed"))) { + list[i].getName().equals("dfsUsed") || + list[i].getName().equals("replicas"))) { continue; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java new file mode 100644 index 0000000000000..f0dab4c01333c --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java @@ -0,0 +1,240 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdfs.protocol; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.Matchers.any; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.concurrent.atomic.AtomicReference; + +import org.apache.hadoop.hdfs.DFSTestUtil; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportReplica; +import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.BlockReportRequestProto; +import org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos.BlockReportResponseProto; +import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; +import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB; +import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState; +import org.apache.hadoop.hdfs.server.datanode.FinalizedReplica; +import org.apache.hadoop.hdfs.server.datanode.Replica; +import org.apache.hadoop.hdfs.server.datanode.ReplicaBeingWritten; +import org.apache.hadoop.hdfs.server.datanode.ReplicaWaitingToBeRecovered; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; +import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; +import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; +import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo; +import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.Capability; +import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; +import org.junit.Test; +import org.mockito.invocation.InvocationOnMock; +import org.mockito.stubbing.Answer; + +import com.google.protobuf.ByteString; +import com.google.protobuf.RpcController; +import com.google.protobuf.ServiceException; + +public class TestBlockListAsLongs { + static Block b1 = new Block(1, 11, 111); + static Block b2 = new Block(2, 22, 222); + static Block b3 = new Block(3, 33, 333); + static Block b4 = new Block(4, 44, 444); + + @Test + public void testEmptyReport() { + BlockListAsLongs blocks = checkReport(); + assertArrayEquals( + new long[] { + 0, 0, + -1, -1, -1 }, + blocks.getBlockListAsLongs()); + } + + @Test + public void testFinalized() { + BlockListAsLongs blocks = checkReport( + new FinalizedReplica(b1, null, null)); + assertArrayEquals( + new long[] { + 1, 0, + 1, 11, 111, + -1, -1, -1 }, + blocks.getBlockListAsLongs()); + } + + @Test + public void testUc() { + BlockListAsLongs blocks = checkReport( + new ReplicaBeingWritten(b1, null, null, null)); + assertArrayEquals( + new long[] { + 0, 1, + -1, -1, -1, + 1, 11, 111, ReplicaState.RBW.getValue() }, + blocks.getBlockListAsLongs()); + } + + @Test + public void testMix() { + BlockListAsLongs blocks = checkReport( + new FinalizedReplica(b1, null, null), + new FinalizedReplica(b2, null, null), + new ReplicaBeingWritten(b3, null, null, null), + new ReplicaWaitingToBeRecovered(b4, null, null)); + assertArrayEquals( + new long[] { + 2, 2, + 1, 11, 111, + 2, 22, 222, + -1, -1, -1, + 3, 33, 333, ReplicaState.RBW.getValue(), + 4, 44, 444, ReplicaState.RWR.getValue() }, + blocks.getBlockListAsLongs()); + } + + @Test + public void testFuzz() throws InterruptedException { + Replica[] replicas = new Replica[100000]; + Random rand = new Random(0); + for (int i=0; i expectedReplicas = new HashMap<>(); + for (Replica replica : replicas) { + expectedReplicas.put(replica.getBlockId(), replica); + } + expectedReplicas = Collections.unmodifiableMap(expectedReplicas); + + // encode the blocks and extract the buffers + BlockListAsLongs blocks = + BlockListAsLongs.encode(expectedReplicas.values()); + List buffers = blocks.getBlocksBuffers(); + + // convert to old-style list of longs + List longs = new ArrayList(); + for (long value : blocks.getBlockListAsLongs()) { + longs.add(value); + } + + // decode the buffers and verify its contents + BlockListAsLongs decodedBlocks = + BlockListAsLongs.decodeBuffers(expectedReplicas.size(), buffers); + checkReplicas(expectedReplicas, decodedBlocks); + + // decode the long and verify its contents + BlockListAsLongs decodedList = BlockListAsLongs.decodeLongs(longs); + checkReplicas(expectedReplicas, decodedList); + return blocks; + } + + private void checkReplicas(Map expectedReplicas, + BlockListAsLongs decodedBlocks) { + assertEquals(expectedReplicas.size(), decodedBlocks.getNumberOfBlocks()); + + Map reportReplicas = new HashMap<>(expectedReplicas); + for (BlockReportReplica replica : decodedBlocks) { + assertNotNull(replica); + Replica expected = reportReplicas.remove(replica.getBlockId()); + assertNotNull(expected); + assertEquals("wrong bytes", + expected.getNumBytes(), replica.getNumBytes()); + assertEquals("wrong genstamp", + expected.getGenerationStamp(), replica.getGenerationStamp()); + assertEquals("wrong replica state", + expected.getState(), replica.getState()); + } + assertTrue(reportReplicas.isEmpty()); + } + + @Test + public void testDatanodeDetect() throws ServiceException, IOException { + final AtomicReference request = + new AtomicReference<>(); + + // just capture the outgoing PB + DatanodeProtocolPB mockProxy = mock(DatanodeProtocolPB.class); + doAnswer(new Answer() { + public BlockReportResponseProto answer(InvocationOnMock invocation) { + Object[] args = invocation.getArguments(); + request.set((BlockReportRequestProto) args[1]); + return BlockReportResponseProto.newBuilder().build(); + } + }).when(mockProxy).blockReport(any(RpcController.class), + any(BlockReportRequestProto.class)); + + @SuppressWarnings("resource") + DatanodeProtocolClientSideTranslatorPB nn = + new DatanodeProtocolClientSideTranslatorPB(mockProxy); + + DatanodeRegistration reg = DFSTestUtil.getLocalDatanodeRegistration(); + NamespaceInfo nsInfo = new NamespaceInfo(1, "cluster", "bp", 1); + reg.setNamespaceInfo(nsInfo); + + Replica r = new FinalizedReplica(new Block(1, 2, 3), null, null); + BlockListAsLongs bbl = BlockListAsLongs.encode(Collections.singleton(r)); + DatanodeStorage storage = new DatanodeStorage("s1"); + StorageBlockReport[] sbr = { new StorageBlockReport(storage, bbl) }; + + // check DN sends new-style BR + request.set(null); + nsInfo.setCapabilities(Capability.STORAGE_BLOCK_REPORT_BUFFERS.getMask()); + nn.blockReport(reg, "pool", sbr, + new BlockReportContext(1, 0, System.nanoTime())); + BlockReportRequestProto proto = request.get(); + assertNotNull(proto); + assertTrue(proto.getReports(0).getBlocksList().isEmpty()); + assertFalse(proto.getReports(0).getBlocksBuffersList().isEmpty()); + + // back up to prior version and check DN sends old-style BR + request.set(null); + nsInfo.setCapabilities(Capability.UNKNOWN.getMask()); + nn.blockReport(reg, "pool", sbr, + new BlockReportContext(1, 0, System.nanoTime())); + proto = request.get(); + assertNotNull(proto); + assertFalse(proto.getReports(0).getBlocksList().isEmpty()); + assertTrue(proto.getReports(0).getBlocksBuffersList().isEmpty()); + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java index 0236288a42024..c7233bd5da9a6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocolPB/TestPBHelper.java @@ -478,10 +478,11 @@ private LocatedBlock createLocatedBlockNoStorageMedia() { AdminStates.NORMAL) }; LocatedBlock lb = new LocatedBlock( - new ExtendedBlock("bp12", 12345, 10, 53), dnInfos, 5, false); + new ExtendedBlock("bp12", 12345, 10, 53), dnInfos); lb.setBlockToken(new Token( "identifier".getBytes(), "password".getBytes(), new Text("kind"), new Text("service"))); + lb.setStartOffset(5); return lb; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java index 5b5c450be37d0..2e84499e4bb08 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java @@ -270,6 +270,7 @@ public void testAcceptRecoveryBehavior() throws Exception { public void testFailToStartWithBadConfig() throws Exception { Configuration conf = new Configuration(); conf.set(DFSConfigKeys.DFS_JOURNALNODE_EDITS_DIR_KEY, "non-absolute-path"); + conf.set(DFSConfigKeys.DFS_JOURNALNODE_HTTP_ADDRESS_KEY, "0.0.0.0:0"); assertJNFailsToStart(conf, "should be an absolute path"); // Existing file which is not a directory diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java index 79b02e1d3f495..dff8fa4ba9a1e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java @@ -67,7 +67,6 @@ public class TestDelegationToken { @Before public void setUp() throws Exception { config = new HdfsConfiguration(); - config.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); config.setLong(DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, 10000); config.setLong(DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY, 5000); config.setBoolean(DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY, true); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java index e6493a24ff3c8..3bbd6f15a9153 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationTokenForProxyUser.java @@ -97,7 +97,6 @@ private static void configureSuperUserIPAddresses(Configuration conf, @BeforeClass public static void setUp() throws Exception { config = new HdfsConfiguration(); - config.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); config.setLong( DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, 10000); config.setLong( diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java index 9b624676d99b6..8b2d11e0560cd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java @@ -262,7 +262,7 @@ static void waitForHeartBeat(long expectedUsedSpace, throws IOException, TimeoutException { long timeout = TIMEOUT; long failtime = (timeout <= 0L) ? Long.MAX_VALUE - : Time.now() + timeout; + : Time.monotonicNow() + timeout; while (true) { long[] status = client.getStats(); @@ -274,7 +274,7 @@ static void waitForHeartBeat(long expectedUsedSpace, && usedSpaceVariance < CAPACITY_ALLOWED_VARIANCE) break; //done - if (Time.now() > failtime) { + if (Time.monotonicNow() > failtime) { throw new TimeoutException("Cluster failed to reached expected values of " + "totalSpace (current: " + status[0] + ", expected: " + expectedTotalSpace @@ -369,7 +369,7 @@ static void waitForBalancer(long totalUsedSpace, long totalCapacity, int expectedExcludedNodes) throws IOException, TimeoutException { long timeout = TIMEOUT; long failtime = (timeout <= 0L) ? Long.MAX_VALUE - : Time.now() + timeout; + : Time.monotonicNow() + timeout; if (!p.nodesToBeIncluded.isEmpty()) { totalCapacity = p.nodesToBeIncluded.size() * CAPACITY; } @@ -399,7 +399,7 @@ static void waitForBalancer(long totalUsedSpace, long totalCapacity, } if (Math.abs(avgUtilization - nodeUtilization) > BALANCE_ALLOWED_VARIANCE) { balanced = false; - if (Time.now() > failtime) { + if (Time.monotonicNow() > failtime) { throw new TimeoutException( "Rebalancing expected avg utilization to become " + avgUtilization + ", but on datanode " + datanode diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java index fccd308e70a1e..23e610f841538 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java @@ -22,6 +22,7 @@ import java.util.Collection; import java.util.HashSet; import java.util.Set; +import java.util.concurrent.ExecutionException; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.DFSTestUtil; @@ -185,7 +186,7 @@ public static void noticeDeadDatanode(NameNode nn, String dnName) { Assert.assertNotNull("Could not find DN with name: " + dnName, theDND); synchronized (hbm) { - theDND.setLastUpdate(0); + DFSTestUtil.setDatanodeDead(theDND); hbm.heartbeatCheck(); } } finally { @@ -300,9 +301,8 @@ public static StorageReport[] getStorageReportsForDatanode( * Have DatanodeManager check decommission state. * @param dm the DatanodeManager to manipulate */ - public static void checkDecommissionState(DatanodeManager dm, - DatanodeDescriptor node) { - dm.checkDecommissionState(node); + public static void recheckDecommissionState(DatanodeManager dm) + throws ExecutionException, InterruptedException { + dm.getDecomManager().runMonitor(); } - } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java index 453f411ac21e6..a7ba29399dcad 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoUnderConstruction.java @@ -23,6 +23,7 @@ import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.server.common.GenerationStamp; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState; +import org.apache.hadoop.util.Time; import org.junit.Test; /** @@ -46,40 +47,34 @@ public void testInitializeBlockRecovery() throws Exception { new DatanodeStorageInfo[] {s1, s2, s3}); // Recovery attempt #1. - long currentTime = System.currentTimeMillis(); - dd1.setLastUpdate(currentTime - 3 * 1000); - dd2.setLastUpdate(currentTime - 1 * 1000); - dd3.setLastUpdate(currentTime - 2 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, -3 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, -1 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, -2 * 1000); blockInfo.initializeBlockRecovery(1); BlockInfoContiguousUnderConstruction[] blockInfoRecovery = dd2.getLeaseRecoveryCommand(1); assertEquals(blockInfoRecovery[0], blockInfo); // Recovery attempt #2. - currentTime = System.currentTimeMillis(); - dd1.setLastUpdate(currentTime - 2 * 1000); - dd2.setLastUpdate(currentTime - 1 * 1000); - dd3.setLastUpdate(currentTime - 3 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, -2 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, -1 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, -3 * 1000); blockInfo.initializeBlockRecovery(2); blockInfoRecovery = dd1.getLeaseRecoveryCommand(1); assertEquals(blockInfoRecovery[0], blockInfo); // Recovery attempt #3. - currentTime = System.currentTimeMillis(); - dd1.setLastUpdate(currentTime - 2 * 1000); - dd2.setLastUpdate(currentTime - 1 * 1000); - dd3.setLastUpdate(currentTime - 3 * 1000); - currentTime = System.currentTimeMillis(); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, -2 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, -1 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, -3 * 1000); blockInfo.initializeBlockRecovery(3); blockInfoRecovery = dd3.getLeaseRecoveryCommand(1); assertEquals(blockInfoRecovery[0], blockInfo); // Recovery attempt #4. // Reset everything. And again pick DN with most recent heart beat. - currentTime = System.currentTimeMillis(); - dd1.setLastUpdate(currentTime - 2 * 1000); - dd2.setLastUpdate(currentTime - 1 * 1000); - dd3.setLastUpdate(currentTime); - currentTime = System.currentTimeMillis(); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, -2 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, -1 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, 0); blockInfo.initializeBlockRecovery(3); blockInfoRecovery = dd3.getLeaseRecoveryCommand(1); assertEquals(blockInfoRecovery[0], blockInfo); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java index f9c2bdce22af4..91abb2a5f5c59 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java @@ -534,14 +534,52 @@ public void testHighestPriReplSrcChosenDespiteMaxReplLimit() throws Exception { UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY)); } + @Test + public void testFavorDecomUntilHardLimit() throws Exception { + bm.maxReplicationStreams = 0; + bm.replicationStreamsHardLimit = 1; + + long blockId = 42; // arbitrary + Block aBlock = new Block(blockId, 0, 0); + List origNodes = getNodes(0, 1); + // Add the block to the first node. + addBlockOnNodes(blockId,origNodes.subList(0,1)); + origNodes.get(0).startDecommission(); + + List cntNodes = new LinkedList(); + List liveNodes = new LinkedList(); + + assertNotNull("Chooses decommissioning source node for a normal replication" + + " if all available source nodes have reached their replication" + + " limits below the hard limit.", + bm.chooseSourceDatanode( + aBlock, + cntNodes, + liveNodes, + new NumberReplicas(), + UnderReplicatedBlocks.QUEUE_UNDER_REPLICATED)); + + + // Increase the replication count to test replication count > hard limit + DatanodeStorageInfo targets[] = { origNodes.get(1).getStorageInfos()[0] }; + origNodes.get(0).addBlockToBeReplicated(aBlock, targets); + + assertNull("Does not choose a source decommissioning node for a normal" + + " replication when all available nodes exceed the hard limit.", + bm.chooseSourceDatanode( + aBlock, + cntNodes, + liveNodes, + new NumberReplicas(), + UnderReplicatedBlocks.QUEUE_UNDER_REPLICATED)); + } + + + @Test public void testSafeModeIBR() throws Exception { DatanodeDescriptor node = spy(nodes.get(0)); DatanodeStorageInfo ds = node.getStorageInfos()[0]; - - // TODO: Needs to be fixed. DatanodeUuid is not storageID. - node.setDatanodeUuidForTesting(ds.getStorageID()); - node.isAlive = true; DatanodeRegistration nodeReg = @@ -559,12 +597,12 @@ public void testSafeModeIBR() throws Exception { reset(node); bm.processReport(node, new DatanodeStorage(ds.getStorageID()), - new BlockListAsLongs(null, null)); + BlockListAsLongs.EMPTY, null, false); assertEquals(1, ds.getBlockReportCount()); // send block report again, should NOT be processed reset(node); bm.processReport(node, new DatanodeStorage(ds.getStorageID()), - new BlockListAsLongs(null, null)); + BlockListAsLongs.EMPTY, null, false); assertEquals(1, ds.getBlockReportCount()); // re-register as if node restarted, should update existing node @@ -575,7 +613,7 @@ public void testSafeModeIBR() throws Exception { // send block report, should be processed after restart reset(node); bm.processReport(node, new DatanodeStorage(ds.getStorageID()), - new BlockListAsLongs(null, null)); + BlockListAsLongs.EMPTY, null, false); // Reinitialize as registration with empty storage list pruned // node.storageMap. ds = node.getStorageInfos()[0]; @@ -587,9 +625,6 @@ public void testSafeModeIBRAfterIncremental() throws Exception { DatanodeDescriptor node = spy(nodes.get(0)); DatanodeStorageInfo ds = node.getStorageInfos()[0]; - // TODO: Needs to be fixed. DatanodeUuid is not storageID. - node.setDatanodeUuidForTesting(ds.getStorageID()); - node.isAlive = true; DatanodeRegistration nodeReg = @@ -607,7 +642,7 @@ public void testSafeModeIBRAfterIncremental() throws Exception { reset(node); doReturn(1).when(node).numBlocks(); bm.processReport(node, new DatanodeStorage(ds.getStorageID()), - new BlockListAsLongs(null, null)); + BlockListAsLongs.EMPTY, null, false); assertEquals(1, ds.getBlockReportCount()); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java index fecca4e915c07..5b08f5302f0de 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java @@ -38,7 +38,6 @@ import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.util.VersionInfo; import org.junit.After; -import org.junit.Assert; import org.junit.Before; import org.junit.Test; import org.mockito.internal.util.reflection.Whitebox; @@ -119,10 +118,17 @@ public void testCompInvalidate() throws Exception { public void testDatanodeReformat() throws Exception { namesystem.writeLock(); try { + // Change the datanode UUID to emulate a reformat + String poolId = cluster.getNamesystem().getBlockPoolId(); + DatanodeRegistration dnr = cluster.getDataNode(nodes[0].getIpcPort()) + .getDNRegistrationForBP(poolId); + dnr = new DatanodeRegistration(UUID.randomUUID().toString(), dnr); + cluster.stopDataNode(nodes[0].getXferAddr()); + Block block = new Block(0, 0, GenerationStamp.LAST_RESERVED_STAMP); bm.addToInvalidates(block, nodes[0]); - // Change the datanode UUID to emulate a reformation - nodes[0].setDatanodeUuidForTesting("fortesting"); + bm.getDatanodeManager().registerDatanode(dnr); + // Since UUID has changed, the invalidation work should be skipped assertEquals(0, bm.computeInvalidateWork(1)); assertEquals(0, bm.getPendingDeletionBlocksCount()); @@ -158,8 +164,8 @@ public void testDatanodeReRegistration() throws Exception { // Re-register each DN and see that it wipes the invalidation work for (DataNode dn : cluster.getDataNodes()) { DatanodeID did = dn.getDatanodeId(); - did.setDatanodeUuidForTesting(UUID.randomUUID().toString()); - DatanodeRegistration reg = new DatanodeRegistration(did, + DatanodeRegistration reg = new DatanodeRegistration( + new DatanodeID(UUID.randomUUID().toString(), did), new StorageInfo(HdfsServerConstants.NodeType.DATA_NODE), new ExportedBlockKeys(), VersionInfo.getVersion()); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java index efd1febbda35d..6fc30ba6d6314 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java @@ -23,6 +23,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocol.Block; @@ -38,6 +39,7 @@ import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; +import org.apache.hadoop.util.Time; import org.junit.Test; /** @@ -164,9 +166,9 @@ public void testHeartbeatBlockRecovery() throws Exception { NameNodeAdapter.sendHeartBeat(nodeReg3, dd3, namesystem); // Test with all alive nodes. - dd1.setLastUpdate(System.currentTimeMillis()); - dd2.setLastUpdate(System.currentTimeMillis()); - dd3.setLastUpdate(System.currentTimeMillis()); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, 0); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, 0); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, 0); final DatanodeStorageInfo[] storages = { dd1.getStorageInfos()[0], dd2.getStorageInfos()[0], @@ -189,10 +191,10 @@ public void testHeartbeatBlockRecovery() throws Exception { assertEquals(recoveringNodes[2], dd3); // Test with one stale node. - dd1.setLastUpdate(System.currentTimeMillis()); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, 0); // More than the default stale interval of 30 seconds. - dd2.setLastUpdate(System.currentTimeMillis() - 40 * 1000); - dd3.setLastUpdate(System.currentTimeMillis()); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, -40 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, 0); blockInfo = new BlockInfoContiguousUnderConstruction( new Block(0, 0, GenerationStamp.LAST_RESERVED_STAMP), (short) 3, BlockUCState.UNDER_RECOVERY, storages); @@ -210,10 +212,10 @@ public void testHeartbeatBlockRecovery() throws Exception { assertEquals(recoveringNodes[1], dd3); // Test with all stale node. - dd1.setLastUpdate(System.currentTimeMillis() - 60 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd1, - 60 * 1000); // More than the default stale interval of 30 seconds. - dd2.setLastUpdate(System.currentTimeMillis() - 40 * 1000); - dd3.setLastUpdate(System.currentTimeMillis() - 80 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd2, - 40 * 1000); + DFSTestUtil.resetLastUpdatesWithOffset(dd3, - 80 * 1000); blockInfo = new BlockInfoContiguousUnderConstruction( new Block(0, 0, GenerationStamp.LAST_RESERVED_STAMP), (short) 3, BlockUCState.UNDER_RECOVERY, storages); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHostFileManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHostFileManager.java index 5435572e00ecc..733446ceb17b1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHostFileManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHostFileManager.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdfs.server.blockmanagement; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.HdfsConstants; import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; @@ -142,7 +143,7 @@ public void testIncludeExcludeLists() throws IOException { .DatanodeReportType.DEAD).size()); DatanodeDescriptor spam = new DatanodeDescriptor(new DatanodeID("127.0.0" + ".3", "127.0.0.3", "uuid-spam", 12345, 1020, 1021, 1022)); - spam.setLastUpdate(0); + DFSTestUtil.setDatanodeDead(spam); includedNodes.add(entry("127.0.0.3:12345")); dnMap.put("uuid-spam", spam); Assert.assertEquals(1, dm.getDatanodeListForReport(HdfsConstants diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java index b67ae7a62325e..4b97d01b73ed4 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java @@ -18,26 +18,40 @@ package org.apache.hadoop.hdfs.server.blockmanagement; +import com.google.common.base.Supplier; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.commons.math3.stat.inference.TestUtils; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.DFSTestUtil; +import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocol.DatanodeID; +import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.StorageReport; import org.apache.hadoop.test.GenericTestUtils; +import org.junit.Assert; import org.junit.Test; +import java.io.File; +import java.io.FileOutputStream; import java.io.IOException; +import java.util.Arrays; +import java.util.Iterator; +import java.util.List; import static org.hamcrest.core.Is.is; import static org.junit.Assert.assertThat; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertEquals; public class TestNameNodePrunesMissingStorages { @@ -110,7 +124,9 @@ public void testUnusedStorageIsPruned() throws IOException { } /** - * Verify that the NameNode does not prune storages with blocks. + * Verify that the NameNode does not prune storages with blocks + * simply as a result of a heartbeat being sent missing that storage. + * * @throws IOException */ @Test (timeout=300000) @@ -118,4 +134,119 @@ public void testStorageWithBlocksIsNotPruned() throws IOException { // Run the test with 1 storage, after the text still expect 1 storage. runTest(GenericTestUtils.getMethodName(), true, 1, 1); } + + /** + * Regression test for HDFS-7960.

    + * + * Shutting down a datanode, removing a storage directory, and restarting + * the DataNode should not produce zombie storages. + */ + @Test(timeout=300000) + public void testRemovingStorageDoesNotProduceZombies() throws Exception { + Configuration conf = new HdfsConfiguration(); + conf.setInt(DFSConfigKeys.DFS_DATANODE_FAILED_VOLUMES_TOLERATED_KEY, 1); + final int NUM_STORAGES_PER_DN = 2; + final MiniDFSCluster cluster = new MiniDFSCluster + .Builder(conf).numDataNodes(3) + .storagesPerDatanode(NUM_STORAGES_PER_DN) + .build(); + try { + cluster.waitActive(); + for (DataNode dn : cluster.getDataNodes()) { + assertEquals(NUM_STORAGES_PER_DN, + cluster.getNamesystem().getBlockManager(). + getDatanodeManager().getDatanode(dn.getDatanodeId()). + getStorageInfos().length); + } + // Create a file which will end up on all 3 datanodes. + final Path TEST_PATH = new Path("/foo1"); + DistributedFileSystem fs = cluster.getFileSystem(); + DFSTestUtil.createFile(fs, TEST_PATH, 1024, (short) 3, 0xcafecafe); + for (DataNode dn : cluster.getDataNodes()) { + DataNodeTestUtils.triggerBlockReport(dn); + } + ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, new Path("/foo1")); + cluster.getNamesystem().writeLock(); + final String storageIdToRemove; + String datanodeUuid; + // Find the first storage which this block is in. + try { + Iterator storageInfoIter = + cluster.getNamesystem().getBlockManager(). + getStorages(block.getLocalBlock()).iterator(); + assertTrue(storageInfoIter.hasNext()); + DatanodeStorageInfo info = storageInfoIter.next(); + storageIdToRemove = info.getStorageID(); + datanodeUuid = info.getDatanodeDescriptor().getDatanodeUuid(); + } finally { + cluster.getNamesystem().writeUnlock(); + } + // Find the DataNode which holds that first storage. + final DataNode datanodeToRemoveStorageFrom; + int datanodeToRemoveStorageFromIdx = 0; + while (true) { + if (datanodeToRemoveStorageFromIdx >= cluster.getDataNodes().size()) { + Assert.fail("failed to find datanode with uuid " + datanodeUuid); + datanodeToRemoveStorageFrom = null; + break; + } + DataNode dn = cluster.getDataNodes(). + get(datanodeToRemoveStorageFromIdx); + if (dn.getDatanodeUuid().equals(datanodeUuid)) { + datanodeToRemoveStorageFrom = dn; + break; + } + datanodeToRemoveStorageFromIdx++; + } + // Find the volume within the datanode which holds that first storage. + List volumes = + datanodeToRemoveStorageFrom.getFSDataset().getVolumes(); + assertEquals(NUM_STORAGES_PER_DN, volumes.size()); + String volumeDirectoryToRemove = null; + for (FsVolumeSpi volume : volumes) { + if (volume.getStorageID().equals(storageIdToRemove)) { + volumeDirectoryToRemove = volume.getBasePath(); + } + } + // Shut down the datanode and remove the volume. + // Replace the volume directory with a regular file, which will + // cause a volume failure. (If we merely removed the directory, + // it would be re-initialized with a new storage ID.) + assertNotNull(volumeDirectoryToRemove); + datanodeToRemoveStorageFrom.shutdown(); + FileUtil.fullyDelete(new File(volumeDirectoryToRemove)); + FileOutputStream fos = new FileOutputStream(volumeDirectoryToRemove); + try { + fos.write(1); + } finally { + fos.close(); + } + cluster.restartDataNode(datanodeToRemoveStorageFromIdx); + // Wait for the NameNode to remove the storage. + LOG.info("waiting for the datanode to remove " + storageIdToRemove); + GenericTestUtils.waitFor(new Supplier() { + @Override + public Boolean get() { + final DatanodeDescriptor dnDescriptor = + cluster.getNamesystem().getBlockManager().getDatanodeManager(). + getDatanode(datanodeToRemoveStorageFrom.getDatanodeUuid()); + assertNotNull(dnDescriptor); + DatanodeStorageInfo[] infos = dnDescriptor.getStorageInfos(); + for (DatanodeStorageInfo info : infos) { + if (info.getStorageID().equals(storageIdToRemove)) { + LOG.info("Still found storage " + storageIdToRemove + " on " + + info + "."); + return false; + } + } + assertEquals(NUM_STORAGES_PER_DN - 1, infos.length); + return true; + } + }, 10, 30000); + } finally { + if (cluster != null) { + cluster.shutdown(); + } + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java index f7973819ecbd3..c3726f2a54382 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNodeCount.java @@ -137,7 +137,7 @@ public void testNodeCount() throws Exception { void initializeTimeout(long timeout) { this.timeout = timeout; - this.failtime = Time.now() + this.failtime = Time.monotonicNow() + ((timeout <= 0) ? Long.MAX_VALUE : timeout); } @@ -148,7 +148,7 @@ void checkTimeout(String testLabel) throws TimeoutException { /* check for timeout, then wait for cycleTime msec */ void checkTimeout(String testLabel, long cycleTime) throws TimeoutException { - if (Time.now() > failtime) { + if (Time.monotonicNow() > failtime) { throw new TimeoutException("Timeout: " + testLabel + " for block " + lastBlock + " after " + timeout + " msec. Last counts: live = " + lastNum.liveReplicas() diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java index 6bbb0c3662f0c..a86b57347f35a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java @@ -17,7 +17,7 @@ */ package org.apache.hadoop.hdfs.server.blockmanagement; -import static org.apache.hadoop.util.Time.now; +import static org.apache.hadoop.util.Time.monotonicNow; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; @@ -42,6 +42,7 @@ import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; import org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; +import org.apache.hadoop.util.Time; import org.junit.Test; public class TestOverReplicatedBlocks { @@ -171,10 +172,10 @@ public void testChooseReplicaToDelete() throws Exception { long waitTime = DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_DEFAULT * 1000 * (DFSConfigKeys.DFS_NAMENODE_TOLERATE_HEARTBEAT_MULTIPLIER_DEFAULT + 1); do { - nodeInfo = - namesystem.getBlockManager().getDatanodeManager().getDatanode(dnReg); - lastHeartbeat = nodeInfo.getLastUpdate(); - } while(now() - lastHeartbeat < waitTime); + nodeInfo = namesystem.getBlockManager().getDatanodeManager() + .getDatanode(dnReg); + lastHeartbeat = nodeInfo.getLastUpdateMonotonic(); + } while (monotonicNow() - lastHeartbeat < waitTime); fs.setReplication(fileName, (short)3); BlockLocation locs[] = fs.getFileBlockLocations( diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java index 24fd81d782121..32fae45a09a4e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java @@ -50,6 +50,7 @@ import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.TestBlockStoragePolicy; import org.apache.hadoop.hdfs.protocol.Block; +import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.HdfsConstants; import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.StatefulBlockInfo; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState; @@ -564,7 +565,7 @@ private boolean containsWithinRange(DatanodeDescriptor target, @Test public void testChooseTargetWithStaleNodes() throws Exception { // Set dataNodes[0] as stale - dataNodes[0].setLastUpdate(Time.now() - staleInterval - 1); + DFSTestUtil.resetLastUpdatesWithOffset(dataNodes[0], -(staleInterval + 1)); namenode.getNamesystem().getBlockManager() .getDatanodeManager().getHeartbeatManager().heartbeatCheck(); assertTrue(namenode.getNamesystem().getBlockManager() @@ -584,7 +585,7 @@ public void testChooseTargetWithStaleNodes() throws Exception { assertFalse(isOnSameRack(targets[0], dataNodes[0])); // reset - dataNodes[0].setLastUpdate(Time.now()); + DFSTestUtil.resetLastUpdatesWithOffset(dataNodes[0], 0); namenode.getNamesystem().getBlockManager() .getDatanodeManager().getHeartbeatManager().heartbeatCheck(); } @@ -601,7 +602,8 @@ public void testChooseTargetWithStaleNodes() throws Exception { public void testChooseTargetWithHalfStaleNodes() throws Exception { // Set dataNodes[0], dataNodes[1], and dataNodes[2] as stale for (int i = 0; i < 3; i++) { - dataNodes[i].setLastUpdate(Time.now() - staleInterval - 1); + DFSTestUtil + .resetLastUpdatesWithOffset(dataNodes[i], -(staleInterval + 1)); } namenode.getNamesystem().getBlockManager() .getDatanodeManager().getHeartbeatManager().heartbeatCheck(); @@ -633,7 +635,7 @@ public void testChooseTargetWithHalfStaleNodes() throws Exception { assertTrue(containsWithinRange(dataNodes[5], targets, 0, 3)); for (int i = 0; i < dataNodes.length; i++) { - dataNodes[i].setLastUpdate(Time.now()); + DFSTestUtil.resetLastUpdatesWithOffset(dataNodes[i], 0); } namenode.getNamesystem().getBlockManager() .getDatanodeManager().getHeartbeatManager().heartbeatCheck(); @@ -659,9 +661,10 @@ public void testChooseTargetWithMoreThanHalfStaleNodes() throws Exception { for (int i = 0; i < 2; i++) { DataNode dn = miniCluster.getDataNodes().get(i); DataNodeTestUtils.setHeartbeatsDisabledForTests(dn, true); - miniCluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager().getDatanode(dn.getDatanodeId()) - .setLastUpdate(Time.now() - staleInterval - 1); + DatanodeDescriptor dnDes = miniCluster.getNameNode().getNamesystem() + .getBlockManager().getDatanodeManager() + .getDatanode(dn.getDatanodeId()); + DFSTestUtil.resetLastUpdatesWithOffset(dnDes, -(staleInterval + 1)); } // Instead of waiting, explicitly call heartbeatCheck to // let heartbeat manager to detect stale nodes @@ -689,9 +692,9 @@ public void testChooseTargetWithMoreThanHalfStaleNodes() throws Exception { for (int i = 0; i < 4; i++) { DataNode dn = miniCluster.getDataNodes().get(i); DataNodeTestUtils.setHeartbeatsDisabledForTests(dn, true); - miniCluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager().getDatanode(dn.getDatanodeId()) - .setLastUpdate(Time.now() - staleInterval - 1); + DatanodeDescriptor dnDesc = miniCluster.getNameNode().getNamesystem().getBlockManager() + .getDatanodeManager().getDatanode(dn.getDatanodeId()); + DFSTestUtil.resetLastUpdatesWithOffset(dnDesc, -(staleInterval + 1)); } // Explicitly call heartbeatCheck miniCluster.getNameNode().getNamesystem().getBlockManager() @@ -710,14 +713,15 @@ public void testChooseTargetWithMoreThanHalfStaleNodes() throws Exception { assertEquals(targets.length, 3); assertTrue(isOnSameRack(targets[0], staleNodeInfo)); - // Step 3. Set 2 stale datanodes back to healthy nodes, + // Step 3. Set 2 stale datanodes back to healthy nodes, // still have 2 stale nodes for (int i = 2; i < 4; i++) { DataNode dn = miniCluster.getDataNodes().get(i); DataNodeTestUtils.setHeartbeatsDisabledForTests(dn, false); - miniCluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager().getDatanode(dn.getDatanodeId()) - .setLastUpdate(Time.now()); + DatanodeDescriptor dnDesc = miniCluster.getNameNode().getNamesystem() + .getBlockManager().getDatanodeManager() + .getDatanode(dn.getDatanodeId()); + DFSTestUtil.resetLastUpdatesWithOffset(dnDesc, 0); } // Explicitly call heartbeatCheck miniCluster.getNameNode().getNamesystem().getBlockManager() @@ -973,7 +977,7 @@ public void testChooseReplicaToDelete() throws Exception { // Refresh the last update time for all the datanodes for (int i = 0; i < dataNodes.length; i++) { - dataNodes[i].setLastUpdate(Time.now()); + DFSTestUtil.resetLastUpdatesWithOffset(dataNodes[i], 0); } List first = new ArrayList(); @@ -1217,7 +1221,7 @@ public void testAddStoredBlockDoesNotCauseSkippedReplication() when(mbc.isUnderConstruction()).thenReturn(true); ContentSummary cs = mock(ContentSummary.class); when(cs.getLength()).thenReturn((long)1); - when(mbc.computeContentSummary()).thenReturn(cs); + when(mbc.computeContentSummary(bm.getStoragePolicySuite())).thenReturn(cs); info.setBlockCollection(mbc); bm.addBlockCollection(info, mbc); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java index d9066e8be507d..d514768a7011a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java @@ -137,7 +137,7 @@ public void testChooseTargetWithDecomNodes() throws IOException { // returns false for (int i = 0; i < 3; i++) { DatanodeDescriptor d = dnManager.getDatanode(dnrList.get(i)); - dnManager.startDecommission(d); + dnManager.getDecomManager().startDecommission(d); d.setDecommissioned(); } assertEquals((double)load/3, dnManager.getFSClusterStats() diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java index 8d9de7b07a233..c4a2d06c0e9c7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/BlockReportTestBase.java @@ -46,12 +46,14 @@ import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportReplica; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants; import org.apache.hadoop.hdfs.server.namenode.NameNode; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; @@ -146,22 +148,32 @@ private static StorageBlockReport[] getBlockReports( // Walk the list of blocks until we find one each to corrupt the // generation stamp and length, if so requested. - for (int i = 0; i < blockList.getNumberOfBlocks(); ++i) { + BlockListAsLongs.Builder builder = BlockListAsLongs.builder(); + for (BlockReportReplica block : blockList) { if (corruptOneBlockGs && !corruptedGs) { - blockList.corruptBlockGSForTesting(i, rand); - LOG.info("Corrupted the GS for block ID " + i); + long gsOld = block.getGenerationStamp(); + long gsNew; + do { + gsNew = rand.nextInt(); + } while (gsNew == gsOld); + block.setGenerationStamp(gsNew); + LOG.info("Corrupted the GS for block ID " + block); corruptedGs = true; } else if (corruptOneBlockLen && !corruptedLen) { - blockList.corruptBlockLengthForTesting(i, rand); - LOG.info("Corrupted the length for block ID " + i); + long lenOld = block.getNumBytes(); + long lenNew; + do { + lenNew = rand.nextInt((int)lenOld - 1); + } while (lenNew == lenOld); + block.setNumBytes(lenNew); + LOG.info("Corrupted the length for block ID " + block); corruptedLen = true; - } else { - break; } + builder.add(new BlockReportReplica(block)); } reports[reportIndex++] = - new StorageBlockReport(dnStorage, blockList.getBlockListAsLongs()); + new StorageBlockReport(dnStorage, builder.build()); } return reports; @@ -602,7 +614,8 @@ protected Object passThrough(InvocationOnMock invocation) .when(spy).blockReport( Mockito.anyObject(), Mockito.anyString(), - Mockito.anyObject()); + Mockito.anyObject(), + Mockito.anyObject()); // Force a block report to be generated. The block report will have // an RBW replica in it. Wait for the RPC to be sent, but block @@ -649,12 +662,12 @@ private void waitForTempReplica(Block bl, int DN_N1) throws IOException { final DataNode dn1 = cluster.getDataNodes().get(DN_N1); String bpid = cluster.getNamesystem().getBlockPoolId(); Replica r = DataNodeTestUtils.fetchReplicaInfo(dn1, bpid, bl.getBlockId()); - long start = Time.now(); + long start = Time.monotonicNow(); int count = 0; while (r == null) { waitTil(5); r = DataNodeTestUtils.fetchReplicaInfo(dn1, bpid, bl.getBlockId()); - long waiting_period = Time.now() - start; + long waiting_period = Time.monotonicNow() - start; if (count++ % 100 == 0) if(LOG.isDebugEnabled()) { LOG.debug("Has been waiting for " + waiting_period + " ms."); @@ -668,7 +681,7 @@ private void waitForTempReplica(Block bl, int DN_N1) throws IOException { if(LOG.isDebugEnabled()) { LOG.debug("Replica state before the loop " + state.getValue()); } - start = Time.now(); + start = Time.monotonicNow(); while (state != HdfsServerConstants.ReplicaState.TEMPORARY) { waitTil(5); state = r.getState(); @@ -676,7 +689,7 @@ private void waitForTempReplica(Block bl, int DN_N1) throws IOException { LOG.debug("Keep waiting for " + bl.getBlockName() + " is in state " + state.getValue()); } - if (Time.now() - start > TIMEOUT) + if (Time.monotonicNow() - start > TIMEOUT) assertTrue("Was waiting too long for a replica to become TEMPORARY", tooLongWait); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java index fd51e523d9cde..9dee724995232 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java @@ -40,7 +40,9 @@ * Utility class for accessing package-private DataNode information during tests. * */ -public class DataNodeTestUtils { +public class DataNodeTestUtils { + private static final String DIR_FAILURE_SUFFIX = ".origin"; + public static DatanodeRegistration getDNRegistrationForBP(DataNode dn, String bpid) throws IOException { return dn.getDNRegistrationForBP(bpid); @@ -159,4 +161,68 @@ public static ReplicaInfo fetchReplicaInfo(final DataNode dn, final String bpid, final long blkId) { return FsDatasetTestUtil.fetchReplicaInfo(dn.getFSDataset(), bpid, blkId); } + + /** + * It injects disk failures to data dirs by replacing these data dirs with + * regular files. + * + * @param dirs data directories. + * @throws IOException on I/O error. + */ + public static void injectDataDirFailure(File... dirs) throws IOException { + for (File dir : dirs) { + File renamedTo = new File(dir.getPath() + DIR_FAILURE_SUFFIX); + if (renamedTo.exists()) { + throw new IOException(String.format( + "Can not inject failure to dir: %s because %s exists.", + dir, renamedTo)); + } + if (!dir.renameTo(renamedTo)) { + throw new IOException(String.format("Failed to rename %s to %s.", + dir, renamedTo)); + } + if (!dir.createNewFile()) { + throw new IOException(String.format( + "Failed to create file %s to inject disk failure.", dir)); + } + } + } + + /** + * Restore the injected data dir failures. + * + * @see {@link #injectDataDirFailures}. + * @param dirs data directories. + * @throws IOException + */ + public static void restoreDataDirFromFailure(File... dirs) + throws IOException { + for (File dir : dirs) { + File renamedDir = new File(dir.getPath() + DIR_FAILURE_SUFFIX); + if (renamedDir.exists()) { + if (dir.exists()) { + if (!dir.isFile()) { + throw new IOException( + "Injected failure data dir is supposed to be file: " + dir); + } + if (!dir.delete()) { + throw new IOException( + "Failed to delete injected failure data dir: " + dir); + } + } + if (!renamedDir.renameTo(dir)) { + throw new IOException(String.format( + "Failed to recover injected failure data dir %s to %s.", + renamedDir, dir)); + } + } + } + } + + public static void runDirectoryScanner(DataNode dn) throws IOException { + DirectoryScanner directoryScanner = dn.getDirectoryScanner(); + if (directoryScanner != null) { + dn.getDirectoryScanner().reconcile(); + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java index 8ae541503df16..160a86c3239ae 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java @@ -30,6 +30,7 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.Set; import javax.management.NotCompliantMBeanException; import javax.management.ObjectName; @@ -51,7 +52,6 @@ import org.apache.hadoop.hdfs.server.datanode.fsdataset.LengthInputStream; import org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams; import org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaOutputStreams; -import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl; import org.apache.hadoop.hdfs.server.datanode.metrics.FSDatasetMBean; import org.apache.hadoop.hdfs.server.protocol.BlockRecoveryCommand.RecoveringBlock; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; @@ -84,7 +84,7 @@ static class Factory extends FsDatasetSpi.Factory { @Override public SimulatedFSDataset newInstance(DataNode datanode, DataStorage storage, Configuration conf) throws IOException { - return new SimulatedFSDataset(storage, conf); + return new SimulatedFSDataset(datanode, storage, conf); } @Override @@ -270,7 +270,7 @@ synchronized public long getVisibleLength() { @Override public ReplicaState getState() { - return null; + return finalized ? ReplicaState.FINALIZED : ReplicaState.RBW; } @Override @@ -509,8 +509,15 @@ public FsDatasetSpi getDataset() { private final SimulatedStorage storage; private final SimulatedVolume volume; private final String datanodeUuid; + private final DataNode datanode; + public SimulatedFSDataset(DataStorage storage, Configuration conf) { + this(null, storage, conf); + } + + public SimulatedFSDataset(DataNode datanode, DataStorage storage, Configuration conf) { + this.datanode = datanode; if (storage != null) { for (int i = 0; i < storage.getNumStorageDirs(); ++i) { storage.createStorageID(storage.getStorageDir(i), false); @@ -528,7 +535,7 @@ public SimulatedFSDataset(DataStorage storage, Configuration conf) { } public synchronized void injectBlocks(String bpid, - Iterable injectBlocks) throws IOException { + Iterable injectBlocks) throws IOException { ExtendedBlock blk = new ExtendedBlock(); if (injectBlocks != null) { for (Block b: injectBlocks) { // if any blocks in list is bad, reject list @@ -581,16 +588,16 @@ public synchronized void unfinalizeBlock(ExtendedBlock b) throws IOException{ } synchronized BlockListAsLongs getBlockReport(String bpid) { - final List blocks = new ArrayList(); + BlockListAsLongs.Builder report = BlockListAsLongs.builder(); final Map map = blockMap.get(bpid); if (map != null) { for (BInfo b : map.values()) { if (b.isFinalized()) { - blocks.add(b); + report.add(b); } } } - return new BlockListAsLongs(blocks, null); + return report.build(); } @Override @@ -737,6 +744,10 @@ public synchronized void invalidate(String bpid, Block[] invalidBlks) } storage.free(bpid, binfo.getNumBytes()); map.remove(b); + if (datanode != null) { + datanode.notifyNamenodeDeletedBlock(new ExtendedBlock(bpid, b), + binfo.getStorageUuid()); + } } if (error) { throw new IOException("Invalidate: Missing blocks."); @@ -959,8 +970,9 @@ public synchronized LengthInputStream getMetaDataInputStream(ExtendedBlock b } @Override - public void checkDataDir() throws DiskErrorException { + public Set checkDataDir() { // nothing to check for simulated data set + return null; } @Override // FsDatasetSpi @@ -1214,7 +1226,7 @@ public void enableTrash(String bpid) { } @Override - public void restoreTrash(String bpid) { + public void clearTrash(String bpid) { } @Override @@ -1281,7 +1293,7 @@ public FsVolumeSpi getVolume(ExtendedBlock b) { } @Override - public synchronized void removeVolumes(Collection volumes) { + public synchronized void removeVolumes(Set volumes, boolean clearFailure) { throw new UnsupportedOperationException(); } @@ -1318,5 +1330,10 @@ public void setPinning(ExtendedBlock b) throws IOException { public boolean getPinning(ExtendedBlock b) throws IOException { return blockMap.get(b.getBlockPoolId()).get(b.getLocalBlock()).pinned; } + + @Override + public boolean isDeletingBlock(String bpid, long blockId) { + throw new UnsupportedOperationException(); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java index bc497934f255e..3aa9a7b7b69a6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java @@ -43,6 +43,7 @@ import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi; import org.apache.hadoop.hdfs.server.datanode.metrics.DataNodeMetrics; import org.apache.hadoop.hdfs.server.protocol.BlockCommand; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; @@ -216,7 +217,8 @@ public void testIgnoreDeletionsFromNonActive() throws Exception { .when(mockNN2).blockReport( Mockito.anyObject(), Mockito.eq(FAKE_BPID), - Mockito.anyObject()); + Mockito.anyObject(), + Mockito.anyObject()); bpos.start(); try { @@ -406,7 +408,8 @@ public Boolean get() { Mockito.verify(mockNN).blockReport( Mockito.anyObject(), Mockito.eq(FAKE_BPID), - Mockito.anyObject()); + Mockito.anyObject(), + Mockito.anyObject()); return true; } catch (Throwable t) { LOG.info("waiting on block report: " + t.getMessage()); @@ -431,7 +434,8 @@ private Boolean get(DatanodeProtocolClientSideTranslatorPB mockNN) { Mockito.verify(mockNN).blockReport( Mockito.anyObject(), Mockito.eq(FAKE_BPID), - Mockito.anyObject()); + Mockito.anyObject(), + Mockito.anyObject()); return true; } catch (Throwable t) { LOG.info("waiting on block report: " + t.getMessage()); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockHasMultipleReplicasOnSameDN.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockHasMultipleReplicasOnSameDN.java index 1152c74477c47..c47209e9eead6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockHasMultipleReplicasOnSameDN.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockHasMultipleReplicasOnSameDN.java @@ -28,6 +28,7 @@ import org.apache.hadoop.hdfs.*; import org.apache.hadoop.hdfs.protocol.*; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; @@ -107,21 +108,23 @@ public void testBlockHasMultipleReplicasOnSameDN() throws IOException { StorageBlockReport reports[] = new StorageBlockReport[cluster.getStoragesPerDatanode()]; - ArrayList blocks = new ArrayList(); + ArrayList blocks = new ArrayList(); for (LocatedBlock locatedBlock : locatedBlocks.getLocatedBlocks()) { - blocks.add(locatedBlock.getBlock().getLocalBlock()); + Block localBlock = locatedBlock.getBlock().getLocalBlock(); + blocks.add(new FinalizedReplica(localBlock, null, null)); } + BlockListAsLongs bll = BlockListAsLongs.encode(blocks); for (int i = 0; i < cluster.getStoragesPerDatanode(); ++i) { - BlockListAsLongs bll = new BlockListAsLongs(blocks); FsVolumeSpi v = dn.getFSDataset().getVolumes().get(i); DatanodeStorage dns = new DatanodeStorage(v.getStorageID()); - reports[i] = new StorageBlockReport(dns, bll.getBlockListAsLongs()); + reports[i] = new StorageBlockReport(dns, bll); } // Should not assert! - cluster.getNameNodeRpc().blockReport(dnReg, bpid, reports); + cluster.getNameNodeRpc().blockReport(dnReg, bpid, reports, + new BlockReportContext(1, 0, System.nanoTime())); // Get the block locations once again. locatedBlocks = client.getLocatedBlocks(filename, 0, BLOCK_SIZE * NUM_BLOCKS); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java index e566f2a983307..f91c0bc3cce48 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java @@ -177,7 +177,7 @@ DatanodeProtocolClientSideTranslatorPB connectToNN( } }; // Trigger a heartbeat so that it acknowledges the NN as active. - dn.getAllBpOs()[0].triggerHeartbeatForTests(); + dn.getAllBpOs().get(0).triggerHeartbeatForTests(); } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java index 1f2c1b7bbe9a1..2c4795a5368c5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java @@ -73,7 +73,7 @@ public void testThrottler() throws IOException { long bandwidthPerSec = 1024*1024L; final long TOTAL_BYTES =6*bandwidthPerSec; long bytesToSend = TOTAL_BYTES; - long start = Time.now(); + long start = Time.monotonicNow(); DataTransferThrottler throttler = new DataTransferThrottler(bandwidthPerSec); long totalBytes = 0L; long bytesSent = 1024*512L; // 0.5MB @@ -86,7 +86,7 @@ public void testThrottler() throws IOException { Thread.sleep(1000); } catch (InterruptedException ignored) {} throttler.throttle(bytesToSend); - long end = Time.now(); + long end = Time.monotonicNow(); assertTrue(totalBytes*1000/(end-start)<=bandwidthPerSec); } @@ -254,7 +254,7 @@ private void checkBlocks(DatanodeInfo[] includeNodes, String fileName, throws IOException, TimeoutException { boolean notDone; final long TIMEOUT = 20000L; - long starttime = Time.now(); + long starttime = Time.monotonicNow(); long failtime = starttime + TIMEOUT; do { try { @@ -279,7 +279,7 @@ private void checkBlocks(DatanodeInfo[] includeNodes, String fileName, } } } - if (Time.now() > failtime) { + if (Time.monotonicNow() > failtime) { String expectedNodesList = ""; String currentNodesList = ""; for (DatanodeInfo dn : includeNodes) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java index 735e9a12b0bf3..14e29e4db7342 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java @@ -162,8 +162,8 @@ private void testVolumeIteratorImpl(int numFiles, boolean testedRewind = false, testedSave = false, testedLoad = false; int blocksProcessed = 0, savedBlocksProcessed = 0; try { - BPOfferService bpos[] = ctx.datanode.getAllBpOs(); - assertEquals(1, bpos.length); + List bpos = ctx.datanode.getAllBpOs(); + assertEquals(1, bpos.size()); BlockIterator iter = volume.newBlockIterator(ctx.bpids[0], "test"); assertEquals(ctx.bpids[0], iter.getBlockPoolId()); iter.setMaxStalenessMs(maxStaleness); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeExit.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeExit.java index c067b07ae67db..da98d26cd5fdf 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeExit.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeExit.java @@ -24,6 +24,7 @@ import static org.junit.Assert.fail; import java.io.IOException; +import java.util.List; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdfs.DFSConfigKeys; @@ -64,11 +65,11 @@ public void tearDown() throws Exception { private void stopBPServiceThreads(int numStopThreads, DataNode dn) throws Exception { - BPOfferService[] bpoList = dn.getAllBpOs(); + List bpoList = dn.getAllBpOs(); int expected = dn.getBpOsCount() - numStopThreads; int index = numStopThreads - 1; while (index >= 0) { - bpoList[index--].stop(); + bpoList.get(index--).stop(); } int iterations = 3000; // Total 30 seconds MAX wait time while(dn.getBpOsCount() != expected && iterations > 0) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java index 3d0bccc47add1..f5772e3a02caa 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java @@ -34,21 +34,22 @@ import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.MiniDFSNNTopology; import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; +import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetTestUtil; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; +import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; -import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; import org.apache.hadoop.test.GenericTestUtils; import org.junit.After; import org.junit.Test; import java.io.File; import java.io.IOException; -import java.io.RandomAccessFile; -import java.nio.channels.FileChannel; -import java.nio.channels.FileLock; -import java.nio.channels.OverlappingFileLockException; import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; @@ -62,6 +63,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.mockito.Mockito; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY; import static org.hamcrest.CoreMatchers.anyOf; @@ -70,10 +72,12 @@ import static org.hamcrest.core.Is.is; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertThat; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.timeout; public class TestDataNodeHotSwapVolumes { private static final Log LOG = LogFactory.getLog( @@ -100,6 +104,8 @@ private void startDFSCluster(int numNameNodes, int numDataNodes) conf.setInt(DFSConfigKeys.DFS_DF_INTERVAL_KEY, 1000); conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 1000); + /* Allow 1 volume failure */ + conf.setInt(DFSConfigKeys.DFS_DATANODE_FAILED_VOLUMES_TOLERATED_KEY, 1); MiniDFSNNTopology nnTopology = MiniDFSNNTopology.simpleFederatedTopology(numNameNodes); @@ -522,7 +528,7 @@ public void testAddVolumeFailures() throws IOException { dn.getConf().get(DFS_DATANODE_DATA_DIR_KEY).split(","); assertEquals(4, effectiveVolumes.length); for (String ev : effectiveVolumes) { - assertThat(new File(ev).getCanonicalPath(), + assertThat(StorageLocation.parse(ev).getFile().getCanonicalPath(), is(not(anyOf(is(newDirs.get(0)), is(newDirs.get(2)))))); } } @@ -538,31 +544,10 @@ public void testAddVolumeFailures() throws IOException { private static void assertFileLocksReleased(Collection dirs) throws IOException { for (String dir: dirs) { - StorageLocation sl = StorageLocation.parse(dir); - File lockFile = new File(sl.getFile(), Storage.STORAGE_FILE_LOCK); - RandomAccessFile raf = null; - FileChannel channel = null; - FileLock lock = null; try { - raf = new RandomAccessFile(lockFile, "rws"); - channel = raf.getChannel(); - lock = channel.tryLock(); - assertNotNull(String.format( - "Lock file at %s appears to be held by a different process.", - lockFile.getAbsolutePath()), lock); - } catch (OverlappingFileLockException e) { - fail(String.format("Must release lock file at %s.", - lockFile.getAbsolutePath())); - } finally { - if (lock != null) { - try { - lock.release(); - } catch (IOException e) { - LOG.warn(String.format("I/O error releasing file lock %s.", - lockFile.getAbsolutePath()), e); - } - } - IOUtils.cleanup(null, channel, raf); + FsDatasetTestUtil.assertFileLockReleased(dir); + } catch (IOException e) { + LOG.warn(e); } } } @@ -672,4 +657,91 @@ public void testAddBackRemovedVolume() // this directory were removed from the previous step. dn.reconfigurePropertyImpl(DFS_DATANODE_DATA_DIR_KEY, oldDataDir); } + + /** Get the FsVolume on the given basePath */ + private FsVolumeImpl getVolume(DataNode dn, File basePath) { + for (FsVolumeSpi vol : dn.getFSDataset().getVolumes()) { + if (vol.getBasePath().equals(basePath.getPath())) { + return (FsVolumeImpl)vol; + } + } + return null; + } + + /** + * Verify that {@link DataNode#checkDiskErrors()} removes all metadata in + * DataNode upon a volume failure. Thus we can run reconfig on the same + * configuration to reload the new volume on the same directory as the failed one. + */ + @Test(timeout=60000) + public void testDirectlyReloadAfterCheckDiskError() + throws IOException, TimeoutException, InterruptedException, + ReconfigurationException { + startDFSCluster(1, 2); + createFile(new Path("/test"), 32, (short)2); + + DataNode dn = cluster.getDataNodes().get(0); + final String oldDataDir = dn.getConf().get(DFS_DATANODE_DATA_DIR_KEY); + File dirToFail = new File(cluster.getDataDirectory(), "data1"); + + FsVolumeImpl failedVolume = getVolume(dn, dirToFail); + assertTrue("No FsVolume was found for " + dirToFail, + failedVolume != null); + long used = failedVolume.getDfsUsed(); + + DataNodeTestUtils.injectDataDirFailure(dirToFail); + // Call and wait DataNode to detect disk failure. + long lastDiskErrorCheck = dn.getLastDiskErrorCheck(); + dn.checkDiskErrorAsync(); + while (dn.getLastDiskErrorCheck() == lastDiskErrorCheck) { + Thread.sleep(100); + } + + createFile(new Path("/test1"), 32, (short)2); + assertEquals(used, failedVolume.getDfsUsed()); + + DataNodeTestUtils.restoreDataDirFromFailure(dirToFail); + dn.reconfigurePropertyImpl(DFS_DATANODE_DATA_DIR_KEY, oldDataDir); + + createFile(new Path("/test2"), 32, (short)2); + FsVolumeImpl restoredVolume = getVolume(dn, dirToFail); + assertTrue(restoredVolume != null); + assertTrue(restoredVolume != failedVolume); + // More data has been written to this volume. + assertTrue(restoredVolume.getDfsUsed() > used); + } + + /** Test that a full block report is sent after hot swapping volumes */ + @Test(timeout=100000) + public void testFullBlockReportAfterRemovingVolumes() + throws IOException, ReconfigurationException { + + Configuration conf = new Configuration(); + conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCK_SIZE); + + // Similar to TestTriggerBlockReport, set a really long value for + // dfs.heartbeat.interval, so that incremental block reports and heartbeats + // won't be sent during this test unless they're triggered + // manually. + conf.setLong(DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_KEY, 10800000L); + conf.setLong(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1080L); + + cluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).build(); + cluster.waitActive(); + + final DataNode dn = cluster.getDataNodes().get(0); + DatanodeProtocolClientSideTranslatorPB spy = + DataNodeTestUtils.spyOnBposToNN(dn, cluster.getNameNode()); + + // Remove a data dir from datanode + File dataDirToKeep = new File(cluster.getDataDirectory(), "data1"); + dn.reconfigurePropertyImpl(DFS_DATANODE_DATA_DIR_KEY, dataDirToKeep.toString()); + + // We should get 1 full report + Mockito.verify(spy, timeout(60000).times(1)).blockReport( + any(DatanodeRegistration.class), + anyString(), + any(StorageBlockReport[].class), + any(BlockReportContext.class)); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java index 8a2bacf90f170..5d27fe67e8a7c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java @@ -72,6 +72,8 @@ public void testDataNodeMetrics() throws Exception { DataNode datanode = datanodes.get(0); MetricsRecordBuilder rb = getMetrics(datanode.getMetrics().name()); assertCounter("BytesWritten", LONG_FILE_LEN, rb); + assertTrue("Expected non-zero number of incremental block reports", + getLongCounter("IncrementalBlockReportsNumOps", rb) > 0); } finally { if (cluster != null) {cluster.shutdown();} } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMultipleRegistrations.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMultipleRegistrations.java index 68881347e16b8..55c7b73dba2c1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMultipleRegistrations.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMultipleRegistrations.java @@ -103,8 +103,8 @@ public void test2NNRegistration() throws IOException { LOG.info("BP: " + bpos); } - BPOfferService bpos1 = dn.getAllBpOs()[0]; - BPOfferService bpos2 = dn.getAllBpOs()[1]; + BPOfferService bpos1 = dn.getAllBpOs().get(0); + BPOfferService bpos2 = dn.getAllBpOs().get(1); // The order of bpos is not guaranteed, so fix the order if (getNNSocketAddress(bpos1).equals(nn2.getNameNodeAddress())) { @@ -173,7 +173,7 @@ public void testFedSingleNN() throws IOException { } // try block report - BPOfferService bpos1 = dn.getAllBpOs()[0]; + BPOfferService bpos1 = dn.getAllBpOs().get(0); bpos1.triggerBlockReportForTests(); assertEquals("wrong nn address", @@ -184,7 +184,7 @@ public void testFedSingleNN() throws IOException { cluster.shutdown(); // Ensure all the BPOfferService threads are shutdown - assertEquals(0, dn.getAllBpOs().length); + assertEquals(0, dn.getAllBpOs().size()); cluster = null; } finally { if (cluster != null) { @@ -202,16 +202,16 @@ public void testClusterIdMismatch() throws Exception { cluster.waitActive(); DataNode dn = cluster.getDataNodes().get(0); - BPOfferService [] bposs = dn.getAllBpOs(); - LOG.info("dn bpos len (should be 2):" + bposs.length); - Assert.assertEquals("should've registered with two namenodes", bposs.length,2); + List bposs = dn.getAllBpOs(); + LOG.info("dn bpos len (should be 2):" + bposs.size()); + Assert.assertEquals("should've registered with two namenodes", bposs.size(),2); // add another namenode cluster.addNameNode(conf, 9938); Thread.sleep(500);// lets wait for the registration to happen bposs = dn.getAllBpOs(); - LOG.info("dn bpos len (should be 3):" + bposs.length); - Assert.assertEquals("should've registered with three namenodes", bposs.length,3); + LOG.info("dn bpos len (should be 3):" + bposs.size()); + Assert.assertEquals("should've registered with three namenodes", bposs.size(),3); // change cluster id and another Namenode StartupOption.FORMAT.setClusterId("DifferentCID"); @@ -221,8 +221,8 @@ public void testClusterIdMismatch() throws Exception { Thread.sleep(500);// lets wait for the registration to happen bposs = dn.getAllBpOs(); - LOG.info("dn bpos len (still should be 3):" + bposs.length); - Assert.assertEquals("should've registered with three namenodes", 3, bposs.length); + LOG.info("dn bpos len (still should be 3):" + bposs.size()); + Assert.assertEquals("should've registered with three namenodes", 3, bposs.size()); } finally { cluster.shutdown(); } @@ -250,7 +250,7 @@ public void testClusterIdMismatchAtStartupWithHA() throws Exception { DataNode dn = cluster.getDataNodes().get(0); assertTrue("Datanode should be running", dn.isDatanodeUp()); assertEquals("Only one BPOfferService should be running", 1, - dn.getAllBpOs().length); + dn.getAllBpOs().size()); } finally { cluster.shutdown(); } @@ -274,7 +274,7 @@ public void testDNWithInvalidStorageWithHA() throws Exception { DataNode dn = cluster.getDataNodes().get(0); assertTrue("Datanode should be running", dn.isDatanodeUp()); assertEquals("BPOfferService should be running", 1, - dn.getAllBpOs().length); + dn.getAllBpOs().size()); DataNodeProperties dnProp = cluster.stopDataNode(0); cluster.getNameNode(0).stop(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java index 7fd8398a28a10..57fee06f254e7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java @@ -19,12 +19,7 @@ package org.apache.hadoop.hdfs.server.datanode; import static org.hamcrest.core.Is.is; -import static org.junit.Assert.assertArrayEquals; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertThat; -import static org.junit.Assert.assertTrue; +import static org.junit.Assert.*; import java.io.File; import java.io.IOException; @@ -43,7 +38,9 @@ import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.MiniDFSCluster.Builder; +import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties; import org.apache.hadoop.hdfs.TestRollingUpgrade; +import org.apache.hadoop.hdfs.client.BlockReportOptions; import org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.LocatedBlocks; @@ -208,29 +205,53 @@ private void rollbackRollingUpgrade() throws Exception { public void testDatanodeRollingUpgradeWithFinalize() throws Exception { try { startCluster(); + rollingUpgradeAndFinalize(); + // Do it again + rollingUpgradeAndFinalize(); + } finally { + shutdownCluster(); + } + } - // Create files in DFS. - Path testFile1 = new Path("/" + GenericTestUtils.getMethodName() + ".01.dat"); - Path testFile2 = new Path("/" + GenericTestUtils.getMethodName() + ".02.dat"); - DFSTestUtil.createFile(fs, testFile1, FILE_SIZE, REPL_FACTOR, SEED); - DFSTestUtil.createFile(fs, testFile2, FILE_SIZE, REPL_FACTOR, SEED); - - startRollingUpgrade(); - File blockFile = getBlockForFile(testFile2, true); - File trashFile = getTrashFileForBlock(blockFile, false); - deleteAndEnsureInTrash(testFile2, blockFile, trashFile); - finalizeRollingUpgrade(); - - // Ensure that delete file testFile2 stays deleted after finalize - assertFalse(isTrashRootPresent()); - assert(!fs.exists(testFile2)); - assert(fs.exists(testFile1)); - + @Test(timeout = 600000) + public void testDatanodeRUwithRegularUpgrade() throws Exception { + try { + startCluster(); + rollingUpgradeAndFinalize(); + DataNodeProperties dn = cluster.stopDataNode(0); + cluster.restartNameNode(0, true, "-upgrade"); + cluster.restartDataNode(dn, true); + cluster.waitActive(); + fs = cluster.getFileSystem(0); + Path testFile3 = new Path("/" + GenericTestUtils.getMethodName() + + ".03.dat"); + DFSTestUtil.createFile(fs, testFile3, FILE_SIZE, REPL_FACTOR, SEED); + cluster.getFileSystem().finalizeUpgrade(); } finally { shutdownCluster(); } } + private void rollingUpgradeAndFinalize() throws IOException, Exception { + // Create files in DFS. + Path testFile1 = new Path("/" + GenericTestUtils.getMethodName() + ".01.dat"); + Path testFile2 = new Path("/" + GenericTestUtils.getMethodName() + ".02.dat"); + DFSTestUtil.createFile(fs, testFile1, FILE_SIZE, REPL_FACTOR, SEED); + DFSTestUtil.createFile(fs, testFile2, FILE_SIZE, REPL_FACTOR, SEED); + + startRollingUpgrade(); + File blockFile = getBlockForFile(testFile2, true); + File trashFile = getTrashFileForBlock(blockFile, false); + cluster.triggerBlockReports(); + deleteAndEnsureInTrash(testFile2, blockFile, trashFile); + finalizeRollingUpgrade(); + + // Ensure that delete file testFile2 stays deleted after finalize + assertFalse(isTrashRootPresent()); + assert(!fs.exists(testFile2)); + assert(fs.exists(testFile1)); + } + @Test (timeout=600000) public void testDatanodeRollingUpgradeWithRollback() throws Exception { try { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java index 6b9c4b18deb9f..41e8d7b45bb04 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java @@ -18,19 +18,20 @@ package org.apache.hadoop.hdfs.server.datanode; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; import static org.junit.Assume.assumeTrue; import java.io.File; -import java.io.FilenameFilter; import java.io.IOException; import java.net.InetSocketAddress; import java.net.Socket; -import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.TimeoutException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; @@ -57,7 +58,12 @@ import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager; import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants; +import org.apache.hadoop.hdfs.server.common.Storage; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi; +import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetTestUtil; import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.DatanodeStorage; import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocols; @@ -116,10 +122,6 @@ public void tearDown() throws Exception { if(cluster != null) { cluster.shutdown(); } - for (int i = 0; i < 3; i++) { - FileUtil.setExecutable(new File(dataDir, "data"+(2*i+1)), true); - FileUtil.setExecutable(new File(dataDir, "data"+(2*i+2)), true); - } } /* @@ -154,7 +156,7 @@ public void testVolumeFailure() throws Exception { !deteteBlocks(failedDir) ) { throw new IOException("Could not delete hdfs directory '" + failedDir + "'"); - } + } data_fail.setReadOnly(); failedDir.setReadOnly(); System.out.println("Deleteing " + failedDir.getPath() + "; exist=" + failedDir.exists()); @@ -181,10 +183,10 @@ public void testVolumeFailure() throws Exception { DatanodeStorage dnStorage = kvPair.getKey(); BlockListAsLongs blockList = kvPair.getValue(); reports[reportIndex++] = - new StorageBlockReport(dnStorage, blockList.getBlockListAsLongs()); + new StorageBlockReport(dnStorage, blockList); } - cluster.getNameNodeRpc().blockReport(dnR, bpid, reports); + cluster.getNameNodeRpc().blockReport(dnR, bpid, reports, null); // verify number of blocks and files... verify(filename, filesize); @@ -200,6 +202,69 @@ public void testVolumeFailure() throws Exception { " is created and replicated"); } + /** + * Test that DataStorage and BlockPoolSliceStorage remove the failed volume + * after failure. + */ + @Test(timeout=150000) + public void testFailedVolumeBeingRemovedFromDataNode() + throws InterruptedException, IOException, TimeoutException { + Path file1 = new Path("/test1"); + DFSTestUtil.createFile(fs, file1, 1024, (short) 2, 1L); + DFSTestUtil.waitReplication(fs, file1, (short) 2); + + File dn0Vol1 = new File(dataDir, "data" + (2 * 0 + 1)); + DataNodeTestUtils.injectDataDirFailure(dn0Vol1); + DataNode dn0 = cluster.getDataNodes().get(0); + long lastDiskErrorCheck = dn0.getLastDiskErrorCheck(); + dn0.checkDiskErrorAsync(); + // Wait checkDiskError thread finish to discover volume failure. + while (dn0.getLastDiskErrorCheck() == lastDiskErrorCheck) { + Thread.sleep(100); + } + + // Verify dn0Vol1 has been completely removed from DN0. + // 1. dn0Vol1 is removed from DataStorage. + DataStorage storage = dn0.getStorage(); + assertEquals(1, storage.getNumStorageDirs()); + for (int i = 0; i < storage.getNumStorageDirs(); i++) { + Storage.StorageDirectory sd = storage.getStorageDir(i); + assertFalse(sd.getRoot().getAbsolutePath().startsWith( + dn0Vol1.getAbsolutePath() + )); + } + final String bpid = cluster.getNamesystem().getBlockPoolId(); + BlockPoolSliceStorage bpsStorage = storage.getBPStorage(bpid); + assertEquals(1, bpsStorage.getNumStorageDirs()); + for (int i = 0; i < bpsStorage.getNumStorageDirs(); i++) { + Storage.StorageDirectory sd = bpsStorage.getStorageDir(i); + assertFalse(sd.getRoot().getAbsolutePath().startsWith( + dn0Vol1.getAbsolutePath() + )); + } + + // 2. dn0Vol1 is removed from FsDataset + FsDatasetSpi data = dn0.getFSDataset(); + for (FsVolumeSpi volume : data.getVolumes()) { + assertNotEquals(new File(volume.getBasePath()).getAbsoluteFile(), + dn0Vol1.getAbsoluteFile()); + } + + // 3. all blocks on dn0Vol1 have been removed. + for (ReplicaInfo replica : FsDatasetTestUtil.getReplicas(data, bpid)) { + assertNotNull(replica.getVolume()); + assertNotEquals( + new File(replica.getVolume().getBasePath()).getAbsoluteFile(), + dn0Vol1.getAbsoluteFile()); + } + + // 4. dn0Vol1 is not in DN0's configuration and dataDirs anymore. + String[] dataDirStrs = + dn0.getConf().get(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY).split(","); + assertEquals(1, dataDirStrs.length); + assertFalse(dataDirStrs[0].contains(dn0Vol1.getAbsolutePath())); + } + /** * Test that there are under replication blocks after vol failures */ @@ -223,8 +288,7 @@ public void testUnderReplicationAfterVolFailure() throws Exception { // Fail the first volume on both datanodes File dn1Vol1 = new File(dataDir, "data"+(2*0+1)); File dn2Vol1 = new File(dataDir, "data"+(2*1+1)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol1, false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, false)); + DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1); Path file2 = new Path("/test2"); DFSTestUtil.createFile(fs, file2, 1024, (short)3, 1L); @@ -319,7 +383,7 @@ private void triggerFailure(String path, long size) throws IOException { private boolean deteteBlocks(File dir) { File [] fileList = dir.listFiles(); for(File f : fileList) { - if(f.getName().startsWith("blk_")) { + if(f.getName().startsWith(Block.BLOCK_FILE_PREFIX)) { if(!f.delete()) return false; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java index a8f799052cf78..aac288aedd7b3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java @@ -34,7 +34,6 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.ReconfigurationException; import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.DFSTestUtil; @@ -87,20 +86,10 @@ public void setUp() throws Exception { @After public void tearDown() throws Exception { - // Restore executable permission on all directories where a failure may have - // been simulated by denying execute access. This is based on the maximum - // number of datanodes and the maximum number of storages per data node used - // throughout the tests in this suite. - int maxDataNodes = 3; - int maxStoragesPerDataNode = 4; - for (int i = 0; i < maxDataNodes; i++) { - for (int j = 1; j <= maxStoragesPerDataNode; j++) { - String subDir = "data" + ((i * maxStoragesPerDataNode) + j); - FileUtil.setExecutable(new File(dataDir, subDir), true); - } - } IOUtils.cleanup(LOG, fs); - cluster.shutdown(); + if (cluster != null) { + cluster.shutdown(); + } } /** @@ -138,8 +127,7 @@ public void testSuccessiveVolumeFailures() throws Exception { * fail. The client does not retry failed nodes even though * perhaps they could succeed because just a single volume failed. */ - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol1, false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, false)); + DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1); /* * Create file1 and wait for 3 replicas (ie all DNs can still @@ -176,7 +164,7 @@ public void testSuccessiveVolumeFailures() throws Exception { * Now fail a volume on the third datanode. We should be able to get * three replicas since we've already identified the other failures. */ - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn3Vol1, false)); + DataNodeTestUtils.injectDataDirFailure(dn3Vol1); Path file2 = new Path("/test2"); DFSTestUtil.createFile(fs, file2, 1024, (short)3, 1L); DFSTestUtil.waitReplication(fs, file2, (short)3); @@ -205,7 +193,7 @@ public void testSuccessiveVolumeFailures() throws Exception { * and that it's no longer up. Only wait for two replicas since * we'll never get a third. */ - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn3Vol2, false)); + DataNodeTestUtils.injectDataDirFailure(dn3Vol2); Path file3 = new Path("/test3"); DFSTestUtil.createFile(fs, file3, 1024, (short)3, 1L); DFSTestUtil.waitReplication(fs, file3, (short)2); @@ -230,10 +218,8 @@ public void testSuccessiveVolumeFailures() throws Exception { * restart, so file creation should be able to succeed after * restoring the data directories and restarting the datanodes. */ - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol1, true)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, true)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn3Vol1, true)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn3Vol2, true)); + DataNodeTestUtils.restoreDataDirFromFailure( + dn1Vol1, dn2Vol1, dn3Vol1, dn3Vol2); cluster.restartDataNodes(); cluster.waitActive(); Path file4 = new Path("/test4"); @@ -272,8 +258,7 @@ public void testVolFailureStatsPreservedOnNNRestart() throws Exception { // third healthy so one node in the pipeline will not fail). File dn1Vol1 = new File(dataDir, "data"+(2*0+1)); File dn2Vol1 = new File(dataDir, "data"+(2*1+1)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol1, false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, false)); + DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1); Path file1 = new Path("/test1"); DFSTestUtil.createFile(fs, file1, 1024, (short)2, 1L); @@ -320,14 +305,7 @@ public void testMultipleVolFailuresOnNode() throws Exception { // Make the first two volume directories on the first two datanodes // non-accessible. - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol1, - false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol2, - false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, - false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol2, - false)); + DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn1Vol2, dn2Vol1, dn2Vol2); // Create file1 and wait for 3 replicas (ie all DNs can still store a block). // Then assert that all DNs are up, despite the volume failures. @@ -377,8 +355,8 @@ public void testDataNodeReconfigureWithVolumeFailures() throws Exception { File dn1Vol2 = new File(dataDir, "data"+(2*0+2)); File dn2Vol1 = new File(dataDir, "data"+(2*1+1)); File dn2Vol2 = new File(dataDir, "data"+(2*1+2)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn1Vol1, false)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, false)); + DataNodeTestUtils.injectDataDirFailure(dn1Vol1); + DataNodeTestUtils.injectDataDirFailure(dn2Vol1); Path file1 = new Path("/test1"); DFSTestUtil.createFile(fs, file1, 1024, (short)2, 1L); @@ -403,23 +381,6 @@ public void testDataNodeReconfigureWithVolumeFailures() throws Exception { checkFailuresAtNameNode(dm, dns.get(0), true, dn1Vol1.getAbsolutePath()); checkFailuresAtNameNode(dm, dns.get(1), true, dn2Vol1.getAbsolutePath()); - // Reconfigure each DataNode to remove its failed volumes. - reconfigureDataNode(dns.get(0), dn1Vol2); - reconfigureDataNode(dns.get(1), dn2Vol2); - - DataNodeTestUtils.triggerHeartbeat(dns.get(0)); - DataNodeTestUtils.triggerHeartbeat(dns.get(1)); - - checkFailuresAtDataNode(dns.get(0), 1, true); - checkFailuresAtDataNode(dns.get(1), 1, true); - - // NN sees reduced capacity, but no volume failures. - DFSTestUtil.waitForDatanodeStatus(dm, 3, 0, 0, - origCapacity - (1*dnCapacity), WAIT_FOR_HEARTBEATS); - checkAggregateFailuresAtNameNode(true, 0); - checkFailuresAtNameNode(dm, dns.get(0), true); - checkFailuresAtNameNode(dm, dns.get(1), true); - // Reconfigure again to try to add back the failed volumes. reconfigureDataNode(dns.get(0), dn1Vol1, dn1Vol2); reconfigureDataNode(dns.get(1), dn2Vol1, dn2Vol2); @@ -460,6 +421,24 @@ public void testDataNodeReconfigureWithVolumeFailures() throws Exception { checkAggregateFailuresAtNameNode(false, 2); checkFailuresAtNameNode(dm, dns.get(0), false, dn1Vol1.getAbsolutePath()); checkFailuresAtNameNode(dm, dns.get(1), false, dn2Vol1.getAbsolutePath()); + + // Replace failed volume with healthy volume and run reconfigure DataNode. + // The failed volume information should be cleared. + DataNodeTestUtils.restoreDataDirFromFailure(dn1Vol1, dn2Vol1); + reconfigureDataNode(dns.get(0), dn1Vol1, dn1Vol2); + reconfigureDataNode(dns.get(1), dn2Vol1, dn2Vol2); + + DataNodeTestUtils.triggerHeartbeat(dns.get(0)); + DataNodeTestUtils.triggerHeartbeat(dns.get(1)); + + checkFailuresAtDataNode(dns.get(0), 1, true); + checkFailuresAtDataNode(dns.get(1), 1, true); + + DFSTestUtil.waitForDatanodeStatus(dm, 3, 0, 0, + origCapacity, WAIT_FOR_HEARTBEATS); + checkAggregateFailuresAtNameNode(true, 0); + checkFailuresAtNameNode(dm, dns.get(0), true); + checkFailuresAtNameNode(dm, dns.get(1), true); } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java index 73dc77c3b2fc8..5b7ac307c0182 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java @@ -76,10 +76,6 @@ public void setUp() throws Exception { @After public void tearDown() throws Exception { - for (int i = 0; i < 3; i++) { - FileUtil.setExecutable(new File(dataDir, "data"+(2*i+1)), true); - FileUtil.setExecutable(new File(dataDir, "data"+(2*i+2)), true); - } cluster.shutdown(); } @@ -152,7 +148,7 @@ public void testConfigureMinValidVolumes() throws Exception { // Fail a volume on the 2nd DN File dn2Vol1 = new File(dataDir, "data"+(2*1+1)); - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, false)); + DataNodeTestUtils.injectDataDirFailure(dn2Vol1); // Should only get two replicas (the first DN and the 3rd) Path file1 = new Path("/test1"); @@ -165,7 +161,7 @@ public void testConfigureMinValidVolumes() throws Exception { // If we restore the volume we should still only be able to get // two replicas since the DN is still considered dead. - assertTrue("Couldn't chmod local vol", FileUtil.setExecutable(dn2Vol1, true)); + DataNodeTestUtils.restoreDataDirFromFailure(dn2Vol1); Path file2 = new Path("/test2"); DFSTestUtil.createFile(fs, file2, 1024, (short)3, 1L); DFSTestUtil.waitReplication(fs, file2, (short)2); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java index da858cd662cee..bf80887313a85 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java @@ -42,6 +42,7 @@ import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.HeartbeatResponse; @@ -136,7 +137,8 @@ public Boolean get() { Mockito.verify(mockNN).blockReport( Mockito.eq(datanodeRegistration), Mockito.eq(POOL_ID), - Mockito.anyObject()); + Mockito.anyObject(), + Mockito.anyObject()); return true; } catch (Throwable t) { LOG.info("waiting on block report: " + t.getMessage()); @@ -173,7 +175,8 @@ public DatanodeRegistration answer(InvocationOnMock invocation) } else { DatanodeRegistration dr = (DatanodeRegistration) invocation.getArguments()[0]; - datanodeRegistration.setDatanodeUuidForTesting(dr.getDatanodeUuid()); + datanodeRegistration = + new DatanodeRegistration(dr.getDatanodeUuid(), dr); LOG.info("mockito succeeded " + datanodeRegistration); return datanodeRegistration; } @@ -226,7 +229,7 @@ DatanodeProtocolClientSideTranslatorPB connectToNN( }; // Trigger a heartbeat so that it acknowledges the NN as active. - dn.getAllBpOs()[0].triggerHeartbeatForTests(); + dn.getAllBpOs().get(0).triggerHeartbeatForTests(); waitForBlockReport(namenode); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDeleteBlockPool.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDeleteBlockPool.java index 755d49922c109..bc15f51a98bd6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDeleteBlockPool.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDeleteBlockPool.java @@ -83,7 +83,7 @@ public void testDeleteBlockPool() throws Exception { Configuration nn1Conf = cluster.getConfiguration(1); nn1Conf.set(DFSConfigKeys.DFS_NAMESERVICES, "namesServerId2"); dn1.refreshNamenodes(nn1Conf); - assertEquals(1, dn1.getAllBpOs().length); + assertEquals(1, dn1.getAllBpOs().size()); try { dn1.deleteBlockPool(bpid1, false); @@ -123,7 +123,7 @@ public void testDeleteBlockPool() throws Exception { } dn2.refreshNamenodes(nn1Conf); - assertEquals(1, dn2.getAllBpOs().length); + assertEquals(1, dn2.getAllBpOs().size()); verifyBlockPoolDirectories(true, dn2StorageDir1, bpid1); verifyBlockPoolDirectories(true, dn2StorageDir2, bpid1); @@ -184,7 +184,7 @@ public void testDfsAdminDeleteBlockPool() throws Exception { Configuration nn1Conf = cluster.getConfiguration(0); nn1Conf.set(DFSConfigKeys.DFS_NAMESERVICES, "namesServerId1"); dn1.refreshNamenodes(nn1Conf); - assertEquals(1, dn1.getAllBpOs().length); + assertEquals(1, dn1.getAllBpOs().size()); DFSAdmin admin = new DFSAdmin(nn1Conf); String dn1Address = dn1.getDatanodeId().getIpAddr() + ":" + dn1.getIpcPort(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDnRespectsBlockReportSplitThreshold.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDnRespectsBlockReportSplitThreshold.java index 7058d71f2e840..aadd9b2020e72 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDnRespectsBlockReportSplitThreshold.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDnRespectsBlockReportSplitThreshold.java @@ -29,6 +29,7 @@ import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; import org.apache.hadoop.hdfs.server.namenode.NameNode; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_SPLIT_THRESHOLD_KEY; @@ -98,7 +99,7 @@ private void verifyCapturedArguments( assertThat(reports.length, is(expectedReportsPerCall)); for (StorageBlockReport report : reports) { - BlockListAsLongs blockList = new BlockListAsLongs(report.getBlocks()); + BlockListAsLongs blockList = report.getBlocks(); numBlocksReported += blockList.getNumberOfBlocks(); } } @@ -133,7 +134,7 @@ public void testAlwaysSplit() throws IOException, InterruptedException { Mockito.verify(nnSpy, times(cluster.getStoragesPerDatanode())).blockReport( any(DatanodeRegistration.class), anyString(), - captor.capture()); + captor.capture(), Mockito.anyObject()); verifyCapturedArguments(captor, 1, BLOCKS_IN_FILE); } @@ -165,7 +166,7 @@ public void testCornerCaseUnderThreshold() throws IOException, InterruptedExcept Mockito.verify(nnSpy, times(1)).blockReport( any(DatanodeRegistration.class), anyString(), - captor.capture()); + captor.capture(), Mockito.anyObject()); verifyCapturedArguments(captor, cluster.getStoragesPerDatanode(), BLOCKS_IN_FILE); } @@ -197,7 +198,7 @@ public void testCornerCaseAtThreshold() throws IOException, InterruptedException Mockito.verify(nnSpy, times(cluster.getStoragesPerDatanode())).blockReport( any(DatanodeRegistration.class), anyString(), - captor.capture()); + captor.capture(), Mockito.anyObject()); verifyCapturedArguments(captor, 1, BLOCKS_IN_FILE); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java index b293075ddd3be..10f371bc3ec8e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java @@ -20,6 +20,7 @@ import static org.apache.hadoop.test.MetricsAsserts.assertCounter; import static org.apache.hadoop.test.MetricsAsserts.getMetrics; +import java.io.IOException; import java.util.EnumSet; import java.util.Random; @@ -30,6 +31,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.hdfs.AppendTestUtil; +import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.io.RandomDatum; @@ -51,15 +53,30 @@ private void checkSyncMetric(MiniDFSCluster cluster, long value) { /** Test basic hsync cases */ @Test public void testHSync() throws Exception { + testHSyncOperation(false); + } + + @Test + public void testHSyncWithAppend() throws Exception { + testHSyncOperation(true); + } + + private void testHSyncOperation(boolean testWithAppend) throws IOException { Configuration conf = new HdfsConfiguration(); MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build(); - final FileSystem fs = cluster.getFileSystem(); + final DistributedFileSystem fs = cluster.getFileSystem(); final Path p = new Path("/testHSync/foo"); final int len = 1 << 16; FSDataOutputStream out = fs.create(p, FsPermission.getDefault(), EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE, CreateFlag.SYNC_BLOCK), 4096, (short) 1, len, null); + if (testWithAppend) { + // re-open the file with append call + out.close(); + out = fs.append(p, EnumSet.of(CreateFlag.APPEND, CreateFlag.SYNC_BLOCK), + 4096, null); + } out.hflush(); // hflush does not sync checkSyncMetric(cluster, 0); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java index b5aa93f6e643e..f27a78e53cf80 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestIncrementalBlockReports.java @@ -69,7 +69,7 @@ public void startCluster() throws IOException { fs = cluster.getFileSystem(); singletonNn = cluster.getNameNode(); singletonDn = cluster.getDataNodes().get(0); - bpos = singletonDn.getAllBpOs()[0]; + bpos = singletonDn.getAllBpOs().get(0); actor = bpos.getBPServiceActors().get(0); storageUuid = singletonDn.getFSDataset().getVolumes().get(0).getStorageID(); } @@ -159,8 +159,8 @@ public void testReportBlockDeleted() throws InterruptedException, IOException { anyString(), any(StorageReceivedDeletedBlocks[].class)); - // Trigger a block report, this also triggers an IBR. - DataNodeTestUtils.triggerBlockReport(singletonDn); + // Trigger a heartbeat, this also triggers an IBR. + DataNodeTestUtils.triggerHeartbeat(singletonDn); Thread.sleep(2000); // Ensure that the deleted block is reported. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesBlockReportPerStorage.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesBlockReportPerStorage.java index 1b03786fa39b3..b150b0d3452a3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesBlockReportPerStorage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesBlockReportPerStorage.java @@ -20,8 +20,10 @@ import java.io.IOException; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; +import org.apache.hadoop.util.Time; /** @@ -33,10 +35,13 @@ public class TestNNHandlesBlockReportPerStorage extends BlockReportTestBase { @Override protected void sendBlockReports(DatanodeRegistration dnR, String poolId, StorageBlockReport[] reports) throws IOException { + int i = 0; for (StorageBlockReport report : reports) { LOG.info("Sending block report for storage " + report.getStorage().getStorageID()); StorageBlockReport[] singletonReport = { report }; - cluster.getNameNodeRpc().blockReport(dnR, poolId, singletonReport); + cluster.getNameNodeRpc().blockReport(dnR, poolId, singletonReport, + new BlockReportContext(reports.length, i, System.nanoTime())); + i++; } } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesCombinedBlockReport.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesCombinedBlockReport.java index 036b550c668e9..dca3c880bf116 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesCombinedBlockReport.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestNNHandlesCombinedBlockReport.java @@ -20,6 +20,7 @@ import java.io.IOException; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; @@ -34,6 +35,7 @@ public class TestNNHandlesCombinedBlockReport extends BlockReportTestBase { protected void sendBlockReports(DatanodeRegistration dnR, String poolId, StorageBlockReport[] reports) throws IOException { LOG.info("Sending combined block reports for " + dnR); - cluster.getNameNodeRpc().blockReport(dnR, poolId, reports); + cluster.getNameNodeRpc().blockReport(dnR, poolId, reports, + new BlockReportContext(1, 0, System.nanoTime())); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRefreshNamenodes.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRefreshNamenodes.java index ee610590cc28a..f8594ca7ebace 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRefreshNamenodes.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRefreshNamenodes.java @@ -59,13 +59,13 @@ public void testRefreshNamenodes() throws IOException { .build(); DataNode dn = cluster.getDataNodes().get(0); - assertEquals(1, dn.getAllBpOs().length); + assertEquals(1, dn.getAllBpOs().size()); cluster.addNameNode(conf, nnPort2); - assertEquals(2, dn.getAllBpOs().length); + assertEquals(2, dn.getAllBpOs().size()); cluster.addNameNode(conf, nnPort3); - assertEquals(3, dn.getAllBpOs().length); + assertEquals(3, dn.getAllBpOs().size()); cluster.addNameNode(conf, nnPort4); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java index efb9d980b35f6..c2348e38b8ad7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java @@ -32,6 +32,7 @@ import org.apache.hadoop.hdfs.client.BlockReportOptions; import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.ReceivedDeletedBlockInfo; import org.apache.hadoop.hdfs.server.protocol.ReceivedDeletedBlockInfo.BlockStatus; @@ -76,7 +77,8 @@ private void testTriggerBlockReport(boolean incremental) throws Exception { Mockito.verify(spy, times(0)).blockReport( any(DatanodeRegistration.class), anyString(), - any(StorageBlockReport[].class)); + any(StorageBlockReport[].class), + Mockito.anyObject()); Mockito.verify(spy, times(1)).blockReceivedAndDeleted( any(DatanodeRegistration.class), anyString(), @@ -89,7 +91,7 @@ private void testTriggerBlockReport(boolean incremental) throws Exception { new Block(5678, 512, 1000), BlockStatus.DELETED_BLOCK, null); DataNode datanode = cluster.getDataNodes().get(0); BPServiceActor actor = - datanode.getAllBpOs()[0].getBPServiceActors().get(0); + datanode.getAllBpOs().get(0).getBPServiceActors().get(0); String storageUuid = datanode.getFSDataset().getVolumes().get(0).getStorageID(); actor.notifyNamenodeDeletedBlock(rdbi, storageUuid); @@ -113,7 +115,8 @@ private void testTriggerBlockReport(boolean incremental) throws Exception { Mockito.verify(spy, timeout(60000)).blockReport( any(DatanodeRegistration.class), anyString(), - any(StorageBlockReport[].class)); + any(StorageBlockReport[].class), + Mockito.anyObject()); } cluster.shutdown(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java index a3c9935ca8140..2c6d868de6aa9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java @@ -61,8 +61,7 @@ public void addVolume(StorageLocation location, List nsInfos) thr } @Override - public void removeVolumes(Collection volumes) { - + public void removeVolumes(Set volumes, boolean clearFailure) { } @Override @@ -196,7 +195,7 @@ public Map getBlockReports(String bpid) { final Map result = new HashMap(); - result.put(storage, new BlockListAsLongs(null, null)); + result.put(storage, BlockListAsLongs.EMPTY); return result; } @@ -243,8 +242,8 @@ public boolean isCached(String bpid, long blockId) { } @Override - public void checkDataDir() throws DiskErrorException { - throw new DiskChecker.DiskErrorException(null); + public Set checkDataDir() { + return null; } @Override @@ -307,7 +306,7 @@ public void enableTrash(String bpid) { } @Override - public void restoreTrash(String bpid) { + public void clearTrash(String bpid) { } @@ -429,4 +428,9 @@ public void setPinning(ExtendedBlock block) throws IOException { public boolean getPinning(ExtendedBlock block) throws IOException { return false; } + + @Override + public boolean isDeletingBlock(String bpid, long blockId) { + return false; + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java index f9e30e12a0a41..7ac9b65db2a0d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java @@ -19,13 +19,24 @@ import java.io.File; import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.channels.FileChannel; +import java.nio.channels.FileLock; +import java.nio.channels.OverlappingFileLockException; import java.util.Collection; +import java.util.Random; import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; +import org.apache.hadoop.hdfs.server.common.Storage; import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.datanode.ReplicaInfo; +import org.apache.hadoop.hdfs.server.datanode.StorageLocation; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi; +import org.apache.hadoop.io.IOUtils; + +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.fail; public class FsDatasetTestUtil { @@ -72,4 +83,36 @@ public static void stopLazyWriter(DataNode dn) { FsDatasetImpl fsDataset = ((FsDatasetImpl) dn.getFSDataset()); ((FsDatasetImpl.LazyWriter) fsDataset.lazyWriter.getRunnable()).stop(); } + + /** + * Asserts that the storage lock file in the given directory has been + * released. This method works by trying to acquire the lock file itself. If + * locking fails here, then the main code must have failed to release it. + * + * @param dir the storage directory to check + * @throws IOException if there is an unexpected I/O error + */ + public static void assertFileLockReleased(String dir) throws IOException { + StorageLocation sl = StorageLocation.parse(dir); + File lockFile = new File(sl.getFile(), Storage.STORAGE_FILE_LOCK); + try (RandomAccessFile raf = new RandomAccessFile(lockFile, "rws"); + FileChannel channel = raf.getChannel()) { + FileLock lock = channel.tryLock(); + assertNotNull(String.format( + "Lock file at %s appears to be held by a different process.", + lockFile.getAbsolutePath()), lock); + if (lock != null) { + try { + lock.release(); + } catch (IOException e) { + FsDatasetImpl.LOG.warn(String.format("I/O error releasing file lock %s.", + lockFile.getAbsolutePath()), e); + throw e; + } + } + } catch (OverlappingFileLockException e) { + fail(String.format("Must release lock file at %s.", + lockFile.getAbsolutePath())); + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java index 8c28033b9eee9..56a4287d631b0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java @@ -18,10 +18,15 @@ package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl; import com.google.common.collect.Lists; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystemTestHelper; +import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.hdfs.HdfsConfiguration; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants; import org.apache.hadoop.hdfs.server.common.Storage; @@ -29,17 +34,22 @@ import org.apache.hadoop.hdfs.server.datanode.BlockScanner; import org.apache.hadoop.hdfs.server.datanode.DNConf; import org.apache.hadoop.hdfs.server.datanode.DataNode; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.hdfs.server.datanode.DataStorage; +import org.apache.hadoop.hdfs.server.datanode.FinalizedReplica; import org.apache.hadoop.hdfs.server.datanode.ReplicaHandler; +import org.apache.hadoop.hdfs.server.datanode.ReplicaInfo; import org.apache.hadoop.hdfs.server.datanode.StorageLocation; import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeReference; import org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy; import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo; +import org.apache.hadoop.io.MultipleIOException; import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.util.DiskChecker; import org.apache.hadoop.util.StringUtils; import org.junit.Before; import org.junit.Test; +import org.mockito.Matchers; import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; @@ -57,12 +67,17 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyList; import static org.mockito.Matchers.anyListOf; +import static org.mockito.Matchers.anyString; import static org.mockito.Matchers.eq; import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doReturn; import static org.mockito.Mockito.doThrow; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; @@ -81,6 +96,8 @@ public class TestFsDatasetImpl { private DataNode datanode; private DataStorage storage; private FsDatasetImpl dataset; + + private final static String BLOCKPOOL = "BP-TEST"; private static Storage.StorageDirectory createStorageDirectory(File root) { Storage.StorageDirectory sd = new Storage.StorageDirectory(root); @@ -95,7 +112,7 @@ private static void createStorageDirs(DataStorage storage, Configuration conf, List dirStrings = new ArrayList(); for (int i = 0; i < numDirs; i++) { File loc = new File(BASE_DIR + "/data" + i); - dirStrings.add(loc.toString()); + dirStrings.add(new Path(loc.toString()).toUri().toString()); loc.mkdirs(); dirs.add(createStorageDirectory(loc)); when(storage.getStorageDir(i)).thenReturn(dirs.get(i)); @@ -142,8 +159,9 @@ public void testAddVolumes() throws IOException { } for (int i = 0; i < numNewVolumes; i++) { String path = BASE_DIR + "/newData" + i; - expectedVolumes.add(path); - StorageLocation loc = StorageLocation.parse(path); + String pathUri = new Path(path).toUri().toString(); + expectedVolumes.add(new File(pathUri).toString()); + StorageLocation loc = StorageLocation.parse(pathUri); Storage.StorageDirectory sd = createStorageDirectory(new File(path)); DataStorage.VolumeBuilder builder = new DataStorage.VolumeBuilder(storage, sd); @@ -162,7 +180,8 @@ public void testAddVolumes() throws IOException { actualVolumes.add( dataset.getVolumes().get(numExistingVolumes + i).getBasePath()); } - assertEquals(actualVolumes, expectedVolumes); + assertEquals(actualVolumes.size(), expectedVolumes.size()); + assertTrue(actualVolumes.containsAll(expectedVolumes)); } @Test(timeout = 30000) @@ -179,10 +198,10 @@ public void testRemoveVolumes() throws IOException { final String[] dataDirs = conf.get(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY).split(","); final String volumePathToRemove = dataDirs[0]; - List volumesToRemove = new ArrayList(); - volumesToRemove.add(StorageLocation.parse(volumePathToRemove)); + Set volumesToRemove = new HashSet<>(); + volumesToRemove.add(StorageLocation.parse(volumePathToRemove).getFile()); - dataset.removeVolumes(volumesToRemove); + dataset.removeVolumes(volumesToRemove, true); int expectedNumVolumes = dataDirs.length - 1; assertEquals("The volume has been removed from the volumeList.", expectedNumVolumes, dataset.getVolumes().size()); @@ -190,7 +209,7 @@ public void testRemoveVolumes() throws IOException { expectedNumVolumes, dataset.storageMap.size()); try { - dataset.asyncDiskService.execute(volumesToRemove.get(0).getFile(), + dataset.asyncDiskService.execute(volumesToRemove.iterator().next(), new Runnable() { @Override public void run() {} @@ -232,8 +251,9 @@ public void testRemoveNewlyAddedVolume() throws IOException { when(storage.getNumStorageDirs()).thenReturn(numExistingVolumes + 1); when(storage.getStorageDir(numExistingVolumes)).thenReturn(sd); - List volumesToRemove = Arrays.asList(loc); - dataset.removeVolumes(volumesToRemove); + Set volumesToRemove = new HashSet<>(); + volumesToRemove.add(loc.getFile()); + dataset.removeVolumes(volumesToRemove, true); assertEquals(numExistingVolumes, dataset.getVolumes().size()); } @@ -262,12 +282,13 @@ public void testChangeVolumeWithRunningCheckDirs() throws IOException { final FsVolumeImpl newVolume = mock(FsVolumeImpl.class); final FsVolumeReference newRef = mock(FsVolumeReference.class); when(newRef.getVolume()).thenReturn(newVolume); + when(newVolume.getBasePath()).thenReturn("data4"); FsVolumeImpl blockedVolume = volumeList.getVolumes().get(1); doAnswer(new Answer() { @Override public Object answer(InvocationOnMock invocationOnMock) throws Throwable { - volumeList.removeVolume(new File("data4")); + volumeList.removeVolume(new File("data4"), false); volumeList.addVolume(newRef); return null; } @@ -291,4 +312,90 @@ public Object answer(InvocationOnMock invocationOnMock) assertFalse(volumeList.getVolumes().contains(brokenVolume)); assertEquals(NUM_VOLUMES - 1, volumeList.getVolumes().size()); } + + @Test + public void testAddVolumeFailureReleasesInUseLock() throws IOException { + FsDatasetImpl spyDataset = spy(dataset); + FsVolumeImpl mockVolume = mock(FsVolumeImpl.class); + File badDir = new File(BASE_DIR, "bad"); + badDir.mkdirs(); + doReturn(mockVolume).when(spyDataset) + .createFsVolume(anyString(), any(File.class), any(StorageType.class)); + doThrow(new IOException("Failed to getVolumeMap()")) + .when(mockVolume).getVolumeMap( + anyString(), + any(ReplicaMap.class), + any(RamDiskReplicaLruTracker.class)); + + Storage.StorageDirectory sd = createStorageDirectory(badDir); + sd.lock(); + DataStorage.VolumeBuilder builder = new DataStorage.VolumeBuilder(storage, sd); + when(storage.prepareVolume(eq(datanode), eq(badDir.getAbsoluteFile()), + Matchers.>any())) + .thenReturn(builder); + + StorageLocation location = StorageLocation.parse(badDir.toString()); + List nsInfos = Lists.newArrayList(); + for (String bpid : BLOCK_POOL_IDS) { + nsInfos.add(new NamespaceInfo(0, CLUSTER_ID, bpid, 1)); + } + + try { + spyDataset.addVolume(location, nsInfos); + fail("Expect to throw MultipleIOException"); + } catch (MultipleIOException e) { + } + + FsDatasetTestUtil.assertFileLockReleased(badDir.toString()); + } + + @Test + public void testDeletingBlocks() throws IOException { + MiniDFSCluster cluster = new MiniDFSCluster.Builder(new HdfsConfiguration()).build(); + try { + cluster.waitActive(); + DataNode dn = cluster.getDataNodes().get(0); + + FsDatasetImpl ds = (FsDatasetImpl) DataNodeTestUtils.getFSDataset(dn); + FsVolumeImpl vol = ds.getVolumes().get(0); + + ExtendedBlock eb; + ReplicaInfo info; + List blockList = new ArrayList(); + for (int i = 1; i <= 63; i++) { + eb = new ExtendedBlock(BLOCKPOOL, i, 1, 1000 + i); + info = new FinalizedReplica( + eb.getLocalBlock(), vol, vol.getCurrentDir().getParentFile()); + ds.volumeMap.add(BLOCKPOOL, info); + info.getBlockFile().createNewFile(); + info.getMetaFile().createNewFile(); + blockList.add(info); + } + ds.invalidate(BLOCKPOOL, blockList.toArray(new Block[0])); + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + // Nothing to do + } + assertTrue(ds.isDeletingBlock(BLOCKPOOL, blockList.get(0).getBlockId())); + + blockList.clear(); + eb = new ExtendedBlock(BLOCKPOOL, 64, 1, 1064); + info = new FinalizedReplica( + eb.getLocalBlock(), vol, vol.getCurrentDir().getParentFile()); + ds.volumeMap.add(BLOCKPOOL, info); + info.getBlockFile().createNewFile(); + info.getMetaFile().createNewFile(); + blockList.add(info); + ds.invalidate(BLOCKPOOL, blockList.toArray(new Block[0])); + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + // Nothing to do + } + assertFalse(ds.isDeletingBlock(BLOCKPOOL, blockList.get(0).getBlockId())); + } finally { + cluster.shutdown(); + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java index 9325cdcf733f5..96a73c64a133b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java @@ -17,14 +17,25 @@ */ package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; + +import java.io.File; import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hdfs.MiniDFSNNTopology; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; +import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.ReplicaState; import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; +import org.apache.hadoop.hdfs.server.datanode.DatanodeUtil; import org.apache.hadoop.hdfs.server.datanode.FinalizedReplica; import org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException; import org.apache.hadoop.hdfs.server.datanode.ReplicaBeingWritten; @@ -34,6 +45,7 @@ import org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException; import org.apache.hadoop.hdfs.server.datanode.ReplicaUnderRecovery; import org.apache.hadoop.hdfs.server.datanode.ReplicaWaitingToBeRecovered; +import org.apache.hadoop.hdfs.server.namenode.NameNode; import org.apache.hadoop.util.DiskChecker.DiskOutOfSpaceException; import org.junit.Assert; import org.junit.Test; @@ -501,4 +513,144 @@ private void testWriteToTemporary(FsDatasetImpl dataSet, ExtendedBlock[] blocks) + "genstamp and replaced it with the newer one: " + blocks[NON_EXISTENT]); } } + + /** + * This is a test to check the replica map before and after the datanode + * quick restart (less than 5 minutes) + * @throws Exception + */ + @Test + public void testReplicaMapAfterDatanodeRestart() throws Exception { + Configuration conf = new HdfsConfiguration(); + MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf) + .nnTopology(MiniDFSNNTopology.simpleFederatedTopology(2)) + .build(); + try { + cluster.waitActive(); + NameNode nn1 = cluster.getNameNode(0); + NameNode nn2 = cluster.getNameNode(1); + assertNotNull("cannot create nn1", nn1); + assertNotNull("cannot create nn2", nn2); + + // check number of volumes in fsdataset + DataNode dn = cluster.getDataNodes().get(0); + FsDatasetImpl dataSet = (FsDatasetImpl)DataNodeTestUtils. + getFSDataset(dn); + ReplicaMap replicaMap = dataSet.volumeMap; + + List volumes = dataSet.getVolumes(); + // number of volumes should be 2 - [data1, data2] + assertEquals("number of volumes is wrong", 2, volumes.size()); + ArrayList bpList = new ArrayList(Arrays.asList( + cluster.getNamesystem(0).getBlockPoolId(), + cluster.getNamesystem(1).getBlockPoolId())); + + Assert.assertTrue("Cluster should have 2 block pools", + bpList.size() == 2); + + createReplicas(bpList, volumes, replicaMap); + ReplicaMap oldReplicaMap = new ReplicaMap(this); + oldReplicaMap.addAll(replicaMap); + + cluster.restartDataNode(0); + cluster.waitActive(); + dn = cluster.getDataNodes().get(0); + dataSet = (FsDatasetImpl) dn.getFSDataset(); + testEqualityOfReplicaMap(oldReplicaMap, dataSet.volumeMap, bpList); + } finally { + cluster.shutdown(); + } + } + + /** + * Compare the replica map before and after the restart + **/ + private void testEqualityOfReplicaMap(ReplicaMap oldReplicaMap, ReplicaMap + newReplicaMap, List bpidList) { + // Traversing through newReplica map and remove the corresponding + // replicaInfo from oldReplicaMap. + for (String bpid: bpidList) { + for (ReplicaInfo info: newReplicaMap.replicas(bpid)) { + assertNotNull("Volume map before restart didn't contain the " + + "blockpool: " + bpid, oldReplicaMap.replicas(bpid)); + + ReplicaInfo oldReplicaInfo = oldReplicaMap.get(bpid, + info.getBlockId()); + // Volume map after restart contains a blockpool id which + assertNotNull("Old Replica Map didnt't contain block with blockId: " + + info.getBlockId(), oldReplicaInfo); + + ReplicaState oldState = oldReplicaInfo.getState(); + // Since after restart, all the RWR, RBW and RUR blocks gets + // converted to RWR + if (info.getState() == ReplicaState.RWR) { + if (oldState == ReplicaState.RWR || oldState == ReplicaState.RBW + || oldState == ReplicaState.RUR) { + oldReplicaMap.remove(bpid, oldReplicaInfo); + } + } else if (info.getState() == ReplicaState.FINALIZED && + oldState == ReplicaState.FINALIZED) { + oldReplicaMap.remove(bpid, oldReplicaInfo); + } + } + } + + // We don't persist the ReplicaInPipeline replica + // and if the old replica map contains any replica except ReplicaInPipeline + // then we didn't persist that replica + for (String bpid: bpidList) { + for (ReplicaInfo replicaInfo: oldReplicaMap.replicas(bpid)) { + if (replicaInfo.getState() != ReplicaState.TEMPORARY) { + Assert.fail("After datanode restart we lost the block with blockId: " + + replicaInfo.getBlockId()); + } + } + } + } + + private void createReplicas(List bpList, List volumes, + ReplicaMap volumeMap) throws IOException { + Assert.assertTrue("Volume map can't be null" , volumeMap != null); + + // Here we create all different type of replicas and add it + // to volume map. + // Created all type of ReplicaInfo, each under Blkpool corresponding volume + long id = 1; // This variable is used as both blockId and genStamp + for (String bpId: bpList) { + for (FsVolumeImpl volume: volumes) { + ReplicaInfo finalizedReplica = new FinalizedReplica(id, 1, id, volume, + DatanodeUtil.idToBlockDir(volume.getFinalizedDir(bpId), id)); + volumeMap.add(bpId, finalizedReplica); + id++; + + ReplicaInfo rbwReplica = new ReplicaBeingWritten(id, 1, id, volume, + volume.getRbwDir(bpId), null, 100); + volumeMap.add(bpId, rbwReplica); + id++; + + ReplicaInfo rwrReplica = new ReplicaWaitingToBeRecovered(id, 1, id, + volume, volume.getRbwDir(bpId)); + volumeMap.add(bpId, rwrReplica); + id++; + + ReplicaInfo ripReplica = new ReplicaInPipeline(id, id, volume, + volume.getTmpDir(bpId), 0); + volumeMap.add(bpId, ripReplica); + id++; + } + } + + for (String bpId: bpList) { + for (ReplicaInfo replicaInfo: volumeMap.replicas(bpId)) { + File parentFile = replicaInfo.getBlockFile().getParentFile(); + if (!parentFile.exists()) { + if (!parentFile.mkdirs()) { + throw new IOException("Failed to mkdirs " + parentFile); + } + } + replicaInfo.getBlockFile().createNewFile(); + replicaInfo.getMetaFile().createNewFile(); + } + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/TestParameterParser.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/TestParameterParser.java index 6a6c5d05d9b18..217d6b572d9b6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/TestParameterParser.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/TestParameterParser.java @@ -23,6 +23,7 @@ import org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier; import org.apache.hadoop.hdfs.web.resources.DelegationParam; import org.apache.hadoop.hdfs.web.resources.NamenodeAddressParam; +import org.apache.hadoop.hdfs.web.resources.OffsetParam; import org.apache.hadoop.security.token.Token; import org.junit.Assert; import org.junit.Test; @@ -55,14 +56,31 @@ public void testDeserializeHAToken() throws IOException { @Test public void testDecodePath() { - final String SCAPED_PATH = "hdfs-6662/test%25251%26%3Dtest?op=OPEN"; - final String EXPECTED_PATH = "/hdfs-6662/test%251&=test"; + final String ESCAPED_PATH = "/test%25+1%26%3Dtest?op=OPEN&foo=bar"; + final String EXPECTED_PATH = "/test%+1&=test"; - Configuration conf = DFSTestUtil.newHAConfiguration(LOGICAL_NAME); + Configuration conf = new Configuration(); QueryStringDecoder decoder = new QueryStringDecoder( - WebHdfsHandler.WEBHDFS_PREFIX + "/" - + SCAPED_PATH); + WebHdfsHandler.WEBHDFS_PREFIX + ESCAPED_PATH); ParameterParser testParser = new ParameterParser(decoder, conf); Assert.assertEquals(EXPECTED_PATH, testParser.path()); } + + @Test + public void testOffset() throws IOException { + final long X = 42; + + long offset = new OffsetParam(Long.toString(X)).getOffset(); + Assert.assertEquals("OffsetParam: ", X, offset); + + offset = new OffsetParam((String) null).getOffset(); + Assert.assertEquals("OffsetParam with null should have defaulted to 0", 0, offset); + + try { + offset = new OffsetParam("abc").getValue(); + Assert.fail("OffsetParam with nondigit value should have thrown IllegalArgumentException"); + } catch (IllegalArgumentException iae) { + // Ignore + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java index f481bc1c64743..002f7c003aba3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java @@ -1560,7 +1560,7 @@ public void testDeDuplication() throws Exception { //restart by loading fsimage cluster.getNameNodeRpc() .setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); cluster.getNameNodeRpc() .setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); cluster.restartNameNode(true); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java index c11abfcc6f529..9e24f7277ec4c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java @@ -26,6 +26,7 @@ import java.util.List; import com.google.common.base.Preconditions; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; @@ -36,6 +37,7 @@ import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs.BlockReportReplica; import org.apache.hadoop.hdfs.protocol.DatanodeID; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; @@ -46,6 +48,7 @@ import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.datanode.DataStorage; import org.apache.hadoop.hdfs.server.protocol.BlockCommand; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; @@ -891,9 +894,9 @@ private static class TinyDatanode implements Comparable { NamespaceInfo nsInfo; DatanodeRegistration dnRegistration; DatanodeStorage storage; //only one storage - final ArrayList blocks; + final ArrayList blocks; int nrBlocks; // actual number of blocks - long[] blockReportList; + BlockListAsLongs blockReportList; final int dnIdx; private static int getNodePort(int num) throws IOException { @@ -904,7 +907,7 @@ private static int getNodePort(int num) throws IOException { TinyDatanode(int dnIdx, int blockCapacity) throws IOException { this.dnIdx = dnIdx; - this.blocks = new ArrayList(blockCapacity); + this.blocks = new ArrayList(blockCapacity); this.nrBlocks = 0; } @@ -934,11 +937,11 @@ void register() throws IOException { //first block reports storage = new DatanodeStorage(DatanodeStorage.generateUuid()); final StorageBlockReport[] reports = { - new StorageBlockReport(storage, - new BlockListAsLongs(null, null).getBlockListAsLongs()) + new StorageBlockReport(storage, BlockListAsLongs.EMPTY) }; nameNodeProto.blockReport(dnRegistration, - nameNode.getNamesystem().getBlockPoolId(), reports); + nameNode.getNamesystem().getBlockPoolId(), reports, + new BlockReportContext(1, 0, System.nanoTime())); } /** @@ -968,19 +971,21 @@ boolean addBlock(Block blk) { } return false; } - blocks.set(nrBlocks, blk); + blocks.set(nrBlocks, new BlockReportReplica(blk)); nrBlocks++; return true; } void formBlockReport() { // fill remaining slots with blocks that do not exist - for(int idx = blocks.size()-1; idx >= nrBlocks; idx--) - blocks.set(idx, new Block(blocks.size() - idx, 0, 0)); - blockReportList = new BlockListAsLongs(blocks).getBlockListAsLongs(); + for (int idx = blocks.size()-1; idx >= nrBlocks; idx--) { + Block block = new Block(blocks.size() - idx, 0, 0); + blocks.set(idx, new BlockReportReplica(block)); + } + blockReportList = BlockListAsLongs.EMPTY; } - long[] getBlockReportList() { + BlockListAsLongs getBlockReportList() { return blockReportList; } @@ -1181,8 +1186,9 @@ long executeOp(int daemonId, int inputIdx, String ignore) throws IOException { long start = Time.now(); StorageBlockReport[] report = { new StorageBlockReport( dn.storage, dn.getBlockReportList()) }; - nameNodeProto.blockReport(dn.dnRegistration, nameNode.getNamesystem() - .getBlockPoolId(), report); + nameNodeProto.blockReport(dn.dnRegistration, + nameNode.getNamesystem().getBlockPoolId(), report, + new BlockReportContext(1, 0, System.nanoTime())); long end = Time.now(); return end-start; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java index fa23fbfe63a3e..25408347d406e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java @@ -83,7 +83,7 @@ public static boolean mkdirs(NameNode namenode, String src, public static void saveNamespace(NameNode namenode) throws AccessControlException, IOException { - namenode.getNamesystem().saveNamespace(); + namenode.getNamesystem().saveNamespace(0, 0); } public static void enterSafeMode(NameNode namenode, boolean resourcesLow) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java index 98297ca91c3ac..7d062418ccf4f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java @@ -114,7 +114,6 @@ public void setupCluster() throws Exception { final long precision = 1L; conf.setLong(DFSConfigKeys.DFS_NAMENODE_ACCESSTIME_PRECISION_KEY, precision); conf.setLong(DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_KEY, 10000L); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); conf.setBoolean(DFSConfigKeys.DFS_NAMENODE_AUDIT_LOG_ASYNC_KEY, useAsyncLog); util = new DFSTestUtil.Builder().setName("TestAuditAllowed"). setNumFiles(20).build(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java index 95da8387cdda0..5a51cb735f6d9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java @@ -1607,7 +1607,7 @@ public void testEditFailureOnFirstCheckpoint() throws IOException { // Make sure the on-disk fsimage on the NN has txid > 0. FSNamesystem fsns = cluster.getNamesystem(); fsns.enterSafeMode(false); - fsns.saveNamespace(); + fsns.saveNamespace(0, 0); fsns.leaveSafeMode(); secondary = startSecondaryNameNode(conf); @@ -2239,7 +2239,7 @@ public void testSecondaryHasVeryOutOfDateImage() throws IOException { NamenodeProtocols nn = cluster.getNameNodeRpc(); nn.setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); for (int i = 0; i < 3; i++) { - nn.saveNamespace(); + nn.saveNamespace(0, 0); } nn.setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); @@ -2324,7 +2324,7 @@ public void testSecondaryNameNodeWithDelegationTokens() throws IOException { // therefore needs to download a new fsimage the next time it performs a // checkpoint. cluster.getNameNodeRpc().setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); cluster.getNameNodeRpc().setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); // Ensure that the 2NN can still perform a checkpoint. @@ -2369,7 +2369,7 @@ public void testSecondaryNameNodeWithSavedLeases() throws IOException { // therefore needs to download a new fsimage the next time it performs a // checkpoint. cluster.getNameNodeRpc().setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); cluster.getNameNodeRpc().setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); // Ensure that the 2NN can still perform a checkpoint. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java index fb1418a0c6e17..92c329e332a53 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java @@ -30,8 +30,10 @@ import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocol.Block; +import org.apache.hadoop.hdfs.protocol.BlockListAsLongs; import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeCommand; import org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; @@ -104,9 +106,10 @@ public void testDeadDatanode() throws Exception { // Ensure blockReport from dead datanode is rejected with IOException StorageBlockReport[] report = { new StorageBlockReport( new DatanodeStorage(reg.getDatanodeUuid()), - new long[] { 0L, 0L, 0L }) }; + BlockListAsLongs.EMPTY) }; try { - dnp.blockReport(reg, poolId, report); + dnp.blockReport(reg, poolId, report, + new BlockReportContext(1, 0, System.nanoTime())); fail("Expected IOException is not thrown"); } catch (IOException ex) { // Expected diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java index a9aba864e94e1..789ee6f4ed55d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java @@ -18,7 +18,6 @@ package org.apache.hadoop.hdfs.server.namenode; import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import java.io.IOException; @@ -29,7 +28,6 @@ import java.util.Iterator; import java.util.List; import java.util.Random; -import java.util.concurrent.TimeoutException; import org.apache.commons.io.output.ByteArrayOutputStream; import org.apache.hadoop.conf.Configuration; @@ -53,7 +51,12 @@ import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil; import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor; import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager; +import org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager; +import org.apache.hadoop.hdfs.server.datanode.DataNode; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.hdfs.tools.DFSAdmin; +import org.apache.log4j.Level; +import org.apache.log4j.Logger; import org.junit.AfterClass; import org.junit.BeforeClass; import org.junit.Test; @@ -90,7 +93,8 @@ public static void setUp() throws Exception { conf.set(DFSConfigKeys.DFS_HOSTS_EXCLUDE, excludeFile.toUri().getPath()); Path includeFile = new Path(dir, "include"); conf.set(DFSConfigKeys.DFS_HOSTS, includeFile.toUri().getPath()); - conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 2000); + conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, + 1000); conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1); conf.setInt(DFSConfigKeys.DFS_NAMENODE_REPLICATION_PENDING_TIMEOUT_SEC_KEY, 4); @@ -104,6 +108,9 @@ public static void setUp() throws Exception { cluster = new MiniDFSCluster.Builder(conf).numDataNodes(numDatanodes).build(); cluster.waitActive(); fileSys = cluster.getFileSystem(); + cluster.getNamesystem().getBlockManager().getDatanodeManager() + .setHeartbeatExpireInterval(3000); + Logger.getLogger(DecommissionManager.class).setLevel(Level.DEBUG); } @AfterClass @@ -186,13 +193,16 @@ private void decommissionNode(FSNamesystem namesystem, private void checkDecommissionStatus(DatanodeDescriptor decommNode, int expectedUnderRep, int expectedDecommissionOnly, int expectedUnderRepInOpenFiles) { - assertEquals(decommNode.decommissioningStatus.getUnderReplicatedBlocks(), - expectedUnderRep); + assertEquals("Unexpected num under-replicated blocks", + expectedUnderRep, + decommNode.decommissioningStatus.getUnderReplicatedBlocks()); + assertEquals("Unexpected number of decom-only replicas", + expectedDecommissionOnly, + decommNode.decommissioningStatus.getDecommissionOnlyReplicas()); assertEquals( - decommNode.decommissioningStatus.getDecommissionOnlyReplicas(), - expectedDecommissionOnly); - assertEquals(decommNode.decommissioningStatus - .getUnderReplicatedInOpenFiles(), expectedUnderRepInOpenFiles); + "Unexpected number of replicas in under-replicated open files", + expectedUnderRepInOpenFiles, + decommNode.decommissioningStatus.getUnderReplicatedInOpenFiles()); } private void checkDFSAdminDecommissionStatus( @@ -244,7 +254,7 @@ private void checkDFSAdminDecommissionStatus( * Tests Decommissioning Status in DFS. */ @Test - public void testDecommissionStatus() throws IOException, InterruptedException { + public void testDecommissionStatus() throws Exception { InetSocketAddress addr = new InetSocketAddress("localhost", cluster .getNameNodePort()); DFSClient client = new DFSClient(addr, conf); @@ -253,7 +263,7 @@ public void testDecommissionStatus() throws IOException, InterruptedException { DistributedFileSystem fileSys = cluster.getFileSystem(); DFSAdmin admin = new DFSAdmin(cluster.getConfiguration(0)); - short replicas = 2; + short replicas = numDatanodes; // // Decommission one node. Verify the decommission status // @@ -263,7 +273,9 @@ public void testDecommissionStatus() throws IOException, InterruptedException { Path file2 = new Path("decommission1.dat"); FSDataOutputStream st1 = writeIncompleteFile(fileSys, file2, replicas); - Thread.sleep(5000); + for (DataNode d: cluster.getDataNodes()) { + DataNodeTestUtils.triggerBlockReport(d); + } FSNamesystem fsn = cluster.getNamesystem(); final DatanodeManager dm = fsn.getBlockManager().getDatanodeManager(); @@ -271,19 +283,22 @@ public void testDecommissionStatus() throws IOException, InterruptedException { String downnode = decommissionNode(fsn, client, localFileSys, iteration); dm.refreshNodes(conf); decommissionedNodes.add(downnode); - Thread.sleep(5000); + BlockManagerTestUtil.recheckDecommissionState(dm); final List decommissioningNodes = dm.getDecommissioningNodes(); if (iteration == 0) { assertEquals(decommissioningNodes.size(), 1); DatanodeDescriptor decommNode = decommissioningNodes.get(0); - checkDecommissionStatus(decommNode, 4, 0, 2); + checkDecommissionStatus(decommNode, 3, 0, 1); checkDFSAdminDecommissionStatus(decommissioningNodes.subList(0, 1), fileSys, admin); } else { assertEquals(decommissioningNodes.size(), 2); DatanodeDescriptor decommNode1 = decommissioningNodes.get(0); DatanodeDescriptor decommNode2 = decommissioningNodes.get(1); - checkDecommissionStatus(decommNode1, 4, 4, 2); + // This one is still 3,3,1 since it passed over the UC block + // earlier, before node 2 was decommed + checkDecommissionStatus(decommNode1, 3, 3, 1); + // This one is 4,4,2 since it has the full state checkDecommissionStatus(decommNode2, 4, 4, 2); checkDFSAdminDecommissionStatus(decommissioningNodes.subList(0, 2), fileSys, admin); @@ -305,8 +320,7 @@ public void testDecommissionStatus() throws IOException, InterruptedException { * the replication process after it rejoins the cluster. */ @Test(timeout=120000) - public void testDecommissionStatusAfterDNRestart() - throws IOException, InterruptedException { + public void testDecommissionStatusAfterDNRestart() throws Exception { DistributedFileSystem fileSys = (DistributedFileSystem)cluster.getFileSystem(); @@ -345,7 +359,7 @@ public void testDecommissionStatusAfterDNRestart() BlockManagerTestUtil.checkHeartbeat(fsn.getBlockManager()); // Force DatanodeManager to check decommission state. - BlockManagerTestUtil.checkDecommissionState(dm, dead.get(0)); + BlockManagerTestUtil.recheckDecommissionState(dm); // Verify that the DN remains in DECOMMISSION_INPROGRESS state. assertTrue("the node should be DECOMMISSION_IN_PROGRESSS", @@ -359,7 +373,7 @@ public void testDecommissionStatusAfterDNRestart() // Delete the under-replicated file, which should let the // DECOMMISSION_IN_PROGRESS node become DECOMMISSIONED cleanupFile(fileSys, f); - BlockManagerTestUtil.checkDecommissionState(dm, dead.get(0)); + BlockManagerTestUtil.recheckDecommissionState(dm); assertTrue("the node should be decommissioned", dead.get(0).isDecommissioned()); @@ -380,8 +394,9 @@ public void testDecommissionStatusAfterDNRestart() * DECOMMISSIONED */ @Test(timeout=120000) - public void testDecommissionDeadDN() - throws IOException, InterruptedException, TimeoutException { + public void testDecommissionDeadDN() throws Exception { + Logger log = Logger.getLogger(DecommissionManager.class); + log.setLevel(Level.DEBUG); DatanodeID dnID = cluster.getDataNodes().get(0).getDatanodeId(); String dnName = dnID.getXferAddr(); DataNodeProperties stoppedDN = cluster.stopDataNode(0); @@ -392,7 +407,7 @@ public void testDecommissionDeadDN() DatanodeDescriptor dnDescriptor = dm.getDatanode(dnID); decommissionNode(fsn, localFileSys, dnName); dm.refreshNodes(conf); - BlockManagerTestUtil.checkDecommissionState(dm, dnDescriptor); + BlockManagerTestUtil.recheckDecommissionState(dm); assertTrue(dnDescriptor.isDecommissioned()); // Add the node back diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java index 281ffb4167296..15ba15ee1710b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java @@ -26,13 +26,18 @@ import org.apache.hadoop.fs.ContentSummary; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.DFSOutputStream; import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.client.HdfsDataOutputStream; +import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException; +import org.apache.hadoop.hdfs.protocol.HdfsConstants; +import org.apache.hadoop.ipc.RemoteException; import org.junit.After; +import org.junit.Assert; import org.junit.Before; import org.junit.Test; @@ -148,12 +153,11 @@ public void testUpdateQuotaForFSync() throws Exception { final Path foo = new Path("/foo"); final Path bar = new Path(foo, "bar"); DFSTestUtil.createFile(dfs, bar, BLOCKSIZE, REPLICATION, 0L); - dfs.setQuota(foo, Long.MAX_VALUE-1, Long.MAX_VALUE-1); + dfs.setQuota(foo, Long.MAX_VALUE - 1, Long.MAX_VALUE - 1); FSDataOutputStream out = dfs.append(bar); out.write(new byte[BLOCKSIZE / 4]); - ((DFSOutputStream) out.getWrappedStream()).hsync(EnumSet - .of(HdfsDataOutputStream.SyncFlag.UPDATE_LENGTH)); + ((DFSOutputStream) out.getWrappedStream()).hsync(EnumSet.of(HdfsDataOutputStream.SyncFlag.UPDATE_LENGTH)); INodeDirectory fooNode = fsdir.getINode4Write(foo.toString()).asDirectory(); QuotaCounts quota = fooNode.getDirectoryWithQuotaFeature() @@ -182,4 +186,126 @@ public void testUpdateQuotaForFSync() throws Exception { assertEquals(2, ns); // foo and bar assertEquals((BLOCKSIZE * 2 + BLOCKSIZE / 2) * REPLICATION, ds); } + + /** + * Test append over storage quota does not mark file as UC or create lease + */ + @Test (timeout=60000) + public void testAppendOverStorageQuota() throws Exception { + final Path dir = new Path("/TestAppendOverQuota"); + final Path file = new Path(dir, "file"); + + // create partial block file + dfs.mkdirs(dir); + DFSTestUtil.createFile(dfs, file, BLOCKSIZE/2, REPLICATION, seed); + + // lower quota to cause exception when appending to partial block + dfs.setQuota(dir, Long.MAX_VALUE - 1, 1); + final INodeDirectory dirNode = fsdir.getINode4Write(dir.toString()) + .asDirectory(); + final long spaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + try { + DFSTestUtil.appendFile(dfs, file, BLOCKSIZE); + Assert.fail("append didn't fail"); + } catch (DSQuotaExceededException e) { + // ignore + } + + // check that the file exists, isn't UC, and has no dangling lease + INodeFile inode = fsdir.getINode(file.toString()).asFile(); + Assert.assertNotNull(inode); + Assert.assertFalse("should not be UC", inode.isUnderConstruction()); + Assert.assertNull("should not have a lease", cluster.getNamesystem().getLeaseManager().getLeaseByPath(file.toString())); + // make sure the quota usage is unchanged + final long newSpaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + assertEquals(spaceUsed, newSpaceUsed); + // make sure edits aren't corrupted + dfs.recoverLease(file); + cluster.restartNameNodes(); + } + + /** + * Test append over a specific type of storage quota does not mark file as + * UC or create a lease + */ + @Test (timeout=60000) + public void testAppendOverTypeQuota() throws Exception { + final Path dir = new Path("/TestAppendOverTypeQuota"); + final Path file = new Path(dir, "file"); + + // create partial block file + dfs.mkdirs(dir); + // set the storage policy on dir + dfs.setStoragePolicy(dir, HdfsConstants.ONESSD_STORAGE_POLICY_NAME); + DFSTestUtil.createFile(dfs, file, BLOCKSIZE/2, REPLICATION, seed); + + // set quota of SSD to 1L + dfs.setQuotaByStorageType(dir, StorageType.SSD, 1L); + final INodeDirectory dirNode = fsdir.getINode4Write(dir.toString()) + .asDirectory(); + final long spaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + try { + DFSTestUtil.appendFile(dfs, file, BLOCKSIZE); + Assert.fail("append didn't fail"); + } catch (RemoteException e) { + assertTrue(e.getClassName().contains("QuotaByStorageTypeExceededException")); + } + + // check that the file exists, isn't UC, and has no dangling lease + INodeFile inode = fsdir.getINode(file.toString()).asFile(); + Assert.assertNotNull(inode); + Assert.assertFalse("should not be UC", inode.isUnderConstruction()); + Assert.assertNull("should not have a lease", cluster.getNamesystem() + .getLeaseManager().getLeaseByPath(file.toString())); + // make sure the quota usage is unchanged + final long newSpaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + assertEquals(spaceUsed, newSpaceUsed); + // make sure edits aren't corrupted + dfs.recoverLease(file); + cluster.restartNameNodes(); + } + + /** + * Test truncate over quota does not mark file as UC or create a lease + */ + @Test (timeout=60000) + public void testTruncateOverQuota() throws Exception { + final Path dir = new Path("/TestTruncateOverquota"); + final Path file = new Path(dir, "file"); + + // create partial block file + dfs.mkdirs(dir); + DFSTestUtil.createFile(dfs, file, BLOCKSIZE/2, REPLICATION, seed); + + // lower quota to cause exception when appending to partial block + dfs.setQuota(dir, Long.MAX_VALUE - 1, 1); + final INodeDirectory dirNode = fsdir.getINode4Write(dir.toString()) + .asDirectory(); + final long spaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + try { + dfs.truncate(file, BLOCKSIZE / 2 - 1); + Assert.fail("truncate didn't fail"); + } catch (RemoteException e) { + assertTrue(e.getClassName().contains("DSQuotaExceededException")); + } + + // check that the file exists, isn't UC, and has no dangling lease + INodeFile inode = fsdir.getINode(file.toString()).asFile(); + Assert.assertNotNull(inode); + Assert.assertFalse("should not be UC", inode.isUnderConstruction()); + Assert.assertNull("should not have a lease", cluster.getNamesystem() + .getLeaseManager().getLeaseByPath(file.toString())); + // make sure the quota usage is unchanged + final long newSpaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + assertEquals(spaceUsed, newSpaceUsed); + // make sure edits aren't corrupted + dfs.recoverLease(file); + cluster.restartNameNodes(); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java index 8b3c7ae43de7f..052c23f68a209 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java @@ -291,7 +291,7 @@ public void testSaveNamespace() throws Exception { LOG.info("Save " + i + ": saving namespace"); - namesystem.saveNamespace(); + namesystem.saveNamespace(0, 0); LOG.info("Save " + i + ": leaving safemode"); long savedImageTxId = fsimage.getStorage().getMostRecentCheckpointTxId(); @@ -421,7 +421,7 @@ public Void answer(InvocationOnMock invocation) throws Throwable { assertTrue(et - st > (BLOCK_TIME - 1)*1000); // Once we're in safe mode, save namespace. - namesystem.saveNamespace(); + namesystem.saveNamespace(0, 0); LOG.info("Joining on edit thread..."); doAnEditThread.join(); @@ -515,7 +515,7 @@ public Void answer(InvocationOnMock invocation) throws Throwable { assertTrue(et - st > (BLOCK_TIME - 1)*1000); // Once we're in safe mode, save namespace. - namesystem.saveNamespace(); + namesystem.saveNamespace(0, 0); LOG.info("Joining on edit thread..."); doAnEditThread.join(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java index f7dad183c5608..7b9ea93d119cb 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java @@ -41,6 +41,7 @@ import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction; import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoContiguous; import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.BlockUCState; +import org.apache.hadoop.hdfs.server.common.HdfsServerConstants.StartupOption; import org.apache.hadoop.hdfs.server.namenode.LeaseManager.Lease; import org.apache.hadoop.hdfs.util.MD5FileUtils; import org.apache.hadoop.test.GenericTestUtils; @@ -219,6 +220,7 @@ public void testZeroBlockSize() throws Exception { .manageDataDfsDirs(false) .manageNameDfsDirs(false) .waitSafeMode(false) + .startupOption(StartupOption.UPGRADE) .build(); try { FileSystem fs = cluster.getFileSystem(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java index 883029a4a5946..0154a0360eec1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java @@ -403,7 +403,7 @@ private void addAcl(INodeWithAdditionalFields inode, AclEntry... acl) private void assertPermissionGranted(UserGroupInformation user, String path, FsAction access) throws IOException { INodesInPath iip = dir.getINodesInPath(path, true); - new FSPermissionChecker(SUPERUSER, SUPERGROUP, user).checkPermission(iip, + dir.getPermissionChecker(SUPERUSER, SUPERGROUP, user).checkPermission(iip, false, null, null, access, null, false); } @@ -411,7 +411,7 @@ private void assertPermissionDenied(UserGroupInformation user, String path, FsAction access) throws IOException { try { INodesInPath iip = dir.getINodesInPath(path, true); - new FSPermissionChecker(SUPERUSER, SUPERGROUP, user).checkPermission(iip, + dir.getPermissionChecker(SUPERUSER, SUPERGROUP, user).checkPermission(iip, false, null, null, access, null, false); fail("expected AccessControlException for user + " + user + ", path = " + path + ", access = " + access); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java index 19b5cdedb734a..fbcc73f5c369b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java @@ -166,6 +166,8 @@ public void testMultipleTruncate() throws IOException { LOG.info("newLength=" + newLength + ", isReady=" + isReady); assertEquals("File must be closed for truncating at the block boundary", isReady, newLength % BLOCK_SIZE == 0); + assertEquals("Truncate is not idempotent", + isReady, fs.truncate(p, newLength)); if (!isReady) { checkBlockRecovery(p); } @@ -176,6 +178,36 @@ public void testMultipleTruncate() throws IOException { fs.delete(dir, true); } + /** Truncate the same file multiple times until its size is zero. */ + @Test + public void testSnapshotTruncateThenDeleteSnapshot() throws IOException { + Path dir = new Path("/testSnapshotTruncateThenDeleteSnapshot"); + fs.mkdirs(dir); + fs.allowSnapshot(dir); + final Path p = new Path(dir, "file"); + final byte[] data = new byte[BLOCK_SIZE]; + DFSUtil.getRandom().nextBytes(data); + writeContents(data, data.length, p); + final String snapshot = "s0"; + fs.createSnapshot(dir, snapshot); + Block lastBlock = getLocatedBlocks(p).getLastLocatedBlock() + .getBlock().getLocalBlock(); + final int newLength = data.length - 1; + assert newLength % BLOCK_SIZE != 0 : + " newLength must not be multiple of BLOCK_SIZE"; + final boolean isReady = fs.truncate(p, newLength); + LOG.info("newLength=" + newLength + ", isReady=" + isReady); + assertEquals("File must be closed for truncating at the block boundary", + isReady, newLength % BLOCK_SIZE == 0); + fs.deleteSnapshot(dir, snapshot); + if (!isReady) { + checkBlockRecovery(p); + } + checkFullFile(p, newLength, data); + assertBlockNotPresent(lastBlock); + fs.delete(dir, true); + } + /** * Truncate files and then run other operations such as * rename, set replication, set permission, etc. @@ -647,27 +679,22 @@ public void testTruncateWithDataNodesRestart() throws Exception { boolean isReady = fs.truncate(p, newLength); assertFalse(isReady); } finally { - cluster.restartDataNode(dn); + cluster.restartDataNode(dn, true, true); cluster.waitActive(); - cluster.triggerBlockReports(); } + checkBlockRecovery(p); LocatedBlock newBlock = getLocatedBlocks(p).getLastLocatedBlock(); /* * For non copy-on-truncate, the truncated block id is the same, but the * GS should increase. - * We trigger block report for dn0 after it restarts, since the GS - * of replica for the last block on it is old, so the reported last block - * from dn0 should be marked corrupt on nn and the replicas of last block - * on nn should decrease 1, then the truncated block will be replicated - * to dn0. + * The truncated block will be replicated to dn0 after it restarts. */ assertEquals(newBlock.getBlock().getBlockId(), oldBlock.getBlock().getBlockId()); assertEquals(newBlock.getBlock().getGenerationStamp(), oldBlock.getBlock().getGenerationStamp() + 1); - checkBlockRecovery(p); // Wait replicas come to 3 DFSTestUtil.waitReplication(fs, p, REPLICATION); // Old replica is disregarded and replaced with the truncated one @@ -709,23 +736,21 @@ public void testCopyOnTruncateWithDataNodesRestart() throws Exception { boolean isReady = fs.truncate(p, newLength); assertFalse(isReady); } finally { - cluster.restartDataNode(dn); + cluster.restartDataNode(dn, true, true); cluster.waitActive(); - cluster.triggerBlockReports(); } + checkBlockRecovery(p); LocatedBlock newBlock = getLocatedBlocks(p).getLastLocatedBlock(); /* * For copy-on-truncate, new block is made with new block id and new GS. - * We trigger block report for dn1 after it restarts. The replicas of - * the new block is 2, and then it will be replicated to dn1. + * The replicas of the new block is 2, then it will be replicated to dn1. */ assertNotEquals(newBlock.getBlock().getBlockId(), oldBlock.getBlock().getBlockId()); assertEquals(newBlock.getBlock().getGenerationStamp(), oldBlock.getBlock().getGenerationStamp() + 1); - checkBlockRecovery(p); // Wait replicas come to 3 DFSTestUtil.waitReplication(fs, p, REPLICATION); // New block is replicated to dn1 @@ -768,10 +793,10 @@ public void testTruncateWithDataNodesRestartImmediately() throws Exception { boolean isReady = fs.truncate(p, newLength); assertFalse(isReady); - cluster.restartDataNode(dn0); - cluster.restartDataNode(dn1); + cluster.restartDataNode(dn0, true, true); + cluster.restartDataNode(dn1, true, true); cluster.waitActive(); - cluster.triggerBlockReports(); + checkBlockRecovery(p); LocatedBlock newBlock = getLocatedBlocks(p).getLastLocatedBlock(); /* @@ -783,7 +808,6 @@ public void testTruncateWithDataNodesRestartImmediately() throws Exception { assertEquals(newBlock.getBlock().getGenerationStamp(), oldBlock.getBlock().getGenerationStamp() + 1); - checkBlockRecovery(p); // Wait replicas come to 3 DFSTestUtil.waitReplication(fs, p, REPLICATION); // Old replica is disregarded and replaced with the truncated one on dn0 @@ -827,6 +851,7 @@ public void testTruncateWithDataNodesShutdownImmediately() throws Exception { assertFalse(isReady); cluster.shutdownDataNodes(); + cluster.setDataNodesDead(); try { for(int i = 0; i < SUCCESS_ATTEMPTS && cluster.isDataNodeUp(); i++) { Thread.sleep(SLEEP); @@ -839,6 +864,7 @@ public void testTruncateWithDataNodesShutdownImmediately() throws Exception { StartupOption.REGULAR, null); cluster.waitActive(); } + checkBlockRecovery(p); fs.delete(parent, true); } @@ -1151,8 +1177,13 @@ static void checkBlockRecovery(Path p) throws IOException { public static void checkBlockRecovery(Path p, DistributedFileSystem dfs) throws IOException { + checkBlockRecovery(p, dfs, SUCCESS_ATTEMPTS, SLEEP); + } + + public static void checkBlockRecovery(Path p, DistributedFileSystem dfs, + int attempts, long sleepMs) throws IOException { boolean success = false; - for(int i = 0; i < SUCCESS_ATTEMPTS; i++) { + for(int i = 0; i < attempts; i++) { LocatedBlocks blocks = getLocatedBlocks(p, dfs); boolean noLastBlock = blocks.getLastLocatedBlock() == null; if(!blocks.isUnderConstruction() && @@ -1160,9 +1191,9 @@ public static void checkBlockRecovery(Path p, DistributedFileSystem dfs) success = true; break; } - try { Thread.sleep(SLEEP); } catch (InterruptedException ignored) {} + try { Thread.sleep(sleepMs); } catch (InterruptedException ignored) {} } - assertThat("inode should complete in ~" + SLEEP * SUCCESS_ATTEMPTS + " ms.", + assertThat("inode should complete in ~" + sleepMs * attempts + " ms.", success, is(true)); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java index 409fffcd2df55..70deb1b33699c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java @@ -1305,7 +1305,7 @@ public void testBlockIdCKDecommission() throws Exception { .getBlockManager().getBlockCollection(eb.getLocalBlock()) .getBlocks()[0].getDatanode(0); cluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager().startDecommission(dn); + .getDatanodeManager().getDecomManager().startDecommission(dn); String dnName = dn.getXferAddr(); //wait for decommission start diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestHDFSConcat.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestHDFSConcat.java index ddf5a3e23d83b..e1c3c0f5f67db 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestHDFSConcat.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestHDFSConcat.java @@ -44,6 +44,7 @@ import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocols; import org.apache.hadoop.ipc.RemoteException; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.test.GenericTestUtils; import org.junit.After; import org.junit.Assert; import org.junit.Before; @@ -388,6 +389,22 @@ public void testIllegalArg() throws IOException { } catch (Exception e) { // exspected } + + // the source file's preferred block size cannot be greater than the target + { + final Path src1 = new Path(parentDir, "src1"); + DFSTestUtil.createFile(dfs, src1, fileLen, REPL_FACTOR, 0L); + final Path src2 = new Path(parentDir, "src2"); + // create a file whose preferred block size is greater than the target + DFSTestUtil.createFile(dfs, src2, 1024, fileLen, + dfs.getDefaultBlockSize(trg) * 2, REPL_FACTOR, 0L); + try { + dfs.concat(trg, new Path[] {src1, src2}); + fail("didn't fail for src with greater preferred block size"); + } catch (Exception e) { + GenericTestUtils.assertExceptionContains("preferred block size", e); + } + } } /** diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java new file mode 100644 index 0000000000000..111c67ce652aa --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java @@ -0,0 +1,229 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.server.namenode; + +import java.io.IOException; +import java.security.PrivilegedExceptionAction; +import java.util.HashSet; +import java.util.Set; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.AclEntry; +import org.apache.hadoop.fs.permission.AclEntryType; +import org.apache.hadoop.fs.permission.FsAction; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.hdfs.HdfsConfiguration; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hdfs.server.namenode.INodeAttributeProvider.AccessControlEnforcer; +import org.apache.hadoop.security.AccessControlException; +import org.apache.hadoop.security.UserGroupInformation; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import com.google.common.collect.Lists; + +public class TestINodeAttributeProvider { + private MiniDFSCluster miniDFS; + private static final Set CALLED = new HashSet(); + + public static class MyAuthorizationProvider extends INodeAttributeProvider { + + public static class MyAccessControlEnforcer implements AccessControlEnforcer { + + @Override + public void checkPermission(String fsOwner, String supergroup, + UserGroupInformation ugi, INodeAttributes[] inodeAttrs, + INode[] inodes, byte[][] pathByNameArr, int snapshotId, String path, + int ancestorIndex, boolean doCheckOwner, FsAction ancestorAccess, + FsAction parentAccess, FsAction access, FsAction subAccess, + boolean ignoreEmptyDir) throws AccessControlException { + CALLED.add("checkPermission|" + ancestorAccess + "|" + parentAccess + "|" + access); + } + } + + @Override + public void start() { + CALLED.add("start"); + } + + @Override + public void stop() { + CALLED.add("stop"); + } + + @Override + public INodeAttributes getAttributes(String[] pathElements, + final INodeAttributes inode) { + CALLED.add("getAttributes"); + final boolean useDefault = useDefault(pathElements); + return new INodeAttributes() { + @Override + public boolean isDirectory() { + return inode.isDirectory(); + } + + @Override + public byte[] getLocalNameBytes() { + return inode.getLocalNameBytes(); + } + + @Override + public String getUserName() { + return (useDefault) ? inode.getUserName() : "foo"; + } + + @Override + public String getGroupName() { + return (useDefault) ? inode.getGroupName() : "bar"; + } + + @Override + public FsPermission getFsPermission() { + return (useDefault) ? inode.getFsPermission() + : new FsPermission(getFsPermissionShort()); + } + + @Override + public short getFsPermissionShort() { + return (useDefault) ? inode.getFsPermissionShort() + : (short) getPermissionLong(); + } + + @Override + public long getPermissionLong() { + return (useDefault) ? inode.getPermissionLong() : 0770; + } + + @Override + public AclFeature getAclFeature() { + AclFeature f; + if (useDefault) { + f = inode.getAclFeature(); + } else { + AclEntry acl = new AclEntry.Builder().setType(AclEntryType.GROUP). + setPermission(FsAction.ALL).setName("xxx").build(); + f = new AclFeature(AclEntryStatusFormat.toInt( + Lists.newArrayList(acl))); + } + return f; + } + + @Override + public XAttrFeature getXAttrFeature() { + return (useDefault) ? inode.getXAttrFeature() : null; + } + + @Override + public long getModificationTime() { + return (useDefault) ? inode.getModificationTime() : 0; + } + + @Override + public long getAccessTime() { + return (useDefault) ? inode.getAccessTime() : 0; + } + }; + + } + + @Override + public AccessControlEnforcer getExternalAccessControlEnforcer( + AccessControlEnforcer deafultEnforcer) { + return new MyAccessControlEnforcer(); + } + + private boolean useDefault(String[] pathElements) { + return (pathElements.length < 2) || + !(pathElements[0].equals("user") && pathElements[1].equals("authz")); + } + + } + + @Before + public void setUp() throws IOException { + CALLED.clear(); + Configuration conf = new HdfsConfiguration(); + conf.set(DFSConfigKeys.DFS_NAMENODE_INODE_ATTRIBUTES_PROVIDER_KEY, + MyAuthorizationProvider.class.getName()); + conf.setBoolean(DFSConfigKeys.DFS_NAMENODE_ACLS_ENABLED_KEY, true); + EditLogFileOutputStream.setShouldSkipFsyncForTesting(true); + miniDFS = new MiniDFSCluster.Builder(conf).build(); + } + + @After + public void cleanUp() throws IOException { + CALLED.clear(); + if (miniDFS != null) { + miniDFS.shutdown(); + } + Assert.assertTrue(CALLED.contains("stop")); + } + + @Test + public void testDelegationToProvider() throws Exception { + Assert.assertTrue(CALLED.contains("start")); + FileSystem fs = FileSystem.get(miniDFS.getConfiguration(0)); + fs.mkdirs(new Path("/tmp")); + fs.setPermission(new Path("/tmp"), new FsPermission((short) 0777)); + UserGroupInformation ugi = UserGroupInformation.createUserForTesting("u1", + new String[]{"g1"}); + ugi.doAs(new PrivilegedExceptionAction() { + @Override + public Void run() throws Exception { + FileSystem fs = FileSystem.get(miniDFS.getConfiguration(0)); + CALLED.clear(); + fs.mkdirs(new Path("/tmp/foo")); + Assert.assertTrue(CALLED.contains("getAttributes")); + Assert.assertTrue(CALLED.contains("checkPermission|null|null|null")); + Assert.assertTrue(CALLED.contains("checkPermission|WRITE|null|null")); + CALLED.clear(); + fs.listStatus(new Path("/tmp/foo")); + Assert.assertTrue(CALLED.contains("getAttributes")); + Assert.assertTrue( + CALLED.contains("checkPermission|null|null|READ_EXECUTE")); + CALLED.clear(); + fs.getAclStatus(new Path("/tmp/foo")); + Assert.assertTrue(CALLED.contains("getAttributes")); + Assert.assertTrue(CALLED.contains("checkPermission|null|null|null")); + return null; + } + }); + } + + @Test + public void testCustomProvider() throws Exception { + FileSystem fs = FileSystem.get(miniDFS.getConfiguration(0)); + fs.mkdirs(new Path("/user/xxx")); + FileStatus status = fs.getFileStatus(new Path("/user/xxx")); + Assert.assertEquals(System.getProperty("user.name"), status.getOwner()); + Assert.assertEquals("supergroup", status.getGroup()); + Assert.assertEquals(new FsPermission((short)0755), status.getPermission()); + fs.mkdirs(new Path("/user/authz")); + status = fs.getFileStatus(new Path("/user/authz")); + Assert.assertEquals("foo", status.getOwner()); + Assert.assertEquals("bar", status.getGroup()); + Assert.assertEquals(new FsPermission((short) 0770), status.getPermission()); + } + +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java index 61d2b3e00360e..daac442a5ded5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java @@ -474,7 +474,7 @@ public void testInodeId() throws IOException { // Apply editlogs to fsimage, ensure inodeUnderConstruction is handled fsn.enterSafeMode(false); - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); fsn.leaveSafeMode(); outStream.close(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java index 92ea111495ebe..5d319b43a3230 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java @@ -41,6 +41,8 @@ import org.apache.hadoop.hdfs.MiniDFSCluster; import org.apache.hadoop.hdfs.protocol.Block; import org.apache.hadoop.hdfs.protocol.HdfsConstants; +import org.apache.hadoop.hdfs.server.datanode.DataNode; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.util.StringUtils; import org.junit.Test; import org.slf4j.Logger; @@ -483,6 +485,10 @@ public void testMaxCorruptFiles() throws Exception { } } + // Run the direcrtoryScanner to update the Datanodes volumeMap + DataNode dn = cluster.getDataNodes().get(0); + DataNodeTestUtils.runDirectoryScanner(dn); + // Occasionally the BlockPoolSliceScanner can run before we have removed // the blocks. Restart the Datanode to trigger the scanner into running // once more. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java index dfd878e59e1b8..b8dc44e89f1c8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionFunctional.java @@ -152,7 +152,7 @@ public void testPurgingWithNameEditsDirAfterFailure() private static void doSaveNamespace(NameNode nn) throws IOException { LOG.info("Saving namespace..."); nn.getRpcServer().setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - nn.getRpcServer().saveNamespace(); + nn.getRpcServer().saveNamespace(0, 0); nn.getRpcServer().setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java index 571d719c8542b..55926cc9d3993 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java @@ -193,7 +193,6 @@ public void testHttpBindHostKey() throws IOException { private static void setupSsl() throws Exception { Configuration conf = new Configuration(); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); conf.set(DFSConfigKeys.DFS_HTTP_POLICY_KEY, HttpConfig.Policy.HTTPS_ONLY.name()); conf.set(DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_KEY, "localhost:0"); conf.set(DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_KEY, "localhost:0"); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java index 426563b355ece..fd611ce1c1e26 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java @@ -30,11 +30,10 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.DF; -import org.apache.hadoop.fs.FSDataOutputStream; -import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DFSConfigKeys; import org.apache.hadoop.hdfs.DFSOutputStream; +import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.DFSUtil; import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.HdfsConfiguration; @@ -201,7 +200,7 @@ public void testXceiverCount() throws Exception { DataNode dn = datanodes.get(i); DatanodeDescriptor dnd = dnm.getDatanode(dn.getDatanodeId()); dn.shutdown(); - dnd.setLastUpdate(0L); + DFSTestUtil.setDatanodeDead(dnd); BlockManagerTestUtil.checkHeartbeat(namesystem.getBlockManager()); expectedInServiceNodes--; assertEquals(expectedInServiceNodes, namesystem.getNumLiveDataNodes()); @@ -240,7 +239,7 @@ public void testXceiverCount() throws Exception { DatanodeDescriptor dnd = dnm.getDatanode(datanodes.get(i).getDatanodeId()); expectedInServiceLoad -= dnd.getXceiverCount(); - dnm.startDecommission(dnd); + dnm.getDecomManager().startDecommission(dnd); DataNodeTestUtils.triggerHeartbeat(datanodes.get(i)); Thread.sleep(100); checkClusterHealth(nodes, namesystem, expectedTotalLoad, expectedInServiceNodes, expectedInServiceLoad); @@ -280,7 +279,7 @@ public void testXceiverCount() throws Exception { dn.shutdown(); // force it to appear dead so live count decreases DatanodeDescriptor dnDesc = dnm.getDatanode(dn.getDatanodeId()); - dnDesc.setLastUpdate(0L); + DFSTestUtil.setDatanodeDead(dnDesc); BlockManagerTestUtil.checkHeartbeat(namesystem.getBlockManager()); assertEquals(nodes-1-i, namesystem.getNumLiveDataNodes()); // first few nodes are already out of service diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestParallelImageWrite.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestParallelImageWrite.java index 420026103e6da..86ae642fb8a8c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestParallelImageWrite.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestParallelImageWrite.java @@ -112,7 +112,7 @@ public void testRestartDFS() throws Exception { files.cleanup(fs, dir); files.createFiles(fs, dir); fsn.setSafeMode(SafeModeAction.SAFEMODE_ENTER); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); final String checkAfterModify = checkImages(fsn, numNamenodeDirs); assertFalse("Modified namespace should change fsimage contents. " + "was: " + checkAfterRestart + " now: " + checkAfterModify, diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java index 168ebb988ac68..37abc5b726d87 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java @@ -33,6 +33,7 @@ import org.apache.hadoop.hdfs.MiniDFSCluster.DataNodeProperties; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.server.blockmanagement.NumberReplicas; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.junit.Test; public class TestProcessCorruptBlocks { @@ -269,6 +270,8 @@ private void corruptBlock(MiniDFSCluster cluster, FileSystem fs, final Path file // But the datadirectory will not change assertTrue(cluster.corruptReplica(dnIndex, block)); + // Run directory scanner to update the DN's volume map + DataNodeTestUtils.runDirectoryScanner(cluster.getDataNodes().get(0)); DataNodeProperties dnProps = cluster.stopDataNode(0); // Each datanode has multiple data dirs, check each diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java index aee756f074e73..6d3893791d09b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java @@ -24,6 +24,7 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; + import org.apache.hadoop.fs.ContentSummary; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.hdfs.DFSConfigKeys; @@ -156,6 +157,11 @@ public void testQuotaByStorageTypeWithFileCreateAppend() throws Exception { ssdConsumed = fnode.asDirectory().getDirectoryWithQuotaFeature() .getSpaceConsumed().getTypeSpaces().get(StorageType.SSD); assertEquals(file1Len, ssdConsumed); + + ContentSummary cs = dfs.getContentSummary(foo); + assertEquals(cs.getSpaceConsumed(), file1Len * REPLICATION); + assertEquals(cs.getTypeConsumed(StorageType.SSD), file1Len); + assertEquals(cs.getTypeConsumed(StorageType.DISK), file1Len * 2); } @Test(timeout = 60000) @@ -192,6 +198,11 @@ public void testQuotaByStorageTypeWithFileCreateDelete() throws Exception { fnode.computeQuotaUsage(fsn.getBlockManager().getStoragePolicySuite(), counts, true); assertEquals(fnode.dumpTreeRecursively().toString(), 0, counts.getTypeSpaces().get(StorageType.SSD)); + + ContentSummary cs = dfs.getContentSummary(foo); + assertEquals(cs.getSpaceConsumed(), 0); + assertEquals(cs.getTypeConsumed(StorageType.SSD), 0); + assertEquals(cs.getTypeConsumed(StorageType.DISK), 0); } @Test(timeout = 60000) @@ -233,6 +244,11 @@ public void testQuotaByStorageTypeWithFileCreateRename() throws Exception { } catch (Throwable t) { LOG.info("Got expected exception ", t); } + + ContentSummary cs = dfs.getContentSummary(foo); + assertEquals(cs.getSpaceConsumed(), file1Len * REPLICATION); + assertEquals(cs.getTypeConsumed(StorageType.SSD), file1Len); + assertEquals(cs.getTypeConsumed(StorageType.DISK), file1Len * 2); } /** @@ -554,6 +570,11 @@ public void testQuotaByStorageTypeWithSnapshot() throws Exception { assertEquals(sub1Node.dumpTreeRecursively().toString(), file1Len, counts1.getTypeSpaces().get(StorageType.SSD)); + ContentSummary cs1 = dfs.getContentSummary(sub1); + assertEquals(cs1.getSpaceConsumed(), file1Len * REPLICATION); + assertEquals(cs1.getTypeConsumed(StorageType.SSD), file1Len); + assertEquals(cs1.getTypeConsumed(StorageType.DISK), file1Len * 2); + // Delete the snapshot s1 dfs.deleteSnapshot(sub1, "s1"); @@ -566,6 +587,11 @@ public void testQuotaByStorageTypeWithSnapshot() throws Exception { sub1Node.computeQuotaUsage(fsn.getBlockManager().getStoragePolicySuite(), counts2, true); assertEquals(sub1Node.dumpTreeRecursively().toString(), 0, counts2.getTypeSpaces().get(StorageType.SSD)); + + ContentSummary cs2 = dfs.getContentSummary(sub1); + assertEquals(cs2.getSpaceConsumed(), 0); + assertEquals(cs2.getTypeConsumed(StorageType.SSD), 0); + assertEquals(cs2.getTypeConsumed(StorageType.DISK), 0); } @Test(timeout = 60000) @@ -601,6 +627,11 @@ public void testQuotaByStorageTypeWithFileCreateTruncate() throws Exception { ssdConsumed = fnode.asDirectory().getDirectoryWithQuotaFeature() .getSpaceConsumed().getTypeSpaces().get(StorageType.SSD); assertEquals(newFile1Len, ssdConsumed); + + ContentSummary cs = dfs.getContentSummary(foo); + assertEquals(cs.getSpaceConsumed(), newFile1Len * REPLICATION); + assertEquals(cs.getTypeConsumed(StorageType.SSD), newFile1Len); + assertEquals(cs.getTypeConsumed(StorageType.DISK), newFile1Len * 2); } @Test @@ -701,6 +732,55 @@ public void testQuotaByStorageTypePersistenceInFsImage() throws IOException { .getDirectoryWithQuotaFeature() .getSpaceConsumed().getTypeSpaces().get(StorageType.SSD); assertEquals(file1Len, ssdConsumedAfterNNRestart); + } + + @Test(timeout = 60000) + public void testContentSummaryWithoutQuotaByStorageType() throws Exception { + final Path foo = new Path(dir, "foo"); + Path createdFile1 = new Path(foo, "created_file1.data"); + dfs.mkdirs(foo); + + // set storage policy on directory "foo" to ONESSD + dfs.setStoragePolicy(foo, HdfsConstants.ONESSD_STORAGE_POLICY_NAME); + INode fnode = fsdir.getINode4Write(foo.toString()); + assertTrue(fnode.isDirectory()); + assertTrue(!fnode.isQuotaSet()); + + // Create file of size 2 * BLOCKSIZE under directory "foo" + long file1Len = BLOCKSIZE * 2; + int bufLen = BLOCKSIZE / 16; + DFSTestUtil.createFile(dfs, createdFile1, bufLen, file1Len, BLOCKSIZE, REPLICATION, seed); + + // Verify getContentSummary without any quota set + ContentSummary cs = dfs.getContentSummary(foo); + assertEquals(cs.getSpaceConsumed(), file1Len * REPLICATION); + assertEquals(cs.getTypeConsumed(StorageType.SSD), file1Len); + assertEquals(cs.getTypeConsumed(StorageType.DISK), file1Len * 2); + } + + @Test(timeout = 60000) + public void testContentSummaryWithoutStoragePolicy() throws Exception { + final Path foo = new Path(dir, "foo"); + Path createdFile1 = new Path(foo, "created_file1.data"); + dfs.mkdirs(foo); + + INode fnode = fsdir.getINode4Write(foo.toString()); + assertTrue(fnode.isDirectory()); + assertTrue(!fnode.isQuotaSet()); + + // Create file of size 2 * BLOCKSIZE under directory "foo" + long file1Len = BLOCKSIZE * 2; + int bufLen = BLOCKSIZE / 16; + DFSTestUtil.createFile(dfs, createdFile1, bufLen, file1Len, BLOCKSIZE, REPLICATION, seed); + + // Verify getContentSummary without any quota set + // Expect no type quota and usage information available + ContentSummary cs = dfs.getContentSummary(foo); + assertEquals(cs.getSpaceConsumed(), file1Len * REPLICATION); + for (StorageType t : StorageType.values()) { + assertEquals(cs.getTypeConsumed(t), 0); + assertEquals(cs.getTypeQuota(t), -1); + } } } \ No newline at end of file diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java index 1821e9856aa79..f43edfba0f634 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java @@ -60,6 +60,7 @@ import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.test.GenericTestUtils.DelayAnswer; import org.apache.log4j.Level; +import org.junit.Assert; import org.junit.Test; import org.mockito.Mockito; import org.mockito.internal.util.reflection.Whitebox; @@ -184,7 +185,7 @@ private void saveNamespaceWithInjectedFault(Fault fault) throws Exception { // Save namespace - this may fail, depending on fault injected fsn.setSafeMode(SafeModeAction.SAFEMODE_ENTER); try { - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); if (shouldFail) { fail("Did not fail!"); } @@ -256,7 +257,7 @@ public void testReinsertnamedirsInSavenamespace() throws Exception { // Save namespace - should mark the first storage dir as faulty // since it's not traversable. LOG.info("Doing the first savenamespace."); - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); LOG.info("First savenamespace sucessful."); assertTrue("Savenamespace should have marked one directory as bad." + @@ -270,7 +271,7 @@ public void testReinsertnamedirsInSavenamespace() throws Exception { // erroneous directory back to fs.name.dir. This command should // be successful. LOG.info("Doing the second savenamespace."); - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); LOG.warn("Second savenamespace sucessful."); assertTrue("Savenamespace should have been successful in removing " + " bad directories from Image." + @@ -393,7 +394,7 @@ public void doTestFailedSaveNamespace(boolean restoreStorageAfterFailure) // Save namespace fsn.setSafeMode(SafeModeAction.SAFEMODE_ENTER); try { - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); fail("saveNamespace did not fail even when all directories failed!"); } catch (IOException ioe) { LOG.info("Got expected exception", ioe); @@ -403,7 +404,7 @@ public void doTestFailedSaveNamespace(boolean restoreStorageAfterFailure) if (restoreStorageAfterFailure) { Mockito.reset(spyImage); spyStorage.setRestoreFailedStorage(true); - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); checkEditExists(fsn, 1); } @@ -441,7 +442,7 @@ public void testSaveWhileEditsRolled() throws Exception { // Save namespace fsn.setSafeMode(SafeModeAction.SAFEMODE_ENTER); - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); // Now shut down and restart the NN fsn.close(); @@ -475,7 +476,7 @@ public void testTxIdPersistence() throws Exception { assertEquals(2, fsn.getEditLog().getLastWrittenTxId()); fsn.setSafeMode(SafeModeAction.SAFEMODE_ENTER); - fsn.saveNamespace(); + fsn.saveNamespace(0, 0); // 2 more txns: END the first segment, BEGIN a new one assertEquals(4, fsn.getEditLog().getLastWrittenTxId()); @@ -597,7 +598,7 @@ public void testSaveNamespaceWithRenamedLease() throws Exception { fs.rename(new Path("/test-source/"), new Path("/test-target/")); fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE); } finally { IOUtils.cleanup(LOG, out, fs); @@ -616,7 +617,7 @@ public void testSaveNamespaceWithDanglingLease() throws Exception { try { cluster.getNamesystem().leaseManager.addLease("me", "/non-existent"); fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE); } finally { if (cluster != null) { @@ -625,6 +626,54 @@ public void testSaveNamespaceWithDanglingLease() throws Exception { } } + @Test + public void testSaveNamespaceBeforeShutdown() throws Exception { + Configuration conf = new HdfsConfiguration(); + MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf) + .numDataNodes(0).build(); + cluster.waitActive(); + DistributedFileSystem fs = cluster.getFileSystem(); + + try { + final FSImage fsimage = cluster.getNameNode().getFSImage(); + final long before = fsimage.getStorage().getMostRecentCheckpointTxId(); + + fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER); + // set the timewindow to 1 hour and tx gap to 1000, which means that if + // there is a checkpoint during the past 1 hour or the tx number happening + // after the latest checkpoint is <= 1000, this saveNamespace request + // will be ignored + cluster.getNameNodeRpc().saveNamespace(3600, 1000); + + // make sure no new checkpoint was done + long after = fsimage.getStorage().getMostRecentCheckpointTxId(); + Assert.assertEquals(before, after); + + Thread.sleep(1000); + // do another checkpoint. this time set the timewindow to 1s + // we should see a new checkpoint + cluster.getNameNodeRpc().saveNamespace(1, 1000); + fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE); + + after = fsimage.getStorage().getMostRecentCheckpointTxId(); + Assert.assertTrue(after > before); + + fs.mkdirs(new Path("/foo/bar/baz")); // 3 new tx + + fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER); + cluster.getNameNodeRpc().saveNamespace(3600, 5); // 3 + end/start segment + long after2 = fsimage.getStorage().getMostRecentCheckpointTxId(); + // no checkpoint should be made + Assert.assertEquals(after, after2); + cluster.getNameNodeRpc().saveNamespace(3600, 3); + after2 = fsimage.getStorage().getMostRecentCheckpointTxId(); + // a new checkpoint should be done + Assert.assertTrue(after2 > after); + } finally { + cluster.shutdown(); + } + } + private void doAnEdit(FSNamesystem fsn, int id) throws IOException { // Make an edit fsn.mkdirs( diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java index 8b903af3fcebf..01621ada93ac7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java @@ -425,7 +425,7 @@ public void testCompression() throws IOException { NamenodeProtocols nnRpc = namenode.getRpcServer(); assertTrue(nnRpc.getFileInfo("/test").isDir()); nnRpc.setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - nnRpc.saveNamespace(); + nnRpc.saveNamespace(0, 0); namenode.stop(); namenode.join(); @@ -455,7 +455,7 @@ private void checkNameSpace(Configuration conf) throws IOException { NamenodeProtocols nnRpc = namenode.getRpcServer(); assertTrue(nnRpc.getFileInfo("/test").isDir()); nnRpc.setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - nnRpc.saveNamespace(); + nnRpc.saveNamespace(0, 0); namenode.stop(); namenode.join(); } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTruncateQuotaUpdate.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTruncateQuotaUpdate.java new file mode 100644 index 0000000000000..a4cb97fc88cb7 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTruncateQuotaUpdate.java @@ -0,0 +1,248 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.server.namenode; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.StorageType; +import org.apache.hadoop.hdfs.DFSConfigKeys; +import org.apache.hadoop.hdfs.DFSTestUtil; +import org.apache.hadoop.hdfs.DistributedFileSystem; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hdfs.protocol.HdfsConstants; +import org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +/** + * Make sure we correctly update the quota usage for truncate. + * We need to cover the following cases: + * 1. No snapshot, truncate to 0 + * 2. No snapshot, truncate at block boundary + * 3. No snapshot, not on block boundary + * 4~6. With snapshot, all the current blocks are included in latest + * snapshots, repeat 1~3 + * 7~9. With snapshot, blocks in the latest snapshot and blocks in the current + * file diverged, repeat 1~3 + */ +public class TestTruncateQuotaUpdate { + private static final int BLOCKSIZE = 1024; + private static final short REPLICATION = 4; + private static final long DISKQUOTA = BLOCKSIZE * 20; + static final long seed = 0L; + private static final Path dir = new Path("/TestTruncateQuotaUpdate"); + private static final Path file = new Path(dir, "file"); + + private MiniDFSCluster cluster; + private FSDirectory fsdir; + private DistributedFileSystem dfs; + + @Before + public void setUp() throws Exception { + final Configuration conf = new Configuration(); + conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCKSIZE); + cluster = new MiniDFSCluster.Builder(conf).numDataNodes(REPLICATION) + .build(); + cluster.waitActive(); + + fsdir = cluster.getNamesystem().getFSDirectory(); + dfs = cluster.getFileSystem(); + + dfs.mkdirs(dir); + dfs.setQuota(dir, Long.MAX_VALUE - 1, DISKQUOTA); + dfs.setQuotaByStorageType(dir, StorageType.DISK, DISKQUOTA); + dfs.setStoragePolicy(dir, HdfsConstants.HOT_STORAGE_POLICY_NAME); + } + + @After + public void tearDown() throws Exception { + if (cluster != null) { + cluster.shutdown(); + } + } + + @Test + public void testTruncateQuotaUpdate() throws Exception { + + } + + public interface TruncateCase { + public void prepare() throws Exception; + public void run() throws Exception; + } + + private void testTruncate(long newLength, long expectedDiff, + long expectedUsage) throws Exception { + // before doing the real truncation, make sure the computation is correct + final INodesInPath iip = fsdir.getINodesInPath4Write(file.toString()); + final INodeFile fileNode = iip.getLastINode().asFile(); + fileNode.recordModification(iip.getLatestSnapshotId(), true); + final long diff = fileNode.computeQuotaDeltaForTruncate(newLength); + Assert.assertEquals(expectedDiff, diff); + + // do the real truncation + dfs.truncate(file, newLength); + // wait for truncate to finish + TestFileTruncate.checkBlockRecovery(file, dfs); + final INodeDirectory dirNode = fsdir.getINode4Write(dir.toString()) + .asDirectory(); + final long spaceUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getStorageSpace(); + final long diskUsed = dirNode.getDirectoryWithQuotaFeature() + .getSpaceConsumed().getTypeSpaces().get(StorageType.DISK); + Assert.assertEquals(expectedUsage, spaceUsed); + Assert.assertEquals(expectedUsage, diskUsed); + } + + /** + * case 1~3 + */ + private class TruncateWithoutSnapshot implements TruncateCase { + @Override + public void prepare() throws Exception { + // original file size: 2.5 block + DFSTestUtil.createFile(dfs, file, BLOCKSIZE * 2 + BLOCKSIZE / 2, + REPLICATION, 0L); + } + + @Override + public void run() throws Exception { + // case 1: first truncate to 1.5 blocks + long newLength = BLOCKSIZE + BLOCKSIZE / 2; + // we truncate 1 blocks, but not on the boundary, thus the diff should + // be -block + (block - 0.5 block) = -0.5 block + long diff = -BLOCKSIZE / 2; + // the new quota usage should be BLOCKSIZE * 1.5 * replication + long usage = (BLOCKSIZE + BLOCKSIZE / 2) * REPLICATION; + testTruncate(newLength, diff, usage); + + // case 2: truncate to 1 block + newLength = BLOCKSIZE; + // the diff should be -0.5 block since this is not on boundary + diff = -BLOCKSIZE / 2; + // after truncation the quota usage should be BLOCKSIZE * replication + usage = BLOCKSIZE * REPLICATION; + testTruncate(newLength, diff, usage); + + // case 3: truncate to 0 + testTruncate(0, -BLOCKSIZE, 0); + } + } + + /** + * case 4~6 + */ + private class TruncateWithSnapshot implements TruncateCase { + @Override + public void prepare() throws Exception { + DFSTestUtil.createFile(dfs, file, BLOCKSIZE * 2 + BLOCKSIZE / 2, + REPLICATION, 0L); + SnapshotTestHelper.createSnapshot(dfs, dir, "s1"); + } + + @Override + public void run() throws Exception { + // case 4: truncate to 1.5 blocks + long newLength = BLOCKSIZE + BLOCKSIZE / 2; + // all the blocks are in snapshot. truncate need to allocate a new block + // diff should be +BLOCKSIZE + long diff = BLOCKSIZE; + // the new quota usage should be BLOCKSIZE * 3 * replication + long usage = BLOCKSIZE * 3 * REPLICATION; + testTruncate(newLength, diff, usage); + + // case 5: truncate to 1 block + newLength = BLOCKSIZE; + // the block for truncation is not in snapshot, diff should be -0.5 block + diff = -BLOCKSIZE / 2; + // after truncation the quota usage should be 2.5 block * repl + usage = (BLOCKSIZE * 2 + BLOCKSIZE / 2) * REPLICATION; + testTruncate(newLength, diff, usage); + + // case 6: truncate to 0 + testTruncate(0, 0, usage); + } + } + + /** + * case 7~9 + */ + private class TruncateWithSnapshot2 implements TruncateCase { + @Override + public void prepare() throws Exception { + // original size: 2.5 blocks + DFSTestUtil.createFile(dfs, file, BLOCKSIZE * 2 + BLOCKSIZE / 2, + REPLICATION, 0L); + SnapshotTestHelper.createSnapshot(dfs, dir, "s1"); + + // truncate to 1.5 block + dfs.truncate(file, BLOCKSIZE + BLOCKSIZE / 2); + TestFileTruncate.checkBlockRecovery(file, dfs); + + // append another 1 BLOCK + DFSTestUtil.appendFile(dfs, file, BLOCKSIZE); + } + + @Override + public void run() throws Exception { + // case 8: truncate to 2 blocks + long newLength = BLOCKSIZE * 2; + // the original 2.5 blocks are in snapshot. the block truncated is not + // in snapshot. diff should be -0.5 block + long diff = -BLOCKSIZE / 2; + // the new quota usage should be BLOCKSIZE * 3.5 * replication + long usage = (BLOCKSIZE * 3 + BLOCKSIZE / 2) * REPLICATION; + testTruncate(newLength, diff, usage); + + // case 7: truncate to 1.5 block + newLength = BLOCKSIZE + BLOCKSIZE / 2; + // the block for truncation is not in snapshot, diff should be + // -0.5 block + (block - 0.5block) = 0 + diff = 0; + // after truncation the quota usage should be 3 block * repl + usage = (BLOCKSIZE * 3) * REPLICATION; + testTruncate(newLength, diff, usage); + + // case 9: truncate to 0 + testTruncate(0, -BLOCKSIZE / 2, + (BLOCKSIZE * 2 + BLOCKSIZE / 2) * REPLICATION); + } + } + + private void testTruncateQuotaUpdate(TruncateCase t) throws Exception { + t.prepare(); + t.run(); + } + + @Test + public void testQuotaNoSnapshot() throws Exception { + testTruncateQuotaUpdate(new TruncateWithoutSnapshot()); + } + + @Test + public void testQuotaWithSnapshot() throws Exception { + testTruncateQuotaUpdate(new TruncateWithSnapshot()); + } + + @Test + public void testQuotaWithSnapshot2() throws Exception { + testTruncateQuotaUpdate(new TruncateWithSnapshot2()); + } +} diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java index fa7a3079b1579..74358bbd82986 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java @@ -53,6 +53,7 @@ import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.hdfs.server.namenode.NameNode; import org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter; +import org.apache.hadoop.hdfs.server.protocol.BlockReportContext; import org.apache.hadoop.hdfs.server.protocol.DatanodeRegistration; import org.apache.hadoop.hdfs.server.protocol.StorageBlockReport; import org.apache.hadoop.io.IOUtils; @@ -547,7 +548,8 @@ protected Object passThrough(InvocationOnMock invocation) .when(spy).blockReport( Mockito.anyObject(), Mockito.anyString(), - Mockito.anyObject()); + Mockito.anyObject(), + Mockito.anyObject()); dn.scheduleAllBlockReport(0); delayer.waitForCall(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java index c5aad9c1bc4c1..86f3e7b61472d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java @@ -498,7 +498,7 @@ private static void assertSafeMode(NameNode nn, int safe, int total, + nodeThresh + ". In safe mode extension. " + "Safe mode will be turned off automatically")); } else { - int additional = total - safe; + int additional = (int) (total * 0.9990) - safe; assertTrue("Bad safemode status: '" + status + "'", status.startsWith( "Safe mode is ON. " + diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java index 4d4fed63aa610..443500cdc266a 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java @@ -36,6 +36,7 @@ import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.protocol.HdfsConstants.DatanodeReportType; import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor; +import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.util.ThreadUtil; import org.junit.Test; @@ -69,6 +70,8 @@ public void testChangedStorageId() throws IOException, URISyntaxException, ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, filePath); assertTrue(cluster.changeGenStampOfBlock(0, block, 900)); + // Run directory dsscanner to update Datanode's volumeMap + DataNodeTestUtils.runDirectoryScanner(cluster.getDataNodes().get(0)); // Stop the DN so the replica with the changed gen stamp will be reported // when this DN starts up. DataNodeProperties dnProps = cluster.stopDataNode(0); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java index 6771ad8284c68..b39039150895d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java @@ -22,12 +22,16 @@ import static org.apache.hadoop.test.MetricsAsserts.assertGauge; import static org.apache.hadoop.test.MetricsAsserts.assertQuantileGauges; import static org.apache.hadoop.test.MetricsAsserts.getMetrics; +import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import java.io.DataInputStream; import java.io.IOException; import java.util.Random; +import com.google.common.collect.ImmutableList; +import com.google.common.io.Files; +import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.commons.logging.impl.Log4JLogger; import org.apache.hadoop.conf.Configuration; @@ -39,14 +43,18 @@ import org.apache.hadoop.hdfs.DistributedFileSystem; import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hdfs.MiniDFSNNTopology; import org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction; import org.apache.hadoop.hdfs.protocol.LocatedBlock; import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager; import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil; +import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor; import org.apache.hadoop.hdfs.server.datanode.DataNode; import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils; import org.apache.hadoop.hdfs.server.namenode.FSNamesystem; +import org.apache.hadoop.hdfs.server.namenode.NameNode; import org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter; +import org.apache.hadoop.hdfs.server.namenode.ha.HATestUtil; import org.apache.hadoop.hdfs.server.namenode.top.TopAuditLogger; import org.apache.hadoop.metrics2.MetricsRecordBuilder; import org.apache.hadoop.metrics2.MetricsSource; @@ -68,6 +76,7 @@ public class TestNameNodeMetrics { new Path("/testNameNodeMetrics"); private static final String NN_METRICS = "NameNodeActivity"; private static final String NS_METRICS = "FSNamesystem"; + public static final Log LOG = LogFactory.getLog(TestNameNodeMetrics.class); // Number of datanodes in the cluster private static final int DATANODE_COUNT = 3; @@ -162,9 +171,10 @@ public void testStaleNodes() throws Exception { long staleInterval = CONF.getLong( DFSConfigKeys.DFS_NAMENODE_STALE_DATANODE_INTERVAL_KEY, DFSConfigKeys.DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT); - cluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager().getDatanode(dn.getDatanodeId()) - .setLastUpdate(Time.now() - staleInterval - 1); + DatanodeDescriptor dnDes = cluster.getNameNode().getNamesystem() + .getBlockManager().getDatanodeManager() + .getDatanode(dn.getDatanodeId()); + DFSTestUtil.resetLastUpdatesWithOffset(dnDes, -(staleInterval + 1)); } // Let HeartbeatManager to check heartbeat BlockManagerTestUtil.checkHeartbeat(cluster.getNameNode().getNamesystem() @@ -175,9 +185,10 @@ public void testStaleNodes() throws Exception { for (int i = 0; i < 2; i++) { DataNode dn = cluster.getDataNodes().get(i); DataNodeTestUtils.setHeartbeatsDisabledForTests(dn, false); - cluster.getNameNode().getNamesystem().getBlockManager() - .getDatanodeManager().getDatanode(dn.getDatanodeId()) - .setLastUpdate(Time.now()); + DatanodeDescriptor dnDes = cluster.getNameNode().getNamesystem() + .getBlockManager().getDatanodeManager() + .getDatanode(dn.getDatanodeId()); + DFSTestUtil.resetLastUpdatesWithOffset(dnDes, 0); } // Let HeartbeatManager to refresh @@ -269,11 +280,16 @@ public void testCorruptBlock() throws Exception { public void testExcessBlocks() throws Exception { Path file = getTestPath("testExcessBlocks"); createFile(file, 100, (short)2); - long totalBlocks = 1; NameNodeAdapter.setReplication(namesystem, file.toString(), (short)1); MetricsRecordBuilder rb = getMetrics(NS_METRICS); - assertGauge("ExcessBlocks", totalBlocks, rb); + assertGauge("ExcessBlocks", 1L, rb); + + // verify ExcessBlocks metric is decremented and + // excessReplicateMap is cleared after deleting a file fs.delete(file, true); + rb = getMetrics(NS_METRICS); + assertGauge("ExcessBlocks", 0L, rb); + assertTrue(bm.excessReplicateMap.isEmpty()); } /** Test to ensure metrics reflects missing blocks */ @@ -396,6 +412,82 @@ public void testGetBlockLocationMetric() throws Exception { assertCounter("GetBlockLocations", 3L, getMetrics(NN_METRICS)); } + /** + * Testing TransactionsSinceLastCheckpoint. Need a new cluster as + * the other tests in here don't use HA. See HDFS-7501. + */ + @Test(timeout = 300000) + public void testTransactionSinceLastCheckpointMetrics() throws Exception { + Random random = new Random(); + int retryCount = 0; + while (retryCount < 5) { + try { + int basePort = 10060 + random.nextInt(100) * 2; + MiniDFSNNTopology topology = new MiniDFSNNTopology() + .addNameservice(new MiniDFSNNTopology.NSConf("ns1") + .addNN(new MiniDFSNNTopology.NNConf("nn1").setHttpPort(basePort)) + .addNN(new MiniDFSNNTopology.NNConf("nn2").setHttpPort(basePort + 1))); + + HdfsConfiguration conf2 = new HdfsConfiguration(); + // Lower the checkpoint condition for purpose of testing. + conf2.setInt( + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_KEY, + 100); + // Check for checkpoint condition very often, for purpose of testing. + conf2.setInt( + DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_CHECK_PERIOD_KEY, + 1); + // Poll and follow ANN txns very often, for purpose of testing. + conf2.setInt(DFSConfigKeys.DFS_HA_TAILEDITS_PERIOD_KEY, 1); + MiniDFSCluster cluster2 = new MiniDFSCluster.Builder(conf2) + .nnTopology(topology).numDataNodes(1).build(); + cluster2.waitActive(); + DistributedFileSystem fs2 = cluster2.getFileSystem(0); + NameNode nn0 = cluster2.getNameNode(0); + NameNode nn1 = cluster2.getNameNode(1); + cluster2.transitionToActive(0); + fs2.mkdirs(new Path("/tmp-t1")); + fs2.mkdirs(new Path("/tmp-t2")); + HATestUtil.waitForStandbyToCatchUp(nn0, nn1); + // Test to ensure tracking works before the first-ever + // checkpoint. + assertEquals("SBN failed to track 2 transactions pre-checkpoint.", + 4L, // 2 txns added further when catch-up is called. + cluster2.getNameNode(1).getNamesystem() + .getTransactionsSinceLastCheckpoint()); + // Complete up to the boundary required for + // an auto-checkpoint. Using 94 to expect fsimage + // rounded at 100, as 4 + 94 + 2 (catch-up call) = 100. + for (int i = 1; i <= 94; i++) { + fs2.mkdirs(new Path("/tmp-" + i)); + } + HATestUtil.waitForStandbyToCatchUp(nn0, nn1); + // Assert 100 transactions in checkpoint. + HATestUtil.waitForCheckpoint(cluster2, 1, ImmutableList.of(100)); + // Test to ensure number tracks the right state of + // uncheckpointed edits, and does not go negative + // (as fixed in HDFS-7501). + assertEquals("Should be zero right after the checkpoint.", + 0L, + cluster2.getNameNode(1).getNamesystem() + .getTransactionsSinceLastCheckpoint()); + fs2.mkdirs(new Path("/tmp-t3")); + fs2.mkdirs(new Path("/tmp-t4")); + HATestUtil.waitForStandbyToCatchUp(nn0, nn1); + // Test to ensure we track the right numbers after + // the checkpoint resets it to zero again. + assertEquals("SBN failed to track 2 added txns after the ckpt.", + 4L, + cluster2.getNameNode(1).getNamesystem() + .getTransactionsSinceLastCheckpoint()); + cluster2.shutdown(); + break; + } catch (Exception e) { + LOG.warn("Unable to set up HA cluster, exception thrown: " + e); + retryCount++; + } + } + } /** * Test NN checkpoint and transaction-related metrics. */ @@ -424,7 +516,7 @@ public void testTransactionAndCheckpointMetrics() throws Exception { assertGauge("TransactionsSinceLastLogRoll", 1L, getMetrics(NS_METRICS)); cluster.getNameNodeRpc().setSafeMode(SafeModeAction.SAFEMODE_ENTER, false); - cluster.getNameNodeRpc().saveNamespace(); + cluster.getNameNodeRpc().saveNamespace(0, 0); cluster.getNameNodeRpc().setSafeMode(SafeModeAction.SAFEMODE_LEAVE, false); long newLastCkptTime = MetricsAsserts.getLongGauge("LastCheckpointTime", diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java index a215beed29305..09bd2dcf3c5f4 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java @@ -519,8 +519,8 @@ private void restartClusterAndCheckImage(boolean compareQuota) File fsnAfter = new File(testDir, "dumptree_after"); SnapshotTestHelper.dumpTree2File(fsdir, fsnBefore); - - cluster.shutdown(); + + cluster.shutdown(false, false); cluster = new MiniDFSCluster.Builder(conf).format(false) .numDataNodes(REPL).build(); cluster.waitActive(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java index c6c8dadac0bfc..85072d1fc0fa1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java @@ -431,6 +431,6 @@ public void testDeletionOfLaterBlocksWithZeroSizeFirstBlock() throws Exception { // Now make sure that the NN can still save an fsimage successfully. cluster.getNameNode().getRpcServer().setSafeMode( SafeModeAction.SAFEMODE_ENTER, false); - cluster.getNameNode().getRpcServer().saveNamespace(); + cluster.getNameNode().getRpcServer().saveNamespace(0, 0); } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java index 98aafc1b3087b..d53140f14ac3e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java @@ -20,8 +20,8 @@ import java.io.ByteArrayOutputStream; import java.io.PrintStream; - import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.FileChecksum; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.hdfs.AppendTestUtil; import org.apache.hadoop.hdfs.DFSConfigKeys; @@ -29,8 +29,9 @@ import org.junit.Before; import org.junit.Test; import static org.hamcrest.CoreMatchers.is; -import static org.junit.Assert.*; - +import static org.hamcrest.CoreMatchers.not; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertThat; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; @@ -103,17 +104,35 @@ public void testSnapshotfileLength() throws Exception { Path file1snap1 = SnapshotTestHelper.getSnapshotPath(sub, snapshot1, file1Name); + final FileChecksum snapChksum1 = hdfs.getFileChecksum(file1snap1); + assertThat("file and snapshot file checksums are not equal", + hdfs.getFileChecksum(file1), is(snapChksum1)); + // Append to the file. FSDataOutputStream out = hdfs.append(file1); + // Nothing has been appended yet. All checksums should still be equal. + assertThat("file and snapshot checksums (open for append) are not equal", + hdfs.getFileChecksum(file1), is(snapChksum1)); + assertThat("snapshot checksum (post-open for append) has changed", + hdfs.getFileChecksum(file1snap1), is(snapChksum1)); try { AppendTestUtil.write(out, 0, toAppend); // Test reading from snapshot of file that is open for append byte[] dataFromSnapshot = DFSTestUtil.readFileBuffer(hdfs, file1snap1); assertThat("Wrong data size in snapshot.", dataFromSnapshot.length, is(origLen)); + // Verify that checksum didn't change + assertThat("snapshot file checksum (pre-close) has changed", + hdfs.getFileChecksum(file1), is(snapChksum1)); + assertThat("snapshot checksum (post-append) has changed", + hdfs.getFileChecksum(file1snap1), is(snapChksum1)); } finally { out.close(); } + assertThat("file and snapshot file checksums (post-close) are equal", + hdfs.getFileChecksum(file1), not(snapChksum1)); + assertThat("snapshot file checksum (post-close) has changed", + hdfs.getFileChecksum(file1snap1), is(snapChksum1)); // Make sure we can read the entire file via its non-snapshot path. fileStatus = hdfs.getFileStatus(file1); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java index bfa871c40f29a..7daabd0c4c87b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitCache.java @@ -36,13 +36,16 @@ import java.util.Iterator; import java.util.Map; +import com.google.common.collect.HashMultimap; import org.apache.commons.lang.mutable.MutableBoolean; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.BlockReaderFactory; import org.apache.hadoop.hdfs.BlockReaderTestUtil; +import org.apache.hadoop.hdfs.DFSClient; import org.apache.hadoop.hdfs.DFSInputStream; import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.DistributedFileSystem; @@ -52,11 +55,14 @@ import org.apache.hadoop.hdfs.protocol.DatanodeInfo; import org.apache.hadoop.hdfs.protocol.ExtendedBlock; import org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader; +import org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry; +import org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.RegisteredShm; import org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.PerDatanodeVisitorInfo; import org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.Visitor; import org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor; import org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.ShortCircuitReplicaCreator; import org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.Slot; +import org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm.ShmId; import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.net.unix.DomainSocket; import org.apache.hadoop.net.unix.TemporarySocketDirectory; @@ -615,4 +621,61 @@ public void visit(HashMap segments, + HashMultimap slots) { + Assert.assertEquals(1, segments.size()); + Assert.assertEquals(1, slots.size()); + } + }); + cluster.shutdown(); + sockDir.close(); + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java index 0e605acb27409..2ad7b6072ba93 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java @@ -32,6 +32,7 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hdfs.DFSTestUtil; import org.apache.hadoop.hdfs.server.namenode.FSEditLogOpCodes; +import org.apache.hadoop.hdfs.server.namenode.NameNodeLayoutVersion; import org.apache.hadoop.hdfs.server.namenode.OfflineEditsViewerHelper; import org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.Flags; import org.apache.hadoop.test.PathUtils; @@ -140,8 +141,8 @@ public void testRecoveryMode() throws IOException { assertEquals(0, runOev(editsReparsed, editsParsedXml2, "xml", false)); // judgment time - assertTrue("Test round trip", - filesEqualIgnoreTrailingZeros(editsParsedXml, editsParsedXml2)); + assertTrue("Test round trip", FileUtils.contentEqualsIgnoreEOL( + new File(editsParsedXml), new File(editsParsedXml2), "UTF-8")); os.close(); } @@ -238,6 +239,10 @@ private boolean filesEqualIgnoreTrailingZeros(String filenameSmall, ByteBuffer small = ByteBuffer.wrap(DFSTestUtil.loadFile(filenameSmall)); ByteBuffer large = ByteBuffer.wrap(DFSTestUtil.loadFile(filenameLarge)); + // OEV outputs with the latest layout version, so tweak the old file's + // contents to have latest version so checkedin binary files don't + // require frequent updates + small.put(3, (byte)NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION); // now correct if it's otherwise if (small.capacity() > large.capacity()) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java index 80369fd0efc1e..20b25f47de8f7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java @@ -71,7 +71,6 @@ protected FileSystem createFileSystem() throws Exception { @BeforeClass public static void setupCluster() { final Configuration conf = new Configuration(); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024); try { cluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).build(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestHttpsFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestHttpsFileSystem.java index 7612de341b4c7..3405c686af808 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestHttpsFileSystem.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestHttpsFileSystem.java @@ -52,7 +52,6 @@ public class TestHttpsFileSystem { @BeforeClass public static void setUp() throws Exception { conf = new Configuration(); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); conf.set(DFSConfigKeys.DFS_HTTP_POLICY_KEY, HttpConfig.Policy.HTTPS_ONLY.name()); conf.set(DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_KEY, "localhost:0"); conf.set(DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_KEY, "localhost:0"); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java index 3eba7db9c2cb2..0ed38f2610f7c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java @@ -42,9 +42,10 @@ import org.apache.hadoop.hdfs.protocol.HdfsFileStatus; import org.apache.hadoop.hdfs.server.namenode.INodeId; import org.apache.hadoop.util.Time; +import org.codehaus.jackson.map.ObjectMapper; +import org.codehaus.jackson.map.ObjectReader; import org.junit.Assert; import org.junit.Test; -import org.mortbay.util.ajax.JSON; import com.google.common.collect.Lists; @@ -58,7 +59,7 @@ static FileStatus toFileStatus(HdfsFileStatus f, String parent) { } @Test - public void testHdfsFileStatus() { + public void testHdfsFileStatus() throws IOException { final long now = Time.now(); final String parent = "/dir"; final HdfsFileStatus status = new HdfsFileStatus(1001L, false, 3, 1L << 26, @@ -70,7 +71,9 @@ public void testHdfsFileStatus() { System.out.println("fstatus = " + fstatus); final String json = JsonUtil.toJsonString(status, true); System.out.println("json = " + json.replace(",", ",\n ")); - final HdfsFileStatus s2 = JsonUtil.toFileStatus((Map)JSON.parse(json), true); + ObjectReader reader = new ObjectMapper().reader(Map.class); + final HdfsFileStatus s2 = + JsonUtil.toFileStatus((Map) reader.readValue(json), true); final FileStatus fs2 = toFileStatus(s2, parent); System.out.println("s2 = " + s2); System.out.println("fs2 = " + fs2); @@ -153,10 +156,11 @@ public void testToDatanodeInfoWithName() throws Exception { } @Test - public void testToAclStatus() { + public void testToAclStatus() throws IOException { String jsonString = "{\"AclStatus\":{\"entries\":[\"user::rwx\",\"user:user1:rw-\",\"group::rw-\",\"other::r-x\"],\"group\":\"supergroup\",\"owner\":\"testuser\",\"stickyBit\":false}}"; - Map json = (Map) JSON.parse(jsonString); + ObjectReader reader = new ObjectMapper().reader(Map.class); + Map json = reader.readValue(jsonString); List aclSpec = Lists.newArrayList(aclEntry(ACCESS, USER, ALL), @@ -215,7 +219,8 @@ public void testToXAttrMap() throws IOException { String jsonString = "{\"XAttrs\":[{\"name\":\"user.a1\",\"value\":\"0x313233\"}," + "{\"name\":\"user.a2\",\"value\":\"0x313131\"}]}"; - Map json = (Map)JSON.parse(jsonString); + ObjectReader reader = new ObjectMapper().reader(Map.class); + Map json = reader.readValue(jsonString); XAttr xAttr1 = (new XAttr.Builder()).setNameSpace(XAttr.NameSpace.USER). setName("a1").setValue(XAttrCodec.decodeValue("0x313233")).build(); XAttr xAttr2 = (new XAttr.Builder()).setNameSpace(XAttr.NameSpace.USER). @@ -240,8 +245,9 @@ public void testGetXAttrFromJson() throws IOException { String jsonString = "{\"XAttrs\":[{\"name\":\"user.a1\",\"value\":\"0x313233\"}," + "{\"name\":\"user.a2\",\"value\":\"0x313131\"}]}"; - Map json = (Map) JSON.parse(jsonString); - + ObjectReader reader = new ObjectMapper().reader(Map.class); + Map json = reader.readValue(jsonString); + // Get xattr: user.a2 byte[] value = JsonUtil.getXAttr(json, "user.a2"); Assert.assertArrayEquals(XAttrCodec.decodeValue("0x313131"), value); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java index eec49d848bbe1..2d8892c957f51 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java @@ -21,10 +21,15 @@ import static org.junit.Assert.fail; import java.io.IOException; +import java.io.OutputStream; +import java.net.HttpURLConnection; +import java.net.InetSocketAddress; import java.net.URISyntaxException; +import java.net.URL; import java.security.PrivilegedExceptionAction; import java.util.Random; +import org.apache.commons.io.IOUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.commons.logging.impl.Log4JLogger; @@ -45,6 +50,9 @@ import org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper; import org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods; import org.apache.hadoop.hdfs.server.protocol.NamenodeProtocols; +import org.apache.hadoop.hdfs.web.resources.LengthParam; +import org.apache.hadoop.hdfs.web.resources.OffsetParam; +import org.apache.hadoop.hdfs.web.resources.Param; import org.apache.hadoop.ipc.RetriableException; import org.apache.hadoop.security.AccessControlException; import org.apache.hadoop.security.UserGroupInformation; @@ -324,18 +332,6 @@ public void testCreateWithNoDN() throws Exception { } } } - - /** - * WebHdfs should be enabled by default after HDFS-5532 - * - * @throws Exception - */ - @Test - public void testWebHdfsEnabledByDefault() throws Exception { - Configuration conf = new HdfsConfiguration(); - Assert.assertTrue(conf.getBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, - false)); - } /** * Test snapshot creation through WebHdfs @@ -523,4 +519,41 @@ public void testDTInInsecureCluster() throws Exception { } } } + + @Test + public void testWebHdfsOffsetAndLength() throws Exception{ + MiniDFSCluster cluster = null; + final Configuration conf = WebHdfsTestUtil.createConf(); + final int OFFSET = 42; + final int LENGTH = 512; + final String PATH = "/foo"; + byte[] CONTENTS = new byte[1024]; + RANDOM.nextBytes(CONTENTS); + try { + cluster = new MiniDFSCluster.Builder(conf).numDataNodes(1).build(); + final WebHdfsFileSystem fs = + WebHdfsTestUtil.getWebHdfsFileSystem(conf, WebHdfsFileSystem.SCHEME); + try (OutputStream os = fs.create(new Path(PATH))) { + os.write(CONTENTS); + } + InetSocketAddress addr = cluster.getNameNode().getHttpAddress(); + URL url = new URL("http", addr.getHostString(), addr + .getPort(), WebHdfsFileSystem.PATH_PREFIX + PATH + "?op=OPEN" + + Param.toSortedString("&", new OffsetParam((long) OFFSET), + new LengthParam((long) LENGTH)) + ); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.setInstanceFollowRedirects(true); + Assert.assertEquals(LENGTH, conn.getContentLength()); + byte[] subContents = new byte[LENGTH]; + byte[] realContents = new byte[LENGTH]; + System.arraycopy(CONTENTS, OFFSET, subContents, 0, LENGTH); + IOUtils.readFully(conn.getInputStream(), realContents); + Assert.assertArrayEquals(subContents, realContents); + } finally { + if (cluster != null) { + cluster.shutdown(); + } + } + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java index 027fda08c0990..b2250fe5793d7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java @@ -60,7 +60,6 @@ public class TestWebHdfsFileSystemContract extends FileSystemContractBaseTest { private UserGroupInformation ugi; static { - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); try { cluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).build(); cluster.waitActive(); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java index d55f2b1c758e9..db0832505efdf 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java @@ -205,7 +205,6 @@ public void testLazyTokenFetchForSWebhdfs() throws Exception { String keystoresDir; String sslConfDir; - clusterConf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); clusterConf.set(DFSConfigKeys.DFS_HTTP_POLICY_KEY, HttpConfig.Policy.HTTPS_ONLY.name()); clusterConf.set(DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_KEY, "localhost:0"); clusterConf.set(DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_KEY, "localhost:0"); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java index 11abd2c070bd9..aeda32ccad742 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithMultipleNameNodes.java @@ -71,8 +71,6 @@ private static void setupCluster(final int nNameNodes, final int nDataNodes) throws Exception { LOG.info("nNameNodes=" + nNameNodes + ", nDataNodes=" + nDataNodes); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); - cluster = new MiniDFSCluster.Builder(conf) .nnTopology(MiniDFSNNTopology.simpleFederatedTopology(nNameNodes)) .numDataNodes(nDataNodes) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java index 369285dc2b28a..70f9735407938 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/WebHdfsTestUtil.java @@ -42,7 +42,6 @@ public class WebHdfsTestUtil { public static Configuration createConf() { final Configuration conf = new Configuration(); - conf.setBoolean(DFSConfigKeys.DFS_WEBHDFS_ENABLED_KEY, true); return conf; } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java index 0bbd5b4c52233..3720abe660312 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java @@ -88,7 +88,10 @@ public void testWriteTraceHooks() throws Exception { "ClientNamenodeProtocol#fsync", "org.apache.hadoop.hdfs.protocol.ClientProtocol.complete", "ClientNamenodeProtocol#complete", - "DFSOutputStream", + "newStreamForCreate", + "DFSOutputStream#writeChunk", + "DFSOutputStream#close", + "dataStreamer", "OpWriteBlockProto", "org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock", "ClientNamenodeProtocol#addBlock" @@ -102,10 +105,25 @@ public void testWriteTraceHooks() throws Exception { long spanStart = s.getStartTimeMillis(); long spanEnd = s.getStopTimeMillis(); - // There should only be one trace id as it should all be homed in the - // top trace. - for (Span span : SetSpanReceiver.SetHolder.spans.values()) { - Assert.assertEquals(ts.getSpan().getTraceId(), span.getTraceId()); + // Spans homed in the top trace shoud have same trace id. + // Spans having multiple parents (e.g. "dataStreamer" added by HDFS-7054) + // and children of them are exception. + String[] spansInTopTrace = { + "testWriteTraceHooks", + "org.apache.hadoop.hdfs.protocol.ClientProtocol.create", + "ClientNamenodeProtocol#create", + "org.apache.hadoop.hdfs.protocol.ClientProtocol.fsync", + "ClientNamenodeProtocol#fsync", + "org.apache.hadoop.hdfs.protocol.ClientProtocol.complete", + "ClientNamenodeProtocol#complete", + "newStreamForCreate", + "DFSOutputStream#writeChunk", + "DFSOutputStream#close", + }; + for (String desc : spansInTopTrace) { + for (Span span : map.get(desc)) { + Assert.assertEquals(ts.getSpan().getTraceId(), span.getTraceId()); + } } } diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testXAttrConf.xml b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testXAttrConf.xml index 9c66cba848736..c2e836c58b027 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testXAttrConf.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testXAttrConf.xml @@ -420,8 +420,8 @@ - ExactComparator - # file: /file1#LF# + SubstringComparator + At least one of the attributes provided was not found diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt index ccd24a62b49d5..f5d2d1a3d0863 100644 --- a/hadoop-mapreduce-project/CHANGES.txt +++ b/hadoop-mapreduce-project/CHANGES.txt @@ -9,6 +9,9 @@ Trunk (Unreleased) MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) + MAPREDUCE-4424. 'mapred job -list' command should show the job name + as well. (Avinash Kujur via aajisaka) + NEW FEATURES MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk) @@ -245,6 +248,79 @@ Trunk (Unreleased) MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang) +Release 2.8.0 - UNRELEASED + + INCOMPATIBLE CHANGES + + NEW FEATURES + + IMPROVEMENTS + + MAPREDUCE-6291. Correct mapred queue usage command. + (Brahma Reddu Battula via harsh) + + MAPREDUCE-579. Streaming "slowmatch" documentation. (harsh) + + MAPREDUCE-6287. Deprecated methods in org.apache.hadoop.examples.Sort + (Chao Zhang via harsh) + + MAPREDUCE-5190. Unnecessary condition test in RandomSampler. + (Jingguo Yao via harsh) + + MAPREDUCE-6239. Consolidate TestJobConf classes in + hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core + (Varun Saxena via harsh) + + MAPREDUCE-5807. Print usage by TeraSort job. (Rohith via harsh) + + MAPREDUCE-4653. TestRandomAlgorithm has an unused "import" statement. + (Amir Sanjar via harsh) + + MAPREDUCE-6100. replace "mapreduce.job.credentials.binary" with + MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY for better readability. + (Zhihai Xu via harsh) + + MAPREDUCE-6105. Inconsistent configuration in property + mapreduce.reduce.shuffle.merge.percent. (Ray Chiang via harsh) + + MAPREDUCE-4414. Add main methods to JobConf and YarnConfiguration, + for debug purposes. (Plamen Jeliazkov via harsh) + + MAPREDUCE-5755. MapTask.MapOutputBuffer#compare/swap should have + @Override annotation. (ozawa) + + MAPREDUCE-6282. Reuse historyFileAbsolute.getFileSystem in + CompletedJob#loadFullHistoryData for code optimization. + (zxu via rkanter) + + OPTIMIZATIONS + + BUG FIXES + + MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with hidden + files/folders. (Maysam Yabandeh via harsh) + + MAPREDUCE-6213. NullPointerException caused by job history server addr not + resolvable. (Peng Zhang via harsh) + + MAPREDUCE-6281. Fix javadoc in Terasort. (Albert Chu via ozawa) + + MAPREDUCE-6242. Progress report log is incredibly excessive in + application master. (Varun Saxena via devaraj) + + MAPREDUCE-6294. Remove an extra parameter described in Javadoc of + TockenCache. (Brahma Reddy Battula via ozawa) + + MAPREDUCE-5875. Make Counter limits consistent across JobClient, + MRAppMaster, and YarnChild. (Gera Shegalov via kasha) + + MAPREDUCE-6199. AbstractCounters are not reset completely on + deserialization (adhoot via rkanter) + + MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which + causes counter limits are not reset correctly. + (Zhihai Xu via harsh) + Release 2.7.0 - UNRELEASED INCOMPATIBLE CHANGES @@ -255,6 +331,9 @@ Release 2.7.0 - UNRELEASED MAPREDUCE-6228. Add truncate operation to SLive. (Plamen Jeliazkov via shv) + MAPREDUCE-5583. Ability to limit running map and reduce tasks. + (Jason Lowe via junping_du) + IMPROVEMENTS MAPREDUCE-6149. Document override log4j.properties in MR job. @@ -314,6 +393,21 @@ Release 2.7.0 - UNRELEASED MAPREDUCE-5612. Add javadoc for TaskCompletionEvent.Status. (Chris Palmer via aajisaka) + MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public + API in DistCp. (Jing Zhao via vinodkv) + + MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own + class. (Chris Trezzo via kasha) + + MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the + application and forcefully kill. (Eric Payne via junping_du) + + MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable + to better control to launch/kill containers. (Zhihai Xu via ozawa) + + MAPREDUCE-6292. Use org.junit package instead of junit.framework in + TestCombineFileInputFormat. (aajisaka) + OPTIMIZATIONS MAPREDUCE-6169. MergeQueue should release reference to the current item @@ -322,6 +416,9 @@ Release 2.7.0 - UNRELEASED MAPREDUCE-6059. Speed up history server startup time (Siqi Li via aw) + MAPREDUCE-4815. Speed up FileOutputCommitter#commitJob for many output + files. (Siqi Li via gera) + BUG FIXES MAPREDUCE-6210. Use getApplicationAttemptId() instead of getApplicationId() @@ -357,9 +454,6 @@ Release 2.7.0 - UNRELEASED MAPREDUCE-6045. need close the DataInputStream after open it in TestMapReduce.java (zxu via rkanter) - MAPREDUCE-6199. AbstractCounters are not reset completely on - deserialization (adhoot via rkanter) - MAPREDUCE-6206. TestAggregatedTransferRate fails on non-US systems (Jens Rabe via jlowe) @@ -399,6 +493,26 @@ Release 2.7.0 - UNRELEASED MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) + MAPREDUCE-6268. Fix typo in Task Attempt API's URL. (Ryu Kobayashi + via ozawa) + + MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. (Brahma + Reddy Battula via ozawa) + + MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc + comments. (Akira AJISAKA and Andrew Purtell via ozawa) + + MAPREDUCE-4742. Fix typo in nnbench#displayUsage. (Liang Xie via ozawa) + + MAPREDUCE-6277. Job can post multiple history files if attempt loses + connection to the RM (Chang Li via jlowe) + + MAPREDUCE-6275. Race condition in FileOutputCommitter v2 for + user-specified task output subdirs (Gera Shegalov and Siqi Li via jlowe) + + MAPREDUCE-6285. ClientServiceDelegate should not retry upon + AuthenticationException. (Jonathan Eagles via ozawa) + Release 2.6.1 - UNRELEASED INCOMPATIBLE CHANGES @@ -610,9 +724,6 @@ Release 2.6.0 - 2014-11-18 MAPREDUCE-6123. TestCombineFileInputFormat incorrectly starts 2 MiniDFSCluster instances. (cnauroth) - MAPREDUCE-5875. Make Counter limits consistent across JobClient, - MRAppMaster, and YarnChild. (Gera Shegalov via kasha) - MAPREDUCE-6125. TestContainerLauncherImpl sometimes fails (Mit Desai via jlowe) diff --git a/hadoop-mapreduce-project/bin/mapred b/hadoop-mapreduce-project/bin/mapred index d1991289569ff..5afe02e76dea4 100755 --- a/hadoop-mapreduce-project/bin/mapred +++ b/hadoop-mapreduce-project/bin/mapred @@ -78,7 +78,7 @@ case ${COMMAND} in HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_CLIENT_OPTS}" ;; classpath) - hadoop_do_classpath_subcommand "$@" + hadoop_do_classpath_subcommand CLASS "$@" ;; distcp) CLASS=org.apache.hadoop.tools.DistCp diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java index 5f39edd72e894..c8f2427fcd0a3 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java @@ -174,7 +174,7 @@ public InetSocketAddress getAddress() { /** * Child checking whether it can commit. * - *
    + *
    * Commit is a two-phased protocol. First the attempt informs the * ApplicationMaster that it is * {@link #commitPending(TaskAttemptID, TaskStatus)}. Then it repeatedly polls @@ -208,7 +208,7 @@ public boolean canCommit(TaskAttemptID taskAttemptID) throws IOException { * TaskAttempt is reporting that it is in commit_pending and it is waiting for * the commit Response * - *
    + *
    * Commit it a two-phased protocol. First the attempt informs the * ApplicationMaster that it is * {@link #commitPending(TaskAttemptID, TaskStatus)}. Then it repeatedly polls diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java index 981e6ffb4b42e..05bb40bb9d909 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java @@ -44,7 +44,6 @@ * proxy if needed

  • * The URL may contain sentinels which will be replaced by jobId and jobStatus * (eg. SUCCEEDED/KILLED/FAILED)
    • - *

    */ public class JobEndNotifier implements Configurable { private static final String JOB_ID = "$jobId"; diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java index 8d5be86863063..5d3ad5b27a473 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java @@ -1451,10 +1451,6 @@ public static void main(String[] args) { String jobUserName = System .getenv(ApplicationConstants.Environment.USER.name()); conf.set(MRJobConfig.USER_NAME, jobUserName); - // Do not automatically close FileSystem objects so that in case of - // SIGTERM I have a chance to write out the job history. I'll be closing - // the objects myself. - conf.setBoolean("fs.automatic.close", false); initAndStartAppMaster(appMaster, conf, jobUserName); } catch (Throwable t) { LOG.fatal("Error starting MRAppMaster", t); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java index 666f757b540ca..9c1125d4ec2a2 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java @@ -70,7 +70,7 @@ public class ContainerLauncherImpl extends AbstractService implements new ConcurrentHashMap(); private final AppContext context; protected ThreadPoolExecutor launcherPool; - protected static final int INITIAL_POOL_SIZE = 10; + protected int initialPoolSize; private int limitOnPoolSize; private Thread eventHandlingThread; protected BlockingQueue eventQueue = @@ -246,6 +246,12 @@ protected void serviceInit(Configuration conf) throws Exception { MRJobConfig.MR_AM_CONTAINERLAUNCHER_THREAD_COUNT_LIMIT, MRJobConfig.DEFAULT_MR_AM_CONTAINERLAUNCHER_THREAD_COUNT_LIMIT); LOG.info("Upper limit on the thread pool size is " + this.limitOnPoolSize); + + this.initialPoolSize = conf.getInt( + MRJobConfig.MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE, + MRJobConfig.DEFAULT_MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE); + LOG.info("The thread pool initial size is " + this.initialPoolSize); + super.serviceInit(conf); cmProxy = new ContainerManagementProtocolProxy(conf); } @@ -256,7 +262,7 @@ protected void serviceStart() throws Exception { "ContainerLauncher #%d").setDaemon(true).build(); // Start with a default core-pool size of 10 and change it dynamically. - launcherPool = new ThreadPoolExecutor(INITIAL_POOL_SIZE, + launcherPool = new ThreadPoolExecutor(initialPoolSize, Integer.MAX_VALUE, 1, TimeUnit.HOURS, new LinkedBlockingQueue(), tf); @@ -289,11 +295,11 @@ public void run() { int idealPoolSize = Math.min(limitOnPoolSize, numNodes); if (poolSize < idealPoolSize) { - // Bump up the pool size to idealPoolSize+INITIAL_POOL_SIZE, the + // Bump up the pool size to idealPoolSize+initialPoolSize, the // later is just a buffer so we are not always increasing the // pool-size int newPoolSize = Math.min(limitOnPoolSize, idealPoolSize - + INITIAL_POOL_SIZE); + + initialPoolSize); LOG.info("Setting ContainerLauncher pool size to " + newPoolSize + " as number-of-nodes to talk to is " + numNodes); launcherPool.setCorePoolSize(newPoolSize); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java index 1acfeec163b5e..8cdcaa840bf01 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java @@ -99,9 +99,9 @@ public class RMContainerAllocator extends RMContainerRequestor public static final float DEFAULT_COMPLETED_MAPS_PERCENT_FOR_REDUCE_SLOWSTART = 0.05f; - private static final Priority PRIORITY_FAST_FAIL_MAP; - private static final Priority PRIORITY_REDUCE; - private static final Priority PRIORITY_MAP; + static final Priority PRIORITY_FAST_FAIL_MAP; + static final Priority PRIORITY_REDUCE; + static final Priority PRIORITY_MAP; @VisibleForTesting public static final String RAMPDOWN_DIAGNOSTIC = "Reducer preempted " @@ -166,6 +166,8 @@ added to the pending and are ramped up (added to scheduled) based */ private long allocationDelayThresholdMs = 0; private float reduceSlowStart = 0; + private int maxRunningMaps = 0; + private int maxRunningReduces = 0; private long retryInterval; private long retrystartTime; private Clock clock; @@ -201,6 +203,10 @@ protected void serviceInit(Configuration conf) throws Exception { allocationDelayThresholdMs = conf.getInt( MRJobConfig.MR_JOB_REDUCER_PREEMPT_DELAY_SEC, MRJobConfig.DEFAULT_MR_JOB_REDUCER_PREEMPT_DELAY_SEC) * 1000;//sec -> ms + maxRunningMaps = conf.getInt(MRJobConfig.JOB_RUNNING_MAP_LIMIT, + MRJobConfig.DEFAULT_JOB_RUNNING_MAP_LIMIT); + maxRunningReduces = conf.getInt(MRJobConfig.JOB_RUNNING_REDUCE_LIMIT, + MRJobConfig.DEFAULT_JOB_RUNNING_REDUCE_LIMIT); RackResolver.init(conf); retryInterval = getConfig().getLong(MRJobConfig.MR_AM_TO_RM_WAIT_INTERVAL_MS, MRJobConfig.DEFAULT_MR_AM_TO_RM_WAIT_INTERVAL_MS); @@ -664,6 +670,8 @@ public void rampDownReduces(int rampDown) { @SuppressWarnings("unchecked") private List getResources() throws Exception { + applyConcurrentTaskLimits(); + // will be null the first time Resource headRoom = getAvailableResources() == null ? Resources.none() : @@ -700,7 +708,7 @@ private List getResources() throws Exception { if (System.currentTimeMillis() - retrystartTime >= retryInterval) { LOG.error("Could not contact RM after " + retryInterval + " milliseconds."); eventHandler.handle(new JobEvent(this.getJob().getID(), - JobEventType.INTERNAL_ERROR)); + JobEventType.JOB_AM_REBOOT)); throw new YarnRuntimeException("Could not contact RM after " + retryInterval + " milliseconds."); } @@ -778,6 +786,43 @@ private List getResources() throws Exception { return newContainers; } + private void applyConcurrentTaskLimits() { + int numScheduledMaps = scheduledRequests.maps.size(); + if (maxRunningMaps > 0 && numScheduledMaps > 0) { + int maxRequestedMaps = Math.max(0, + maxRunningMaps - assignedRequests.maps.size()); + int numScheduledFailMaps = scheduledRequests.earlierFailedMaps.size(); + int failedMapRequestLimit = Math.min(maxRequestedMaps, + numScheduledFailMaps); + int normalMapRequestLimit = Math.min( + maxRequestedMaps - failedMapRequestLimit, + numScheduledMaps - numScheduledFailMaps); + setRequestLimit(PRIORITY_FAST_FAIL_MAP, mapResourceRequest, + failedMapRequestLimit); + setRequestLimit(PRIORITY_MAP, mapResourceRequest, normalMapRequestLimit); + } + + int numScheduledReduces = scheduledRequests.reduces.size(); + if (maxRunningReduces > 0 && numScheduledReduces > 0) { + int maxRequestedReduces = Math.max(0, + maxRunningReduces - assignedRequests.reduces.size()); + int reduceRequestLimit = Math.min(maxRequestedReduces, + numScheduledReduces); + setRequestLimit(PRIORITY_REDUCE, reduceResourceRequest, + reduceRequestLimit); + } + } + + private boolean canAssignMaps() { + return (maxRunningMaps <= 0 + || assignedRequests.maps.size() < maxRunningMaps); + } + + private boolean canAssignReduces() { + return (maxRunningReduces <= 0 + || assignedRequests.reduces.size() < maxRunningReduces); + } + private void updateAMRMToken(Token token) throws IOException { org.apache.hadoop.security.token.Token amrmToken = new org.apache.hadoop.security.token.Token(token @@ -1046,8 +1091,7 @@ reduceResourceRequest, getSchedulerResourceTypes()) <= 0 it = allocatedContainers.iterator(); while (it.hasNext()) { Container allocated = it.next(); - LOG.info("Releasing unassigned and invalid container " - + allocated + ". RM may have assignment issues"); + LOG.info("Releasing unassigned container " + allocated); containerNotAssigned(allocated); } } @@ -1150,7 +1194,8 @@ else if (PRIORITY_REDUCE.equals(priority)) { private ContainerRequest assignToFailedMap(Container allocated) { //try to assign to earlierFailedMaps if present ContainerRequest assigned = null; - while (assigned == null && earlierFailedMaps.size() > 0) { + while (assigned == null && earlierFailedMaps.size() > 0 + && canAssignMaps()) { TaskAttemptId tId = earlierFailedMaps.removeFirst(); if (maps.containsKey(tId)) { assigned = maps.remove(tId); @@ -1168,7 +1213,7 @@ private ContainerRequest assignToFailedMap(Container allocated) { private ContainerRequest assignToReduce(Container allocated) { ContainerRequest assigned = null; //try to assign to reduces if present - if (assigned == null && reduces.size() > 0) { + if (assigned == null && reduces.size() > 0 && canAssignReduces()) { TaskAttemptId tId = reduces.keySet().iterator().next(); assigned = reduces.remove(tId); LOG.info("Assigned to reduce"); @@ -1180,7 +1225,7 @@ private ContainerRequest assignToReduce(Container allocated) { private void assignMapsWithLocality(List allocatedContainers) { // try to assign to all nodes first to match node local Iterator it = allocatedContainers.iterator(); - while(it.hasNext() && maps.size() > 0){ + while(it.hasNext() && maps.size() > 0 && canAssignMaps()){ Container allocated = it.next(); Priority priority = allocated.getPriority(); assert PRIORITY_MAP.equals(priority); @@ -1212,7 +1257,7 @@ private void assignMapsWithLocality(List allocatedContainers) { // try to match all rack local it = allocatedContainers.iterator(); - while(it.hasNext() && maps.size() > 0){ + while(it.hasNext() && maps.size() > 0 && canAssignMaps()){ Container allocated = it.next(); Priority priority = allocated.getPriority(); assert PRIORITY_MAP.equals(priority); @@ -1242,7 +1287,7 @@ private void assignMapsWithLocality(List allocatedContainers) { // assign remaining it = allocatedContainers.iterator(); - while(it.hasNext() && maps.size() > 0){ + while(it.hasNext() && maps.size() > 0 && canAssignMaps()){ Container allocated = it.next(); Priority priority = allocated.getPriority(); assert PRIORITY_MAP.equals(priority); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java index bb9ad029f3bd4..166686485f61b 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java @@ -22,6 +22,7 @@ import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; +import java.util.Iterator; import java.util.Map; import java.util.Set; import java.util.TreeMap; @@ -44,6 +45,7 @@ import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.api.records.ResourceBlacklistRequest; import org.apache.hadoop.yarn.api.records.ResourceRequest; +import org.apache.hadoop.yarn.api.records.ResourceRequest.ResourceRequestComparator; import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; import org.apache.hadoop.yarn.factories.RecordFactory; @@ -58,6 +60,8 @@ public abstract class RMContainerRequestor extends RMCommunicator { private static final Log LOG = LogFactory.getLog(RMContainerRequestor.class); + private static final ResourceRequestComparator RESOURCE_REQUEST_COMPARATOR = + new ResourceRequestComparator(); protected int lastResponseID; private Resource availableResources; @@ -77,12 +81,18 @@ public abstract class RMContainerRequestor extends RMCommunicator { // use custom comparator to make sure ResourceRequest objects differing only in // numContainers dont end up as duplicates private final Set ask = new TreeSet( - new org.apache.hadoop.yarn.api.records.ResourceRequest.ResourceRequestComparator()); + RESOURCE_REQUEST_COMPARATOR); private final Set release = new TreeSet(); // pendingRelease holds history or release requests.request is removed only if // RM sends completedContainer. // How it different from release? --> release is for per allocate() request. protected Set pendingRelease = new TreeSet(); + + private final Map requestLimits = + new TreeMap(RESOURCE_REQUEST_COMPARATOR); + private final Set requestLimitsToUpdate = + new TreeSet(RESOURCE_REQUEST_COMPARATOR); + private boolean nodeBlacklistingEnabled; private int blacklistDisablePercent; private AtomicBoolean ignoreBlacklisting = new AtomicBoolean(false); @@ -178,6 +188,7 @@ protected void serviceInit(Configuration conf) throws Exception { protected AllocateResponse makeRemoteRequest() throws YarnException, IOException { + applyRequestLimits(); ResourceBlacklistRequest blacklistRequest = ResourceBlacklistRequest.newInstance(new ArrayList(blacklistAdditions), new ArrayList(blacklistRemovals)); @@ -190,13 +201,14 @@ protected AllocateResponse makeRemoteRequest() throws YarnException, availableResources = allocateResponse.getAvailableResources(); lastClusterNmCount = clusterNmCount; clusterNmCount = allocateResponse.getNumClusterNodes(); + int numCompletedContainers = + allocateResponse.getCompletedContainersStatuses().size(); if (ask.size() > 0 || release.size() > 0) { LOG.info("getResources() for " + applicationId + ":" + " ask=" + ask.size() + " release= " + release.size() + " newContainers=" + allocateResponse.getAllocatedContainers().size() - + " finishedContainers=" - + allocateResponse.getCompletedContainersStatuses().size() + + " finishedContainers=" + numCompletedContainers + " resourcelimit=" + availableResources + " knownNMs=" + clusterNmCount); } @@ -204,6 +216,12 @@ protected AllocateResponse makeRemoteRequest() throws YarnException, ask.clear(); release.clear(); + if (numCompletedContainers > 0) { + // re-send limited requests when a container completes to trigger asking + // for more containers + requestLimitsToUpdate.addAll(requestLimits.keySet()); + } + if (blacklistAdditions.size() > 0 || blacklistRemovals.size() > 0) { LOG.info("Update the blacklist for " + applicationId + ": blacklistAdditions=" + blacklistAdditions.size() + @@ -214,6 +232,36 @@ protected AllocateResponse makeRemoteRequest() throws YarnException, return allocateResponse; } + private void applyRequestLimits() { + Iterator iter = requestLimits.values().iterator(); + while (iter.hasNext()) { + ResourceRequest reqLimit = iter.next(); + int limit = reqLimit.getNumContainers(); + Map> remoteRequests = + remoteRequestsTable.get(reqLimit.getPriority()); + Map reqMap = (remoteRequests != null) + ? remoteRequests.get(ResourceRequest.ANY) : null; + ResourceRequest req = (reqMap != null) + ? reqMap.get(reqLimit.getCapability()) : null; + if (req == null) { + continue; + } + // update an existing ask or send a new one if updating + if (ask.remove(req) || requestLimitsToUpdate.contains(req)) { + ResourceRequest newReq = req.getNumContainers() > limit + ? reqLimit : req; + ask.add(newReq); + LOG.info("Applying ask limit of " + newReq.getNumContainers() + + " for priority:" + reqLimit.getPriority() + + " and capability:" + reqLimit.getCapability()); + } + if (limit == Integer.MAX_VALUE) { + iter.remove(); + } + } + requestLimitsToUpdate.clear(); + } + protected void addOutstandingRequestOnResync() { for (Map> rr : remoteRequestsTable .values()) { @@ -229,6 +277,7 @@ protected void addOutstandingRequestOnResync() { if (!pendingRelease.isEmpty()) { release.addAll(pendingRelease); } + requestLimitsToUpdate.addAll(requestLimits.keySet()); } // May be incorrect if there's multiple NodeManagers running on a single host. @@ -459,10 +508,8 @@ private void decResourceRequest(Priority priority, String resourceName, private void addResourceRequestToAsk(ResourceRequest remoteRequest) { // because objects inside the resource map can be deleted ask can end up // containing an object that matches new resource object but with different - // numContainers. So exisintg values must be replaced explicitly - if(ask.contains(remoteRequest)) { - ask.remove(remoteRequest); - } + // numContainers. So existing values must be replaced explicitly + ask.remove(remoteRequest); ask.add(remoteRequest); } @@ -490,6 +537,19 @@ protected ContainerRequest getFilteredContainerRequest(ContainerRequest orig) { return newReq; } + protected void setRequestLimit(Priority priority, Resource capability, + int limit) { + if (limit < 0) { + limit = Integer.MAX_VALUE; + } + ResourceRequest newReqLimit = ResourceRequest.newInstance(priority, + ResourceRequest.ANY, capability, limit); + ResourceRequest oldReqLimit = requestLimits.put(newReqLimit, newReqLimit); + if (oldReqLimit == null || oldReqLimit.getNumContainers() < limit) { + requestLimitsToUpdate.add(newReqLimit); + } + } + public Set getBlacklistedNodes() { return blacklistedNodes; } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java index de35d840b9473..43e3dbe4ae93e 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java @@ -464,7 +464,7 @@ public void testTimelineEventHandling() throws Exception { t.appAttemptId, 200, t.containerId, "nmhost", 3000, 4000), currentTime - 10)); TimelineEntities entities = ts.getEntities("MAPREDUCE_JOB", null, null, - null, null, null, null, null, null); + null, null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); TimelineEntity tEntity = entities.getEntities().get(0); Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId()); @@ -480,7 +480,7 @@ public void testTimelineEventHandling() throws Exception { new HashMap(), "default"), currentTime + 10)); entities = ts.getEntities("MAPREDUCE_JOB", null, null, null, - null, null, null, null, null); + null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); tEntity = entities.getEntities().get(0); Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId()); @@ -498,7 +498,7 @@ public void testTimelineEventHandling() throws Exception { new JobQueueChangeEvent(TypeConverter.fromYarn(t.jobId), "q2"), currentTime - 20)); entities = ts.getEntities("MAPREDUCE_JOB", null, null, null, - null, null, null, null, null); + null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); tEntity = entities.getEntities().get(0); Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId()); @@ -520,7 +520,7 @@ public void testTimelineEventHandling() throws Exception { new JobFinishedEvent(TypeConverter.fromYarn(t.jobId), 0, 0, 0, 0, 0, new Counters(), new Counters(), new Counters()), currentTime)); entities = ts.getEntities("MAPREDUCE_JOB", null, null, null, - null, null, null, null, null); + null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); tEntity = entities.getEntities().get(0); Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId()); @@ -546,7 +546,7 @@ public void testTimelineEventHandling() throws Exception { new JobUnsuccessfulCompletionEvent(TypeConverter.fromYarn(t.jobId), 0, 0, 0, JobStateInternal.KILLED.toString()), currentTime + 20)); entities = ts.getEntities("MAPREDUCE_JOB", null, null, null, - null, null, null, null, null); + null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); tEntity = entities.getEntities().get(0); Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId()); @@ -575,7 +575,7 @@ public void testTimelineEventHandling() throws Exception { handleEvent(jheh, new JobHistoryEvent(t.jobId, new TaskStartedEvent(t.taskID, 0, TaskType.MAP, ""))); entities = ts.getEntities("MAPREDUCE_TASK", null, null, null, - null, null, null, null, null); + null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); tEntity = entities.getEntities().get(0); Assert.assertEquals(t.taskID.toString(), tEntity.getEntityId()); @@ -588,7 +588,7 @@ public void testTimelineEventHandling() throws Exception { handleEvent(jheh, new JobHistoryEvent(t.jobId, new TaskStartedEvent(t.taskID, 0, TaskType.REDUCE, ""))); entities = ts.getEntities("MAPREDUCE_TASK", null, null, null, - null, null, null, null, null); + null, null, null, null, null, null); Assert.assertEquals(1, entities.getEntities().size()); tEntity = entities.getEntities().get(0); Assert.assertEquals(t.taskID.toString(), tEntity.getEntityId()); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java index dc1d72f89f0f7..41ee65dcf9732 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java @@ -90,7 +90,7 @@ public class TestContainerLauncher { static final Log LOG = LogFactory.getLog(TestContainerLauncher.class); - @Test (timeout = 5000) + @Test (timeout = 10000) public void testPoolSize() throws InterruptedException { ApplicationId appId = ApplicationId.newInstance(12345, 67); @@ -108,12 +108,14 @@ public void testPoolSize() throws InterruptedException { ThreadPoolExecutor threadPool = containerLauncher.getThreadPool(); // No events yet + Assert.assertEquals(containerLauncher.initialPoolSize, + MRJobConfig.DEFAULT_MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE); Assert.assertEquals(0, threadPool.getPoolSize()); - Assert.assertEquals(ContainerLauncherImpl.INITIAL_POOL_SIZE, + Assert.assertEquals(containerLauncher.initialPoolSize, threadPool.getCorePoolSize()); Assert.assertNull(containerLauncher.foundErrors); - containerLauncher.expectedCorePoolSize = ContainerLauncherImpl.INITIAL_POOL_SIZE; + containerLauncher.expectedCorePoolSize = containerLauncher.initialPoolSize; for (int i = 0; i < 10; i++) { ContainerId containerId = ContainerId.newContainerId(appAttemptId, i); TaskAttemptId taskAttemptId = MRBuilderUtils.newTaskAttemptId(taskId, i); @@ -152,7 +154,7 @@ public void testPoolSize() throws InterruptedException { // Different hosts, there should be an increase in core-thread-pool size to // 21(11hosts+10buffer) // Core pool size should be 21 but the live pool size should be only 11. - containerLauncher.expectedCorePoolSize = 11 + ContainerLauncherImpl.INITIAL_POOL_SIZE; + containerLauncher.expectedCorePoolSize = 11 + containerLauncher.initialPoolSize; containerLauncher.finishEventHandling = false; ContainerId containerId = ContainerId.newContainerId(appAttemptId, 21); TaskAttemptId taskAttemptId = MRBuilderUtils.newTaskAttemptId(taskId, 21); @@ -164,6 +166,15 @@ public void testPoolSize() throws InterruptedException { Assert.assertNull(containerLauncher.foundErrors); containerLauncher.stop(); + + // change configuration MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE + // and verify initialPoolSize value. + Configuration conf = new Configuration(); + conf.setInt(MRJobConfig.MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE, + 20); + containerLauncher = new CustomContainerLauncher(context); + containerLauncher.init(conf); + Assert.assertEquals(containerLauncher.initialPoolSize, 20); } @Test(timeout = 5000) @@ -187,7 +198,7 @@ public void testPoolLimits() throws InterruptedException { ThreadPoolExecutor threadPool = containerLauncher.getThreadPool(); // 10 different hosts - containerLauncher.expectedCorePoolSize = ContainerLauncherImpl.INITIAL_POOL_SIZE; + containerLauncher.expectedCorePoolSize = containerLauncher.initialPoolSize; for (int i = 0; i < 10; i++) { containerLauncher.handle(new ContainerLauncherEvent(taskAttemptId, containerId, "host" + i + ":1234", null, diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java index 4759693a9164e..3b7a4323238c9 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java @@ -31,9 +31,11 @@ import static org.mockito.Mockito.when; import java.io.IOException; +import java.nio.ByteBuffer; import java.security.PrivilegedExceptionAction; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.EnumSet; import java.util.HashMap; import java.util.HashSet; @@ -63,6 +65,8 @@ import org.apache.hadoop.mapreduce.v2.app.job.Task; import org.apache.hadoop.mapreduce.v2.app.job.TaskAttempt; import org.apache.hadoop.mapreduce.v2.app.job.TaskAttemptStateInternal; +import org.apache.hadoop.mapreduce.v2.app.job.event.JobEvent; +import org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType; import org.apache.hadoop.mapreduce.v2.app.job.event.JobUpdatedNodesEvent; import org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptContainerAssignedEvent; import org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEvent; @@ -81,7 +85,13 @@ import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.util.Time; import org.apache.hadoop.yarn.api.ApplicationMasterProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest; import org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse; +import org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterRequest; +import org.apache.hadoop.yarn.api.protocolrecords.FinishApplicationMasterResponse; +import org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterRequest; +import org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse; +import org.apache.hadoop.yarn.api.records.ApplicationAccessType; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.Container; @@ -89,6 +99,10 @@ import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerState; import org.apache.hadoop.yarn.api.records.ContainerStatus; +import org.apache.hadoop.yarn.api.records.NMToken; +import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.api.records.NodeReport; +import org.apache.hadoop.yarn.api.records.Priority; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.api.records.ResourceRequest; import org.apache.hadoop.yarn.conf.YarnConfiguration; @@ -1597,6 +1611,7 @@ private static class MyContainerAllocator extends RMContainerAllocator { = new ArrayList(); static final List jobUpdatedNodeEvents = new ArrayList(); + static final List jobEvents = new ArrayList(); private MyResourceManager rm; private boolean isUnregistered = false; private AllocateResponse allocateResponse; @@ -1619,6 +1634,8 @@ public void handle(Event event) { taskAttemptKillEvents.add((TaskAttemptKillEvent)event); } else if (event instanceof JobUpdatedNodesEvent) { jobUpdatedNodeEvents.add((JobUpdatedNodesEvent)event); + } else if (event instanceof JobEvent) { + jobEvents.add((JobEvent)event); } } }); @@ -1773,6 +1790,18 @@ protected AllocateResponse makeRemoteRequest() throws IOException, } } + private static class MyContainerAllocator2 extends MyContainerAllocator { + public MyContainerAllocator2(MyResourceManager rm, Configuration conf, + ApplicationAttemptId appAttemptId, Job job) { + super(rm, conf, appAttemptId, job); + } + @Override + protected AllocateResponse makeRemoteRequest() throws IOException, + YarnException { + throw new YarnRuntimeException("for testing"); + } + } + @Test public void testReduceScheduling() throws Exception { int totalMaps = 10; @@ -2300,6 +2329,50 @@ public void testRMContainerAllocatorResendsRequestsOnRMRestart() } + @Test + public void testRMUnavailable() + throws Exception { + Configuration conf = new Configuration(); + conf.setInt( + MRJobConfig.MR_AM_TO_RM_WAIT_INTERVAL_MS, 0); + MyResourceManager rm1 = new MyResourceManager(conf); + rm1.start(); + DrainDispatcher dispatcher = + (DrainDispatcher) rm1.getRMContext().getDispatcher(); + RMApp app = rm1.submitApp(1024); + dispatcher.await(); + + MockNM nm1 = new MockNM("h1:1234", 15120, rm1.getResourceTrackerService()); + nm1.registerNode(); + nm1.nodeHeartbeat(true); + dispatcher.await(); + + ApplicationAttemptId appAttemptId = + app.getCurrentAppAttempt().getAppAttemptId(); + rm1.sendAMLaunched(appAttemptId); + dispatcher.await(); + + JobId jobId = MRBuilderUtils.newJobId(appAttemptId.getApplicationId(), 0); + Job mockJob = mock(Job.class); + when(mockJob.getReport()).thenReturn( + MRBuilderUtils.newJobReport(jobId, "job", "user", JobState.RUNNING, 0, + 0, 0, 0, 0, 0, 0, "jobfile", null, false, "")); + MyContainerAllocator2 allocator = + new MyContainerAllocator2(rm1, conf, appAttemptId, mockJob); + allocator.jobEvents.clear(); + try { + allocator.schedule(); + Assert.fail("Should Have Exception"); + } catch (YarnRuntimeException e) { + Assert.assertTrue(e.getMessage().contains("Could not contact RM after")); + } + dispatcher.await(); + Assert.assertEquals("Should Have 1 Job Event", 1, + allocator.jobEvents.size()); + JobEvent event = allocator.jobEvents.get(0); + Assert.assertTrue("Should Reboot", event.getType().equals(JobEventType.JOB_AM_REBOOT)); + } + @Test(timeout=60000) public void testAMRMTokenUpdate() throws Exception { LOG.info("Running testAMRMTokenUpdate"); @@ -2387,6 +2460,208 @@ public Token run() throws Exception { new Text(rmAddr), ugiToken.getService()); } + @Test + public void testConcurrentTaskLimits() throws Exception { + final int MAP_LIMIT = 3; + final int REDUCE_LIMIT = 1; + LOG.info("Running testConcurrentTaskLimits"); + Configuration conf = new Configuration(); + conf.setInt(MRJobConfig.JOB_RUNNING_MAP_LIMIT, MAP_LIMIT); + conf.setInt(MRJobConfig.JOB_RUNNING_REDUCE_LIMIT, REDUCE_LIMIT); + conf.setFloat(MRJobConfig.COMPLETED_MAPS_FOR_REDUCE_SLOWSTART, 1.0f); + ApplicationId appId = ApplicationId.newInstance(1, 1); + ApplicationAttemptId appAttemptId = ApplicationAttemptId.newInstance( + appId, 1); + JobId jobId = MRBuilderUtils.newJobId(appAttemptId.getApplicationId(), 0); + Job mockJob = mock(Job.class); + when(mockJob.getReport()).thenReturn( + MRBuilderUtils.newJobReport(jobId, "job", "user", JobState.RUNNING, 0, + 0, 0, 0, 0, 0, 0, "jobfile", null, false, "")); + final MockScheduler mockScheduler = new MockScheduler(appAttemptId); + MyContainerAllocator allocator = new MyContainerAllocator(null, conf, + appAttemptId, mockJob) { + @Override + protected void register() { + } + + @Override + protected ApplicationMasterProtocol createSchedulerProxy() { + return mockScheduler; + } + }; + + // create some map requests + ContainerRequestEvent[] reqMapEvents = new ContainerRequestEvent[5]; + for (int i = 0; i < reqMapEvents.length; ++i) { + reqMapEvents[i] = createReq(jobId, i, 1024, new String[] { "h" + i }); + } + allocator.sendRequests(Arrays.asList(reqMapEvents)); + + // create some reduce requests + ContainerRequestEvent[] reqReduceEvents = new ContainerRequestEvent[2]; + for (int i = 0; i < reqReduceEvents.length; ++i) { + reqReduceEvents[i] = createReq(jobId, i, 1024, new String[] {}, + false, true); + } + allocator.sendRequests(Arrays.asList(reqReduceEvents)); + allocator.schedule(); + + // verify all of the host-specific asks were sent plus one for the + // default rack and one for the ANY request + Assert.assertEquals(reqMapEvents.length + 2, mockScheduler.lastAsk.size()); + + // verify AM is only asking for the map limit overall + Assert.assertEquals(MAP_LIMIT, mockScheduler.lastAnyAskMap); + + // assign a map task and verify we do not ask for any more maps + ContainerId cid0 = mockScheduler.assignContainer("h0", false); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(2, mockScheduler.lastAnyAskMap); + + // complete the map task and verify that we ask for one more + mockScheduler.completeContainer(cid0); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(3, mockScheduler.lastAnyAskMap); + + // assign three more maps and verify we ask for no more maps + ContainerId cid1 = mockScheduler.assignContainer("h1", false); + ContainerId cid2 = mockScheduler.assignContainer("h2", false); + ContainerId cid3 = mockScheduler.assignContainer("h3", false); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(0, mockScheduler.lastAnyAskMap); + + // complete two containers and verify we only asked for one more + // since at that point all maps should be scheduled/completed + mockScheduler.completeContainer(cid2); + mockScheduler.completeContainer(cid3); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(1, mockScheduler.lastAnyAskMap); + + // allocate the last container and complete the first one + // and verify there are no more map asks. + mockScheduler.completeContainer(cid1); + ContainerId cid4 = mockScheduler.assignContainer("h4", false); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(0, mockScheduler.lastAnyAskMap); + + // complete the last map + mockScheduler.completeContainer(cid4); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(0, mockScheduler.lastAnyAskMap); + + // verify only reduce limit being requested + Assert.assertEquals(REDUCE_LIMIT, mockScheduler.lastAnyAskReduce); + + // assign a reducer and verify ask goes to zero + cid0 = mockScheduler.assignContainer("h0", true); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(0, mockScheduler.lastAnyAskReduce); + + // complete the reducer and verify we ask for another + mockScheduler.completeContainer(cid0); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(1, mockScheduler.lastAnyAskReduce); + + // assign a reducer and verify ask goes to zero + cid0 = mockScheduler.assignContainer("h0", true); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(0, mockScheduler.lastAnyAskReduce); + + // complete the reducer and verify no more reducers + mockScheduler.completeContainer(cid0); + allocator.schedule(); + allocator.schedule(); + Assert.assertEquals(0, mockScheduler.lastAnyAskReduce); + allocator.close(); + } + + private static class MockScheduler implements ApplicationMasterProtocol { + ApplicationAttemptId attemptId; + long nextContainerId = 10; + List lastAsk = null; + int lastAnyAskMap = 0; + int lastAnyAskReduce = 0; + List containersToComplete = + new ArrayList(); + List containersToAllocate = new ArrayList(); + + public MockScheduler(ApplicationAttemptId attemptId) { + this.attemptId = attemptId; + } + + @Override + public RegisterApplicationMasterResponse registerApplicationMaster( + RegisterApplicationMasterRequest request) throws YarnException, + IOException { + return RegisterApplicationMasterResponse.newInstance( + Resource.newInstance(512, 1), + Resource.newInstance(512000, 1024), + Collections.emptyMap(), + ByteBuffer.wrap("fake_key".getBytes()), + Collections.emptyList(), + "default", + Collections.emptyList()); + } + + @Override + public FinishApplicationMasterResponse finishApplicationMaster( + FinishApplicationMasterRequest request) throws YarnException, + IOException { + return FinishApplicationMasterResponse.newInstance(false); + } + + @Override + public AllocateResponse allocate(AllocateRequest request) + throws YarnException, IOException { + lastAsk = request.getAskList(); + for (ResourceRequest req : lastAsk) { + if (ResourceRequest.ANY.equals(req.getResourceName())) { + Priority priority = req.getPriority(); + if (priority.equals(RMContainerAllocator.PRIORITY_MAP)) { + lastAnyAskMap = req.getNumContainers(); + } else if (priority.equals(RMContainerAllocator.PRIORITY_REDUCE)){ + lastAnyAskReduce = req.getNumContainers(); + } + } + } + AllocateResponse response = AllocateResponse.newInstance( + request.getResponseId(), + containersToComplete, containersToAllocate, + Collections.emptyList(), + Resource.newInstance(512000, 1024), null, 10, null, + Collections.emptyList()); + containersToComplete.clear(); + containersToAllocate.clear(); + return response; + } + + public ContainerId assignContainer(String nodeName, boolean isReduce) { + ContainerId containerId = + ContainerId.newContainerId(attemptId, nextContainerId++); + Priority priority = isReduce ? RMContainerAllocator.PRIORITY_REDUCE + : RMContainerAllocator.PRIORITY_MAP; + Container container = Container.newInstance(containerId, + NodeId.newInstance(nodeName, 1234), nodeName + ":5678", + Resource.newInstance(1024, 1), priority, null); + containersToAllocate.add(container); + return containerId; + } + + public void completeContainer(ContainerId containerId) { + containersToComplete.add(ContainerStatus.newInstance(containerId, + ContainerState.COMPLETE, "", 0)); + } + } + public static void main(String[] args) throws Exception { TestRMContainerAllocator t = new TestRMContainerAllocator(); t.testSimple(); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java index 1520fc8139bf3..e4b43b5a703d8 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java @@ -345,7 +345,7 @@ public static void setJobClassLoader(Configuration conf) * {@link MRJobConfig#MAPREDUCE_JOB_CLASSLOADER} is set to true, and * the APP_CLASSPATH environment variable is set. * @param conf - * @returns the created job classloader, or null if the job classloader is not + * @return the created job classloader, or null if the job classloader is not * enabled or the APP_CLASSPATH environment variable is not set * @throws IOException */ diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java index cac01191fcd6d..d367060eb19a5 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRWebAppUtil.java @@ -137,8 +137,9 @@ public static String getApplicationWebURLOnJHSWithoutScheme(Configuration conf, hsAddress, getDefaultJHSWebappPort(), getDefaultJHSWebappURLWithoutScheme()); StringBuffer sb = new StringBuffer(); - if (address.getAddress().isAnyLocalAddress() || - address.getAddress().isLoopbackAddress()) { + if (address.getAddress() != null && + (address.getAddress().isAnyLocalAddress() || + address.getAddress().isLoopbackAddress())) { sb.append(InetAddress.getLocalHost().getCanonicalHostName()); } else { sb.append(address.getHostName()); @@ -171,4 +172,4 @@ private static String getDefaultJHSWebappURLWithoutScheme() { public static String getAMWebappScheme(Configuration conf) { return "http://"; } -} \ No newline at end of file +} diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java index 370d67deb64db..0783eb50f8476 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java @@ -113,7 +113,7 @@ * } * } * - *

    + * * * It is also very common to use the DistributedCache by using * {@link org.apache.hadoop.util.GenericOptionsParser}. diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java index 8b567872b1aa6..904897b13265c 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java @@ -48,7 +48,7 @@ * Task capacity of the cluster. * *
  • - * The number of currently running map & reduce tasks. + * The number of currently running map and reduce tasks. *
    • *
  • * State of the JobTracker. @@ -56,7 +56,7 @@ *
  • * Details regarding black listed trackers. *
    • - *

    + * * *

    Clients can query for the latest ClusterStatus, via * {@link JobClient#getClusterStatus()}.

    diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java index 721c8a846375c..821c1e8943f58 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java @@ -179,7 +179,7 @@ public static Path getOutputPath(JobConf conf) { * Get the {@link Path} to the task's temporary output directory * for the map-reduce job * - *

    Tasks' Side-Effect Files

    + * Tasks' Side-Effect Files * *

    Note: The following is valid only if the {@link OutputCommitter} * is {@link FileOutputCommitter}. If OutputCommitter is not diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java index 30ebd6b8ca3fe..32e07e7b9fbef 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java @@ -47,7 +47,7 @@ import org.apache.commons.logging.LogFactory; /** - * IFile is the simple format + * IFile is the simple <key-len, value-len, key, value> format * for the intermediate map-outputs in Map-Reduce. * * There is a Writer to write out map-outputs in this format and diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobACLsManager.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobACLsManager.java index 37633ab504a37..0dbbe5a86c317 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobACLsManager.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobACLsManager.java @@ -101,7 +101,6 @@ boolean isMRAdmin(UserGroupInformation callerUGI) { * @param jobOperation * @param jobOwner * @param jobACL - * @throws AccessControlException */ public boolean checkAccess(UserGroupInformation callerUGI, JobACL jobOperation, String jobOwner, AccessControlList jobACL) { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java index 89a966eb84c11..e91fbfed37cfa 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java @@ -79,7 +79,7 @@ * Submitting the job to the cluster and optionally monitoring * it's status. * - *

    + * * * Normally the user creates the application, describes various facets of the * job via {@link JobConf} and then uses the JobClient to submit @@ -101,9 +101,9 @@ * * // Submit the job, then poll for progress until the job is complete * JobClient.runJob(job); - *

    + * * - *

    Job Control

    + * Job Control * *

    At times clients would chain map-reduce jobs to accomplish complex tasks * which cannot be done via a single map-reduce job. This is fairly easy since @@ -127,7 +127,7 @@ * {@link JobConf#setJobEndNotificationURI(String)} : setup a notification * on job-completion, thus avoiding polling. * - *

    + * * * @see JobConf * @see ClusterStatus diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java index 315c82998721b..9cac685db2f23 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java @@ -74,7 +74,7 @@ * more complex for the user to control finely * (e.g. {@link #setNumMapTasks(int)}). * - *

    + * * *

    JobConf typically specifies the {@link Mapper}, combiner * (if any), {@link Partitioner}, {@link Reducer}, {@link InputFormat} and @@ -105,7 +105,7 @@ * * job.setInputFormat(SequenceFileInputFormat.class); * job.setOutputFormat(SequenceFileOutputFormat.class); - *

    + * * * @see JobClient * @see ClusterStatus @@ -486,7 +486,7 @@ public JobConf(Path config) { /** A new map/reduce configuration where the behavior of reading from the * default resources can be turned off. - *

    + *

    * If the parameter {@code loadDefaults} is false, the new instance * will not load resources from the default files. * @@ -993,19 +993,19 @@ public RawComparator getOutputValueGroupingComparator() { /** * Set the user defined {@link RawComparator} comparator for * grouping keys in the input to the combiner. - *

    + * *

    This comparator should be provided if the equivalence rules for keys * for sorting the intermediates are different from those for grouping keys * before each call to * {@link Reducer#reduce(Object, java.util.Iterator, OutputCollector, Reporter)}.

    - *

    + * *

    For key-value pairs (K1,V1) and (K2,V2), the values (V1, V2) are passed * in a single call to the reduce function if K1 and K2 compare as equal.

    - *

    + * *

    Since {@link #setOutputKeyComparatorClass(Class)} can be used to control * how keys are sorted, this can be used in conjunction to simulate * secondary sort on values.

    - *

    + * *

    Note: This is not a guarantee of the combiner sort being * stable in any sense. (In any case, with the order of available * map-outputs to the combiner being non-deterministic, it wouldn't make @@ -1210,7 +1210,7 @@ public Class getCombinerClass() { *

  • be side-effect free
    • *
  • have the same input and output key types and the same input and * output value types
    • - *

    + * * *

    Typically the combiner is same as the Reducer for the * job i.e. {@link #setReducerClass(Class)}.

    @@ -1309,7 +1309,7 @@ public void setReduceSpeculativeExecution(boolean speculativeExecution) { * A custom {@link InputFormat} is typically used to accurately control * the number of map tasks for the job.

    * - *

    How many maps?

    + * How many maps? * *

    The number of maps is usually driven by the total size of the inputs * i.e. total number of blocks of the input files.

    @@ -1350,7 +1350,7 @@ public void setReduceSpeculativeExecution(boolean speculativeExecution) { /** * Set the requisite number of reduce tasks for this job. * - *

    How many reduces?

    + * How many reduces? * *

    The right number of reduces seems to be 0.95 or * 1.75 multiplied by (<no. of nodes> * @@ -1370,7 +1370,7 @@ public void setReduceSpeculativeExecution(boolean speculativeExecution) { * reserve a few reduce slots in the framework for speculative-tasks, failures * etc.

    * - *

    Reducer NONE

    + * Reducer NONE * *

    It is legal to set the number of reduce-tasks to zero.

    * @@ -1693,9 +1693,9 @@ public void setProfileTaskRange(boolean isMap, String newValue) { * given task's stdout, stderr, syslog, jobconf files as arguments.

    * *

    The debug command, run on the node where the map failed, is:

    - * 

     
+   * 
        * $script $stdout $stderr $syslog $jobconf.
-   *

    + * * *

    The script file is distributed through {@link DistributedCache} * APIs. The script needs to be symlinked.

    @@ -1705,7 +1705,7 @@ public void setProfileTaskRange(boolean isMap, String newValue) { * job.setMapDebugScript("./myscript"); * DistributedCache.createSymlink(job); * DistributedCache.addCacheFile("/debug/scripts/myscript#myscript"); - *

    + * * * @param mDbgScript the script name */ @@ -1730,9 +1730,9 @@ public String getMapDebugScript() { * is given task's stdout, stderr, syslog, jobconf files as arguments.

    * *

    The debug command, run on the node where the map failed, is:

    - * 

     
+   * 
        * $script $stdout $stderr $syslog $jobconf.
-   *

    + * * *

    The script file is distributed through {@link DistributedCache} * APIs. The script file needs to be symlinked

    @@ -1742,7 +1742,7 @@ public String getMapDebugScript() { * job.setReduceDebugScript("./myscript"); * DistributedCache.createSymlink(job); * DistributedCache.addCacheFile("/debug/scripts/myscript#myscript"); - *

    + * * * @param rDbgScript the script name */ @@ -1785,8 +1785,6 @@ public String getJobEndNotificationURI() { * * @param uri the job end notification uri * @see JobStatus - * @see Job Completion and Chaining */ public void setJobEndNotificationURI(String uri) { set(JobContext.MR_JOB_END_NOTIFICATION_URL, uri); @@ -1816,7 +1814,7 @@ public String getJobLocalDir() { * * If a value is specified in the configuration, it is returned. * Else, it returns {@link JobContext#DEFAULT_MAP_MEMORY_MB}. - *

    + *

    * For backward compatibility, if the job configuration sets the * key {@link #MAPRED_TASK_MAXVMEM_PROPERTY} to a value different * from {@link #DISABLED_MEMORY_LIMIT}, that value will be used @@ -1842,7 +1840,7 @@ public void setMemoryForMapTask(long mem) { * * If a value is specified in the configuration, it is returned. * Else, it returns {@link JobContext#DEFAULT_REDUCE_MEMORY_MB}. - *

    + *

    * For backward compatibility, if the job configuration sets the * key {@link #MAPRED_TASK_MAXVMEM_PROPERTY} to a value different * from {@link #DISABLED_MEMORY_LIMIT}, that value will be used @@ -1915,7 +1913,6 @@ public static long normalizeMemoryConfigValue(long val) { * * @param my_class the class to find. * @return a jar file that contains the class, or null. - * @throws IOException */ public static String findContainingJar(Class my_class) { return ClassUtil.findContainingJar(my_class); @@ -1924,10 +1921,10 @@ public static String findContainingJar(Class my_class) { /** * Get the memory required to run a task of this job, in bytes. See * {@link #MAPRED_TASK_MAXVMEM_PROPERTY} - *

    + *

    * This method is deprecated. Now, different memory limits can be * set for map and reduce tasks of a job, in MB. - *

    + *

    * For backward compatibility, if the job configuration sets the * key {@link #MAPRED_TASK_MAXVMEM_PROPERTY}, that value is returned. * Otherwise, this method will return the larger of the values returned by @@ -1953,7 +1950,7 @@ public long getMaxVirtualMemoryForTask() { /** * Set the maximum amount of memory any task of this job can use. See * {@link #MAPRED_TASK_MAXVMEM_PROPERTY} - *

    + *

    * mapred.task.maxvmem is split into * mapreduce.map.memory.mb * and mapreduce.map.memory.mb,mapred @@ -2073,7 +2070,7 @@ public String getTaskJavaOpts(TaskType taskType) { /** * Parse the Maximum heap size from the java opts as specified by the -Xmx option - * Format: -Xmx[g|G|m|M|k|K] + * Format: -Xmx<size>[g|G|m|M|k|K] * @param javaOpts String to parse to read maximum heap size * @return Maximum heap size in MB or -1 if not specified */ @@ -2143,5 +2140,10 @@ public int getMemoryRequired(TaskType taskType) { } } + /* For debugging. Dump configurations to system output as XML format. */ + public static void main(String[] args) throws Exception { + new JobConf(new Configuration()).writeXml(System.out); + } + } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java index 097e338a92d71..81f6140124655 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java @@ -224,7 +224,7 @@ private void displayQueueAclsInfoForCurrentUser() throws IOException { } private void displayUsage(String cmd) { - String prefix = "Usage: JobQueueClient "; + String prefix = "Usage: queue "; if ("-queueinfo".equals(cmd)) { System.err.println(prefix + "[" + cmd + " [-showJobs]]"); } else { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java index 80943178298cf..c4957b755179d 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java @@ -1255,6 +1255,7 @@ int offsetFor(int metapos) { * Compare by partition, then by key. * @see IndexedSortable#compare */ + @Override public int compare(final int mi, final int mj) { final int kvi = offsetFor(mi % maxRec); final int kvj = offsetFor(mj % maxRec); @@ -1278,6 +1279,7 @@ public int compare(final int mi, final int mj) { * Swap metadata for items i, j * @see IndexedSortable#swap */ + @Override public void swap(final int mi, final int mj) { int iOff = (mi % maxRec) * METASIZE; int jOff = (mj % maxRec) * METASIZE; diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Mapper.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Mapper.java index eaa6c2b72c497..ac2c96df5f0a2 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Mapper.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Mapper.java @@ -117,7 +117,7 @@ * output.collect(key, val); * } * } - *

    + * * *

    Applications may write a custom {@link MapRunnable} to exert greater * control on map processing e.g. multi-threaded Mappers etc.

    diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java index 39fae2a531245..794c55da1eaff 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java @@ -46,20 +46,20 @@ /** * Class that exposes information about queues maintained by the Hadoop * Map/Reduce framework. - *

    + *

    * The Map/Reduce framework can be configured with one or more queues, * depending on the scheduler it is configured with. While some * schedulers work only with one queue, some schedulers support multiple * queues. Some schedulers also support the notion of queues within * queues - a feature called hierarchical queues. - *

    + *

    * Queue names are unique, and used as a key to lookup queues. Hierarchical * queues are named by a 'fully qualified name' such as q1:q2:q3, where * q2 is a child queue of q1 and q3 is a child queue of q2. - *

    + *

    * Leaf level queues are queues that contain no queues within them. Jobs * can be submitted only to leaf level queues. - *

    + *

    * Queues can be configured with various properties. Some of these * properties are common to all schedulers, and those are handled by this * class. Schedulers might also associate several custom properties with @@ -69,11 +69,11 @@ * provided by the framework, but define their own mechanisms. In such cases, * it is likely that the name of the queue will be used to relate the * common properties of a queue with scheduler specific properties. - *

    + *

    * Information related to a queue, such as its name, properties, scheduling * information and children are exposed by this class via a serializable * class called {@link JobQueueInfo}. - *

    + *

    * Queues are configured in the configuration file mapred-queues.xml. * To support backwards compatibility, queues can also be configured * in mapred-site.xml. However, when configured in the latter, there is @@ -102,7 +102,7 @@ public class QueueManager { /** * Factory method to create an appropriate instance of a queue * configuration parser. - *

    + *

    * Returns a parser that can parse either the deprecated property * style queue configuration in mapred-site.xml, or one that can * parse hierarchical queues in mapred-queues.xml. First preference @@ -157,7 +157,7 @@ static QueueConfigurationParser getQueueConfigurationParser( /** * Construct a new QueueManager using configuration specified in the passed * in {@link org.apache.hadoop.conf.Configuration} object. - *

    + *

    * This instance supports queue configuration specified in mapred-site.xml, * but without support for hierarchical queues. If no queue configuration * is found in mapred-site.xml, it will then look for site configuration @@ -173,7 +173,7 @@ public QueueManager(Configuration clusterConf) { /** * Create an instance that supports hierarchical queues, defined in * the passed in configuration file. - *

    + *

    * This is mainly used for testing purposes and should not called from * production code. * @@ -208,7 +208,7 @@ private void initialize(QueueConfigurationParser cp) { /** * Return the set of leaf level queues configured in the system to * which jobs are submitted. - *

    + *

    * The number of queues configured should be dependent on the Scheduler * configured. Note that some schedulers work with only one queue, whereas * others can support multiple queues. @@ -222,7 +222,7 @@ public synchronized Set getLeafQueueNames() { /** * Return true if the given user is part of the ACL for the given * {@link QueueACL} name for the given queue. - *

    + *

    * An operation is allowed if all users are provided access for this * operation, or if either the user or any of the groups specified is * provided access. @@ -283,7 +283,7 @@ synchronized boolean isRunning(String queueName) { /** * Set a generic Object that represents scheduling information relevant * to a queue. - *

    + *

    * A string representation of this Object will be used by the framework * to display in user facing applications like the JobTracker web UI and * the hadoop CLI. @@ -323,7 +323,7 @@ public synchronized Object getSchedulerInfo(String queueName) { /** * Refresh acls, state and scheduler properties for the configured queues. - *

    + *

    * This method reloads configuration related to queues, but does not * support changes to the list of queues or hierarchy. The expected usage * is that an administrator can modify the queue configuration file and @@ -431,7 +431,7 @@ synchronized JobQueueInfo getJobQueueInfo(String queue) { /** * JobQueueInfo for all the queues. - *

    + *

    * Contribs can use this data structure to either create a hierarchy or for * traversing. * They can also use this to refresh properties in case of refreshQueues @@ -450,7 +450,7 @@ synchronized Map getJobQueueInfoMapping() { /** * Generates the array of QueueAclsInfo object. - *

    + *

    * The array consists of only those queues for which user has acls. * * @return QueueAclsInfo[] diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RecordReader.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RecordReader.java index 0c95a147dc69b..6e2c89fd8ca1f 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RecordReader.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/RecordReader.java @@ -29,7 +29,7 @@ * *

    RecordReader, typically, converts the byte-oriented view of * the input, provided by the InputSplit, and presents a - * record-oriented view for the {@link Mapper} & {@link Reducer} tasks for + * record-oriented view for the {@link Mapper} and {@link Reducer} tasks for * processing. It thus assumes the responsibility of processing record * boundaries and presenting the tasks with keys and values.

    * diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Reducer.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Reducer.java index 3fefa4bed7134..962e195b89b3b 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Reducer.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Reducer.java @@ -42,7 +42,7 @@ *
      *
    1. * - *

      Shuffle

      + * Shuffle * *

      Reducer is input the grouped output of a {@link Mapper}. * In the phase the framework, for each Reducer, fetches the @@ -51,7 +51,7 @@ *

      2. * *
    2. - *

      Sort

      + * Sort * *

      The framework groups Reducer inputs by keys * (since different Mappers may have output the same key) in this @@ -60,7 +60,7 @@ *

      The shuffle and sort phases occur simultaneously i.e. while outputs are * being fetched they are merged.

      * - *
      SecondarySort
      + * SecondarySort * *

      If equivalence rules for keys while grouping the intermediates are * different from those for grouping keys before reduction, then one may @@ -86,11 +86,11 @@ *

      3. * *
    3. - *

      Reduce

      + * Reduce * *

      In this phase the * {@link #reduce(Object, Iterator, OutputCollector, Reporter)} - * method is called for each <key, (list of values)> pair in + * method is called for each <key, (list of values)> pair in * the grouped inputs.

      *

      The output of the reduce task is typically written to the * {@link FileSystem} via @@ -156,7 +156,7 @@ * } * } * } - *

      + * * * @see Mapper * @see Partitioner @@ -171,7 +171,7 @@ public interface Reducer extends JobConfigurable, Closeable { * Reduces values for a given key. * *

      The framework calls this method for each - * <key, (list of values)> pair in the grouped inputs. + * <key, (list of values)> pair in the grouped inputs. * Output values must be of the same type as input values. Input keys must * not be altered. The framework will reuse the key and value objects * that are passed into the reduce, therefore the application should clone diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java index 7fa5d02682f7d..80881bc8b4433 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java @@ -171,7 +171,7 @@ static synchronized String getOutputName(int partition) { skipRanges.skipRangeIterator(); private ResourceCalculatorProcessTree pTree; - private long initCpuCumulativeTime = 0; + private long initCpuCumulativeTime = ResourceCalculatorProcessTree.UNAVAILABLE; protected JobConf conf; protected MapOutputFile mapOutputFile; @@ -229,6 +229,11 @@ public Task(String jobFile, TaskAttemptID taskId, int partition, gcUpdater = new GcTimeUpdater(); } + @VisibleForTesting + void setTaskDone() { + taskDone.set(true); + } + //////////////////////////////////////////// // Accessors //////////////////////////////////////////// @@ -536,9 +541,6 @@ public void localizeConfiguration(JobConf conf) throws IOException { public abstract void run(JobConf job, TaskUmbilicalProtocol umbilical) throws IOException, ClassNotFoundException, InterruptedException; - /** The number of milliseconds between progress reports. */ - public static final int PROGRESS_INTERVAL = 3000; - private transient Progress taskProgress = new Progress(); // Current counters @@ -714,6 +716,9 @@ public void run() { int remainingRetries = MAX_RETRIES; // get current flag value and reset it as well boolean sendProgress = resetProgressFlag(); + long taskProgressInterval = + conf.getLong(MRJobConfig.TASK_PROGRESS_REPORT_INTERVAL, + MRJobConfig.DEFAULT_TASK_PROGRESS_REPORT_INTERVAL); while (!taskDone.get()) { synchronized (lock) { done = false; @@ -726,7 +731,7 @@ public void run() { if (taskDone.get()) { break; } - lock.wait(PROGRESS_INTERVAL); + lock.wait(taskProgressInterval); } if (taskDone.get()) { break; @@ -861,13 +866,25 @@ void updateResourceCounters() { } pTree.updateProcessTree(); long cpuTime = pTree.getCumulativeCpuTime(); - long pMem = pTree.getCumulativeRssmem(); - long vMem = pTree.getCumulativeVmem(); + long pMem = pTree.getRssMemorySize(); + long vMem = pTree.getVirtualMemorySize(); // Remove the CPU time consumed previously by JVM reuse - cpuTime -= initCpuCumulativeTime; - counters.findCounter(TaskCounter.CPU_MILLISECONDS).setValue(cpuTime); - counters.findCounter(TaskCounter.PHYSICAL_MEMORY_BYTES).setValue(pMem); - counters.findCounter(TaskCounter.VIRTUAL_MEMORY_BYTES).setValue(vMem); + if (cpuTime != ResourceCalculatorProcessTree.UNAVAILABLE && + initCpuCumulativeTime != ResourceCalculatorProcessTree.UNAVAILABLE) { + cpuTime -= initCpuCumulativeTime; + } + + if (cpuTime != ResourceCalculatorProcessTree.UNAVAILABLE) { + counters.findCounter(TaskCounter.CPU_MILLISECONDS).setValue(cpuTime); + } + + if (pMem != ResourceCalculatorProcessTree.UNAVAILABLE) { + counters.findCounter(TaskCounter.PHYSICAL_MEMORY_BYTES).setValue(pMem); + } + + if (vMem != ResourceCalculatorProcessTree.UNAVAILABLE) { + counters.findCounter(TaskCounter.VIRTUAL_MEMORY_BYTES).setValue(vMem); + } } /** diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskUmbilicalProtocol.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskUmbilicalProtocol.java index 5df02c7b5b14e..c3678d67065e2 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskUmbilicalProtocol.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskUmbilicalProtocol.java @@ -178,7 +178,6 @@ void preempted(TaskAttemptID taskId, TaskStatus taskStatus) * * @param taskID task's id * @return the most recent checkpoint (if any) for this task - * @throws IOException */ TaskCheckpointID getCheckpointID(TaskID taskID); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainMapper.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainMapper.java index 14f040af96a5b..723a234d301db 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainMapper.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainMapper.java @@ -29,61 +29,61 @@ /** * The ChainMapper class allows to use multiple Mapper classes within a single * Map task. - *

      + *

      * The Mapper classes are invoked in a chained (or piped) fashion, the output of * the first becomes the input of the second, and so on until the last Mapper, * the output of the last Mapper will be written to the task's output. - *

      + *

      * The key functionality of this feature is that the Mappers in the chain do not * need to be aware that they are executed in a chain. This enables having * reusable specialized Mappers that can be combined to perform composite * operations within a single task. - *

      + *

      * Special care has to be taken when creating chains that the key/values output * by a Mapper are valid for the following Mapper in the chain. It is assumed * all Mappers and the Reduce in the chain use maching output and input key and * value classes as no conversion is done by the chaining code. - *

      + *

      * Using the ChainMapper and the ChainReducer classes is possible to compose * Map/Reduce jobs that look like [MAP+ / REDUCE MAP*]. And * immediate benefit of this pattern is a dramatic reduction in disk IO. - *

      + *

      * IMPORTANT: There is no need to specify the output key/value classes for the * ChainMapper, this is done by the addMapper for the last mapper in the chain. - *

      + *

      * ChainMapper usage pattern: - *

      + *

      * 

        * ...
  * conf.setJobName("chain");
  * conf.setInputFormat(TextInputFormat.class);
  * conf.setOutputFormat(TextOutputFormat.class);
- * 
      
+ *
  * JobConf mapAConf = new JobConf(false);
  * ...
  * ChainMapper.addMapper(conf, AMap.class, LongWritable.class, Text.class,
  *   Text.class, Text.class, true, mapAConf);
- * 
      
+ *
  * JobConf mapBConf = new JobConf(false);
  * ...
  * ChainMapper.addMapper(conf, BMap.class, Text.class, Text.class,
  *   LongWritable.class, Text.class, false, mapBConf);
- * 
      
+ *
  * JobConf reduceConf = new JobConf(false);
  * ...
  * ChainReducer.setReducer(conf, XReduce.class, LongWritable.class, Text.class,
  *   Text.class, Text.class, true, reduceConf);
- * 
      
+ *
  * ChainReducer.addMapper(conf, CMap.class, Text.class, Text.class,
  *   LongWritable.class, Text.class, false, null);
- * 
      
+ *
  * ChainReducer.addMapper(conf, DMap.class, LongWritable.class, Text.class,
  *   LongWritable.class, LongWritable.class, true, null);
- * 
      
+ *
  * FileInputFormat.setInputPaths(conf, inDir);
  * FileOutputFormat.setOutputPath(conf, outDir);
  * ...
- * 
      
+ *
  * JobClient jc = new JobClient(conf);
  * RunningJob job = jc.submitJob(conf);
  * ...
@@ -95,21 +95,21 @@ public class ChainMapper implements Mapper {
 
   /**
    * Adds a Mapper class to the chain job's JobConf.
-   * 
      
+   * 
      
    * It has to be specified how key and values are passed from one element of
    * the chain to the next, by value or by reference. If a Mapper leverages the
    * assumed semantics that the key and values are not modified by the collector
    * 'by value' must be used. If the Mapper does not expect this semantics, as
    * an optimization to avoid serialization and deserialization 'by reference'
    * can be used.
-   * 
      
+   * 
      
    * For the added Mapper the configuration given for it,
    * mapperConf, have precedence over the job's JobConf. This
    * precedence is in effect when the task is running.
-   * 
      
+   * 
      
    * IMPORTANT: There is no need to specify the output key/value classes for the
    * ChainMapper, this is done by the addMapper for the last mapper in the chain
-   * 
      
+   * 
      
    *
    * @param job              job's JobConf to add the Mapper class.
    * @param klass            the Mapper class to add.
@@ -148,7 +148,7 @@ public ChainMapper() {
 
   /**
    * Configures the ChainMapper and all the Mappers in the chain.
-   * 
      
+   * 
      
    * If this method is overriden super.configure(...) should be
    * invoked at the beginning of the overwriter method.
    */
@@ -171,7 +171,7 @@ public void map(Object key, Object value, OutputCollector output,
 
   /**
    * Closes  the ChainMapper and all the Mappers in the chain.
-   * 
      
+   * 
      
    * If this method is overriden super.close() should be
    * invoked at the end of the overwriter method.
    */
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainReducer.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainReducer.java
index 641d82c0839aa..6f5b7cde2e52f 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainReducer.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/ChainReducer.java
@@ -27,63 +27,63 @@
 /**
  * The ChainReducer class allows to chain multiple Mapper classes after a
  * Reducer within the Reducer task.
- * 
      
+ * 
      
  * For each record output by the Reducer, the Mapper classes are invoked in a
  * chained (or piped) fashion, the output of the first becomes the input of the
  * second, and so on until the last Mapper, the output of the last Mapper will
  * be written to the task's output.
- * 
      
+ * 
      
  * The key functionality of this feature is that the Mappers in the chain do not
  * need to be aware that they are executed after the Reducer or in a chain.
  * This enables having reusable specialized Mappers that can be combined to
  * perform composite operations within a single task.
- * 
      
+ * 
      
  * Special care has to be taken when creating chains that the key/values output
  * by a Mapper are valid for the following Mapper in the chain. It is assumed
  * all Mappers and the Reduce in the chain use maching output and input key and
  * value classes as no conversion is done by the chaining code.
- * 
      
+ * 
      
  * Using the ChainMapper and the ChainReducer classes is possible to compose
  * Map/Reduce jobs that look like [MAP+ / REDUCE MAP*]. And
  * immediate benefit of this pattern is a dramatic reduction in disk IO.
- * 
      
+ * 
      
  * IMPORTANT: There is no need to specify the output key/value classes for the
  * ChainReducer, this is done by the setReducer or the addMapper for the last
  * element in the chain.
- * 
      
+ * 
      
  * ChainReducer usage pattern:
- * 
      
+ * 
      
  * 
        * ...
  * conf.setJobName("chain");
  * conf.setInputFormat(TextInputFormat.class);
  * conf.setOutputFormat(TextOutputFormat.class);
- * 
      
+ *
  * JobConf mapAConf = new JobConf(false);
  * ...
  * ChainMapper.addMapper(conf, AMap.class, LongWritable.class, Text.class,
  *   Text.class, Text.class, true, mapAConf);
- * 
      
+ *
  * JobConf mapBConf = new JobConf(false);
  * ...
  * ChainMapper.addMapper(conf, BMap.class, Text.class, Text.class,
  *   LongWritable.class, Text.class, false, mapBConf);
- * 
      
+ *
  * JobConf reduceConf = new JobConf(false);
  * ...
  * ChainReducer.setReducer(conf, XReduce.class, LongWritable.class, Text.class,
  *   Text.class, Text.class, true, reduceConf);
- * 
      
+ *
  * ChainReducer.addMapper(conf, CMap.class, Text.class, Text.class,
  *   LongWritable.class, Text.class, false, null);
- * 
      
+ *
  * ChainReducer.addMapper(conf, DMap.class, LongWritable.class, Text.class,
  *   LongWritable.class, LongWritable.class, true, null);
- * 
      
+ *
  * FileInputFormat.setInputPaths(conf, inDir);
  * FileOutputFormat.setOutputPath(conf, outDir);
  * ...
- * 
      
+ *
  * JobClient jc = new JobClient(conf);
  * RunningJob job = jc.submitJob(conf);
  * ...
@@ -95,18 +95,18 @@ public class ChainReducer implements Reducer {
 
   /**
    * Sets the Reducer class to the chain job's JobConf.
-   * 
      
+   * 
      
    * It has to be specified how key and values are passed from one element of
    * the chain to the next, by value or by reference. If a Reducer leverages the
    * assumed semantics that the key and values are not modified by the collector
    * 'by value' must be used. If the Reducer does not expect this semantics, as
    * an optimization to avoid serialization and deserialization 'by reference'
    * can be used.
-   * 
      
+   * 
      
    * For the added Reducer the configuration given for it,
    * reducerConf, have precedence over the job's JobConf. This
    * precedence is in effect when the task is running.
-   * 
      
+   * 
      
    * IMPORTANT: There is no need to specify the output key/value classes for the
    * ChainReducer, this is done by the setReducer or the addMapper for the last
    * element in the chain.
@@ -139,18 +139,18 @@ public static  void setReducer(JobConf job,
 
   /**
    * Adds a Mapper class to the chain job's JobConf.
-   * 
      
+   * 
      
    * It has to be specified how key and values are passed from one element of
    * the chain to the next, by value or by reference. If a Mapper leverages the
    * assumed semantics that the key and values are not modified by the collector
    * 'by value' must be used. If the Mapper does not expect this semantics, as
    * an optimization to avoid serialization and deserialization 'by reference'
    * can be used.
-   * 
      
+   * 
      
    * For the added Mapper the configuration given for it,
    * mapperConf, have precedence over the job's JobConf. This
    * precedence is in effect when the task is running.
-   * 
      
+   * 
      
    * IMPORTANT: There is no need to specify the output key/value classes for the
    * ChainMapper, this is done by the addMapper for the last mapper in the chain
    * .
@@ -191,7 +191,7 @@ public ChainReducer() {
 
   /**
    * Configures the ChainReducer, the Reducer and all the Mappers in the chain.
-   * 
      
+   * 
      
    * If this method is overriden super.configure(...) should be
    * invoked at the beginning of the overwriter method.
    */
@@ -215,7 +215,7 @@ public void reduce(Object key, Iterator values, OutputCollector output,
 
   /**
    * Closes  the ChainReducer, the Reducer and all the Mappers in the chain.
-   * 
      
+   * 
      
    * If this method is overriden super.close() should be
    * invoked at the end of the overwriter method.
    */
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/MultipleOutputs.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/MultipleOutputs.java
index 39e80f9a16b52..f0f3652beb13c 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/MultipleOutputs.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/MultipleOutputs.java
@@ -31,29 +31,29 @@
  * than the job default output via the OutputCollector passed to
  * the map() and reduce() methods of the
  * Mapper and Reducer implementations.
- * 
      
+ * 
      
  * Each additional output, or named output, may be configured with its own
  * OutputFormat, with its own key class and with its own value
  * class.
- * 
      
+ * 
      
  * A named output can be a single file or a multi file. The later is refered as
  * a multi named output.
- * 
      
+ * 
      
  * A multi named output is an unbound set of files all sharing the same
  * OutputFormat, key class and value class configuration.
- * 
      
+ * 
      
  * When named outputs are used within a Mapper implementation,
  * key/values written to a name output are not part of the reduce phase, only
  * key/values written to the job OutputCollector are part of the
  * reduce phase.
- * 
      
+ * 
      
  * MultipleOutputs supports counters, by default the are disabled. The counters
  * group is the {@link MultipleOutputs} class name.
  * 
      
  * The names of the counters are the same as the named outputs. For multi
  * named outputs the name of the counter is the concatenation of the named
  * output, and underscore '_' and the multiname.
- * 
      
+ * 
      
  * Job configuration usage pattern is:
  * 
        *
@@ -82,7 +82,7 @@
  *
  * ...
  * 
      
- * 
      
+ * 
      
  * Job configuration usage pattern is:
  * 
        *
@@ -271,7 +271,6 @@ public static Class getNamedOutputValueClass(JobConf conf,
 
   /**
    * Adds a named output for the job.
-   * 
      
    *
    * @param conf              job conf to add the named output
    * @param namedOutput       named output name, it has to be a word, letters
@@ -291,7 +290,6 @@ public static void addNamedOutput(JobConf conf, String namedOutput,
 
   /**
    * Adds a multi named output for the job.
-   * 
      
    *
    * @param conf              job conf to add the named output
    * @param namedOutput       named output name, it has to be a word, letters
@@ -311,7 +309,6 @@ public static void addMultiNamedOutput(JobConf conf, String namedOutput,
 
   /**
    * Adds a named output for the job.
-   * 
      
    *
    * @param conf              job conf to add the named output
    * @param namedOutput       named output name, it has to be a word, letters
@@ -339,9 +336,9 @@ private static void addNamedOutput(JobConf conf, String namedOutput,
 
   /**
    * Enables or disables counters for the named outputs.
-   * 
      
+   * 
      
    * By default these counters are disabled.
-   * 
      
+   * 
      
    * MultipleOutputs supports counters, by default the are disabled.
    * The counters group is the {@link MultipleOutputs} class name.
    * 
      
@@ -358,9 +355,9 @@ public static void setCountersEnabled(JobConf conf, boolean enabled) {
 
   /**
    * Returns if the counters for the named outputs are enabled or not.
-   * 
      
+   * 
      
    * By default these counters are disabled.
-   * 
      
+   * 
      
    * MultipleOutputs supports counters, by default the are disabled.
    * The counters group is the {@link MultipleOutputs} class name.
    * 
      
@@ -465,7 +462,6 @@ public void close(Reporter reporter) throws IOException {
 
   /**
    * Gets the output collector for a named output.
-   * 
      
    *
    * @param namedOutput the named output name
    * @param reporter    the reporter
@@ -480,7 +476,6 @@ public OutputCollector getCollector(String namedOutput, Reporter reporter)
 
   /**
    * Gets the output collector for a multi named output.
-   * 
      
    *
    * @param namedOutput the named output name
    * @param multiName   the multi name part
@@ -525,7 +520,7 @@ public void collect(Object key, Object value) throws IOException {
 
   /**
    * Closes all the opened named outputs.
-   * 
      
+   * 
      
    * If overriden subclasses must invoke super.close() at the
    * end of their close()
    *
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TokenCountMapper.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TokenCountMapper.java
index 8e884cee8bbc8..75179e110f97c 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TokenCountMapper.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TokenCountMapper.java
@@ -32,7 +32,7 @@
 
 
 /** 
- * A {@link Mapper} that maps text values into  pairs.  Uses
+ * A {@link Mapper} that maps text values into <token,freq> pairs.  Uses
  * {@link StringTokenizer} to break text into tokens. 
  */
 @InterfaceAudience.Public
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java
index 8c20723b711bd..6251925ea2e15 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorJob.java
@@ -60,7 +60,7 @@
  * The developer using Aggregate will need only to provide a plugin class
  * conforming to the following interface:
  * 
- * public interface ValueAggregatorDescriptor { public ArrayList
+ * public interface ValueAggregatorDescriptor { public ArrayList<Entry>
  * generateKeyValPairs(Object key, Object value); public void
  * configure(JobConfjob); }
  * 
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorReducer.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorReducer.java
index a6b357324dad3..2738968d6b106 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorReducer.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/aggregate/ValueAggregatorReducer.java
@@ -45,7 +45,8 @@ public class ValueAggregatorReducer values,
                      OutputCollector output, Reporter reporter) throws IOException {
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/db/DBInputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/db/DBInputFormat.java
index 2715705407ef6..159919f5a04de 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/db/DBInputFormat.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/db/DBInputFormat.java
@@ -195,8 +195,8 @@ public InputSplit[] getSplits(JobConf job, int chunks) throws IOException {
    * @param inputClass the class object implementing DBWritable, which is the 
    * Java object holding tuple fields.
    * @param tableName The table to read data from
-   * @param conditions The condition which to select data with, eg. '(updated >
-   * 20070101 AND length > 0)'
+   * @param conditions The condition which to select data with, eg. '(updated >
+   * 20070101 AND length > 0)'
    * @param orderBy the fieldNames in the orderBy clause.
    * @param fieldNames The field names in the table
    * @see #setInput(JobConf, Class, String, String)
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java
index 8f4259e3ec157..4f5b6a1c9c49b 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java
@@ -363,7 +363,7 @@ Parser createParser() {
     void printUsage() {
       // The CLI package should do this for us, but I can't figure out how
       // to make it print something reasonable.
-      System.out.println("bin/hadoop pipes");
+      System.out.println("Usage: pipes ");
       System.out.println("  [-input ] // Input directory");
       System.out.println("  [-output ] // Output directory");
       System.out.println("  [-jar  // jar filename");
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
index 60ff715cb8332..34353acb87853 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
@@ -134,6 +134,7 @@ Configuration getConf() {
   
   /**
    * Close the Cluster.
+   * @throws IOException
    */
   public synchronized void close() throws IOException {
     clientProtocolProvider.close(client);
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/ClusterMetrics.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/ClusterMetrics.java
index c4c2778dd0b2b..b5e54b5eaf28e 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/ClusterMetrics.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/ClusterMetrics.java
@@ -40,15 +40,15 @@
  *   Slot capacity of the cluster. 
  *
      4. *
    4. - * The number of currently occupied/reserved map & reduce slots. + * The number of currently occupied/reserved map and reduce slots. *
      5. *
    5. - * The number of currently running map & reduce tasks. + * The number of currently running map and reduce tasks. *
      6. *
    6. * The number of job submissions. *
      7. - *

    + * * *

    Clients can query for the latest ClusterMetrics, via * {@link Cluster#getClusterStatus()}.

    diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java index 184cdf0618f91..ef0617690931f 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java @@ -123,11 +123,11 @@ public static FSDataOutputStream wrapIfNecessary(Configuration conf, * "mapreduce.job.encrypted-intermediate-data.buffer.kb" Job configuration * variable. * - * If the value of 'length' is > -1, The InputStream is additionally wrapped - * in a LimitInputStream. CryptoStreams are late buffering in nature. This - * means they will always try to read ahead if they can. The LimitInputStream - * will ensure that the CryptoStream does not read past the provided length - * from the given Input Stream. + * If the value of 'length' is > -1, The InputStream is additionally + * wrapped in a LimitInputStream. CryptoStreams are late buffering in nature. + * This means they will always try to read ahead if they can. The + * LimitInputStream will ensure that the CryptoStream does not read past the + * provided length from the given Input Stream. * * @param conf * @param in diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java index 470290c266be5..9eea4cc76b977 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java @@ -69,7 +69,7 @@ * * // Submit the job, then poll for progress until the job is complete * job.waitForCompletion(true); - *

    + * * * */ @@ -98,6 +98,7 @@ public static enum JobState {DEFINE, RUNNING}; "mapreduce.client.genericoptionsparser.used"; public static final String SUBMIT_REPLICATION = "mapreduce.client.submit.file.replication"; + public static final int DEFAULT_SUBMIT_REPLICATION = 10; @InterfaceStability.Evolving public static enum TaskStatusFilter { NONE, KILLED, FAILED, SUCCEEDED, ALL } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java index 836f1829079af..6bd2d1f0d5118 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java @@ -289,7 +289,6 @@ public Class> getPartitionerClass() * Get the timestamps of the archives. Used by internal * DistributedCache and MapReduce code. * @return a string array of timestamps - * @throws IOException */ public String[] getArchiveTimestamps(); @@ -297,7 +296,6 @@ public Class> getPartitionerClass() * Get the timestamps of the files. Used by internal * DistributedCache and MapReduce code. * @return a string array of timestamps - * @throws IOException */ public String[] getFileTimestamps(); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java new file mode 100644 index 0000000000000..eebdf88311612 --- /dev/null +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java @@ -0,0 +1,363 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.mapreduce; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.InetAddress; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.UnknownHostException; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.permission.FsPermission; +import org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager; +import org.apache.hadoop.mapreduce.filecache.DistributedCache; + +@InterfaceAudience.Private +@InterfaceStability.Unstable +class JobResourceUploader { + protected static final Log LOG = LogFactory.getLog(JobResourceUploader.class); + private FileSystem jtFs; + + JobResourceUploader(FileSystem submitFs) { + this.jtFs = submitFs; + } + + /** + * Upload and configure files, libjars, jobjars, and archives pertaining to + * the passed job. + * + * @param job the job containing the files to be uploaded + * @param submitJobDir the submission directory of the job + * @throws IOException + */ + public void uploadFiles(Job job, Path submitJobDir) throws IOException { + Configuration conf = job.getConfiguration(); + short replication = + (short) conf.getInt(Job.SUBMIT_REPLICATION, + Job.DEFAULT_SUBMIT_REPLICATION); + + if (!(conf.getBoolean(Job.USED_GENERIC_PARSER, false))) { + LOG.warn("Hadoop command-line option parsing not performed. " + + "Implement the Tool interface and execute your application " + + "with ToolRunner to remedy this."); + } + + // get all the command line arguments passed in by the user conf + String files = conf.get("tmpfiles"); + String libjars = conf.get("tmpjars"); + String archives = conf.get("tmparchives"); + String jobJar = job.getJar(); + + // + // Figure out what fs the JobTracker is using. Copy the + // job to it, under a temporary name. This allows DFS to work, + // and under the local fs also provides UNIX-like object loading + // semantics. (that is, if the job file is deleted right after + // submission, we can still run the submission to completion) + // + + // Create a number of filenames in the JobTracker's fs namespace + LOG.debug("default FileSystem: " + jtFs.getUri()); + if (jtFs.exists(submitJobDir)) { + throw new IOException("Not submitting job. Job directory " + submitJobDir + + " already exists!! This is unexpected.Please check what's there in" + + " that directory"); + } + submitJobDir = jtFs.makeQualified(submitJobDir); + submitJobDir = new Path(submitJobDir.toUri().getPath()); + FsPermission mapredSysPerms = + new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION); + FileSystem.mkdirs(jtFs, submitJobDir, mapredSysPerms); + Path filesDir = JobSubmissionFiles.getJobDistCacheFiles(submitJobDir); + Path archivesDir = JobSubmissionFiles.getJobDistCacheArchives(submitJobDir); + Path libjarsDir = JobSubmissionFiles.getJobDistCacheLibjars(submitJobDir); + // add all the command line files/ jars and archive + // first copy them to jobtrackers filesystem + + if (files != null) { + FileSystem.mkdirs(jtFs, filesDir, mapredSysPerms); + String[] fileArr = files.split(","); + for (String tmpFile : fileArr) { + URI tmpURI = null; + try { + tmpURI = new URI(tmpFile); + } catch (URISyntaxException e) { + throw new IllegalArgumentException(e); + } + Path tmp = new Path(tmpURI); + Path newPath = copyRemoteFiles(filesDir, tmp, conf, replication); + try { + URI pathURI = getPathURI(newPath, tmpURI.getFragment()); + DistributedCache.addCacheFile(pathURI, conf); + } catch (URISyntaxException ue) { + // should not throw a uri exception + throw new IOException("Failed to create uri for " + tmpFile, ue); + } + } + } + + if (libjars != null) { + FileSystem.mkdirs(jtFs, libjarsDir, mapredSysPerms); + String[] libjarsArr = libjars.split(","); + for (String tmpjars : libjarsArr) { + Path tmp = new Path(tmpjars); + Path newPath = copyRemoteFiles(libjarsDir, tmp, conf, replication); + DistributedCache.addFileToClassPath( + new Path(newPath.toUri().getPath()), conf); + } + } + + if (archives != null) { + FileSystem.mkdirs(jtFs, archivesDir, mapredSysPerms); + String[] archivesArr = archives.split(","); + for (String tmpArchives : archivesArr) { + URI tmpURI; + try { + tmpURI = new URI(tmpArchives); + } catch (URISyntaxException e) { + throw new IllegalArgumentException(e); + } + Path tmp = new Path(tmpURI); + Path newPath = copyRemoteFiles(archivesDir, tmp, conf, replication); + try { + URI pathURI = getPathURI(newPath, tmpURI.getFragment()); + DistributedCache.addCacheArchive(pathURI, conf); + } catch (URISyntaxException ue) { + // should not throw an uri excpetion + throw new IOException("Failed to create uri for " + tmpArchives, ue); + } + } + } + + if (jobJar != null) { // copy jar to JobTracker's fs + // use jar name if job is not named. + if ("".equals(job.getJobName())) { + job.setJobName(new Path(jobJar).getName()); + } + Path jobJarPath = new Path(jobJar); + URI jobJarURI = jobJarPath.toUri(); + // If the job jar is already in a global fs, + // we don't need to copy it from local fs + if (jobJarURI.getScheme() == null || jobJarURI.getScheme().equals("file")) { + copyJar(jobJarPath, JobSubmissionFiles.getJobJar(submitJobDir), + replication); + job.setJar(JobSubmissionFiles.getJobJar(submitJobDir).toString()); + } + } else { + LOG.warn("No job jar file set. User classes may not be found. " + + "See Job or Job#setJar(String)."); + } + + addLog4jToDistributedCache(job, submitJobDir); + + // set the timestamps of the archives and files + // set the public/private visibility of the archives and files + ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(conf); + // get DelegationToken for cached file + ClientDistributedCacheManager.getDelegationTokens(conf, + job.getCredentials()); + } + + // copies a file to the jobtracker filesystem and returns the path where it + // was copied to + private Path copyRemoteFiles(Path parentDir, Path originalPath, + Configuration conf, short replication) throws IOException { + // check if we do not need to copy the files + // is jt using the same file system. + // just checking for uri strings... doing no dns lookups + // to see if the filesystems are the same. This is not optimal. + // but avoids name resolution. + + FileSystem remoteFs = null; + remoteFs = originalPath.getFileSystem(conf); + if (compareFs(remoteFs, jtFs)) { + return originalPath; + } + // this might have name collisions. copy will throw an exception + // parse the original path to create new path + Path newPath = new Path(parentDir, originalPath.getName()); + FileUtil.copy(remoteFs, originalPath, jtFs, newPath, false, conf); + jtFs.setReplication(newPath, replication); + return newPath; + } + + /* + * see if two file systems are the same or not. + */ + private boolean compareFs(FileSystem srcFs, FileSystem destFs) { + URI srcUri = srcFs.getUri(); + URI dstUri = destFs.getUri(); + if (srcUri.getScheme() == null) { + return false; + } + if (!srcUri.getScheme().equals(dstUri.getScheme())) { + return false; + } + String srcHost = srcUri.getHost(); + String dstHost = dstUri.getHost(); + if ((srcHost != null) && (dstHost != null)) { + try { + srcHost = InetAddress.getByName(srcHost).getCanonicalHostName(); + dstHost = InetAddress.getByName(dstHost).getCanonicalHostName(); + } catch (UnknownHostException ue) { + return false; + } + if (!srcHost.equals(dstHost)) { + return false; + } + } else if (srcHost == null && dstHost != null) { + return false; + } else if (srcHost != null && dstHost == null) { + return false; + } + // check for ports + if (srcUri.getPort() != dstUri.getPort()) { + return false; + } + return true; + } + + private void copyJar(Path originalJarPath, Path submitJarFile, + short replication) throws IOException { + jtFs.copyFromLocalFile(originalJarPath, submitJarFile); + jtFs.setReplication(submitJarFile, replication); + jtFs.setPermission(submitJarFile, new FsPermission( + JobSubmissionFiles.JOB_FILE_PERMISSION)); + } + + private void addLog4jToDistributedCache(Job job, Path jobSubmitDir) + throws IOException { + Configuration conf = job.getConfiguration(); + String log4jPropertyFile = + conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, ""); + if (!log4jPropertyFile.isEmpty()) { + short replication = (short) conf.getInt(Job.SUBMIT_REPLICATION, 10); + copyLog4jPropertyFile(job, jobSubmitDir, replication); + } + } + + private URI getPathURI(Path destPath, String fragment) + throws URISyntaxException { + URI pathURI = destPath.toUri(); + if (pathURI.getFragment() == null) { + if (fragment == null) { + pathURI = new URI(pathURI.toString() + "#" + destPath.getName()); + } else { + pathURI = new URI(pathURI.toString() + "#" + fragment); + } + } + return pathURI; + } + + // copy user specified log4j.property file in local + // to HDFS with putting on distributed cache and adding its parent directory + // to classpath. + @SuppressWarnings("deprecation") + private void copyLog4jPropertyFile(Job job, Path submitJobDir, + short replication) throws IOException { + Configuration conf = job.getConfiguration(); + + String file = + validateFilePath( + conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE), conf); + LOG.debug("default FileSystem: " + jtFs.getUri()); + FsPermission mapredSysPerms = + new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION); + if (!jtFs.exists(submitJobDir)) { + throw new IOException("Cannot find job submission directory! " + + "It should just be created, so something wrong here."); + } + + Path fileDir = JobSubmissionFiles.getJobLog4jFile(submitJobDir); + + // first copy local log4j.properties file to HDFS under submitJobDir + if (file != null) { + FileSystem.mkdirs(jtFs, fileDir, mapredSysPerms); + URI tmpURI = null; + try { + tmpURI = new URI(file); + } catch (URISyntaxException e) { + throw new IllegalArgumentException(e); + } + Path tmp = new Path(tmpURI); + Path newPath = copyRemoteFiles(fileDir, tmp, conf, replication); + DistributedCache.addFileToClassPath(new Path(newPath.toUri().getPath()), + conf); + } + } + + /** + * takes input as a path string for file and verifies if it exist. It defaults + * for file:/// if the files specified do not have a scheme. it returns the + * paths uri converted defaulting to file:///. So an input of /home/user/file1 + * would return file:///home/user/file1 + * + * @param file + * @param conf + * @return + */ + private String validateFilePath(String file, Configuration conf) + throws IOException { + if (file == null) { + return null; + } + if (file.isEmpty()) { + throw new IllegalArgumentException("File name can't be empty string"); + } + String finalPath; + URI pathURI; + try { + pathURI = new URI(file); + } catch (URISyntaxException e) { + throw new IllegalArgumentException(e); + } + Path path = new Path(pathURI); + FileSystem localFs = FileSystem.getLocal(conf); + if (pathURI.getScheme() == null) { + // default to the local file system + // check if the file exists or not first + if (!localFs.exists(path)) { + throw new FileNotFoundException("File " + file + " does not exist."); + } + finalPath = + path.makeQualified(localFs.getUri(), localFs.getWorkingDirectory()) + .toString(); + } else { + // check if the file exists in this file system + // we need to recreate this filesystem object to copy + // these files to the file system ResourceManager is running + // on. + FileSystem fs = path.getFileSystem(conf); + if (!fs.exists(path)) { + throw new FileNotFoundException("File " + file + " does not exist."); + } + finalPath = + path.makeQualified(fs.getUri(), fs.getWorkingDirectory()).toString(); + } + return finalPath; + } +} diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java index 516e661133be2..712507774739e 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java @@ -100,7 +100,7 @@ public static Path getJobDistCacheLibjars(Path jobSubmitDir) { /** * Initializes the staging directory and returns the path. It also - * keeps track of all necessary ownership & permissions + * keeps track of all necessary ownership and permissions * @param cluster * @param conf */ diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java index 75357f73c4b8b..023bd63f86fa8 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java @@ -86,297 +86,6 @@ class JobSubmitter { this.submitClient = submitClient; this.jtFs = submitFs; } - /* - * see if two file systems are the same or not. - */ - private boolean compareFs(FileSystem srcFs, FileSystem destFs) { - URI srcUri = srcFs.getUri(); - URI dstUri = destFs.getUri(); - if (srcUri.getScheme() == null) { - return false; - } - if (!srcUri.getScheme().equals(dstUri.getScheme())) { - return false; - } - String srcHost = srcUri.getHost(); - String dstHost = dstUri.getHost(); - if ((srcHost != null) && (dstHost != null)) { - try { - srcHost = InetAddress.getByName(srcHost).getCanonicalHostName(); - dstHost = InetAddress.getByName(dstHost).getCanonicalHostName(); - } catch(UnknownHostException ue) { - return false; - } - if (!srcHost.equals(dstHost)) { - return false; - } - } else if (srcHost == null && dstHost != null) { - return false; - } else if (srcHost != null && dstHost == null) { - return false; - } - //check for ports - if (srcUri.getPort() != dstUri.getPort()) { - return false; - } - return true; - } - - // copies a file to the jobtracker filesystem and returns the path where it - // was copied to - private Path copyRemoteFiles(Path parentDir, - Path originalPath, Configuration conf, short replication) - throws IOException { - //check if we do not need to copy the files - // is jt using the same file system. - // just checking for uri strings... doing no dns lookups - // to see if the filesystems are the same. This is not optimal. - // but avoids name resolution. - - FileSystem remoteFs = null; - remoteFs = originalPath.getFileSystem(conf); - if (compareFs(remoteFs, jtFs)) { - return originalPath; - } - // this might have name collisions. copy will throw an exception - //parse the original path to create new path - Path newPath = new Path(parentDir, originalPath.getName()); - FileUtil.copy(remoteFs, originalPath, jtFs, newPath, false, conf); - jtFs.setReplication(newPath, replication); - return newPath; - } - - // configures -files, -libjars and -archives. - private void copyAndConfigureFiles(Job job, Path submitJobDir, - short replication) throws IOException { - Configuration conf = job.getConfiguration(); - if (!(conf.getBoolean(Job.USED_GENERIC_PARSER, false))) { - LOG.warn("Hadoop command-line option parsing not performed. " + - "Implement the Tool interface and execute your application " + - "with ToolRunner to remedy this."); - } - - // get all the command line arguments passed in by the user conf - String files = conf.get("tmpfiles"); - String libjars = conf.get("tmpjars"); - String archives = conf.get("tmparchives"); - String jobJar = job.getJar(); - - // - // Figure out what fs the JobTracker is using. Copy the - // job to it, under a temporary name. This allows DFS to work, - // and under the local fs also provides UNIX-like object loading - // semantics. (that is, if the job file is deleted right after - // submission, we can still run the submission to completion) - // - - // Create a number of filenames in the JobTracker's fs namespace - LOG.debug("default FileSystem: " + jtFs.getUri()); - if (jtFs.exists(submitJobDir)) { - throw new IOException("Not submitting job. Job directory " + submitJobDir - +" already exists!! This is unexpected.Please check what's there in" + - " that directory"); - } - submitJobDir = jtFs.makeQualified(submitJobDir); - submitJobDir = new Path(submitJobDir.toUri().getPath()); - FsPermission mapredSysPerms = new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION); - FileSystem.mkdirs(jtFs, submitJobDir, mapredSysPerms); - Path filesDir = JobSubmissionFiles.getJobDistCacheFiles(submitJobDir); - Path archivesDir = JobSubmissionFiles.getJobDistCacheArchives(submitJobDir); - Path libjarsDir = JobSubmissionFiles.getJobDistCacheLibjars(submitJobDir); - // add all the command line files/ jars and archive - // first copy them to jobtrackers filesystem - - if (files != null) { - FileSystem.mkdirs(jtFs, filesDir, mapredSysPerms); - String[] fileArr = files.split(","); - for (String tmpFile: fileArr) { - URI tmpURI = null; - try { - tmpURI = new URI(tmpFile); - } catch (URISyntaxException e) { - throw new IllegalArgumentException(e); - } - Path tmp = new Path(tmpURI); - Path newPath = copyRemoteFiles(filesDir, tmp, conf, replication); - try { - URI pathURI = getPathURI(newPath, tmpURI.getFragment()); - DistributedCache.addCacheFile(pathURI, conf); - } catch(URISyntaxException ue) { - //should not throw a uri exception - throw new IOException("Failed to create uri for " + tmpFile, ue); - } - } - } - - if (libjars != null) { - FileSystem.mkdirs(jtFs, libjarsDir, mapredSysPerms); - String[] libjarsArr = libjars.split(","); - for (String tmpjars: libjarsArr) { - Path tmp = new Path(tmpjars); - Path newPath = copyRemoteFiles(libjarsDir, tmp, conf, replication); - DistributedCache.addFileToClassPath( - new Path(newPath.toUri().getPath()), conf); - } - } - - if (archives != null) { - FileSystem.mkdirs(jtFs, archivesDir, mapredSysPerms); - String[] archivesArr = archives.split(","); - for (String tmpArchives: archivesArr) { - URI tmpURI; - try { - tmpURI = new URI(tmpArchives); - } catch (URISyntaxException e) { - throw new IllegalArgumentException(e); - } - Path tmp = new Path(tmpURI); - Path newPath = copyRemoteFiles(archivesDir, tmp, conf, - replication); - try { - URI pathURI = getPathURI(newPath, tmpURI.getFragment()); - DistributedCache.addCacheArchive(pathURI, conf); - } catch(URISyntaxException ue) { - //should not throw an uri excpetion - throw new IOException("Failed to create uri for " + tmpArchives, ue); - } - } - } - - if (jobJar != null) { // copy jar to JobTracker's fs - // use jar name if job is not named. - if ("".equals(job.getJobName())){ - job.setJobName(new Path(jobJar).getName()); - } - Path jobJarPath = new Path(jobJar); - URI jobJarURI = jobJarPath.toUri(); - // If the job jar is already in a global fs, - // we don't need to copy it from local fs - if ( jobJarURI.getScheme() == null - || jobJarURI.getScheme().equals("file")) { - copyJar(jobJarPath, JobSubmissionFiles.getJobJar(submitJobDir), - replication); - job.setJar(JobSubmissionFiles.getJobJar(submitJobDir).toString()); - } - } else { - LOG.warn("No job jar file set. User classes may not be found. "+ - "See Job or Job#setJar(String)."); - } - - addLog4jToDistributedCache(job, submitJobDir); - - // set the timestamps of the archives and files - // set the public/private visibility of the archives and files - ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(conf); - // get DelegationToken for cached file - ClientDistributedCacheManager.getDelegationTokens(conf, job - .getCredentials()); - } - - // copy user specified log4j.property file in local - // to HDFS with putting on distributed cache and adding its parent directory - // to classpath. - @SuppressWarnings("deprecation") - private void copyLog4jPropertyFile(Job job, Path submitJobDir, - short replication) throws IOException { - Configuration conf = job.getConfiguration(); - - String file = validateFilePath( - conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE), conf); - LOG.debug("default FileSystem: " + jtFs.getUri()); - FsPermission mapredSysPerms = - new FsPermission(JobSubmissionFiles.JOB_DIR_PERMISSION); - if (!jtFs.exists(submitJobDir)) { - throw new IOException("Cannot find job submission directory! " - + "It should just be created, so something wrong here."); - } - - Path fileDir = JobSubmissionFiles.getJobLog4jFile(submitJobDir); - - // first copy local log4j.properties file to HDFS under submitJobDir - if (file != null) { - FileSystem.mkdirs(jtFs, fileDir, mapredSysPerms); - URI tmpURI = null; - try { - tmpURI = new URI(file); - } catch (URISyntaxException e) { - throw new IllegalArgumentException(e); - } - Path tmp = new Path(tmpURI); - Path newPath = copyRemoteFiles(fileDir, tmp, conf, replication); - DistributedCache.addFileToClassPath(new Path(newPath.toUri().getPath()), conf); - } - } - - /** - * takes input as a path string for file and verifies if it exist. - * It defaults for file:/// if the files specified do not have a scheme. - * it returns the paths uri converted defaulting to file:///. - * So an input of /home/user/file1 would return file:///home/user/file1 - * @param file - * @param conf - * @return - */ - private String validateFilePath(String file, Configuration conf) - throws IOException { - if (file == null) { - return null; - } - if (file.isEmpty()) { - throw new IllegalArgumentException("File name can't be empty string"); - } - String finalPath; - URI pathURI; - try { - pathURI = new URI(file); - } catch (URISyntaxException e) { - throw new IllegalArgumentException(e); - } - Path path = new Path(pathURI); - FileSystem localFs = FileSystem.getLocal(conf); - if (pathURI.getScheme() == null) { - //default to the local file system - //check if the file exists or not first - if (!localFs.exists(path)) { - throw new FileNotFoundException("File " + file + " does not exist."); - } - finalPath = path.makeQualified(localFs.getUri(), - localFs.getWorkingDirectory()).toString(); - } - else { - // check if the file exists in this file system - // we need to recreate this filesystem object to copy - // these files to the file system ResourceManager is running - // on. - FileSystem fs = path.getFileSystem(conf); - if (!fs.exists(path)) { - throw new FileNotFoundException("File " + file + " does not exist."); - } - finalPath = path.makeQualified(fs.getUri(), - fs.getWorkingDirectory()).toString(); - } - return finalPath; - } - - private URI getPathURI(Path destPath, String fragment) - throws URISyntaxException { - URI pathURI = destPath.toUri(); - if (pathURI.getFragment() == null) { - if (fragment == null) { - pathURI = new URI(pathURI.toString() + "#" + destPath.getName()); - } else { - pathURI = new URI(pathURI.toString() + "#" + fragment); - } - } - return pathURI; - } - - private void copyJar(Path originalJarPath, Path submitJarFile, - short replication) throws IOException { - jtFs.copyFromLocalFile(originalJarPath, submitJarFile); - jtFs.setReplication(submitJarFile, replication); - jtFs.setPermission(submitJarFile, new FsPermission(JobSubmissionFiles.JOB_FILE_PERMISSION)); - } /** * configure the jobconf of the user with the command line options of @@ -386,9 +95,8 @@ private void copyJar(Path originalJarPath, Path submitJarFile, */ private void copyAndConfigureFiles(Job job, Path jobSubmitDir) throws IOException { - Configuration conf = job.getConfiguration(); - short replication = (short)conf.getInt(Job.SUBMIT_REPLICATION, 10); - copyAndConfigureFiles(job, jobSubmitDir, replication); + JobResourceUploader rUploader = new JobResourceUploader(jtFs); + rUploader.uploadFiles(job, jobSubmitDir); // Get the working directory. If not set, sets it to filesystem working dir // This code has been added so that working directory reset before running @@ -396,8 +104,8 @@ private void copyAndConfigureFiles(Job job, Path jobSubmitDir) // might use the public API JobConf#setWorkingDirectory to reset the working // directory. job.getWorkingDirectory(); - } + /** * Internal method for submitting jobs to the system. * @@ -484,10 +192,7 @@ JobStatus submitJobInternal(Job job, Cluster cluster) } copyAndConfigureFiles(job, submitJobDir); - - - Path submitJobFile = JobSubmissionFiles.getJobConfPath(submitJobDir); // Create the splits for the job @@ -678,7 +383,7 @@ private void readTokensFromFiles(Configuration conf, Credentials credentials) throws IOException { // add tokens and secrets coming from a token storage file String binaryTokenFilename = - conf.get("mapreduce.job.credentials.binary"); + conf.get(MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY); if (binaryTokenFilename != null) { Credentials binary = Credentials.readTokenStorageFile( FileSystem.getLocal(conf).makeQualified( @@ -766,15 +471,4 @@ private static void addMRFrameworkToDistributedCache(Configuration conf) DistributedCache.addCacheArchive(uri, conf); } } - - private void addLog4jToDistributedCache(Job job, - Path jobSubmitDir) throws IOException { - Configuration conf = job.getConfiguration(); - String log4jPropertyFile = - conf.get(MRJobConfig.MAPREDUCE_JOB_LOG4J_PROPERTIES_FILE, ""); - if (!log4jPropertyFile.isEmpty()) { - short replication = (short)conf.getInt(Job.SUBMIT_REPLICATION, 10); - copyLog4jPropertyFile(job, jobSubmitDir, replication); - } - } } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java index d06b075e2f478..947c81426107a 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java @@ -49,6 +49,11 @@ public interface MRJobConfig { public static final String TASK_CLEANUP_NEEDED = "mapreduce.job.committer.task.cleanup.needed"; + public static final String TASK_PROGRESS_REPORT_INTERVAL = + "mapreduce.task.progress-report.interval"; + /** The number of milliseconds between progress reports. */ + public static final int DEFAULT_TASK_PROGRESS_REPORT_INTERVAL = 3000; + public static final String JAR = "mapreduce.job.jar"; public static final String ID = "mapreduce.job.id"; @@ -305,6 +310,7 @@ public interface MRJobConfig { = "mapreduce.reduce.shuffle.memory.limit.percent"; public static final String SHUFFLE_MERGE_PERCENT = "mapreduce.reduce.shuffle.merge.percent"; + public static final float DEFAULT_SHUFFLE_MERGE_PERCENT = 0.66f; public static final String REDUCE_FAILURES_MAXPERCENT = "mapreduce.reduce.failures.maxpercent"; @@ -373,6 +379,14 @@ public interface MRJobConfig { public static final String DEFAULT_JOB_ACL_MODIFY_JOB = " "; + public static final String JOB_RUNNING_MAP_LIMIT = + "mapreduce.job.running.map.limit"; + public static final int DEFAULT_JOB_RUNNING_MAP_LIMIT = 0; + + public static final String JOB_RUNNING_REDUCE_LIMIT = + "mapreduce.job.running.reduce.limit"; + public static final int DEFAULT_JOB_RUNNING_REDUCE_LIMIT = 0; + /* config for tracking the local file where all the credentials for the job * credentials. */ @@ -496,6 +510,14 @@ public interface MRJobConfig { public static final int DEFAULT_MR_AM_CONTAINERLAUNCHER_THREAD_COUNT_LIMIT = 500; + /** + * The initial size of thread pool to launch containers in the app master + */ + public static final String MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE = + MR_AM_PREFIX+"containerlauncher.threadpool-initial-size"; + public static final int DEFAULT_MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE = + 10; + /** Number of threads to handle job client RPC requests.*/ public static final String MR_AM_JOB_CLIENT_THREAD_COUNT = MR_AM_PREFIX + "job.client.thread-count"; @@ -636,6 +658,11 @@ public interface MRJobConfig { public static final int DEFAULT_MR_AM_HISTORY_USE_BATCHED_FLUSH_QUEUE_SIZE_THRESHOLD = 50; + public static final String MR_AM_HARD_KILL_TIMEOUT_MS = + MR_AM_PREFIX + "hard-kill-timeout-ms"; + public static final long DEFAULT_MR_AM_HARD_KILL_TIMEOUT_MS = + 10 * 1000l; + /** * The threshold in terms of seconds after which an unsatisfied mapper request * triggers reducer preemption to free space. Default 0 implies that the reduces diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Mapper.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Mapper.java index 3a6186b9b9158..6b4147b900bff 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Mapper.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Mapper.java @@ -42,9 +42,9 @@ * *

    The framework first calls * {@link #setup(org.apache.hadoop.mapreduce.Mapper.Context)}, followed by - * {@link #map(Object, Object, Context)} + * {@link #map(Object, Object, org.apache.hadoop.mapreduce.Mapper.Context)} * for each key/value pair in the InputSplit. Finally - * {@link #cleanup(Context)} is called.

    + * {@link #cleanup(org.apache.hadoop.mapreduce.Mapper.Context)} is called.

    * *

    All intermediate values associated with a given output key are * subsequently grouped by the framework, and passed to a {@link Reducer} to @@ -84,9 +84,10 @@ * } * } * } - *

    + * * - *

    Applications may override the {@link #run(Context)} method to exert + *

    Applications may override the + * {@link #run(org.apache.hadoop.mapreduce.Mapper.Context)} method to exert * greater control on map processing e.g. multi-threaded Mappers * etc.

    * diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java index ddf67e18abb2a..ab67ab05734e6 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java @@ -39,14 +39,14 @@ *
      *
    1. * - *

      Shuffle

      + * Shuffle * *

      The Reducer copies the sorted output from each * {@link Mapper} using HTTP across the network.

      *
      2. * *
    2. - *

      Sort

      + * Sort * *

      The framework merge sorts Reducer inputs by * keys @@ -55,7 +55,7 @@ *

      The shuffle and sort phases occur simultaneously i.e. while outputs are * being fetched they are merged.

      * - *
      SecondarySort
      + * SecondarySort * *

      To achieve a secondary sort on the values returned by the value * iterator, the application should extend the key with the secondary @@ -83,10 +83,10 @@ *

      3. * *
    3. - *

      Reduce

      + * Reduce * *

      In this phase the - * {@link #reduce(Object, Iterable, Context)} + * {@link #reduce(Object, Iterable, org.apache.hadoop.mapreduce.Reducer.Context)} * method is called for each <key, (collection of values)> in * the sorted inputs.

      *

      The output of the reduce task is typically written to a @@ -113,7 +113,7 @@ * context.write(key, result); * } * } - *

      + * * * @see Mapper * @see Partitioner diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java index 06737c993967d..51fe69a72127a 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java @@ -115,7 +115,7 @@ * } * } * - *

      + * * * It is also very common to use the DistributedCache by using * {@link org.apache.hadoop.util.GenericOptionsParser}. @@ -235,7 +235,6 @@ private static long[] parseTimestamps(String[] strs) { * DistributedCache and MapReduce code. * @param conf The configuration which stored the timestamps * @return a long array of timestamps - * @throws IOException * @deprecated Use {@link JobContext#getArchiveTimestamps()} instead */ @Deprecated @@ -250,7 +249,6 @@ public static long[] getArchiveTimestamps(Configuration conf) { * DistributedCache and MapReduce code. * @param conf The configuration which stored the timestamps * @return a long array of timestamps - * @throws IOException * @deprecated Use {@link JobContext#getFileTimestamps()} instead */ @Deprecated @@ -434,7 +432,6 @@ private static boolean[] parseBooleans(String[] strs) { * internal DistributedCache and MapReduce code. * @param conf The configuration which stored the timestamps * @return a string array of booleans - * @throws IOException */ public static boolean[] getFileVisibilities(Configuration conf) { return parseBooleans(conf.getStrings(MRJobConfig.CACHE_FILE_VISIBILITIES)); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java index 43b2df290d560..f343d7c2e2d7c 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java @@ -93,7 +93,7 @@ public HistoryViewer(String historyFile, final Configuration jobConf = new Configuration(conf); try { jobConf.addResource(fs.open(jobConfPath), jobConfPath.toString()); - Limits.reset(conf); + Limits.reset(jobConf); } catch (FileNotFoundException fnf) { if (LOG.isWarnEnabled()) { LOG.warn("Missing job conf in history", fnf); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorJob.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorJob.java index d8833da52558c..de25f6490620d 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorJob.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorJob.java @@ -60,7 +60,7 @@ * The developer using Aggregate will need only to provide a plugin class * conforming to the following interface: * - * public interface ValueAggregatorDescriptor { public ArrayList + * public interface ValueAggregatorDescriptor { public ArrayList<Entry> * generateKeyValPairs(Object key, Object value); public void * configure(Configuration conf); } * diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java index 208616b66b5ac..1dad13e153688 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/Chain.java @@ -600,7 +600,7 @@ protected static Configuration getChainElementConf(Configuration jobConf, /** * Adds a Mapper class to the chain job. * - *

      + *

      * The configuration properties of the chain job have precedence over the * configuration properties of the Mapper. * @@ -738,7 +738,7 @@ protected static void setMapperConf(boolean isMap, Configuration jobConf, /** * Sets the Reducer class to the chain job. * - *

      + *

      * The configuration properties of the chain job have precedence over the * configuration properties of the Reducer. * diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainMapper.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainMapper.java index c042ff0769c9f..c3bf0120153e6 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainMapper.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainMapper.java @@ -57,24 +57,24 @@ * ChainMapper, this is done by the addMapper for the last mapper in the chain. *

      * ChainMapper usage pattern: - *

      + *

      * * 

        * ...
  * Job = new Job(conf);
- * 
      
+ *
  * Configuration mapAConf = new Configuration(false);
  * ...
  * ChainMapper.addMapper(job, AMap.class, LongWritable.class, Text.class,
  *   Text.class, Text.class, true, mapAConf);
- * 
      
+ *
  * Configuration mapBConf = new Configuration(false);
  * ...
  * ChainMapper.addMapper(job, BMap.class, Text.class, Text.class,
  *   LongWritable.class, Text.class, false, mapBConf);
- * 
      
+ *
  * ...
- * 
      
+ *
  * job.waitForComplettion(true);
  * ...
  *
      diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainReducer.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainReducer.java index dc03d5d8cf69d..1c37587905233 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainReducer.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/chain/ChainReducer.java @@ -50,7 +50,7 @@ * all Mappers and the Reduce in the chain use matching output and input key and * value classes as no conversion is done by the chaining code. *

      - *

      Using the ChainMapper and the ChainReducer classes is possible to + *

      Using the ChainMapper and the ChainReducer classes is possible to * compose Map/Reduce jobs that look like [MAP+ / REDUCE MAP*]. And * immediate benefit of this pattern is a dramatic reduction in disk IO.

      *

      @@ -59,26 +59,26 @@ * element in the chain. *

      * ChainReducer usage pattern: - *

      + *

      * * 

        * ...
  * Job = new Job(conf);
  * ....
- * 
      
+ *
  * Configuration reduceConf = new Configuration(false);
  * ...
  * ChainReducer.setReducer(job, XReduce.class, LongWritable.class, Text.class,
  *   Text.class, Text.class, true, reduceConf);
- * 
      
+ *
  * ChainReducer.addMapper(job, CMap.class, Text.class, Text.class,
  *   LongWritable.class, Text.class, false, null);
- * 
      
+ *
  * ChainReducer.addMapper(job, DMap.class, LongWritable.class, Text.class,
  *   LongWritable.class, LongWritable.class, true, null);
- * 
      
+ *
  * ...
- * 
      
+ *
  * job.waitForCompletion(true);
  * ...
  *
      diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java index a6953b7598abe..78c3a0fd8e811 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java @@ -319,7 +319,7 @@ protected String getCountQuery() { * Java object holding tuple fields. * @param tableName The table to read data from * @param conditions The condition which to select data with, - * eg. '(updated > 20070101 AND length > 0)' + * eg. '(updated > 20070101 AND length > 0)' * @param orderBy the fieldNames in the orderBy clause. * @param fieldNames The field names in the table * @see #setInput(Job, Class, String, String) diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBWritable.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBWritable.java index cc0d30a525b89..5753a3b1cd38c 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBWritable.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBWritable.java @@ -73,7 +73,7 @@ * timestamp = resultSet.getLong(2); * } * } - *

      + * */ @InterfaceAudience.Public @InterfaceStability.Stable diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java index af6b3f0358807..2990ca99d36c7 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java @@ -144,7 +144,7 @@ public void remove() { /** * Convert Tuple to String as in the following. - * [,,...,] + * [<child1>,<child2>,...,<childn>] */ public String toString() { StringBuffer buf = new StringBuffer("["); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/map/MultithreadedMapper.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/map/MultithreadedMapper.java index 814e49451230e..733b18c9ec4f3 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/map/MultithreadedMapper.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/map/MultithreadedMapper.java @@ -44,15 +44,15 @@ * Multithreaded implementation for @link org.apache.hadoop.mapreduce.Mapper. *

      * It can be used instead of the default implementation, - * @link org.apache.hadoop.mapred.MapRunner, when the Map operation is not CPU + * {@link org.apache.hadoop.mapred.MapRunner}, when the Map operation is not CPU * bound in order to improve throughput. *

      * Mapper implementations using this MapRunnable must be thread-safe. *

      * The Map-Reduce job has to be configured with the mapper to use via - * {@link #setMapperClass(Configuration, Class)} and + * {@link #setMapperClass(Job, Class)} and * the number of thread the thread-pool can use with the - * {@link #getNumberOfThreads(Configuration) method. The default + * {@link #getNumberOfThreads(JobContext)} method. The default * value is 10 threads. *

      */ diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java index 55252f04f9f6c..6e5d0a1e627bc 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java @@ -18,6 +18,7 @@ package org.apache.hadoop.mapreduce.lib.output; +import java.io.FileNotFoundException; import java.io.IOException; import org.apache.commons.logging.Log; @@ -25,6 +26,7 @@ import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; @@ -57,10 +59,14 @@ public class FileOutputCommitter extends OutputCommitter { @Deprecated protected static final String TEMP_DIR_NAME = PENDING_DIR_NAME; public static final String SUCCEEDED_FILE_NAME = "_SUCCESS"; - public static final String SUCCESSFUL_JOB_OUTPUT_DIR_MARKER = - "mapreduce.fileoutputcommitter.marksuccessfuljobs"; + public static final String SUCCESSFUL_JOB_OUTPUT_DIR_MARKER = + "mapreduce.fileoutputcommitter.marksuccessfuljobs"; + public static final String FILEOUTPUTCOMMITTER_ALGORITHM_VERSION = + "mapreduce.fileoutputcommitter.algorithm.version"; + public static final int FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT = 1; private Path outputPath = null; private Path workPath = null; + private final int algorithmVersion; /** * Create a file output committer @@ -87,6 +93,14 @@ public FileOutputCommitter(Path outputPath, @Private public FileOutputCommitter(Path outputPath, JobContext context) throws IOException { + Configuration conf = context.getConfiguration(); + algorithmVersion = + conf.getInt(FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, + FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT); + LOG.info("File Output Committer Algorithm version is " + algorithmVersion); + if (algorithmVersion != 1 && algorithmVersion != 2) { + throw new IOException("Only 1 or 2 algorithm version is supported"); + } if (outputPath != null) { FileSystem fs = outputPath.getFileSystem(context.getConfiguration()); this.outputPath = fs.makeQualified(outputPath); @@ -248,14 +262,14 @@ private static Path getCommittedTaskPath(int appAttemptId, TaskAttemptContext co return new Path(getJobAttemptPath(appAttemptId, out), String.valueOf(context.getTaskAttemptID().getTaskID())); } - + private static class CommittedTaskFilter implements PathFilter { @Override public boolean accept(Path path) { return !PENDING_DIR_NAME.equals(path.getName()); } } - + /** * Get a list of all paths where output from committed tasks are stored. * @param context the context of the current job @@ -268,7 +282,7 @@ private FileStatus[] getAllCommittedTaskPaths(JobContext context) FileSystem fs = jobAttemptPath.getFileSystem(context.getConfiguration()); return fs.listStatus(jobAttemptPath, new CommittedTaskFilter()); } - + /** * Get the directory that the task should write results into. * @return the work directory @@ -295,7 +309,7 @@ public void setupJob(JobContext context) throws IOException { LOG.warn("Output Path is null in setupJob()"); } } - + /** * The job has completed so move all committed tasks to the final output dir. * Delete the temporary directory, including all of the work directories. @@ -306,8 +320,11 @@ public void commitJob(JobContext context) throws IOException { if (hasOutputPath()) { Path finalOutput = getOutputPath(); FileSystem fs = finalOutput.getFileSystem(context.getConfiguration()); - for(FileStatus stat: getAllCommittedTaskPaths(context)) { - mergePaths(fs, stat, finalOutput); + + if (algorithmVersion == 1) { + for (FileStatus stat: getAllCommittedTaskPaths(context)) { + mergePaths(fs, stat, finalOutput); + } } // delete the _temporary folder and create a _done file in the o/p folder @@ -331,44 +348,61 @@ public void commitJob(JobContext context) throws IOException { * @param to the path data is going to. * @throws IOException on any error */ - private static void mergePaths(FileSystem fs, final FileStatus from, - final Path to) - throws IOException { - LOG.debug("Merging data from "+from+" to "+to); - if(from.isFile()) { - if(fs.exists(to)) { - if(!fs.delete(to, true)) { - throw new IOException("Failed to delete "+to); - } - } + private void mergePaths(FileSystem fs, final FileStatus from, + final Path to) throws IOException { + if (LOG.isDebugEnabled()) { + LOG.debug("Merging data from " + from + " to " + to); + } + FileStatus toStat; + try { + toStat = fs.getFileStatus(to); + } catch (FileNotFoundException fnfe) { + toStat = null; + } + + if (from.isFile()) { + if (toStat != null) { + if (!fs.delete(to, true)) { + throw new IOException("Failed to delete " + to); + } + } - if(!fs.rename(from.getPath(), to)) { - throw new IOException("Failed to rename "+from+" to "+to); - } - } else if(from.isDirectory()) { - if(fs.exists(to)) { - FileStatus toStat = fs.getFileStatus(to); - if(!toStat.isDirectory()) { - if(!fs.delete(to, true)) { - throw new IOException("Failed to delete "+to); - } - if(!fs.rename(from.getPath(), to)) { - throw new IOException("Failed to rename "+from+" to "+to); - } - } else { - //It is a directory so merge everything in the directories - for(FileStatus subFrom: fs.listStatus(from.getPath())) { - Path subTo = new Path(to, subFrom.getPath().getName()); - mergePaths(fs, subFrom, subTo); - } - } - } else { - //it does not exist just rename - if(!fs.rename(from.getPath(), to)) { - throw new IOException("Failed to rename "+from+" to "+to); - } - } - } + if (!fs.rename(from.getPath(), to)) { + throw new IOException("Failed to rename " + from + " to " + to); + } + } else if (from.isDirectory()) { + if (toStat != null) { + if (!toStat.isDirectory()) { + if (!fs.delete(to, true)) { + throw new IOException("Failed to delete " + to); + } + renameOrMerge(fs, from, to); + } else { + //It is a directory so merge everything in the directories + for (FileStatus subFrom : fs.listStatus(from.getPath())) { + Path subTo = new Path(to, subFrom.getPath().getName()); + mergePaths(fs, subFrom, subTo); + } + } + } else { + renameOrMerge(fs, from, to); + } + } + } + + private void renameOrMerge(FileSystem fs, FileStatus from, Path to) + throws IOException { + if (algorithmVersion == 1) { + if (!fs.rename(from.getPath(), to)) { + throw new IOException("Failed to rename " + from + " to " + to); + } + } else { + fs.mkdirs(to); + for (FileStatus subFrom : fs.listStatus(from.getPath())) { + Path subTo = new Path(to, subFrom.getPath().getName()); + mergePaths(fs, subFrom, subTo); + } + } } @Override @@ -417,27 +451,42 @@ public void commitTask(TaskAttemptContext context) @Private public void commitTask(TaskAttemptContext context, Path taskAttemptPath) - throws IOException { + throws IOException { + TaskAttemptID attemptId = context.getTaskAttemptID(); if (hasOutputPath()) { context.progress(); if(taskAttemptPath == null) { taskAttemptPath = getTaskAttemptPath(context); } - Path committedTaskPath = getCommittedTaskPath(context); FileSystem fs = taskAttemptPath.getFileSystem(context.getConfiguration()); - if (fs.exists(taskAttemptPath)) { - if(fs.exists(committedTaskPath)) { - if(!fs.delete(committedTaskPath, true)) { - throw new IOException("Could not delete " + committedTaskPath); + FileStatus taskAttemptDirStatus; + try { + taskAttemptDirStatus = fs.getFileStatus(taskAttemptPath); + } catch (FileNotFoundException e) { + taskAttemptDirStatus = null; + } + + if (taskAttemptDirStatus != null) { + if (algorithmVersion == 1) { + Path committedTaskPath = getCommittedTaskPath(context); + if (fs.exists(committedTaskPath)) { + if (!fs.delete(committedTaskPath, true)) { + throw new IOException("Could not delete " + committedTaskPath); + } } + if (!fs.rename(taskAttemptPath, committedTaskPath)) { + throw new IOException("Could not rename " + taskAttemptPath + " to " + + committedTaskPath); + } + LOG.info("Saved output of task '" + attemptId + "' to " + + committedTaskPath); + } else { + // directly merge everything from taskAttemptPath to output directory + mergePaths(fs, taskAttemptDirStatus, outputPath); + LOG.info("Saved output of task '" + attemptId + "' to " + + outputPath); } - if(!fs.rename(taskAttemptPath, committedTaskPath)) { - throw new IOException("Could not rename " + taskAttemptPath + " to " - + committedTaskPath); - } - LOG.info("Saved output of task '" + attemptId + "' to " + - committedTaskPath); } else { LOG.warn("No Output found for " + attemptId); } @@ -511,32 +560,44 @@ public void recoverTask(TaskAttemptContext context) throw new IOException ("Cannot recover task output for first attempt..."); } - Path committedTaskPath = getCommittedTaskPath(context); Path previousCommittedTaskPath = getCommittedTaskPath( previousAttempt, context); - FileSystem fs = committedTaskPath.getFileSystem(context.getConfiguration()); - - LOG.debug("Trying to recover task from " + previousCommittedTaskPath - + " into " + committedTaskPath); - if (fs.exists(previousCommittedTaskPath)) { - if(fs.exists(committedTaskPath)) { - if(!fs.delete(committedTaskPath, true)) { - throw new IOException("Could not delete "+committedTaskPath); + FileSystem fs = previousCommittedTaskPath.getFileSystem(context.getConfiguration()); + if (LOG.isDebugEnabled()) { + LOG.debug("Trying to recover task from " + previousCommittedTaskPath); + } + if (algorithmVersion == 1) { + if (fs.exists(previousCommittedTaskPath)) { + Path committedTaskPath = getCommittedTaskPath(context); + if (fs.exists(committedTaskPath)) { + if (!fs.delete(committedTaskPath, true)) { + throw new IOException("Could not delete "+committedTaskPath); + } } + //Rename can fail if the parent directory does not yet exist. + Path committedParent = committedTaskPath.getParent(); + fs.mkdirs(committedParent); + if (!fs.rename(previousCommittedTaskPath, committedTaskPath)) { + throw new IOException("Could not rename " + previousCommittedTaskPath + + " to " + committedTaskPath); + } + } else { + LOG.warn(attemptId+" had no output to recover."); } - //Rename can fail if the parent directory does not yet exist. - Path committedParent = committedTaskPath.getParent(); - fs.mkdirs(committedParent); - if(!fs.rename(previousCommittedTaskPath, committedTaskPath)) { - throw new IOException("Could not rename " + previousCommittedTaskPath + - " to " + committedTaskPath); - } - LOG.info("Saved output of " + attemptId + " to " + committedTaskPath); } else { - LOG.warn(attemptId+" had no output to recover."); + // essentially a no-op, but for backwards compatibility + // after upgrade to the new fileOutputCommitter, + // check if there are any output left in committedTaskPath + if (fs.exists(previousCommittedTaskPath)) { + LOG.info("Recovering task for upgrading scenario, moving files from " + + previousCommittedTaskPath + " to " + outputPath); + FileStatus from = fs.getFileStatus(previousCommittedTaskPath); + mergePaths(fs, from, outputPath); + } + LOG.info("Done recovering task " + attemptId); } } else { LOG.warn("Output Path is null in recoverTask()"); } } -} +} \ No newline at end of file diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java index fa3708e7948dc..2c6954275f2d4 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java @@ -181,7 +181,7 @@ public static Path getOutputPath(JobContext job) { * Get the {@link Path} to the task's temporary output directory * for the map-reduce job * - *

      Tasks' Side-Effect Files

      + * Tasks' Side-Effect Files * *

      Some applications need to create/write-to side-files, which differ from * the actual job-outputs. diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java index b8cb997c09c90..da33770b4bab9 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MapFileOutputFormat.java @@ -24,6 +24,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.FileUtil; +import org.apache.hadoop.fs.PathFilter; import org.apache.hadoop.io.MapFile; import org.apache.hadoop.io.WritableComparable; @@ -88,7 +89,16 @@ public void close(TaskAttemptContext context) throws IOException { public static MapFile.Reader[] getReaders(Path dir, Configuration conf) throws IOException { FileSystem fs = dir.getFileSystem(conf); - Path[] names = FileUtil.stat2Paths(fs.listStatus(dir)); + PathFilter filter = new PathFilter() { + @Override + public boolean accept(Path path) { + String name = path.getName(); + if (name.startsWith("_") || name.startsWith(".")) + return false; + return true; + } + }; + Path[] names = FileUtil.stat2Paths(fs.listStatus(dir, filter)); // sort names, so that hash partitioning works Arrays.sort(names); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java index 24baa596b31ef..c31cab7ddd8b7 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java @@ -81,7 +81,7 @@ *

      * Usage in Reducer: * 

      - *  String generateFileName(K k, V v) {
+ * <K, V> String generateFileName(K k, V v) {
  *   return k.toString() + "_" + v.toString();
  * }
  * 
@@ -124,16 +124,16 @@
  * 
      
  * 
  * 
      - * private MultipleOutputs out;
+ * private MultipleOutputs<Text, Text> out;
  * 
  * public void setup(Context context) {
- *   out = new MultipleOutputs(context);
+ *   out = new MultipleOutputs<Text, Text>(context);
  *   ...
  * }
  * 
- * public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
+ * public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
  * for (Text t : values) {
- *   out.write(key, t, generateFileName(<parameter list...>));
+ *   out.write(key, t, generateFileName(<parameter list...>));
  *   }
  * }
  * 
@@ -294,7 +294,6 @@ private static Class getNamedOutputValueClass(
 
   /**
    * Adds a named output for the job.
-   * 
      
    *
    * @param job               job to add the named output
    * @param namedOutput       named output name, it has to be a word, letters
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/BinaryPartitioner.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/BinaryPartitioner.java
index 4a40840924413..2a89908119f27 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/BinaryPartitioner.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/BinaryPartitioner.java
@@ -64,7 +64,7 @@
  *   
    4. {@link #setOffsets}
      5. 
  *   
    5. {@link #setLeftOffset}
      6. 
  *   
    6. {@link #setRightOffset}
      7. 
- * 
      
+ * 
  */
 @InterfaceAudience.Public
 @InterfaceStability.Evolving
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java
index 4668f49cc236a..cce9f37838293 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java
@@ -230,10 +230,8 @@ public K[] getSample(InputFormat inf, Job job)
               // to reflect the possibility of existing elements being
               // pushed out
               int ind = r.nextInt(numSamples);
-              if (ind != numSamples) {
-                samples.set(ind, ReflectionUtils.copy(job.getConfiguration(),
-                                 reader.getCurrentKey(), null));
-              }
+              samples.set(ind, ReflectionUtils.copy(job.getConfiguration(),
+                               reader.getCurrentKey(), null));
               freq *= (numSamples - 1) / (double) numSamples;
             }
           }
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
index 7b1f657ec40b1..6c0de1b07a880 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
@@ -105,7 +105,6 @@ static void obtainTokensForNamenodesInternal(Credentials credentials,
    * get delegation token for a specific FS
    * @param fs
    * @param credentials
-   * @param p
    * @param conf
    * @throws IOException
    */
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/JobContextImpl.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/JobContextImpl.java
index 247c2f2029b85..b9014ef7861af 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/JobContextImpl.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/JobContextImpl.java
@@ -374,7 +374,6 @@ private static String[] toTimestampStrs(long[] timestamps) {
    * Get the timestamps of the archives.  Used by internal
    * DistributedCache and MapReduce code.
    * @return a string array of timestamps 
-   * @throws IOException
    */
   public String[] getArchiveTimestamps() {
     return toTimestampStrs(DistributedCache.getArchiveTimestamps(conf));
@@ -384,7 +383,6 @@ public String[] getArchiveTimestamps() {
    * Get the timestamps of the files.  Used by internal
    * DistributedCache and MapReduce code.
    * @return a string array of timestamps 
-   * @throws IOException
    */
   public String[] getFileTimestamps() {
     return toTimestampStrs(DistributedCache.getFileTimestamps(conf));
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
index a4b1aa82e8f3f..8bf17ef75f84b 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
@@ -191,8 +191,9 @@ public MergeManagerImpl(TaskAttemptID reduceId, JobConf jobConf,
     this.memToMemMergeOutputsThreshold = 
             jobConf.getInt(MRJobConfig.REDUCE_MEMTOMEM_THRESHOLD, ioSortFactor);
     this.mergeThreshold = (long)(this.memoryLimit * 
-                          jobConf.getFloat(MRJobConfig.SHUFFLE_MERGE_PERCENT, 
-                                           0.90f));
+                          jobConf.getFloat(
+                            MRJobConfig.SHUFFLE_MERGE_PERCENT,
+                            MRJobConfig.DEFAULT_SHUFFLE_MERGE_PERCENT));
     LOG.info("MergerManager: memoryLimit=" + memoryLimit + ", " +
              "maxSingleShuffleLimit=" + maxSingleShuffleLimit + ", " +
              "mergeThreshold=" + mergeThreshold + ", " + 
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
index 3630c642b6522..a350829c93fc1 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
@@ -421,7 +421,7 @@ private String getTaskTypes() {
    * Display usage of the command-line tool and terminate execution.
    */
   private void displayUsage(String cmd) {
-    String prefix = "Usage: CLI ";
+    String prefix = "Usage: job ";
     String jobPriorityValues = getJobPriorityNames();
     String taskStates = "running, completed";
     
@@ -616,17 +616,19 @@ public void displayJobList(JobStatus[] jobs)
   }
 
   @Private
-  public static String headerPattern = "%23s\t%10s\t%14s\t%12s\t%12s\t%10s\t%15s\t%15s\t%8s\t%8s\t%10s\t%10s\n";
+  public static String headerPattern = "%23s\t%20s\t%10s\t%14s\t%12s\t%12s" +
+      "\t%10s\t%15s\t%15s\t%8s\t%8s\t%10s\t%10s\n";
   @Private
-  public static String dataPattern   = "%23s\t%10s\t%14d\t%12s\t%12s\t%10s\t%15s\t%15s\t%8s\t%8s\t%10s\t%10s\n";
+  public static String dataPattern   = "%23s\t%20s\t%10s\t%14d\t%12s\t%12s" +
+      "\t%10s\t%15s\t%15s\t%8s\t%8s\t%10s\t%10s\n";
   private static String memPattern   = "%dM";
   private static String UNAVAILABLE  = "N/A";
 
   @Private
   public void displayJobList(JobStatus[] jobs, PrintWriter writer) {
     writer.println("Total jobs:" + jobs.length);
-    writer.printf(headerPattern, "JobId", "State", "StartTime", "UserName",
-      "Queue", "Priority", "UsedContainers",
+    writer.printf(headerPattern, "JobId", "JobName", "State", "StartTime",
+      "UserName", "Queue", "Priority", "UsedContainers",
       "RsvdContainers", "UsedMem", "RsvdMem", "NeededMem", "AM info");
     for (JobStatus job : jobs) {
       int numUsedSlots = job.getNumUsedSlots();
@@ -634,10 +636,11 @@ public void displayJobList(JobStatus[] jobs, PrintWriter writer) {
       int usedMem = job.getUsedMem();
       int rsvdMem = job.getReservedMem();
       int neededMem = job.getNeededMem();
-      writer.printf(dataPattern,
-          job.getJobID().toString(), job.getState(), job.getStartTime(),
-          job.getUsername(), job.getQueue(), 
-          job.getPriority().name(),
+      int jobNameLength = job.getJobName().length();
+      writer.printf(dataPattern, job.getJobID().toString(),
+          job.getJobName().substring(0, jobNameLength > 20 ? 20 : jobNameLength),
+          job.getState(), job.getStartTime(), job.getUsername(),
+          job.getQueue(), job.getPriority().name(),
           numUsedSlots < 0 ? UNAVAILABLE : numUsedSlots,
           numReservedSlots < 0 ? UNAVAILABLE : numReservedSlots,
           usedMem < 0 ? UNAVAILABLE : String.format(memPattern, usedMem),
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
index 6e8067976f49d..820c1ac8d80bd 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
@@ -82,6 +82,22 @@
   
 
 
+
+  mapreduce.job.running.map.limit
+  0
+  The maximum number of simultaneous map tasks per job.
+  There is no limit if this value is 0 or negative.
+  
+
+
+
+  mapreduce.job.running.reduce.limit
+  0
+  The maximum number of simultaneous reduce tasks per job.
+  There is no limit if this value is 0 or negative.
+  
+
+
 
   mapreduce.job.reducer.preempt.delay.sec
   0
@@ -1301,6 +1317,60 @@
   contact with the RM has been re-established.
 
 
+
+  mapreduce.fileoutputcommitter.algorithm.version
+  1
+  The file output committer algorithm version
+  valid algorithm version number: 1 or 2
+  default to 1, which is the original algorithm
+
+  In algorithm version 1,
+
+  1. commitTask will rename directory
+  $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
+  to
+  $joboutput/_temporary/$appAttemptID/$taskID/
+
+  2. recoverTask will also do a rename
+  $joboutput/_temporary/$appAttemptID/$taskID/
+  to
+  $joboutput/_temporary/($appAttemptID + 1)/$taskID/
+
+  3. commitJob will merge every task output file in
+  $joboutput/_temporary/$appAttemptID/$taskID/
+  to
+  $joboutput/, then it will delete $joboutput/_temporary/
+  and write $joboutput/_SUCCESS
+
+  It has a performance regression, which is discussed in MAPREDUCE-4815.
+  If a job generates many files to commit then the commitJob
+  method call at the end of the job can take minutes.
+  the commit is single-threaded and waits until all
+  tasks have completed before commencing.
+
+  algorithm version 2 will change the behavior of commitTask,
+  recoverTask, and commitJob.
+
+  1. commitTask will rename all files in
+  $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
+  to $joboutput/
+
+  2. recoverTask actually doesn't require to do anything, but for
+  upgrade from version 1 to version 2 case, it will check if there
+  are any files in
+  $joboutput/_temporary/($appAttemptID - 1)/$taskID/
+  and rename them to $joboutput/
+
+  3. commitJob can simply delete $joboutput/_temporary and write
+  $joboutput/_SUCCESS
+
+  This algorithm will reduce the output commit time for
+  large jobs by having the tasks commit directly to the final
+  output directory as they were completing and commitJob had
+  very little to do.
+  
+
+
 
   yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
   1000
@@ -1343,6 +1413,14 @@
   
 
 
+
+  yarn.app.mapreduce.am.hard-kill-timeout-ms
+  10000
+  
+     Number of milliseconds to wait before the job client kills the application.
+  
+
+
 
   CLASSPATH for MR applications. A comma-separated list
   of CLASSPATH entries. If mapreduce.application.framework is set then this
@@ -1616,4 +1694,12 @@
     calculated as (heapSize / mapreduce.heap.memory-mb.ratio).
   
 
+
+
+  yarn.app.mapreduce.am.containerlauncher.threadpool-initial-size
+  10
+  The initial size of thread pool to launch containers in the
+    app master.
+  
+
 
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
index 0859571d1f2bd..3207a71efe500 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java
@@ -50,7 +50,6 @@ public class TestFileOutputCommitter extends TestCase {
   private Text val1 = new Text("val1");
   private Text val2 = new Text("val2");
 
-  
   private void writeOutput(RecordWriter theRecordWriter,
       TaskAttemptContext context) throws IOException, InterruptedException {
     NullWritable nullWritable = NullWritable.get();
@@ -83,12 +82,16 @@ private void writeMapFileOutput(RecordWriter theRecordWriter,
       theRecordWriter.close(null);
     }
   }
-  
-  public void testRecovery() throws Exception {
-    JobConf conf = new JobConf();
+
+  private void testRecoveryInternal(int commitVersion, int recoveryVersion)
+      throws Exception {
+  JobConf conf = new JobConf();
     FileOutputFormat.setOutputPath(conf, outDir);
     conf.set(JobContext.TASK_ATTEMPT_ID, attempt);
     conf.setInt(MRConstants.APPLICATION_ATTEMPT_ID, 1);
+    conf.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        commitVersion);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter();
@@ -99,7 +102,7 @@ public void testRecovery() throws Exception {
 
     // write output
     TextOutputFormat theOutputFormat = new TextOutputFormat();
-    RecordWriter theRecordWriter = 
+    RecordWriter theRecordWriter =
         theOutputFormat.getRecordWriter(null, conf, partFile, null);
     writeOutput(theRecordWriter, tContext);
 
@@ -107,31 +110,59 @@ public void testRecovery() throws Exception {
     if(committer.needsTaskCommit(tContext)) {
       committer.commitTask(tContext);
     }
+
     Path jobTempDir1 = committer.getCommittedTaskPath(tContext);
     File jtd1 = new File(jobTempDir1.toUri().getPath());
-    assertTrue(jtd1.exists());
-    validateContent(jobTempDir1);        
-    
-    //now while running the second app attempt, 
+    if (commitVersion == 1) {
+      assertTrue("Version 1 commits to temporary dir " + jtd1, jtd1.exists());
+      validateContent(jobTempDir1);
+    } else {
+      assertFalse("Version 2 commits to output dir " + jtd1, jtd1.exists());
+    }
+
+    //now while running the second app attempt,
     //recover the task output from first attempt
     JobConf conf2 = new JobConf(conf);
     conf2.set(JobContext.TASK_ATTEMPT_ID, attempt);
     conf2.setInt(MRConstants.APPLICATION_ATTEMPT_ID, 2);
+    conf2.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        recoveryVersion);
     JobContext jContext2 = new JobContextImpl(conf2, taskID.getJobID());
     TaskAttemptContext tContext2 = new TaskAttemptContextImpl(conf2, taskID);
     FileOutputCommitter committer2 = new FileOutputCommitter();
     committer2.setupJob(jContext2);
-    Path jobTempDir2 = committer2.getCommittedTaskPath(tContext2);
-    
+
     committer2.recoverTask(tContext2);
+
+    Path jobTempDir2 = committer2.getCommittedTaskPath(tContext2);
     File jtd2 = new File(jobTempDir2.toUri().getPath());
-    assertTrue(jtd2.exists());
-    validateContent(jobTempDir2);
-    
+    if (recoveryVersion == 1) {
+      assertTrue("Version 1 recovers to " + jtd2, jtd2.exists());
+      validateContent(jobTempDir2);
+    } else {
+        assertFalse("Version 2 commits to output dir " + jtd2, jtd2.exists());
+        if (commitVersion == 1) {
+          assertTrue("Version 2  recovery moves to output dir from "
+              + jtd1 , jtd1.list().length == 0);
+        }
+      }
+
     committer2.commitJob(jContext2);
     validateContent(outDir);
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
+  public void testRecoveryV1() throws Exception {
+    testRecoveryInternal(1, 1);
+  }
+
+  public void testRecoveryV2() throws Exception {
+    testRecoveryInternal(2, 2);
+  }
+
+  public void testRecoveryUpgradeV1V2() throws Exception {
+    testRecoveryInternal(1, 2);
+  }
 
   private void validateContent(Path dir) throws IOException {
     File fdir = new File(dir.toUri().getPath());
@@ -170,11 +201,13 @@ else if (f.getPath().getName().equals(MapFile.DATA_FILE_NAME)) {
     assert(fileCount > 0);
     assert(dataFileFound && indexFileFound);
   }
-  
-  public void testCommitter() throws Exception {
+
+  private void testCommitterInternal(int version) throws Exception {
     JobConf conf = new JobConf();
     FileOutputFormat.setOutputPath(conf, outDir);
     conf.set(JobContext.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter();
@@ -200,21 +233,33 @@ public void testCommitter() throws Exception {
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
 
-  public void testMapFileOutputCommitter() throws Exception {
+  public void testCommitterV1() throws Exception {
+    testCommitterInternal(1);
+  }
+
+  public void testCommitterV2() throws Exception {
+    testCommitterInternal(2);
+  }
+
+  private void testMapFileOutputCommitterInternal(int version)
+      throws Exception {
     JobConf conf = new JobConf();
     FileOutputFormat.setOutputPath(conf, outDir);
     conf.set(JobContext.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
-    FileOutputCommitter committer = new FileOutputCommitter();    
-    
+    FileOutputCommitter committer = new FileOutputCommitter();
+
     // setup
     committer.setupJob(jContext);
     committer.setupTask(tContext);
 
     // write output
     MapFileOutputFormat theOutputFormat = new MapFileOutputFormat();
-    RecordWriter theRecordWriter = theOutputFormat.getRecordWriter(null, conf, partFile, null);
+    RecordWriter theRecordWriter =
+        theOutputFormat.getRecordWriter(null, conf, partFile, null);
     writeMapFileOutput(theRecordWriter, tContext);
 
     // do commit
@@ -227,11 +272,29 @@ public void testMapFileOutputCommitter() throws Exception {
     validateMapFileOutputContent(FileSystem.get(conf), outDir);
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
-  
-  public void testMapOnlyNoOutput() throws Exception {
+
+  public void testMapFileOutputCommitterV1() throws Exception {
+    testMapFileOutputCommitterInternal(1);
+  }
+
+  public void testMapFileOutputCommitterV2() throws Exception {
+    testMapFileOutputCommitterInternal(2);
+  }
+
+  public void testMapOnlyNoOutputV1() throws Exception {
+    testMapOnlyNoOutputInternal(1);
+  }
+
+  public void testMapOnlyNoOutputV2() throws Exception {
+    testMapOnlyNoOutputInternal(2);
+  }
+
+  private void testMapOnlyNoOutputInternal(int version) throws Exception {
     JobConf conf = new JobConf();
     //This is not set on purpose. FileOutputFormat.setOutputPath(conf, outDir);
     conf.set(JobContext.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter();    
@@ -249,11 +312,14 @@ public void testMapOnlyNoOutput() throws Exception {
     // validate output
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
-  
-  public void testAbort() throws IOException, InterruptedException {
+
+  private void testAbortInternal(int version)
+      throws IOException, InterruptedException {
     JobConf conf = new JobConf();
     FileOutputFormat.setOutputPath(conf, outDir);
     conf.set(JobContext.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter();
@@ -283,6 +349,14 @@ public void testAbort() throws IOException, InterruptedException {
     FileUtil.fullyDelete(out);
   }
 
+  public void testAbortV1() throws Exception {
+    testAbortInternal(1);
+  }
+
+  public void testAbortV2() throws Exception {
+    testAbortInternal(2);
+  }
+
   public static class FakeFileSystem extends RawLocalFileSystem {
     public FakeFileSystem() {
       super();
@@ -299,11 +373,14 @@ public boolean delete(Path p, boolean recursive) throws IOException {
   }
 
   
-  public void testFailAbort() throws IOException, InterruptedException {
+  private void testFailAbortInternal(int version)
+      throws IOException, InterruptedException {
     JobConf conf = new JobConf();
     conf.set(FileSystem.FS_DEFAULT_NAME_KEY, "faildel:///");
     conf.setClass("fs.faildel.impl", FakeFileSystem.class, FileSystem.class);
     conf.set(JobContext.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(org.apache.hadoop.mapreduce.lib.output.
+        FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, version);
     conf.setInt(MRConstants.APPLICATION_ATTEMPT_ID, 1);
     FileOutputFormat.setOutputPath(conf, outDir);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
@@ -353,6 +430,13 @@ public void testFailAbort() throws IOException, InterruptedException {
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
 
+  public void testFailAbortV1() throws Exception {
+    testFailAbortInternal(1);
+  }
+
+  public void testFailAbortV2() throws Exception {
+    testFailAbortInternal(2);
+  }
   public static String slurp(File f) throws IOException {
     int len = (int) f.length();
     byte[] buf = new byte[len];
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
index 3d924e1f72d47..0612ade83d1c8 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
@@ -22,6 +22,7 @@
 import static org.junit.Assert.*;
 
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapreduce.MRJobConfig;
 import org.junit.Assert;
 import org.junit.Test;
 
@@ -188,4 +189,176 @@ public void testDeprecatedPropertyNameForTaskVmem() {
     Assert.assertEquals(2048, configuration.getLong(
         JobConf.MAPREDUCE_JOB_REDUCE_MEMORY_MB_PROPERTY, -1));
   }
+
+
+  @Test
+  public void testProfileParamsDefaults() {
+    JobConf configuration = new JobConf();
+    String result = configuration.getProfileParams();
+    Assert.assertNotNull(result);
+    Assert.assertTrue(result.contains("file=%s"));
+    Assert.assertTrue(result.startsWith("-agentlib:hprof"));
+  }
+
+  @Test
+  public void testProfileParamsSetter() {
+    JobConf configuration = new JobConf();
+
+    configuration.setProfileParams("test");
+    Assert.assertEquals("test", configuration.get(MRJobConfig.TASK_PROFILE_PARAMS));
+  }
+
+  @Test
+  public void testProfileParamsGetter() {
+    JobConf configuration = new JobConf();
+
+    configuration.set(MRJobConfig.TASK_PROFILE_PARAMS, "test");
+    Assert.assertEquals("test", configuration.getProfileParams());
+  }
+
+  /**
+   * Testing mapred.task.maxvmem replacement with new values
+   *
+   */
+  @Test
+  public void testMemoryConfigForMapOrReduceTask(){
+    JobConf configuration = new JobConf();
+    configuration.set(MRJobConfig.MAP_MEMORY_MB,String.valueOf(300));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,String.valueOf(300));
+    Assert.assertEquals(configuration.getMemoryForMapTask(),300);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(),300);
+
+    configuration.set("mapred.task.maxvmem" , String.valueOf(2*1024 * 1024));
+    configuration.set(MRJobConfig.MAP_MEMORY_MB,String.valueOf(300));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,String.valueOf(300));
+    Assert.assertEquals(configuration.getMemoryForMapTask(),2);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
+
+    configuration = new JobConf();
+    configuration.set("mapred.task.maxvmem" , "-1");
+    configuration.set(MRJobConfig.MAP_MEMORY_MB,String.valueOf(300));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,String.valueOf(400));
+    Assert.assertEquals(configuration.getMemoryForMapTask(), 300);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(), 400);
+
+    configuration = new JobConf();
+    configuration.set("mapred.task.maxvmem" , String.valueOf(2*1024 * 1024));
+    configuration.set(MRJobConfig.MAP_MEMORY_MB,"-1");
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,"-1");
+    Assert.assertEquals(configuration.getMemoryForMapTask(),2);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
+
+    configuration = new JobConf();
+    configuration.set("mapred.task.maxvmem" , String.valueOf(-1));
+    configuration.set(MRJobConfig.MAP_MEMORY_MB,"-1");
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,"-1");
+    Assert.assertEquals(configuration.getMemoryForMapTask(),
+        MRJobConfig.DEFAULT_MAP_MEMORY_MB);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(),
+        MRJobConfig.DEFAULT_REDUCE_MEMORY_MB);
+
+    configuration = new JobConf();
+    configuration.set("mapred.task.maxvmem" , String.valueOf(2*1024 * 1024));
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, "3");
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "3");
+    Assert.assertEquals(configuration.getMemoryForMapTask(),2);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
+  }
+
+  /**
+   * Test that negative values for MAPRED_TASK_MAXVMEM_PROPERTY cause
+   * new configuration keys' values to be used.
+   */
+  @Test
+  public void testNegativeValueForTaskVmem() {
+    JobConf configuration = new JobConf();
+
+    configuration.set(JobConf.MAPRED_TASK_MAXVMEM_PROPERTY, "-3");
+    Assert.assertEquals(MRJobConfig.DEFAULT_MAP_MEMORY_MB,
+        configuration.getMemoryForMapTask());
+    Assert.assertEquals(MRJobConfig.DEFAULT_REDUCE_MEMORY_MB,
+        configuration.getMemoryForReduceTask());
+
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, "4");
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "5");
+    Assert.assertEquals(4, configuration.getMemoryForMapTask());
+    Assert.assertEquals(5, configuration.getMemoryForReduceTask());
+
+  }
+
+  /**
+   * Test that negative values for new configuration keys get passed through.
+   */
+  @Test
+  public void testNegativeValuesForMemoryParams() {
+    JobConf configuration = new JobConf();
+
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, "-5");
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "-6");
+    Assert.assertEquals(MRJobConfig.DEFAULT_MAP_MEMORY_MB,
+        configuration.getMemoryForMapTask());
+    Assert.assertEquals(MRJobConfig.DEFAULT_REDUCE_MEMORY_MB,
+        configuration.getMemoryForReduceTask());
+  }
+
+  /**
+   *   Test deprecated accessor and mutator method for mapred.task.maxvmem
+   */
+  @Test
+  public void testMaxVirtualMemoryForTask() {
+    JobConf configuration = new JobConf();
+
+    //get test case
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(300));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(-1));
+    Assert.assertEquals(
+      configuration.getMaxVirtualMemoryForTask(), 1024 * 1024 * 1024);
+
+    configuration = new JobConf();
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(-1));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(200));
+    Assert.assertEquals(
+      configuration.getMaxVirtualMemoryForTask(), 1024 * 1024 * 1024);
+
+    configuration = new JobConf();
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(-1));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(-1));
+    configuration.set("mapred.task.maxvmem", String.valueOf(1 * 1024 * 1024));
+    Assert.assertEquals(
+      configuration.getMaxVirtualMemoryForTask(), 1 * 1024 * 1024);
+
+    configuration = new JobConf();
+    configuration.set("mapred.task.maxvmem", String.valueOf(1 * 1024 * 1024));
+    Assert.assertEquals(
+      configuration.getMaxVirtualMemoryForTask(), 1 * 1024 * 1024);
+
+    //set test case
+
+    configuration = new JobConf();
+    configuration.setMaxVirtualMemoryForTask(2 * 1024 * 1024);
+    Assert.assertEquals(configuration.getMemoryForMapTask(), 2);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(), 2);
+
+    configuration = new JobConf();
+    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(300));
+    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(400));
+    configuration.setMaxVirtualMemoryForTask(2 * 1024 * 1024);
+    Assert.assertEquals(configuration.getMemoryForMapTask(), 2);
+    Assert.assertEquals(configuration.getMemoryForReduceTask(), 2);
+  }
+
+  /**
+   * Ensure that by default JobContext.MAX_TASK_FAILURES_PER_TRACKER is less
+   * JobContext.MAP_MAX_ATTEMPTS and JobContext.REDUCE_MAX_ATTEMPTS so that
+   * failed tasks will be retried on other nodes
+   */
+  @Test
+  public void testMaxTaskFailuresPerTracker() {
+    JobConf jobConf = new JobConf(true);
+    Assert.assertTrue("By default JobContext.MAX_TASK_FAILURES_PER_TRACKER was "
+      + "not less than JobContext.MAP_MAX_ATTEMPTS and REDUCE_MAX_ATTEMPTS"
+      ,jobConf.getMaxTaskFailuresPerTracker() < jobConf.getMaxMapAttempts() &&
+      jobConf.getMaxTaskFailuresPerTracker() < jobConf.getMaxReduceAttempts()
+      );
+  }
 }
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskProgressReporter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskProgressReporter.java
new file mode 100644
index 0000000000000..0bceb87e9a07a
--- /dev/null
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskProgressReporter.java
@@ -0,0 +1,160 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements.  See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership.  The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License.  You may obtain a copy of the License at
+*
+*     http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.hadoop.mapred;
+
+import java.io.IOException;
+
+import org.apache.hadoop.ipc.ProtocolSignature;
+import org.apache.hadoop.mapred.SortedRanges.Range;
+import org.apache.hadoop.mapreduce.MRJobConfig;
+import org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestTaskProgressReporter {
+  private static int statusUpdateTimes = 0;
+  private FakeUmbilical fakeUmbilical = new FakeUmbilical();
+
+  private static class DummyTask extends Task {
+    @Override
+    public void run(JobConf job, TaskUmbilicalProtocol umbilical)
+        throws IOException, ClassNotFoundException, InterruptedException {
+    }
+
+    @Override
+    public boolean isMapTask() {
+      return true;
+    }
+  }
+
+  private static class FakeUmbilical implements TaskUmbilicalProtocol {
+    @Override
+    public long getProtocolVersion(String protocol, long clientVersion)
+        throws IOException {
+      return 0;
+    }
+
+    @Override
+    public ProtocolSignature getProtocolSignature(String protocol,
+        long clientVersion, int clientMethodsHash) throws IOException {
+      return null;
+    }
+
+    @Override
+    public JvmTask getTask(JvmContext context) throws IOException {
+      return null;
+    }
+
+    @Override
+    public AMFeedback statusUpdate(TaskAttemptID taskId,
+        TaskStatus taskStatus) throws IOException, InterruptedException {
+      statusUpdateTimes++;
+      AMFeedback feedback = new AMFeedback();
+      feedback.setTaskFound(true);
+      feedback.setPreemption(true);
+      return feedback;
+    }
+
+    @Override
+    public void reportDiagnosticInfo(TaskAttemptID taskid, String trace)
+        throws IOException {
+    }
+
+    @Override
+    public void reportNextRecordRange(TaskAttemptID taskid, Range range)
+        throws IOException {
+    }
+
+    @Override
+    public void done(TaskAttemptID taskid) throws IOException {
+    }
+
+    @Override
+    public void commitPending(TaskAttemptID taskId, TaskStatus taskStatus)
+        throws IOException, InterruptedException {
+    }
+
+    @Override
+    public boolean canCommit(TaskAttemptID taskid) throws IOException {
+      return false;
+    }
+
+    @Override
+    public void shuffleError(TaskAttemptID taskId, String message)
+        throws IOException {
+    }
+
+    @Override
+    public void fsError(TaskAttemptID taskId, String message)
+        throws IOException {
+    }
+
+    @Override
+    public void fatalError(TaskAttemptID taskId, String message)
+        throws IOException {
+    }
+
+    @Override
+    public MapTaskCompletionEventsUpdate getMapCompletionEvents(
+        JobID jobId, int fromIndex, int maxLocs, TaskAttemptID id)
+        throws IOException {
+      return null;
+    }
+
+    @Override
+    public void preempted(TaskAttemptID taskId, TaskStatus taskStatus)
+        throws IOException, InterruptedException {
+    }
+
+    @Override
+    public TaskCheckpointID getCheckpointID(TaskID taskID) {
+      return null;
+    }
+
+    @Override
+    public void setCheckpointID(TaskID tid, TaskCheckpointID cid) {
+    }
+  }
+
+  private class DummyTaskReporter extends Task.TaskReporter {
+    public DummyTaskReporter(Task task) {
+      task.super(task.getProgress(), fakeUmbilical);
+    }
+    @Override
+    public void setProgress(float progress) {
+      super.setProgress(progress);
+    }
+  }
+
+  @Test (timeout=10000)
+  public void testTaskProgress() throws Exception {
+    JobConf job = new JobConf();
+    job.setLong(MRJobConfig.TASK_PROGRESS_REPORT_INTERVAL, 1000);
+    Task task = new DummyTask();
+    task.setConf(job);
+    DummyTaskReporter reporter = new DummyTaskReporter(task);
+    Thread t = new Thread(reporter);
+    t.start();
+    Thread.sleep(2100);
+    task.setTaskDone();
+    reporter.resetDoneFlag();
+    t.join();
+    Assert.assertEquals(statusUpdateTimes, 2);
+  }
+}
\ No newline at end of file
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
index d20480ebba9f2..7678f35363cf5 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java
@@ -22,9 +22,15 @@
 import java.io.FileInputStream;
 import java.io.IOException;
 import java.net.URI;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
 
 import junit.framework.TestCase;
 
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.FileSystem;
@@ -39,6 +45,7 @@
 import org.apache.hadoop.mapreduce.JobContext;
 import org.apache.hadoop.mapreduce.JobStatus;
 import org.apache.hadoop.mapreduce.MRJobConfig;
+import org.apache.hadoop.mapreduce.OutputCommitter;
 import org.apache.hadoop.mapreduce.RecordWriter;
 import org.apache.hadoop.mapreduce.TaskAttemptContext;
 import org.apache.hadoop.mapreduce.TaskAttemptID;
@@ -47,13 +54,25 @@
 
 @SuppressWarnings("unchecked")
 public class TestFileOutputCommitter extends TestCase {
-  private static Path outDir = new Path(System.getProperty("test.build.data",
-      "/tmp"), "output");
+  private static final Path outDir = new Path(
+      System.getProperty("test.build.data",
+          System.getProperty("java.io.tmpdir")),
+      TestFileOutputCommitter.class.getName());
+
+  private final static String SUB_DIR = "SUB_DIR";
+  private final static Path OUT_SUB_DIR = new Path(outDir, SUB_DIR);
+
+  private static final Log LOG =
+      LogFactory.getLog(TestFileOutputCommitter.class);
 
   // A random task attempt id for testing.
-  private static String attempt = "attempt_200707121733_0001_m_000000_0";
-  private static String partFile = "part-m-00000";
-  private static TaskAttemptID taskID = TaskAttemptID.forName(attempt);
+  private static final String attempt = "attempt_200707121733_0001_m_000000_0";
+  private static final String partFile = "part-m-00000";
+  private static final TaskAttemptID taskID = TaskAttemptID.forName(attempt);
+
+  private static final String attempt1 = "attempt_200707121733_0001_m_000001_0";
+  private static final TaskAttemptID taskID1 = TaskAttemptID.forName(attempt1);
+
   private Text key1 = new Text("key1");
   private Text key2 = new Text("key2");
   private Text val1 = new Text("val1");
@@ -109,12 +128,15 @@ private void writeMapFileOutput(RecordWriter theRecordWriter,
     }
   }
   
-  public void testRecovery() throws Exception {
+  private void testRecoveryInternal(int commitVersion, int recoveryVersion)
+      throws Exception {
     Job job = Job.getInstance();
     FileOutputFormat.setOutputPath(job, outDir);
     Configuration conf = job.getConfiguration();
     conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
     conf.setInt(MRJobConfig.APPLICATION_ATTEMPT_ID, 1);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        commitVersion);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter(outDir, tContext);
@@ -130,32 +152,59 @@ public void testRecovery() throws Exception {
 
     // do commit
     committer.commitTask(tContext);
+
     Path jobTempDir1 = committer.getCommittedTaskPath(tContext);
     File jtd = new File(jobTempDir1.toUri().getPath());
-    assertTrue(jtd.exists());
-    validateContent(jtd);    
-    
+    if (commitVersion == 1) {
+      assertTrue("Version 1 commits to temporary dir " + jtd, jtd.exists());
+      validateContent(jtd);
+    } else {
+      assertFalse("Version 2 commits to output dir " + jtd, jtd.exists());
+    }
+
     //now while running the second app attempt, 
     //recover the task output from first attempt
     Configuration conf2 = job.getConfiguration();
     conf2.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
     conf2.setInt(MRJobConfig.APPLICATION_ATTEMPT_ID, 2);
+    conf2.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        recoveryVersion);
     JobContext jContext2 = new JobContextImpl(conf2, taskID.getJobID());
     TaskAttemptContext tContext2 = new TaskAttemptContextImpl(conf2, taskID);
     FileOutputCommitter committer2 = new FileOutputCommitter(outDir, tContext2);
     committer2.setupJob(tContext2);
     Path jobTempDir2 = committer2.getCommittedTaskPath(tContext2);
     File jtd2 = new File(jobTempDir2.toUri().getPath());
-    
+
     committer2.recoverTask(tContext2);
-    assertTrue(jtd2.exists());
-    validateContent(jtd2);
-    
+    if (recoveryVersion == 1) {
+      assertTrue("Version 1 recovers to " + jtd2, jtd2.exists());
+      validateContent(jtd2);
+    } else {
+      assertFalse("Version 2 commits to output dir " + jtd2, jtd2.exists());
+      if (commitVersion == 1) {
+        assertTrue("Version 2  recovery moves to output dir from "
+            + jtd , jtd.list().length == 0);
+      }
+    }
+
     committer2.commitJob(jContext2);
     validateContent(outDir);
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
 
+  public void testRecoveryV1() throws Exception {
+    testRecoveryInternal(1, 1);
+  }
+
+  public void testRecoveryV2() throws Exception {
+    testRecoveryInternal(2, 2);
+  }
+
+  public void testRecoveryUpgradeV1V2() throws Exception {
+    testRecoveryInternal(1, 2);
+  }
+
   private void validateContent(Path dir) throws IOException {
     validateContent(new File(dir.toUri().getPath()));
   }
@@ -197,12 +246,14 @@ else if (f.getPath().getName().equals(MapFile.DATA_FILE_NAME)) {
     assert(fileCount > 0);
     assert(dataFileFound && indexFileFound);
   }
-  
-  public void testCommitter() throws Exception {
+
+  private void testCommitterInternal(int version) throws Exception {
     Job job = Job.getInstance();
     FileOutputFormat.setOutputPath(job, outDir);
     Configuration conf = job.getConfiguration();
     conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter(outDir, tContext);
@@ -225,11 +276,22 @@ public void testCommitter() throws Exception {
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
 
-  public void testMapFileOutputCommitter() throws Exception {
+  public void testCommitterV1() throws Exception {
+    testCommitterInternal(1);
+  }
+
+  public void testCommitterV2() throws Exception {
+    testCommitterInternal(2);
+  }
+
+  private void testMapFileOutputCommitterInternal(int version)
+      throws Exception {
     Job job = Job.getInstance();
     FileOutputFormat.setOutputPath(job, outDir);
     Configuration conf = job.getConfiguration();
     conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());    
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter(outDir, tContext);
@@ -247,16 +309,51 @@ public void testMapFileOutputCommitter() throws Exception {
     committer.commitTask(tContext);
     committer.commitJob(jContext);
 
+    // Ensure getReaders call works and also ignores
+    // hidden filenames (_ or . prefixes)
+    try {
+      MapFileOutputFormat.getReaders(outDir, conf);
+    } catch (Exception e) {
+      fail("Fail to read from MapFileOutputFormat: " + e);
+      e.printStackTrace();
+    }
+
     // validate output
     validateMapFileOutputContent(FileSystem.get(job.getConfiguration()), outDir);
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
+
+  public void testMapFileOutputCommitterV1() throws Exception {
+    testMapFileOutputCommitterInternal(1);
+  }
   
-  public void testAbort() throws IOException, InterruptedException {
+  public void testMapFileOutputCommitterV2() throws Exception {
+    testMapFileOutputCommitterInternal(2);
+  }
+
+  public void testInvalidVersionNumber() throws IOException {
+    Job job = Job.getInstance();
+    FileOutputFormat.setOutputPath(job, outDir);
+    Configuration conf = job.getConfiguration();
+    conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION, 3);
+    TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
+    try {
+      new FileOutputCommitter(outDir, tContext);
+      fail("should've thrown an exception!");
+    } catch (IOException e) {
+      //test passed
+    }
+  }
+
+  private void testAbortInternal(int version)
+      throws IOException, InterruptedException {
     Job job = Job.getInstance();
     FileOutputFormat.setOutputPath(job, outDir);
     Configuration conf = job.getConfiguration();
     conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        version);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
     FileOutputCommitter committer = new FileOutputCommitter(outDir, tContext);
@@ -285,6 +382,14 @@ public void testAbort() throws IOException, InterruptedException {
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
 
+  public void testAbortV1() throws IOException, InterruptedException {
+    testAbortInternal(1);
+  }
+
+  public void testAbortV2() throws IOException, InterruptedException {
+    testAbortInternal(2);
+  }
+
   public static class FakeFileSystem extends RawLocalFileSystem {
     public FakeFileSystem() {
       super();
@@ -301,13 +406,16 @@ public boolean delete(Path p, boolean recursive) throws IOException {
   }
 
   
-  public void testFailAbort() throws IOException, InterruptedException {
+  private void testFailAbortInternal(int version)
+      throws IOException, InterruptedException {
     Job job = Job.getInstance();
     Configuration conf = job.getConfiguration();
     conf.set(FileSystem.FS_DEFAULT_NAME_KEY, "faildel:///");
     conf.setClass("fs.faildel.impl", FakeFileSystem.class, FileSystem.class);
     conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
     conf.setInt(MRJobConfig.APPLICATION_ATTEMPT_ID, 1);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        version);
     FileOutputFormat.setOutputPath(job, outDir);
     JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
     TaskAttemptContext tContext = new TaskAttemptContextImpl(conf, taskID);
@@ -353,6 +461,115 @@ public void testFailAbort() throws IOException, InterruptedException {
     FileUtil.fullyDelete(new File(outDir.toString()));
   }
 
+  public void testFailAbortV1() throws Exception {
+    testFailAbortInternal(1);
+  }
+
+  public void testFailAbortV2() throws Exception {
+    testFailAbortInternal(2);
+  }
+
+  static class RLFS extends RawLocalFileSystem {
+    private final ThreadLocal needNull = new ThreadLocal() {
+      @Override
+      protected Boolean initialValue() {
+        return true;
+      }
+    };
+
+    public RLFS() {
+    }
+
+    @Override
+    public FileStatus getFileStatus(Path f) throws IOException {
+      if (needNull.get() &&
+          OUT_SUB_DIR.toUri().getPath().equals(f.toUri().getPath())) {
+        needNull.set(false); // lie once per thread
+        return null;
+      }
+      return super.getFileStatus(f);
+    }
+  }
+
+  private void testConcurrentCommitTaskWithSubDir(int version)
+      throws Exception {
+    final Job job = Job.getInstance();
+    FileOutputFormat.setOutputPath(job, outDir);
+    final Configuration conf = job.getConfiguration();
+    conf.set(MRJobConfig.TASK_ATTEMPT_ID, attempt);
+    conf.setInt(FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION,
+        version);
+
+    conf.setClass("fs.file.impl", RLFS.class, FileSystem.class);
+    FileSystem.closeAll();
+
+    final JobContext jContext = new JobContextImpl(conf, taskID.getJobID());
+    final FileOutputCommitter amCommitter =
+        new FileOutputCommitter(outDir, jContext);
+    amCommitter.setupJob(jContext);
+
+    final TaskAttemptContext[] taCtx = new TaskAttemptContextImpl[2];
+    taCtx[0] = new TaskAttemptContextImpl(conf, taskID);
+    taCtx[1] = new TaskAttemptContextImpl(conf, taskID1);
+
+    final TextOutputFormat[] tof = new TextOutputFormat[2];
+    for (int i = 0; i < tof.length; i++) {
+      tof[i] = new TextOutputFormat() {
+        @Override
+        public Path getDefaultWorkFile(TaskAttemptContext context,
+            String extension) throws IOException {
+          final FileOutputCommitter foc = (FileOutputCommitter)
+              getOutputCommitter(context);
+          return new Path(new Path(foc.getWorkPath(), SUB_DIR),
+              getUniqueFile(context, getOutputName(context), extension));
+        }
+      };
+    }
+
+    final ExecutorService executor = Executors.newFixedThreadPool(2);
+    try {
+      for (int i = 0; i < taCtx.length; i++) {
+        final int taskIdx = i;
+        executor.submit(new Callable() {
+          @Override
+          public Void call() throws IOException, InterruptedException {
+            final OutputCommitter outputCommitter =
+                tof[taskIdx].getOutputCommitter(taCtx[taskIdx]);
+            outputCommitter.setupTask(taCtx[taskIdx]);
+            final RecordWriter rw =
+                tof[taskIdx].getRecordWriter(taCtx[taskIdx]);
+            writeOutput(rw, taCtx[taskIdx]);
+            outputCommitter.commitTask(taCtx[taskIdx]);
+            return null;
+          }
+        });
+      }
+    } finally {
+      executor.shutdown();
+      while (!executor.awaitTermination(1, TimeUnit.SECONDS)) {
+        LOG.info("Awaiting thread termination!");
+      }
+    }
+
+    amCommitter.commitJob(jContext);
+    final RawLocalFileSystem lfs = new RawLocalFileSystem();
+    lfs.setConf(conf);
+    assertFalse("Must not end up with sub_dir/sub_dir",
+        lfs.exists(new Path(OUT_SUB_DIR, SUB_DIR)));
+
+    // validate output
+    validateContent(OUT_SUB_DIR);
+    FileUtil.fullyDelete(new File(outDir.toString()));
+  }
+
+  public void testConcurrentCommitTaskWithSubDirV1() throws Exception {
+    testConcurrentCommitTaskWithSubDir(1);
+  }
+
+  public void testConcurrentCommitTaskWithSubDirV2() throws Exception {
+    testConcurrentCommitTaskWithSubDir(2);
+  }
+
   public static String slurp(File f) throws IOException {
     int len = (int) f.length();
     byte[] buf = new byte[len];
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
index 1cf63d4a71bf3..6df82611d0005 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
@@ -345,9 +345,7 @@ protected synchronized void loadFullHistoryData(boolean loadTasks,
       JobHistoryParser parser = null;
       try {
         final FileSystem fs = historyFileAbsolute.getFileSystem(conf);
-        parser =
-            new JobHistoryParser(historyFileAbsolute.getFileSystem(conf),
-                historyFileAbsolute);
+        parser = new JobHistoryParser(fs, historyFileAbsolute);
         final Path jobConfPath = new Path(historyFileAbsolute.getParent(),
             JobHistoryUtils.getIntermediateConfFileName(jobId));
         final Configuration conf = new Configuration();
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/markdown/HistoryServerRest.md b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/markdown/HistoryServerRest.md
index 8a78754538556..b4ce00a07e148 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/markdown/HistoryServerRest.md
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/markdown/HistoryServerRest.md
@@ -1889,7 +1889,7 @@ A Task Attempt resource contains information about a particular task attempt wit
 
 Use the following URI to obtain an Task Attempt Object, from a task identified by the attemptid value.
 
-      * http:///ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempt/{attemptid}
+      * http:///ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}
 
 #### HTTP Operations Supported
 
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
index 686fa0c70c599..8517c19737249 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
@@ -64,6 +64,7 @@
 import org.apache.hadoop.mapreduce.v2.util.MRApps;
 import org.apache.hadoop.net.NetUtils;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authorize.AuthorizationException;
 import org.apache.hadoop.security.token.Token;
 import org.apache.hadoop.yarn.api.records.ApplicationId;
 import org.apache.hadoop.yarn.api.records.ApplicationReport;
@@ -328,6 +329,11 @@ private synchronized Object invoke(String method, Class argClass,
         // Force reconnection by setting the proxy to null.
         realProxy = null;
         // HS/AMS shut down
+
+        if (e.getCause() instanceof AuthorizationException) {
+          throw new IOException(e.getTargetException());
+        }
+
         // if it's AM shut down, do not decrement maxClientRetry as we wait for
         // AM to be restarted.
         if (!usingAMProxy.get()) {
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
index 41dc72f2b6e22..8e5760761256f 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
@@ -640,7 +640,10 @@ public void killJob(JobID arg0) throws IOException, InterruptedException {
       clientCache.getClient(arg0).killJob(arg0);
       long currentTimeMillis = System.currentTimeMillis();
       long timeKillIssued = currentTimeMillis;
-      while ((currentTimeMillis < timeKillIssued + 10000L)
+      long killTimeOut =
+          conf.getLong(MRJobConfig.MR_AM_HARD_KILL_TIMEOUT_MS,
+                       MRJobConfig.DEFAULT_MR_AM_HARD_KILL_TIMEOUT_MS);
+      while ((currentTimeMillis < timeKillIssued + killTimeOut)
           && !isJobInTerminalState(status)) {
         try {
           Thread.sleep(1000L);
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
deleted file mode 100644
index f67ba1fb16b24..0000000000000
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestJobConf.java
+++ /dev/null
@@ -1,199 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.conf;
-
-import org.junit.Assert;
-import org.junit.Test;
-import org.apache.hadoop.mapred.JobConf;
-import org.apache.hadoop.mapreduce.MRJobConfig;
-
-public class TestJobConf {
-
-  @Test
-  public void testProfileParamsDefaults() {
-    JobConf configuration = new JobConf();
-    String result = configuration.getProfileParams();
-    Assert.assertNotNull(result);
-    Assert.assertTrue(result.contains("file=%s"));
-    Assert.assertTrue(result.startsWith("-agentlib:hprof"));
-  }
-
-  @Test
-  public void testProfileParamsSetter() {
-    JobConf configuration = new JobConf();
-
-    configuration.setProfileParams("test");
-    Assert.assertEquals("test", configuration.get(MRJobConfig.TASK_PROFILE_PARAMS));
-  }
-
-  @Test
-  public void testProfileParamsGetter() {
-    JobConf configuration = new JobConf();
-
-    configuration.set(MRJobConfig.TASK_PROFILE_PARAMS, "test");
-    Assert.assertEquals("test", configuration.getProfileParams());
-  }
-
-  /**
-   * Testing mapred.task.maxvmem replacement with new values
-   *
-   */
-  @Test
-  public void testMemoryConfigForMapOrReduceTask(){
-    JobConf configuration = new JobConf();
-    configuration.set(MRJobConfig.MAP_MEMORY_MB,String.valueOf(300));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,String.valueOf(300));
-    Assert.assertEquals(configuration.getMemoryForMapTask(),300);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(),300);
-
-    configuration.set("mapred.task.maxvmem" , String.valueOf(2*1024 * 1024));
-    configuration.set(MRJobConfig.MAP_MEMORY_MB,String.valueOf(300));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,String.valueOf(300));
-    Assert.assertEquals(configuration.getMemoryForMapTask(),2);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
-
-    configuration = new JobConf();
-    configuration.set("mapred.task.maxvmem" , "-1");
-    configuration.set(MRJobConfig.MAP_MEMORY_MB,String.valueOf(300));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,String.valueOf(400));
-    Assert.assertEquals(configuration.getMemoryForMapTask(), 300);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(), 400);
-
-    configuration = new JobConf();
-    configuration.set("mapred.task.maxvmem" , String.valueOf(2*1024 * 1024));
-    configuration.set(MRJobConfig.MAP_MEMORY_MB,"-1");
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,"-1");
-    Assert.assertEquals(configuration.getMemoryForMapTask(),2);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
-
-    configuration = new JobConf();
-    configuration.set("mapred.task.maxvmem" , String.valueOf(-1));
-    configuration.set(MRJobConfig.MAP_MEMORY_MB,"-1");
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB,"-1");
-    Assert.assertEquals(configuration.getMemoryForMapTask(),
-        MRJobConfig.DEFAULT_MAP_MEMORY_MB);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(),
-        MRJobConfig.DEFAULT_REDUCE_MEMORY_MB);
-
-    configuration = new JobConf();
-    configuration.set("mapred.task.maxvmem" , String.valueOf(2*1024 * 1024));
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, "3");
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "3");
-    Assert.assertEquals(configuration.getMemoryForMapTask(),2);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(),2);
-  }
-
-  /**
-   * Test that negative values for MAPRED_TASK_MAXVMEM_PROPERTY cause
-   * new configuration keys' values to be used.
-   */
-  @Test
-  public void testNegativeValueForTaskVmem() {
-    JobConf configuration = new JobConf();
-
-    configuration.set(JobConf.MAPRED_TASK_MAXVMEM_PROPERTY, "-3");
-    Assert.assertEquals(MRJobConfig.DEFAULT_MAP_MEMORY_MB,
-        configuration.getMemoryForMapTask());
-    Assert.assertEquals(MRJobConfig.DEFAULT_REDUCE_MEMORY_MB,
-        configuration.getMemoryForReduceTask());
-    
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, "4");
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "5");
-    Assert.assertEquals(4, configuration.getMemoryForMapTask());
-    Assert.assertEquals(5, configuration.getMemoryForReduceTask());
-    
-  }
-  
-  /**
-   * Test that negative values for new configuration keys get passed through.
-   */
-  @Test
-  public void testNegativeValuesForMemoryParams() {
-    JobConf configuration = new JobConf();
-        
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, "-5");
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, "-6");
-    Assert.assertEquals(MRJobConfig.DEFAULT_MAP_MEMORY_MB,
-        configuration.getMemoryForMapTask());
-    Assert.assertEquals(MRJobConfig.DEFAULT_REDUCE_MEMORY_MB,
-        configuration.getMemoryForReduceTask());
-  }
-
-  /**
-   *   Test deprecated accessor and mutator method for mapred.task.maxvmem
-   */
-  @Test
-  public void testMaxVirtualMemoryForTask() {
-    JobConf configuration = new JobConf();
-
-    //get test case
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(300));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(-1));
-    Assert.assertEquals(
-      configuration.getMaxVirtualMemoryForTask(), 1024 * 1024 * 1024);
-
-    configuration = new JobConf();
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(-1));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(200));
-    Assert.assertEquals(
-      configuration.getMaxVirtualMemoryForTask(), 1024 * 1024 * 1024);
-
-    configuration = new JobConf();
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(-1));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(-1));
-    configuration.set("mapred.task.maxvmem", String.valueOf(1 * 1024 * 1024));
-    Assert.assertEquals(
-      configuration.getMaxVirtualMemoryForTask(), 1 * 1024 * 1024);
-
-    configuration = new JobConf();
-    configuration.set("mapred.task.maxvmem", String.valueOf(1 * 1024 * 1024));
-    Assert.assertEquals(
-      configuration.getMaxVirtualMemoryForTask(), 1 * 1024 * 1024);
-
-    //set test case
-
-    configuration = new JobConf();
-    configuration.setMaxVirtualMemoryForTask(2 * 1024 * 1024);
-    Assert.assertEquals(configuration.getMemoryForMapTask(), 2);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(), 2);
-
-    configuration = new JobConf();   
-    configuration.set(MRJobConfig.MAP_MEMORY_MB, String.valueOf(300));
-    configuration.set(MRJobConfig.REDUCE_MEMORY_MB, String.valueOf(400));
-    configuration.setMaxVirtualMemoryForTask(2 * 1024 * 1024);
-    Assert.assertEquals(configuration.getMemoryForMapTask(), 2);
-    Assert.assertEquals(configuration.getMemoryForReduceTask(), 2);
-    
-    
-  }
-
-  /**
-   * Ensure that by default JobContext.MAX_TASK_FAILURES_PER_TRACKER is less
-   * JobContext.MAP_MAX_ATTEMPTS and JobContext.REDUCE_MAX_ATTEMPTS so that
-   * failed tasks will be retried on other nodes
-   */
-  @Test
-  public void testMaxTaskFailuresPerTracker() {
-    JobConf jobConf = new JobConf(true);
-    Assert.assertTrue("By default JobContext.MAX_TASK_FAILURES_PER_TRACKER was "
-      + "not less than JobContext.MAP_MAX_ATTEMPTS and REDUCE_MAX_ATTEMPTS"
-      ,jobConf.getMaxTaskFailuresPerTracker() < jobConf.getMaxMapAttempts() &&
-      jobConf.getMaxTaskFailuresPerTracker() < jobConf.getMaxReduceAttempts()
-      );
-  }
-}
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
index bd46e7c8bdbe7..bb5021391ce99 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
@@ -182,8 +182,8 @@ private static void displayUsage() {
       "\t-reduces \n" +
       "\t-startTime 
      
  * 
  * Equivalently, {@link RandomTextWriter} also supports all the above options
  * and ones supported by {@link Tool} via the command-line.
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/RandomWriter.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/RandomWriter.java
index a326c8ca2be2c..67c9ca8f0f3ad 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/RandomWriter.java
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/RandomWriter.java
@@ -47,7 +47,7 @@
  * random binary sequence file of BytesWritable.
  * In order for this program to generate data for terasort with 10-byte keys
  * and 90-byte values, have the following config:
- * 
+ * <pre>{@code
  * <?xml version="1.0"?>
  * <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  * <configuration>
@@ -71,8 +71,7 @@
  *     <name>mapreduce.randomwriter.totalbytes</name>
  *     <value>1099511627776</value>
  *   </property>
- * </configuration>
- * 
+ * }
      * Equivalently, {@link RandomWriter} also supports all the above options * and ones supported by {@link GenericOptionsParser} via the command-line. */ diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java index fd36285aa27f0..191195bd3bdb5 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java @@ -531,6 +531,49 @@ public void testChangingJobPriority(String jobId, Configuration conf) verifyJobPriority(jobId, "NORMAL", conf, createJobClient()); } + /** + * Test -list option displays job name. + * The name is capped to 20 characters for display. + */ + public void testJobName() throws Exception { + Configuration conf = createJobConf(); + CLI jc = createJobClient(); + Job job = MapReduceTestUtil.createJob(conf, getInputDir(), getOutputDir(), + 1, 1, "short_name"); + job.setJobName("mapreduce"); + job.setPriority(JobPriority.NORMAL); + job.waitForCompletion(true); + String jobId = job.getJobID().toString(); + verifyJobName(jobId, "mapreduce", conf, jc); + Job job2 = MapReduceTestUtil.createJob(conf, getInputDir(), getOutputDir(), + 1, 1, "long_name"); + job2.setJobName("mapreduce_job_with_long_name"); + job2.setPriority(JobPriority.NORMAL); + job2.waitForCompletion(true); + jobId = job2.getJobID().toString(); + verifyJobName(jobId, "mapreduce_job_with_l", conf, jc); + } + + protected void verifyJobName(String jobId, String name, + Configuration conf, CLI jc) throws Exception { + PipedInputStream pis = new PipedInputStream(); + PipedOutputStream pos = new PipedOutputStream(pis); + int exitCode = runTool(conf, jc, + new String[] { "-list", "all" }, pos); + assertEquals("Exit code", 0, exitCode); + BufferedReader br = new BufferedReader(new InputStreamReader(pis)); + String line = null; + while ((line = br.readLine()) != null) { + LOG.info("line = " + line); + if (!line.contains(jobId)) { + continue; + } + assertTrue(line.contains(name)); + break; + } + pis.close(); + } + protected CLI createJobClient() throws IOException { return new CLI(); } diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java index db51ec6a9c59c..85c675c308fb7 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java @@ -29,9 +29,6 @@ import java.util.concurrent.TimeoutException; import java.util.zip.GZIPOutputStream; -import org.junit.Assert; -import junit.framework.TestCase; - import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.BlockLocation; import org.apache.hadoop.fs.CommonConfigurationKeys; @@ -60,7 +57,11 @@ import com.google.common.collect.HashMultiset; -public class TestCombineFileInputFormat extends TestCase { +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +public class TestCombineFileInputFormat { private static final String rack1[] = new String[] { "/r1" @@ -221,6 +222,7 @@ public RecordReader createRecordReader(InputSplit split, } } + @Test public void testRecordReaderInit() throws InterruptedException, IOException { // Test that we properly initialize the child recordreader when // CombineFileInputFormat and CombineFileRecordReader are used. @@ -258,6 +260,7 @@ public void testRecordReaderInit() throws InterruptedException, IOException { rr.getCurrentKey().toString()); } + @Test public void testReinit() throws Exception { // Test that a split containing multiple files works correctly, // with the child RecordReader getting its initialize() method @@ -296,6 +299,7 @@ public void testReinit() throws Exception { assertFalse(rr.nextKeyValue()); } + @Test public void testSplitPlacement() throws Exception { MiniDFSCluster dfs = null; FileSystem fileSys = null; @@ -725,6 +729,7 @@ private static void writeDataAndSetReplication(FileSystem fileSys, Path name, DFSTestUtil.waitReplication(fileSys, name, replication); } + @Test public void testNodeDistribution() throws IOException, InterruptedException { DummyInputFormat inFormat = new DummyInputFormat(); int numBlocks = 60; @@ -774,20 +779,21 @@ public void testNodeDistribution() throws IOException, InterruptedException { maxSplitSize, minSizeNode, minSizeRack, splits); int expectedSplitCount = (int) (totLength / maxSplitSize); - Assert.assertEquals(expectedSplitCount, splits.size()); + assertEquals(expectedSplitCount, splits.size()); // Ensure 90+% of the splits have node local blocks. // 100% locality may not always be achieved. int numLocalSplits = 0; for (InputSplit inputSplit : splits) { - Assert.assertEquals(maxSplitSize, inputSplit.getLength()); + assertEquals(maxSplitSize, inputSplit.getLength()); if (inputSplit.getLocations().length == 1) { numLocalSplits++; } } - Assert.assertTrue(numLocalSplits >= 0.9 * splits.size()); + assertTrue(numLocalSplits >= 0.9 * splits.size()); } - + + @Test public void testNodeInputSplit() throws IOException, InterruptedException { // Regression test for MAPREDUCE-4892. There are 2 nodes with all blocks on // both nodes. The grouping ensures that both nodes get splits instead of @@ -826,18 +832,19 @@ public void testNodeInputSplit() throws IOException, InterruptedException { maxSize, minSizeNode, minSizeRack, splits); int expectedSplitCount = (int)(totLength/maxSize); - Assert.assertEquals(expectedSplitCount, splits.size()); + assertEquals(expectedSplitCount, splits.size()); HashMultiset nodeSplits = HashMultiset.create(); for(int i=0; i { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java index d565098e0a153..25dee6bf5ff05 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java @@ -50,7 +50,7 @@ * where $S=[0,1)^2$ is a unit square, * $x=(x_1,x_2)$ is a 2-dimensional point, * and $f$ is a function describing the inscribed circle of the square $S$, - * $f(x)=1$ if $(2x_1-1)^2+(2x_2-1)^2 <= 1$ and $f(x)=0$, otherwise. + * $f(x)=1$ if $(2x_1-1)^2+(2x_2-1)^2 <= 1$ and $f(x)=0$, otherwise. * It is easy to see that Pi is equal to $4I$. * So an approximation of Pi is obtained once $I$ is evaluated numerically. * @@ -155,7 +155,7 @@ public static class QmcMapper extends /** Map method. * @param offset samples starting from the (offset+1)th sample. * @param size the number of samples for this map - * @param context output {ture->numInside, false->numOutside} + * @param context output {ture->numInside, false->numOutside} */ public void map(LongWritable offset, LongWritable size, diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomTextWriter.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomTextWriter.java index 4d555c606d391..6309ee67e59c2 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomTextWriter.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomTextWriter.java @@ -42,7 +42,7 @@ * random sequence of words. * In order for this program to generate data for terasort with a 5-10 words * per key and 20-100 words per value, have the following config: - * + * <pre>{@code * <?xml version="1.0"?> * <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> * <configuration> @@ -66,7 +66,7 @@ * <name>mapreduce.randomtextwriter.totalbytes</name> * <value>1099511627776</value> * </property> - * </configuration> + * } * * Equivalently, {@link RandomTextWriter} also supports all the above options * and ones supported by {@link Tool} via the command-line. diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomWriter.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomWriter.java index e1c13ecfa2735..8f322b154eeda 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomWriter.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomWriter.java @@ -47,7 +47,7 @@ * random binary sequence file of BytesWritable. * In order for this program to generate data for terasort with 10-byte keys * and 90-byte values, have the following config: - * + * <pre>{@code * <?xml version="1.0"?> * <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> * <configuration> @@ -71,8 +71,7 @@ * <name>mapreduce.randomwriter.totalbytes</name> * <value>1099511627776</value> * </property> - * </configuration> - * + * } * Equivalently, {@link RandomWriter} also supports all the above options * and ones supported by {@link GenericOptionsParser} via the command-line. */ diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/SecondarySort.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/SecondarySort.java index d536ec98c11cd..8841fdcfbb2b1 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/SecondarySort.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/SecondarySort.java @@ -74,7 +74,7 @@ public int getSecond() { } /** * Read the two integers. - * Encoded as: MIN_VALUE -> 0, 0 -> -MIN_VALUE, MAX_VALUE-> -1 + * Encoded as: MIN_VALUE -> 0, 0 -> -MIN_VALUE, MAX_VALUE-> -1 */ @Override public void readFields(DataInput in) throws IOException { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java index a90c02b0e3a77..0382c09c78059 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java @@ -24,7 +24,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; -import org.apache.hadoop.mapreduce.filecache.DistributedCache; +import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.Writable; @@ -160,13 +160,14 @@ public int run(String[] args) throws Exception { System.out.println("Sampling input to effect total-order sort..."); job.setPartitionerClass(TotalOrderPartitioner.class); Path inputDir = FileInputFormat.getInputPaths(job)[0]; - inputDir = inputDir.makeQualified(inputDir.getFileSystem(conf)); + FileSystem fs = inputDir.getFileSystem(conf); + inputDir = inputDir.makeQualified(fs.getUri(), fs.getWorkingDirectory()); Path partitionFile = new Path(inputDir, "_sortPartitioning"); TotalOrderPartitioner.setPartitionFile(conf, partitionFile); InputSampler.writePartitionFile(job, sampler); URI partitionUri = new URI(partitionFile.toString() + "#" + "_sortPartitioning"); - DistributedCache.addCacheFile(partitionUri, conf); + job.addCacheFile(partitionUri); } System.out.println("Running on " + diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/DistBbp.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/DistBbp.java index 4484d20d2c93a..268066cb84820 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/DistBbp.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/DistBbp.java @@ -35,7 +35,7 @@ * A map/reduce program that uses a BBP-type method to compute exact * binary digits of Pi. * This program is designed for computing the n th bit of Pi, - * for large n, say n >= 10^8. + * for large n, say n >= 10^8. * For computing lower bits of Pi, consider using bbp. * * The actually computation is done by DistSum jobs. diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/math/Modular.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/math/Modular.java index 58f859d53526a..1c039a2fea151 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/math/Modular.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/math/Modular.java @@ -78,7 +78,7 @@ public static double addMod(double x, final double a) { return x >= 1? x - 1: x < 0? x + 1: x; } - /** Given 0 < x < y, + /** Given 0 < x < y, * return x^(-1) mod y. */ public static long modInverse(final long x, final long y) { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/GenSort.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/GenSort.java index 94f9baaf89b58..beb07439faede 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/GenSort.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/GenSort.java @@ -28,7 +28,7 @@ /** * A single process data generator for the terasort data. Based on gensort.c - * version 1.1 (3 Mar 2009) from Chris Nyberg . + * version 1.1 (3 Mar 2009) from Chris Nyberg <chris.nyberg@ordinal.com>. */ public class GenSort { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraGen.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraGen.java index e8b65038e7971..d7d751a6f6fee 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraGen.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraGen.java @@ -70,7 +70,6 @@ public class TeraGen extends Configured implements Tool { public static enum Counters {CHECKSUM} - public static final String NUM_ROWS = "mapreduce.terasort.num-rows"; /** * An input format that assigns ranges of longs to each mapper. */ @@ -189,11 +188,12 @@ public List getSplits(JobContext job) { } static long getNumberOfRows(JobContext job) { - return job.getConfiguration().getLong(NUM_ROWS, 0); + return job.getConfiguration().getLong(TeraSortConfigKeys.NUM_ROWS.key(), + TeraSortConfigKeys.DEFAULT_NUM_ROWS); } static void setNumberOfRows(Job job, long numRows) { - job.getConfiguration().setLong(NUM_ROWS, numRows); + job.getConfiguration().setLong(TeraSortConfigKeys.NUM_ROWS.key(), numRows); } /** diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraInputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraInputFormat.java index 88b12dd1ff4c6..20ce8ef2b60de 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraInputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraInputFormat.java @@ -50,10 +50,6 @@ public class TeraInputFormat extends FileInputFormat { static final String PARTITION_FILENAME = "_partition.lst"; - private static final String NUM_PARTITIONS = - "mapreduce.terasort.num.partitions"; - private static final String SAMPLE_SIZE = - "mapreduce.terasort.partitions.sample"; static final int KEY_LENGTH = 10; static final int VALUE_LENGTH = 90; static final int RECORD_LENGTH = KEY_LENGTH + VALUE_LENGTH; @@ -123,11 +119,16 @@ public static void writePartitionFile(final JobContext job, final TeraInputFormat inFormat = new TeraInputFormat(); final TextSampler sampler = new TextSampler(); int partitions = job.getNumReduceTasks(); - long sampleSize = conf.getLong(SAMPLE_SIZE, 100000); + long sampleSize = + conf.getLong(TeraSortConfigKeys.SAMPLE_SIZE.key(), + TeraSortConfigKeys.DEFAULT_SAMPLE_SIZE); final List splits = inFormat.getSplits(job); long t2 = System.currentTimeMillis(); System.out.println("Computing input splits took " + (t2 - t1) + "ms"); - int samples = Math.min(conf.getInt(NUM_PARTITIONS, 10), splits.size()); + int samples = + Math.min(conf.getInt(TeraSortConfigKeys.NUM_PARTITIONS.key(), + TeraSortConfigKeys.DEFAULT_NUM_PARTITIONS), + splits.size()); System.out.println("Sampling " + samples + " splits of " + splits.size()); final long recordsPerSample = sampleSize / samples; final int sampleStep = splits.size() / samples; @@ -294,7 +295,8 @@ public List getSplits(JobContext job) throws IOException { lastResult = super.getSplits(job); t2 = System.currentTimeMillis(); System.out.println("Spent " + (t2 - t1) + "ms computing base-splits."); - if (job.getConfiguration().getBoolean(TeraScheduler.USE, true)) { + if (job.getConfiguration().getBoolean(TeraSortConfigKeys.USE_TERA_SCHEDULER.key(), + TeraSortConfigKeys.DEFAULT_USE_TERA_SCHEDULER)) { TeraScheduler scheduler = new TeraScheduler( lastResult.toArray(new FileSplit[0]), job.getConfiguration()); lastResult = scheduler.getNewFileSplits(); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java index 867f33ef7a393..fd3ea78fdcdb5 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java @@ -40,21 +40,23 @@ * An output format that writes the key and value appended together. */ public class TeraOutputFormat extends FileOutputFormat { - static final String FINAL_SYNC_ATTRIBUTE = "mapreduce.terasort.final.sync"; private OutputCommitter committer = null; /** * Set the requirement for a final sync before the stream is closed. */ static void setFinalSync(JobContext job, boolean newValue) { - job.getConfiguration().setBoolean(FINAL_SYNC_ATTRIBUTE, newValue); + job.getConfiguration().setBoolean( + TeraSortConfigKeys.FINAL_SYNC_ATTRIBUTE.key(), newValue); } /** * Does the user want a final sync at close? */ public static boolean getFinalSync(JobContext job) { - return job.getConfiguration().getBoolean(FINAL_SYNC_ATTRIBUTE, false); + return job.getConfiguration().getBoolean( + TeraSortConfigKeys.FINAL_SYNC_ATTRIBUTE.key(), + TeraSortConfigKeys.DEFAULT_FINAL_SYNC_ATTRIBUTE); } static class TeraRecordWriter extends RecordWriter { diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraScheduler.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraScheduler.java index 7095dd7d28e90..30b50d8cf151f 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraScheduler.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraScheduler.java @@ -31,7 +31,6 @@ import com.google.common.base.Charsets; class TeraScheduler { - static String USE = "mapreduce.terasort.use.terascheduler"; private static final Log LOG = LogFactory.getLog(TeraScheduler.class); private Split[] splits; private List hosts = new ArrayList(); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java index 6022f6c50266b..5d586e67c5549 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java @@ -48,8 +48,6 @@ */ public class TeraSort extends Configured implements Tool { private static final Log LOG = LogFactory.getLog(TeraSort.class); - static String SIMPLE_PARTITIONER = "mapreduce.terasort.simplepartitioner"; - static String OUTPUT_REPLICATION = "mapreduce.terasort.output.replication"; /** * A partitioner that splits text keys into roughly equal partitions @@ -147,7 +145,7 @@ void print(PrintStream strm) throws IOException { * Read the cut points from the given sequence file. * @param fs the file system * @param p the path to read - * @param job the job config + * @param conf the job config * @return the strings to split the partitions on * @throws IOException */ @@ -262,22 +260,40 @@ public int getPartition(Text key, Text value, int numPartitions) { } public static boolean getUseSimplePartitioner(JobContext job) { - return job.getConfiguration().getBoolean(SIMPLE_PARTITIONER, false); + return job.getConfiguration().getBoolean( + TeraSortConfigKeys.USE_SIMPLE_PARTITIONER.key(), + TeraSortConfigKeys.DEFAULT_USE_SIMPLE_PARTITIONER); } public static void setUseSimplePartitioner(Job job, boolean value) { - job.getConfiguration().setBoolean(SIMPLE_PARTITIONER, value); + job.getConfiguration().setBoolean( + TeraSortConfigKeys.USE_SIMPLE_PARTITIONER.key(), value); } public static int getOutputReplication(JobContext job) { - return job.getConfiguration().getInt(OUTPUT_REPLICATION, 1); + return job.getConfiguration().getInt( + TeraSortConfigKeys.OUTPUT_REPLICATION.key(), + TeraSortConfigKeys.DEFAULT_OUTPUT_REPLICATION); } public static void setOutputReplication(Job job, int value) { - job.getConfiguration().setInt(OUTPUT_REPLICATION, value); + job.getConfiguration().setInt(TeraSortConfigKeys.OUTPUT_REPLICATION.key(), + value); + } + + private static void usage() throws IOException { + System.err.println("Usage: terasort [-Dproperty=value] "); + System.err.println("TeraSort configurations are:"); + for (TeraSortConfigKeys teraSortConfigKeys : TeraSortConfigKeys.values()) { + System.err.println(teraSortConfigKeys.toString()); + } } public int run(String[] args) throws Exception { + if (args.length != 2) { + usage(); + return 2; + } LOG.info("starting"); Job job = Job.getInstance(getConf()); Path inputDir = new Path(args[0]); diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSortConfigKeys.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSortConfigKeys.java new file mode 100644 index 0000000000000..0822a50f2dbbf --- /dev/null +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSortConfigKeys.java @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.examples.terasort; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Unstable; + +/** + *

      + * TeraSort configurations. + *

      + */ +@Private +@Unstable +public enum TeraSortConfigKeys { + + NUM_ROWS("mapreduce.terasort.num-rows", + "Number of rows to generate during teragen."), + + NUM_PARTITIONS("mapreduce.terasort.num.partitions", + "Number of partitions used for sampling."), + + SAMPLE_SIZE("mapreduce.terasort.partitions.sample", + "Sample size for each partition."), + + FINAL_SYNC_ATTRIBUTE("mapreduce.terasort.final.sync", + "Perform a disk-persisting hsync at end of every file-write."), + + USE_TERA_SCHEDULER("mapreduce.terasort.use.terascheduler", + "Use TeraScheduler for computing input split distribution."), + + USE_SIMPLE_PARTITIONER("mapreduce.terasort.simplepartitioner", + "Use SimplePartitioner instead of TotalOrderPartitioner."), + + OUTPUT_REPLICATION("mapreduce.terasort.output.replication", + "Replication factor to use for output data files."); + + private String confName; + private String description; + + TeraSortConfigKeys(String configName, String description) { + this.confName = configName; + this.description = description; + } + + public String key() { + return this.confName; + } + + public String toString() { + return "<" + confName + "> " + description; + } + + public static final long DEFAULT_NUM_ROWS = 0L; + public static final int DEFAULT_NUM_PARTITIONS = 10; + public static final long DEFAULT_SAMPLE_SIZE = 100000L; + public static final boolean DEFAULT_FINAL_SYNC_ATTRIBUTE = false; + public static final boolean DEFAULT_USE_TERA_SCHEDULER = true; + public static final boolean DEFAULT_USE_SIMPLE_PARTITIONER = false; + public static final int DEFAULT_OUTPUT_REPLICATION = 1; +} diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/examples/terasort/TestTeraSort.java b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/examples/terasort/TestTeraSort.java index 1956872b7ed57..349208999d3ca 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/examples/terasort/TestTeraSort.java +++ b/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/examples/terasort/TestTeraSort.java @@ -104,4 +104,9 @@ public void testTeraSort() throws Exception { TERA_OUTPUT_PATH); } + public void testTeraSortWithLessThanTwoArgs() throws Exception { + String[] args = new String[1]; + assertEquals(new TeraSort().run(args), 2); + } + } diff --git a/hadoop-mapreduce-project/shellprofile.d/mapreduce b/hadoop-mapreduce-project/shellprofile.d/mapreduce.sh similarity index 100% rename from hadoop-mapreduce-project/shellprofile.d/mapreduce rename to hadoop-mapreduce-project/shellprofile.d/mapreduce.sh diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 2c0f03a5f12c6..6c95cf03c3ffd 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -833,6 +833,10 @@ org.jboss.netty netty + + jline + jline + @@ -902,9 +906,9 @@ - com.microsoft.windowsazure.storage - microsoft-windowsazure-storage-sdk - 0.6.0 + com.microsoft.azure + azure-storage + 2.0.0 diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index 1deba81fc90b8..2e098ef253bd4 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -68,6 +68,7 @@ + @@ -124,7 +125,7 @@ - + @@ -159,7 +160,8 @@ - + + diff --git a/hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/DfsTask.java b/hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/DfsTask.java index 78cb360c253ed..9d0b3a42c22f2 100644 --- a/hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/DfsTask.java +++ b/hadoop-tools/hadoop-ant/src/main/java/org/apache/hadoop/ant/DfsTask.java @@ -41,8 +41,8 @@ public class DfsTask extends Task { /** - * Default sink for {@link java.lang.System.out System.out} - * and {@link java.lang.System.err System.err}. + * Default sink for {@link java.lang.System#out} + * and {@link java.lang.System#err}. */ private static final OutputStream nullOut = new OutputStream() { public void write(int b) { /* ignore */ } @@ -171,7 +171,7 @@ protected int postCmd(int exit_code) { } /** - * Invoke {@link org.apache.hadoop.fs.FsShell#doMain FsShell.doMain} after a + * Invoke {@link org.apache.hadoop.fs.FsShell#main} after a * few cursory checks of the configuration. */ public void execute() throws BuildException { diff --git a/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java b/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java index 18cd972af4e9e..c5c42b1425ff1 100644 --- a/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java +++ b/hadoop-tools/hadoop-archives/src/main/java/org/apache/hadoop/tools/HadoopArchives.java @@ -101,7 +101,7 @@ public class HadoopArchives implements Tool { /** the desired replication degree; default is 10 **/ short repl = 10; - private static final String usage = "archive" + private static final String usage = "Usage: archive" + " -archiveName .har -p [-r ]" + "* " + "\n"; diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java index dda3cf683fa25..8bdfe9ac0773f 100644 --- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/S3FileSystem.java @@ -44,10 +44,9 @@ import org.apache.hadoop.util.Progressable; /** - *

      * A block-based {@link FileSystem} backed by * Amazon S3. - *

      + * * @see NativeS3FileSystem */ @InterfaceAudience.Public @@ -70,7 +69,6 @@ public S3FileSystem(FileSystemStore store) { /** * Return the protocol scheme for the FileSystem. - *

      * * @return s3 */ diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java index 1d4f67b837c84..3486dfbedfd72 100644 --- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java @@ -18,8 +18,12 @@ package org.apache.hadoop.fs.s3a; - public class Constants { + // s3 access key + public static final String ACCESS_KEY = "fs.s3a.access.key"; + + // s3 secret key + public static final String SECRET_KEY = "fs.s3a.secret.key"; // number of simultaneous connections to s3 public static final String MAXIMUM_CONNECTIONS = "fs.s3a.connection.maximum"; @@ -83,6 +87,14 @@ public class Constants { // comma separated list of directories public static final String BUFFER_DIR = "fs.s3a.buffer.dir"; + // should we upload directly from memory rather than using a file buffer + public static final String FAST_UPLOAD = "fs.s3a.fast.upload"; + public static final boolean DEFAULT_FAST_UPLOAD = false; + + //initial size of memory buffer for a fast upload + public static final String FAST_BUFFER_SIZE = "fs.s3a.fast.buffer.size"; + public static final int DEFAULT_FAST_BUFFER_SIZE = 1048576; //1MB + // private | public-read | public-read-write | authenticated-read | // log-delivery-write | bucket-owner-read | bucket-owner-full-control public static final String CANNED_ACL = "fs.s3a.acl.default"; diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java new file mode 100644 index 0000000000000..68195817ee415 --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java @@ -0,0 +1,413 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import com.amazonaws.AmazonClientException; +import com.amazonaws.AmazonServiceException; +import com.amazonaws.event.ProgressEvent; +import com.amazonaws.event.ProgressListener; +import com.amazonaws.services.s3.AmazonS3Client; +import com.amazonaws.services.s3.model.AbortMultipartUploadRequest; +import com.amazonaws.services.s3.model.CannedAccessControlList; +import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest; +import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; +import com.amazonaws.services.s3.model.ObjectMetadata; +import com.amazonaws.services.s3.model.PartETag; +import com.amazonaws.services.s3.model.PutObjectRequest; +import com.amazonaws.services.s3.model.PutObjectResult; +import com.amazonaws.services.s3.model.UploadPartRequest; +import com.google.common.util.concurrent.Futures; +import com.google.common.util.concurrent.ListenableFuture; +import com.google.common.util.concurrent.ListeningExecutorService; +import com.google.common.util.concurrent.MoreExecutors; +import org.apache.commons.lang.StringUtils; +import org.apache.hadoop.classification.InterfaceStability; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.util.Progressable; +import org.slf4j.Logger; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.util.ArrayList; +import java.util.List; + +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ThreadPoolExecutor; + + +/** + * Upload files/parts asap directly from a memory buffer (instead of buffering + * to a file). + *

      + * Uploads are managed low-level rather than through the AWS TransferManager. + * This allows for uploading each part of a multi-part upload as soon as + * the bytes are in memory, rather than waiting until the file is closed. + *

      + * Unstable: statistics and error handling might evolve + */ +@InterfaceStability.Unstable +public class S3AFastOutputStream extends OutputStream { + + private static final Logger LOG = S3AFileSystem.LOG; + private final String key; + private final String bucket; + private final AmazonS3Client client; + private final int partSize; + private final int multiPartThreshold; + private final S3AFileSystem fs; + private final CannedAccessControlList cannedACL; + private final FileSystem.Statistics statistics; + private final String serverSideEncryptionAlgorithm; + private final ProgressListener progressListener; + private final ListeningExecutorService executorService; + private MultiPartUpload multiPartUpload; + private boolean closed; + private ByteArrayOutputStream buffer; + private int bufferLimit; + + + /** + * Creates a fast OutputStream that uploads to S3 from memory. + * For MultiPartUploads, as soon as sufficient bytes have been written to + * the stream a part is uploaded immediately (by using the low-level + * multi-part upload API on the AmazonS3Client). + * + * @param client AmazonS3Client used for S3 calls + * @param fs S3AFilesystem + * @param bucket S3 bucket name + * @param key S3 key name + * @param progress report progress in order to prevent timeouts + * @param statistics track FileSystem.Statistics on the performed operations + * @param cannedACL used CannedAccessControlList + * @param serverSideEncryptionAlgorithm algorithm for server side encryption + * @param partSize size of a single part in a multi-part upload (except + * last part) + * @param multiPartThreshold files at least this size use multi-part upload + * @throws IOException + */ + public S3AFastOutputStream(AmazonS3Client client, S3AFileSystem fs, + String bucket, String key, Progressable progress, + FileSystem.Statistics statistics, CannedAccessControlList cannedACL, + String serverSideEncryptionAlgorithm, long partSize, + long multiPartThreshold, ThreadPoolExecutor threadPoolExecutor) + throws IOException { + this.bucket = bucket; + this.key = key; + this.client = client; + this.fs = fs; + this.cannedACL = cannedACL; + this.statistics = statistics; + this.serverSideEncryptionAlgorithm = serverSideEncryptionAlgorithm; + //Ensure limit as ByteArrayOutputStream size cannot exceed Integer.MAX_VALUE + if (partSize > Integer.MAX_VALUE) { + this.partSize = Integer.MAX_VALUE; + LOG.warn("s3a: MULTIPART_SIZE capped to ~2.14GB (maximum allowed size " + + "when using 'FAST_UPLOAD = true')"); + } else { + this.partSize = (int) partSize; + } + if (multiPartThreshold > Integer.MAX_VALUE) { + this.multiPartThreshold = Integer.MAX_VALUE; + LOG.warn("s3a: MIN_MULTIPART_THRESHOLD capped to ~2.14GB (maximum " + + "allowed size when using 'FAST_UPLOAD = true')"); + } else { + this.multiPartThreshold = (int) multiPartThreshold; + } + this.bufferLimit = this.multiPartThreshold; + this.closed = false; + int initialBufferSize = this.fs.getConf() + .getInt(Constants.FAST_BUFFER_SIZE, Constants.DEFAULT_FAST_BUFFER_SIZE); + if (initialBufferSize < 0) { + LOG.warn("s3a: FAST_BUFFER_SIZE should be a positive number. Using " + + "default value"); + initialBufferSize = Constants.DEFAULT_FAST_BUFFER_SIZE; + } else if (initialBufferSize > this.bufferLimit) { + LOG.warn("s3a: automatically adjusting FAST_BUFFER_SIZE to not " + + "exceed MIN_MULTIPART_THRESHOLD"); + initialBufferSize = this.bufferLimit; + } + this.buffer = new ByteArrayOutputStream(initialBufferSize); + this.executorService = MoreExecutors.listeningDecorator(threadPoolExecutor); + this.multiPartUpload = null; + this.progressListener = new ProgressableListener(progress); + if (LOG.isDebugEnabled()){ + LOG.debug("Initialized S3AFastOutputStream for bucket '{}' key '{}'", + bucket, key); + } + } + + /** + * Writes a byte to the memory buffer. If this causes the buffer to reach + * its limit, the actual upload is submitted to the threadpool. + * @param b the int of which the lowest byte is written + * @throws IOException + */ + @Override + public synchronized void write(int b) throws IOException { + buffer.write(b); + if (buffer.size() == bufferLimit) { + uploadBuffer(); + } + } + + /** + * Writes a range of bytes from to the memory buffer. If this causes the + * buffer to reach its limit, the actual upload is submitted to the + * threadpool and the remainder of the array is written to memory + * (recursively). + * @param b byte array containing + * @param off offset in array where to start + * @param len number of bytes to be written + * @throws IOException + */ + @Override + public synchronized void write(byte b[], int off, int len) + throws IOException { + if (b == null) { + throw new NullPointerException(); + } else if ((off < 0) || (off > b.length) || (len < 0) || + ((off + len) > b.length) || ((off + len) < 0)) { + throw new IndexOutOfBoundsException(); + } else if (len == 0) { + return; + } + if (buffer.size() + len < bufferLimit) { + buffer.write(b, off, len); + } else { + int firstPart = bufferLimit - buffer.size(); + buffer.write(b, off, firstPart); + uploadBuffer(); + this.write(b, off + firstPart, len - firstPart); + } + } + + private synchronized void uploadBuffer() throws IOException { + if (multiPartUpload == null) { + multiPartUpload = initiateMultiPartUpload(); + /* Upload the existing buffer if it exceeds partSize. This possibly + requires multiple parts! */ + final byte[] allBytes = buffer.toByteArray(); + buffer = null; //earlier gc? + if (LOG.isDebugEnabled()) { + LOG.debug("Total length of initial buffer: {}", allBytes.length); + } + int processedPos = 0; + while ((multiPartThreshold - processedPos) >= partSize) { + if (LOG.isDebugEnabled()) { + LOG.debug("Initial buffer: processing from byte {} to byte {}", + processedPos, (processedPos + partSize - 1)); + } + multiPartUpload.uploadPartAsync(new ByteArrayInputStream(allBytes, + processedPos, partSize), partSize); + processedPos += partSize; + } + //resize and reset stream + bufferLimit = partSize; + buffer = new ByteArrayOutputStream(bufferLimit); + buffer.write(allBytes, processedPos, multiPartThreshold - processedPos); + } else { + //upload next part + multiPartUpload.uploadPartAsync(new ByteArrayInputStream(buffer + .toByteArray()), partSize); + buffer.reset(); + } + } + + + @Override + public synchronized void close() throws IOException { + if (closed) { + return; + } + closed = true; + try { + if (multiPartUpload == null) { + putObject(); + } else { + if (buffer.size() > 0) { + //send last part + multiPartUpload.uploadPartAsync(new ByteArrayInputStream(buffer + .toByteArray()), buffer.size()); + } + final List partETags = multiPartUpload + .waitForAllPartUploads(); + multiPartUpload.complete(partETags); + } + statistics.incrementWriteOps(1); + // This will delete unnecessary fake parent directories + fs.finishedWrite(key); + if (LOG.isDebugEnabled()) { + LOG.debug("Upload complete for bucket '{}' key '{}'", bucket, key); + } + } finally { + buffer = null; + super.close(); + } + } + + private ObjectMetadata createDefaultMetadata() { + ObjectMetadata om = new ObjectMetadata(); + if (StringUtils.isNotBlank(serverSideEncryptionAlgorithm)) { + om.setServerSideEncryption(serverSideEncryptionAlgorithm); + } + return om; + } + + private MultiPartUpload initiateMultiPartUpload() throws IOException { + final ObjectMetadata om = createDefaultMetadata(); + final InitiateMultipartUploadRequest initiateMPURequest = + new InitiateMultipartUploadRequest(bucket, key, om); + initiateMPURequest.setCannedACL(cannedACL); + try { + return new MultiPartUpload( + client.initiateMultipartUpload(initiateMPURequest).getUploadId()); + } catch (AmazonServiceException ase) { + throw new IOException("Unable to initiate MultiPartUpload (server side)" + + ": " + ase, ase); + } catch (AmazonClientException ace) { + throw new IOException("Unable to initiate MultiPartUpload (client side)" + + ": " + ace, ace); + } + } + + private void putObject() throws IOException { + if (LOG.isDebugEnabled()) { + LOG.debug("Executing regular upload for bucket '{}' key '{}'", bucket, + key); + } + final ObjectMetadata om = createDefaultMetadata(); + om.setContentLength(buffer.size()); + final PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, + new ByteArrayInputStream(buffer.toByteArray()), om); + putObjectRequest.setCannedAcl(cannedACL); + putObjectRequest.setGeneralProgressListener(progressListener); + ListenableFuture putObjectResult = + executorService.submit(new Callable() { + @Override + public PutObjectResult call() throws Exception { + return client.putObject(putObjectRequest); + } + }); + //wait for completion + try { + putObjectResult.get(); + } catch (InterruptedException ie) { + LOG.warn("Interrupted object upload:" + ie, ie); + Thread.currentThread().interrupt(); + } catch (ExecutionException ee) { + throw new IOException("Regular upload failed", ee.getCause()); + } + } + + private class MultiPartUpload { + private final String uploadId; + private final List> partETagsFutures; + + public MultiPartUpload(String uploadId) { + this.uploadId = uploadId; + this.partETagsFutures = new ArrayList>(); + if (LOG.isDebugEnabled()) { + LOG.debug("Initiated multi-part upload for bucket '{}' key '{}' with " + + "id '{}'", bucket, key, uploadId); + } + } + + public void uploadPartAsync(ByteArrayInputStream inputStream, + int partSize) { + final int currentPartNumber = partETagsFutures.size() + 1; + final UploadPartRequest request = + new UploadPartRequest().withBucketName(bucket).withKey(key) + .withUploadId(uploadId).withInputStream(inputStream) + .withPartNumber(currentPartNumber).withPartSize(partSize); + request.setGeneralProgressListener(progressListener); + ListenableFuture partETagFuture = + executorService.submit(new Callable() { + @Override + public PartETag call() throws Exception { + if (LOG.isDebugEnabled()) { + LOG.debug("Uploading part {} for id '{}'", currentPartNumber, + uploadId); + } + return client.uploadPart(request).getPartETag(); + } + }); + partETagsFutures.add(partETagFuture); + } + + public List waitForAllPartUploads() throws IOException { + try { + return Futures.allAsList(partETagsFutures).get(); + } catch (InterruptedException ie) { + LOG.warn("Interrupted partUpload:" + ie, ie); + Thread.currentThread().interrupt(); + } catch (ExecutionException ee) { + //there is no way of recovering so abort + //cancel all partUploads + for (ListenableFuture future : partETagsFutures) { + future.cancel(true); + } + //abort multipartupload + this.abort(); + throw new IOException("Part upload failed in multi-part upload with " + + "id '" +uploadId + "':" + ee, ee); + } + //should not happen? + return null; + } + + public void complete(List partETags) { + if (LOG.isDebugEnabled()) { + LOG.debug("Completing multi-part upload for key '{}', id '{}'", key, + uploadId); + } + final CompleteMultipartUploadRequest completeRequest = + new CompleteMultipartUploadRequest(bucket, key, uploadId, partETags); + client.completeMultipartUpload(completeRequest); + + } + + public void abort() { + LOG.warn("Aborting multi-part upload with id '{}'", uploadId); + try { + client.abortMultipartUpload(new AbortMultipartUploadRequest(bucket, + key, uploadId)); + } catch (Exception e2) { + LOG.warn("Unable to abort multipart upload, you may need to purge " + + "uploaded parts: " + e2, e2); + } + } + } + + private static class ProgressableListener implements ProgressListener { + private final Progressable progress; + + public ProgressableListener(Progressable progress) { + this.progress = progress; + } + + public void progressChanged(ProgressEvent progressEvent) { + if (progress != null) { + progress.progress(); + } + } + } +} diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java index 2373e7ef02f82..91a606cf1f443 100644 --- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java @@ -32,8 +32,6 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; -import org.apache.hadoop.fs.s3.S3Credentials; - import com.amazonaws.AmazonClientException; import com.amazonaws.AmazonServiceException; import com.amazonaws.ClientConfiguration; @@ -88,7 +86,8 @@ public class S3AFileSystem extends FileSystem { private int maxKeys; private long partSize; private TransferManager transfers; - private int partSizeThreshold; + private ThreadPoolExecutor threadPoolExecutor; + private int multiPartThreshold; public static final Logger LOG = LoggerFactory.getLogger(S3AFileSystem.class); private CannedAccessControlList cannedACL; private String serverSideEncryptionAlgorithm; @@ -158,12 +157,22 @@ public void initialize(URI name, Configuration conf) throws IOException { this.getWorkingDirectory()); // Try to get our credentials or just connect anonymously - S3Credentials s3Credentials = new S3Credentials(); - s3Credentials.initialize(name, conf); + String accessKey = conf.get(ACCESS_KEY, null); + String secretKey = conf.get(SECRET_KEY, null); + + String userInfo = name.getUserInfo(); + if (userInfo != null) { + int index = userInfo.indexOf(':'); + if (index != -1) { + accessKey = userInfo.substring(0, index); + secretKey = userInfo.substring(index + 1); + } else { + accessKey = userInfo; + } + } AWSCredentialsProviderChain credentials = new AWSCredentialsProviderChain( - new BasicAWSCredentialsProvider(s3Credentials.getAccessKey(), - s3Credentials.getSecretAccessKey()), + new BasicAWSCredentialsProvider(accessKey, secretKey), new InstanceProfileCredentialsProvider(), new AnonymousAWSCredentialsProvider() ); @@ -237,7 +246,7 @@ public void initialize(URI name, Configuration conf) throws IOException { maxKeys = conf.getInt(MAX_PAGING_KEYS, DEFAULT_MAX_PAGING_KEYS); partSize = conf.getLong(MULTIPART_SIZE, DEFAULT_MULTIPART_SIZE); - partSizeThreshold = conf.getInt(MIN_MULTIPART_THRESHOLD, + multiPartThreshold = conf.getInt(MIN_MULTIPART_THRESHOLD, DEFAULT_MIN_MULTIPART_THRESHOLD); if (partSize < 5 * 1024 * 1024) { @@ -245,9 +254,9 @@ public void initialize(URI name, Configuration conf) throws IOException { partSize = 5 * 1024 * 1024; } - if (partSizeThreshold < 5 * 1024 * 1024) { + if (multiPartThreshold < 5 * 1024 * 1024) { LOG.error(MIN_MULTIPART_THRESHOLD + " must be at least 5 MB"); - partSizeThreshold = 5 * 1024 * 1024; + multiPartThreshold = 5 * 1024 * 1024; } int maxThreads = conf.getInt(MAX_THREADS, DEFAULT_MAX_THREADS); @@ -262,20 +271,20 @@ public void initialize(URI name, Configuration conf) throws IOException { LinkedBlockingQueue workQueue = new LinkedBlockingQueue<>(maxThreads * conf.getInt(MAX_TOTAL_TASKS, DEFAULT_MAX_TOTAL_TASKS)); - ThreadPoolExecutor tpe = new ThreadPoolExecutor( + threadPoolExecutor = new ThreadPoolExecutor( coreThreads, maxThreads, keepAliveTime, TimeUnit.SECONDS, workQueue, newDaemonThreadFactory("s3a-transfer-shared-")); - tpe.allowCoreThreadTimeOut(true); + threadPoolExecutor.allowCoreThreadTimeOut(true); TransferManagerConfiguration transferConfiguration = new TransferManagerConfiguration(); transferConfiguration.setMinimumUploadPartSize(partSize); - transferConfiguration.setMultipartUploadThreshold(partSizeThreshold); + transferConfiguration.setMultipartUploadThreshold(multiPartThreshold); - transfers = new TransferManager(s3, tpe); + transfers = new TransferManager(s3, threadPoolExecutor); transfers.setConfiguration(transferConfiguration); String cannedACLName = conf.get(CANNED_ACL, DEFAULT_CANNED_ACL); @@ -391,7 +400,12 @@ public FSDataOutputStream create(Path f, FsPermission permission, boolean overwr if (!overwrite && exists(f)) { throw new FileAlreadyExistsException(f + " already exists"); } - + if (getConf().getBoolean(FAST_UPLOAD, DEFAULT_FAST_UPLOAD)) { + return new FSDataOutputStream(new S3AFastOutputStream(s3, this, bucket, + key, progress, statistics, cannedACL, + serverSideEncryptionAlgorithm, partSize, (long)multiPartThreshold, + threadPoolExecutor), statistics); + } // We pass null to FSDataOutputStream so it won't count writes that are being buffered to a file return new FSDataOutputStream(new S3AOutputStream(getConf(), transfers, this, bucket, key, progress, cannedACL, statistics, diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java index acc5500d7f5bb..a2f9805ffaa73 100644 --- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java +++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java @@ -62,24 +62,29 @@ import org.slf4j.LoggerFactory; /** - *

      * A {@link FileSystem} for reading and writing files stored on * Amazon S3. * Unlike {@link org.apache.hadoop.fs.s3.S3FileSystem} this implementation * stores files on S3 in their * native form so they can be read by other S3 tools. - * + *

      * A note about directories. S3 of course has no "native" support for them. * The idiom we choose then is: for any directory created by this class, * we use an empty object "#{dirpath}_$folder$" as a marker. * Further, to interoperate with other S3 tools, we also accept the following: - * - an object "#{dirpath}/' denoting a directory marker - * - if there exists any objects with the prefix "#{dirpath}/", then the - * directory is said to exist - * - if both a file with the name of a directory and a marker for that - * directory exists, then the *file masks the directory*, and the directory - * is never returned. - *

      + *
        + *
      • an object "#{dirpath}/' denoting a directory marker
        • + *
      • + * if there exists any objects with the prefix "#{dirpath}/", then the + * directory is said to exist + *
        • + *
      • + * if both a file with the name of a directory and a marker for that + * directory exists, then the *file masks the directory*, and the directory + * is never returned. + *
        • + *
      + * * @see org.apache.hadoop.fs.s3.S3FileSystem */ @InterfaceAudience.Public @@ -308,7 +313,6 @@ public NativeS3FileSystem(NativeFileSystemStore store) { /** * Return the protocol scheme for the FileSystem. - *

      * * @return s3n */ diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md index 8e80b92593cb3..e0389c05caa4d 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md @@ -141,12 +141,12 @@ If you do any of these: change your credentials immediately! ### Authentication properties - fs.s3a.awsAccessKeyId + fs.s3a.access.key AWS access key ID. Omit for Role-based authentication. - fs.s3a.awsSecretAccessKey + fs.s3a.secret.key AWS secret key. Omit for Role-based authentication. @@ -213,13 +213,13 @@ If you do any of these: change your credentials immediately! fs.s3a.connection.establish.timeout 5000 - Socket connection setup timeout in seconds. + Socket connection setup timeout in milliseconds. fs.s3a.connection.timeout 50000 - Socket connection timeout in seconds. + Socket connection timeout in milliseconds. @@ -292,7 +292,7 @@ If you do any of these: change your credentials immediately! fs.s3a.buffer.dir ${hadoop.tmp.dir}/s3a Comma separated list of directories that will be used to buffer file - uploads to. + uploads to. No effect if fs.s3a.fast.upload is true. @@ -301,6 +301,40 @@ If you do any of these: change your credentials immediately! The implementation class of the S3A Filesystem +### S3AFastOutputStream + **Warning: NEW in hadoop 2.7. UNSTABLE, EXPERIMENTAL: use at own risk** + + + fs.s3a.fast.upload + false + Upload directly from memory instead of buffering to + disk first. Memory usage and parallelism can be controlled as up to + fs.s3a.multipart.size memory is consumed for each (part)upload actively + uploading (fs.s3a.threads.max) or queueing (fs.s3a.max.total.tasks) + + + + fs.s3a.fast.buffer.size + 1048576 + Size (in bytes) of initial memory buffer allocated for an + upload. No effect if fs.s3a.fast.upload is false. + + +Writes are buffered in memory instead of to a file on local disk. This +removes the throughput bottleneck of the local disk write and read cycle +before starting the actual upload. Furthermore, it allows handling files that +are larger than the remaining local disk space. + +However, non-trivial memory tuning is needed for optimal results and careless +settings could cause memory overflow. Up to `fs.s3a.threads.max` parallel +(part)uploads are active. Furthermore, up to `fs.s3a.max.total.tasks` +additional part(uploads) can be waiting (and thus memory buffers are created). +The memory buffer is uploaded as a single upload if it is not larger than +`fs.s3a.multipart.threshold`. Else, a multi-part upload is initiatated and +parts of size `fs.s3a.multipart.size` are used to protect against overflowing +the available memory. These settings should be tuned to the envisioned +workflow (some large files, many small ones, ...) and the physical +limitations of the machine and cluster (memory, network bandwidth). ## Testing the S3 filesystem clients @@ -334,7 +368,7 @@ each filesystem for its testing. The contents of each bucket will be destroyed during the test process: do not use the bucket for any purpose other than testing. Furthermore, for s3a, all in-progress multi-part uploads to the bucket will be aborted at the -start of a test (by forcing fs.s3a.multipart.purge=true) to clean up the +start of a test (by forcing `fs.s3a.multipart.purge=true`) to clean up the temporary state of previously failed tests. Example: @@ -377,13 +411,13 @@ Example: - fs.s3a.awsAccessKeyId + fs.s3a.access.key AWS access key ID. Omit for Role-based authentication. - DONOTPCOMMITTHISKEYTOSCM + DONOTCOMMITTHISKEYTOSCM - fs.s3a.awsSecretAccessKey + fs.s3a.secret.key AWS secret key. Omit for Role-based authentication. DONOTEVERSHARETHISSECRETKEY! @@ -392,14 +426,14 @@ Example: ## File `contract-test-options.xml` The file `hadoop-tools/hadoop-aws/src/test/resources/contract-test-options.xml` -must be created and configured for the test fileystems. +must be created and configured for the test filesystems. If a specific file `fs.contract.test.fs.*` test path is not defined for any of the filesystems, those tests will be skipped. The standard S3 authentication details must also be provided. This can be through copy-and-paste of the `auth-keys.xml` credentials, or it can be -through direct XInclude inclustion. +through direct XInclude inclusion. #### s3:// diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AFastOutputStream.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AFastOutputStream.java new file mode 100644 index 0000000000000..e507cf6cf7b34 --- /dev/null +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AFastOutputStream.java @@ -0,0 +1,74 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.s3a; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.contract.ContractTestUtils; +import org.junit.After; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.Timeout; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; + +import java.io.IOException; + +/** + * Tests regular and multi-part upload functionality for S3AFastOutputStream. + * File sizes are kept small to reduce test duration on slow connections + */ +public class TestS3AFastOutputStream { + private FileSystem fs; + + + @Rule + public Timeout testTimeout = new Timeout(30 * 60 * 1000); + + @Before + public void setUp() throws Exception { + Configuration conf = new Configuration(); + conf.setLong(Constants.MIN_MULTIPART_THRESHOLD, 5 * 1024 * 1024); + conf.setInt(Constants.MULTIPART_SIZE, 5 * 1024 * 1024); + conf.setBoolean(Constants.FAST_UPLOAD, true); + fs = S3ATestUtils.createTestFileSystem(conf); + } + + @After + public void tearDown() throws Exception { + if (fs != null) { + fs.delete(getTestPath(), true); + } + } + + protected Path getTestPath() { + return new Path("/tests3a"); + } + + @Test + public void testRegularUpload() throws IOException { + ContractTestUtils.createAndVerifyFile(fs, getTestPath(), 1024 * 1024); + } + + @Test + public void testMultiPartUpload() throws IOException { + ContractTestUtils.createAndVerifyFile(fs, getTestPath(), 6 * 1024 * + 1024); + } +} diff --git a/hadoop-tools/hadoop-azure/pom.xml b/hadoop-tools/hadoop-azure/pom.xml index d39dd769e1af0..e9b3af775f2af 100644 --- a/hadoop-tools/hadoop-azure/pom.xml +++ b/hadoop-tools/hadoop-azure/pom.xml @@ -140,12 +140,13 @@ httpclient compile - + - com.microsoft.windowsazure.storage - microsoft-windowsazure-storage-sdk + com.microsoft.azure + azure-storage compile + com.google.guava diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java index c0c03b3fce8f5..5dc096397d244 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java @@ -65,23 +65,23 @@ import org.mortbay.util.ajax.JSON; import com.google.common.annotations.VisibleForTesting; -import com.microsoft.windowsazure.storage.CloudStorageAccount; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RetryExponentialRetry; -import com.microsoft.windowsazure.storage.RetryNoRetry; -import com.microsoft.windowsazure.storage.StorageCredentials; -import com.microsoft.windowsazure.storage.StorageCredentialsAccountAndKey; -import com.microsoft.windowsazure.storage.StorageCredentialsSharedAccessSignature; -import com.microsoft.windowsazure.storage.StorageErrorCode; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.BlobListingDetails; -import com.microsoft.windowsazure.storage.blob.BlobProperties; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; -import com.microsoft.windowsazure.storage.blob.CloudBlob; -import com.microsoft.windowsazure.storage.blob.CopyStatus; -import com.microsoft.windowsazure.storage.blob.DeleteSnapshotsOption; -import com.microsoft.windowsazure.storage.blob.ListBlobItem; -import com.microsoft.windowsazure.storage.core.Utility; +import com.microsoft.azure.storage.CloudStorageAccount; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RetryExponentialRetry; +import com.microsoft.azure.storage.RetryNoRetry; +import com.microsoft.azure.storage.StorageCredentials; +import com.microsoft.azure.storage.StorageCredentialsAccountAndKey; +import com.microsoft.azure.storage.StorageCredentialsSharedAccessSignature; +import com.microsoft.azure.storage.StorageErrorCode; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.BlobListingDetails; +import com.microsoft.azure.storage.blob.BlobProperties; +import com.microsoft.azure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.blob.CopyStatus; +import com.microsoft.azure.storage.blob.DeleteSnapshotsOption; +import com.microsoft.azure.storage.blob.ListBlobItem; +import com.microsoft.azure.storage.core.Utility; /** * Core implementation of Windows Azure Filesystem for Hadoop. @@ -134,6 +134,15 @@ public class AzureNativeFileSystemStore implements NativeFileSystemStore { private static final String KEY_MAX_BACKOFF_INTERVAL = "fs.azure.io.retry.max.backoff.interval"; private static final String KEY_BACKOFF_INTERVAL = "fs.azure.io.retry.backoff.interval"; private static final String KEY_MAX_IO_RETRIES = "fs.azure.io.retry.max.retries"; + + private static final String KEY_COPYBLOB_MIN_BACKOFF_INTERVAL = + "fs.azure.io.copyblob.retry.min.backoff.interval"; + private static final String KEY_COPYBLOB_MAX_BACKOFF_INTERVAL = + "fs.azure.io.copyblob.retry.max.backoff.interval"; + private static final String KEY_COPYBLOB_BACKOFF_INTERVAL = + "fs.azure.io.copyblob.retry.backoff.interval"; + private static final String KEY_COPYBLOB_MAX_IO_RETRIES = + "fs.azure.io.copyblob.retry.max.retries"; private static final String KEY_SELF_THROTTLE_ENABLE = "fs.azure.selfthrottling.enable"; private static final String KEY_SELF_THROTTLE_READ_FACTOR = "fs.azure.selfthrottling.read.factor"; @@ -199,6 +208,11 @@ public class AzureNativeFileSystemStore implements NativeFileSystemStore { private static final int DEFAULT_MAX_BACKOFF_INTERVAL = 30 * 1000; // 30s private static final int DEFAULT_BACKOFF_INTERVAL = 1 * 1000; // 1s private static final int DEFAULT_MAX_RETRY_ATTEMPTS = 15; + + private static final int DEFAULT_COPYBLOB_MIN_BACKOFF_INTERVAL = 3 * 1000; + private static final int DEFAULT_COPYBLOB_MAX_BACKOFF_INTERVAL = 90 * 1000; + private static final int DEFAULT_COPYBLOB_BACKOFF_INTERVAL = 30 * 1000; + private static final int DEFAULT_COPYBLOB_MAX_RETRY_ATTEMPTS = 15; // Self-throttling defaults. Allowed range = (0,1.0] // Value of 1.0 means no self-throttling. @@ -586,8 +600,6 @@ private String getHTTPScheme() { * Azure. * * @throws AzureException - * @throws ConfigurationException - * */ private void configureAzureStorageSession() throws AzureException { @@ -691,7 +703,7 @@ private void configureAzureStorageSession() throws AzureException { * raised on errors communicating with Azure storage. * @throws IOException * raised on errors performing I/O or setting up the session. - * @throws URISyntaxExceptions + * @throws URISyntaxException * raised on creating mal-formed URI's. */ private void connectUsingAnonymousCredentials(final URI uri) @@ -1022,7 +1034,6 @@ private Set getDirectorySet(final String configVar) /** * Checks if the given key in Azure Storage should be stored as a page * blob instead of block blob. - * @throws URISyntaxException */ public boolean isPageBlobKey(String key) { return isKeyForDirectorySet(key, pageBlobDirs); @@ -1741,7 +1752,7 @@ private String normalizeKey(CloudBlobWrapper blob) { * the path and returns a path relative to the root directory of the * container. * - * @param blob + * @param directory * - adjust the key to this directory to a path relative to the root * directory * @@ -2128,14 +2139,10 @@ private PartialListing list(String prefix, String delimiter, * uses a in-order first traversal of blob directory structures to maintain * the sorted order of the blob names. * - * @param dir - * -- Azure blob directory - * - * @param list - * -- a list of file metadata objects for each non-directory blob. - * - * @param maxListingLength - * -- maximum length of the built up list. + * @param aCloudBlobDirectory Azure blob directory + * @param aFileMetadataList a list of file metadata objects for each + * non-directory blob. + * @param maxListingCount maximum length of the built up list. */ private void buildUpList(CloudBlobDirectoryWrapper aCloudBlobDirectory, ArrayList aFileMetadataList, final int maxListingCount, @@ -2306,8 +2313,7 @@ private long getDataLength(CloudBlobWrapper blob, BlobProperties properties) * swallow the error since what most probably happened is that * the first operation succeeded on the server. * @param blob The blob to delete. - * @param leaseID A string identifying the lease, or null if no - * lease is to be used. + * @param lease Azure blob lease, or null if no lease is to be used. * @throws StorageException */ private void safeDelete(CloudBlobWrapper blob, SelfRenewingLease lease) throws StorageException { @@ -2435,11 +2441,46 @@ public void rename(String srcKey, String dstKey, boolean acquireLease, // Rename the source blob to the destination blob by copying it to // the destination blob then deleting it. // - dstBlob.startCopyFromBlob(srcUri, getInstrumentedContext()); + // Copy blob operation in Azure storage is very costly. It will be highly + // likely throttled during Azure storage gc. Short term fix will be using + // a more intensive exponential retry policy when the cluster is getting + // throttled. + try { + dstBlob.startCopyFromBlob(srcUri, null, getInstrumentedContext()); + } catch (StorageException se) { + if (se.getErrorCode().equals( + StorageErrorCode.SERVER_BUSY.toString())) { + int copyBlobMinBackoff = sessionConfiguration.getInt( + KEY_COPYBLOB_MIN_BACKOFF_INTERVAL, + DEFAULT_COPYBLOB_MIN_BACKOFF_INTERVAL); + + int copyBlobMaxBackoff = sessionConfiguration.getInt( + KEY_COPYBLOB_MAX_BACKOFF_INTERVAL, + DEFAULT_COPYBLOB_MAX_BACKOFF_INTERVAL); + + int copyBlobDeltaBackoff = sessionConfiguration.getInt( + KEY_COPYBLOB_BACKOFF_INTERVAL, + DEFAULT_COPYBLOB_BACKOFF_INTERVAL); + + int copyBlobMaxRetries = sessionConfiguration.getInt( + KEY_COPYBLOB_MAX_IO_RETRIES, + DEFAULT_COPYBLOB_MAX_RETRY_ATTEMPTS); + + BlobRequestOptions options = new BlobRequestOptions(); + options.setRetryPolicyFactory(new RetryExponentialRetry( + copyBlobMinBackoff, copyBlobDeltaBackoff, copyBlobMaxBackoff, + copyBlobMaxRetries)); + dstBlob.startCopyFromBlob(srcUri, options, getInstrumentedContext()); + } else { + throw se; + } + } waitForCopyToComplete(dstBlob, getInstrumentedContext()); - safeDelete(srcBlob, lease); - } catch (Exception e) { + } catch (StorageException e) { + // Re-throw exception as an Azure storage exception. + throw new AzureException(e); + } catch (URISyntaxException e) { // Re-throw exception as an Azure storage exception. throw new AzureException(e); } @@ -2543,7 +2584,8 @@ public void updateFolderLastModifiedTime(String key, Date lastModified, try { checkContainer(ContainerAccessType.ReadThenWrite); CloudBlobWrapper blob = getBlobReference(key); - blob.getProperties().setLastModified(lastModified); + //setLastModified function is not available in 2.0.0 version. blob.uploadProperties automatically updates last modified + //timestamp to current time blob.uploadProperties(getInstrumentedContext(), folderLease); } catch (Exception e) { diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java index 0248b85a283a5..623645a414d8a 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java @@ -74,19 +74,17 @@ import org.codehaus.jackson.map.ObjectMapper; import com.google.common.annotations.VisibleForTesting; -import com.microsoft.windowsazure.storage.AccessCondition; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.CloudBlob; -import com.microsoft.windowsazure.storage.core.*; +import com.microsoft.azure.storage.AccessCondition; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.core.*; /** - *

      * A {@link FileSystem} for reading and writing files stored on Windows Azure. This implementation is * blob-based and stores files on Azure in their native form so they can be read * by other Azure tools. - *

      */ @InterfaceAudience.Public @InterfaceStability.Stable @@ -218,9 +216,11 @@ public SelfRenewingLease getFolderLease() { } /** - * Write to disk the information needed to redo folder rename, in JSON format. - * The file name will be wasb:///folderName-RenamePending.json + * Write to disk the information needed to redo folder rename, + * in JSON format. The file name will be + * {@code wasb:///folderName-RenamePending.json} * The file format will be: + * 
      {@code
      * {
      *   FormatVersion: "1.0",
      *   OperationTime: "",
@@ -239,7 +239,7 @@ public SelfRenewingLease getFolderLease() {
      *    "innerFile",
      *    "innerFile2"
      *  ]
-     * }
+     * } }
      * @throws IOException */ public void writeFile(FileSystem fs) throws IOException { @@ -913,9 +913,6 @@ public void setEncodedKey(String anEncodedKey) { * The create also includes the name of the original key value which is * stored in the m_key member variable. This method should only be called * when the stream is closed. - * - * @param anEncodedKey - * Encoding of the original key stored in m_key member. */ private void restoreKey() throws IOException { store.rename(getEncodedKey(), getKey()); @@ -1796,7 +1793,7 @@ private static enum UMaskApplyMode { * * @param permission * The permission to mask. - * @param applyDefaultUmask + * @param applyMode * Whether to also apply the default umask. * @return The masked persmission. */ @@ -2409,7 +2406,6 @@ protected void finalize() throws Throwable { * recover the original key. * * @param aKey - * @param numBuckets * @return Encoded version of the original key. */ private static String encodeKey(String aKey) { diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobFormatHelpers.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobFormatHelpers.java index ad11aacbcf5ac..9a316a51bd26e 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobFormatHelpers.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobFormatHelpers.java @@ -20,7 +20,7 @@ import java.nio.ByteBuffer; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.BlobRequestOptions; /** * Constants and helper methods for ASV's custom data format in page blobs. diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java index 62b47ee4f4931..468ac65d76b2d 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java @@ -33,10 +33,10 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.azure.StorageInterface.CloudPageBlobWrapper; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; -import com.microsoft.windowsazure.storage.blob.PageRange; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.PageRange; /** * An input stream that reads file data from a page blob stored diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java index 4d1d5c8e6aff8..2b8846c7f73ec 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java @@ -44,10 +44,10 @@ import org.apache.hadoop.conf.Configuration; import com.google.common.annotations.VisibleForTesting; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; -import com.microsoft.windowsazure.storage.blob.CloudPageBlob; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.CloudPageBlob; /** diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java index bda6006d60225..06f32ce361879 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfRenewingLease.java @@ -22,9 +22,9 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.azure.StorageInterface.CloudBlobWrapper; -import com.microsoft.windowsazure.storage.AccessCondition; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.AccessCondition; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.CloudBlob; import java.util.concurrent.atomic.AtomicInteger; diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java index d18a14406edf0..a9e3df907f956 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SelfThrottlingIntercept.java @@ -25,11 +25,11 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RequestResult; -import com.microsoft.windowsazure.storage.ResponseReceivedEvent; -import com.microsoft.windowsazure.storage.SendingRequestEvent; -import com.microsoft.windowsazure.storage.StorageEvent; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RequestResult; +import com.microsoft.azure.storage.ResponseReceivedEvent; +import com.microsoft.azure.storage.SendingRequestEvent; +import com.microsoft.azure.storage.StorageEvent; /* * Self throttling is implemented by hooking into send & response callbacks diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java index 18f173ebbf7b7..4d564d5338e86 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java @@ -25,12 +25,13 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience; -import com.microsoft.windowsazure.storage.Constants.HeaderConstants; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.SendingRequestEvent; -import com.microsoft.windowsazure.storage.StorageCredentials; -import com.microsoft.windowsazure.storage.StorageEvent; -import com.microsoft.windowsazure.storage.StorageException; +import com.microsoft.azure.storage.Constants.HeaderConstants; +import com.microsoft.azure.storage.core.StorageCredentialsHelper; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.SendingRequestEvent; +import com.microsoft.azure.storage.StorageCredentials; +import com.microsoft.azure.storage.StorageEvent; +import com.microsoft.azure.storage.StorageException; /** * Manages the lifetime of binding on the operation contexts to intercept send @@ -146,7 +147,8 @@ && isOutOfBandIoAllowed()) { try { // Sign the request. GET's have no payload so the content length is // zero. - getCredentials().signBlobAndQueueRequest(urlConnection, -1L, getOperationContext()); + StorageCredentialsHelper.signBlobAndQueueRequest(getCredentials(), + urlConnection, -1L, getOperationContext()); } catch (InvalidKeyException e) { // Log invalid key exception to track signing error before the send // fails. diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java index 8d0229d8cf725..e89151d709893 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterface.java @@ -29,18 +29,18 @@ import org.apache.hadoop.classification.InterfaceAudience; -import com.microsoft.windowsazure.storage.CloudStorageAccount; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RetryPolicyFactory; -import com.microsoft.windowsazure.storage.StorageCredentials; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.BlobListingDetails; -import com.microsoft.windowsazure.storage.blob.BlobProperties; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; -import com.microsoft.windowsazure.storage.blob.CloudBlob; -import com.microsoft.windowsazure.storage.blob.CopyState; -import com.microsoft.windowsazure.storage.blob.ListBlobItem; -import com.microsoft.windowsazure.storage.blob.PageRange; +import com.microsoft.azure.storage.CloudStorageAccount; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RetryPolicyFactory; +import com.microsoft.azure.storage.StorageCredentials; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.BlobListingDetails; +import com.microsoft.azure.storage.blob.BlobProperties; +import com.microsoft.azure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.blob.CopyState; +import com.microsoft.azure.storage.blob.ListBlobItem; +import com.microsoft.azure.storage.blob.PageRange; /** * This is a very thin layer over the methods exposed by the Windows Azure @@ -383,6 +383,10 @@ public interface CloudBlobWrapper extends ListBlobItem { * * @param source * A java.net.URI The URI of a source blob. + * @param options + * A {@link BlobRequestOptions} object that specifies any additional options for the request. Specifying + * null will use the default request options from the associated service client ( + * {@link CloudBlobClient}). * @param opContext * An {@link OperationContext} object that represents the context for the current operation. This object * is used to track requests to the storage service, and to provide additional runtime information about @@ -394,7 +398,7 @@ public interface CloudBlobWrapper extends ListBlobItem { * */ public abstract void startCopyFromBlob(URI source, - OperationContext opContext) + BlobRequestOptions options, OperationContext opContext) throws StorageException, URISyntaxException; /** diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java index e44823c82f8ed..90d4d8838ad50 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java @@ -30,26 +30,26 @@ import org.apache.hadoop.classification.InterfaceAudience; -import com.microsoft.windowsazure.storage.AccessCondition; -import com.microsoft.windowsazure.storage.CloudStorageAccount; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RetryPolicyFactory; -import com.microsoft.windowsazure.storage.StorageCredentials; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.StorageUri; -import com.microsoft.windowsazure.storage.blob.BlobListingDetails; -import com.microsoft.windowsazure.storage.blob.BlobProperties; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; -import com.microsoft.windowsazure.storage.blob.CloudBlob; -import com.microsoft.windowsazure.storage.blob.CloudBlobClient; -import com.microsoft.windowsazure.storage.blob.CloudBlobContainer; -import com.microsoft.windowsazure.storage.blob.CloudBlobDirectory; -import com.microsoft.windowsazure.storage.blob.CloudBlockBlob; -import com.microsoft.windowsazure.storage.blob.CloudPageBlob; -import com.microsoft.windowsazure.storage.blob.CopyState; -import com.microsoft.windowsazure.storage.blob.DeleteSnapshotsOption; -import com.microsoft.windowsazure.storage.blob.ListBlobItem; -import com.microsoft.windowsazure.storage.blob.PageRange; +import com.microsoft.azure.storage.AccessCondition; +import com.microsoft.azure.storage.CloudStorageAccount; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RetryPolicyFactory; +import com.microsoft.azure.storage.StorageCredentials; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.StorageUri; +import com.microsoft.azure.storage.blob.BlobListingDetails; +import com.microsoft.azure.storage.blob.BlobProperties; +import com.microsoft.azure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.blob.CloudBlobClient; +import com.microsoft.azure.storage.blob.CloudBlobContainer; +import com.microsoft.azure.storage.blob.CloudBlobDirectory; +import com.microsoft.azure.storage.blob.CloudBlockBlob; +import com.microsoft.azure.storage.blob.CloudPageBlob; +import com.microsoft.azure.storage.blob.CopyState; +import com.microsoft.azure.storage.blob.DeleteSnapshotsOption; +import com.microsoft.azure.storage.blob.ListBlobItem; +import com.microsoft.azure.storage.blob.PageRange; /** * A real implementation of the Azure interaction layer that just redirects @@ -61,12 +61,14 @@ class StorageInterfaceImpl extends StorageInterface { @Override public void setRetryPolicyFactory(final RetryPolicyFactory retryPolicyFactory) { - serviceClient.setRetryPolicyFactory(retryPolicyFactory); + serviceClient.getDefaultRequestOptions().setRetryPolicyFactory( + retryPolicyFactory); } @Override public void setTimeoutInMs(int timeoutInMs) { - serviceClient.setTimeoutInMs(timeoutInMs); + serviceClient.getDefaultRequestOptions().setTimeoutIntervalInMs( + timeoutInMs); } @Override @@ -391,11 +393,11 @@ public CopyState getCopyState() { } @Override - public void startCopyFromBlob(URI source, + public void startCopyFromBlob(URI source, BlobRequestOptions options, OperationContext opContext) throws StorageException, URISyntaxException { getBlob().startCopyFromBlob(source, - null, null, null, opContext); + null, null, options, opContext); } @Override diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ErrorMetricUpdater.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ErrorMetricUpdater.java index d33e8c447fe93..dc23354e7b253 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ErrorMetricUpdater.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ErrorMetricUpdater.java @@ -24,10 +24,10 @@ import org.apache.hadoop.classification.InterfaceAudience; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RequestResult; -import com.microsoft.windowsazure.storage.ResponseReceivedEvent; -import com.microsoft.windowsazure.storage.StorageEvent; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RequestResult; +import com.microsoft.azure.storage.ResponseReceivedEvent; +import com.microsoft.azure.storage.StorageEvent; /** diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ResponseReceivedMetricUpdater.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ResponseReceivedMetricUpdater.java index 676adb941ceb2..de503bf19072d 100644 --- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ResponseReceivedMetricUpdater.java +++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/metrics/ResponseReceivedMetricUpdater.java @@ -24,11 +24,11 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience; -import com.microsoft.windowsazure.storage.Constants.HeaderConstants; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RequestResult; -import com.microsoft.windowsazure.storage.ResponseReceivedEvent; -import com.microsoft.windowsazure.storage.StorageEvent; +import com.microsoft.azure.storage.Constants.HeaderConstants; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RequestResult; +import com.microsoft.azure.storage.ResponseReceivedEvent; +import com.microsoft.azure.storage.StorageEvent; /** diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java index b8ff912b4e156..635c024185e12 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java @@ -40,20 +40,20 @@ import org.apache.hadoop.metrics2.MetricsTag; import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; -import com.microsoft.windowsazure.storage.AccessCondition; -import com.microsoft.windowsazure.storage.CloudStorageAccount; -import com.microsoft.windowsazure.storage.StorageCredentials; -import com.microsoft.windowsazure.storage.StorageCredentialsAccountAndKey; -import com.microsoft.windowsazure.storage.StorageCredentialsAnonymous; -import com.microsoft.windowsazure.storage.blob.BlobContainerPermissions; -import com.microsoft.windowsazure.storage.blob.BlobContainerPublicAccessType; -import com.microsoft.windowsazure.storage.blob.BlobOutputStream; -import com.microsoft.windowsazure.storage.blob.CloudBlobClient; -import com.microsoft.windowsazure.storage.blob.CloudBlobContainer; -import com.microsoft.windowsazure.storage.blob.CloudBlockBlob; -import com.microsoft.windowsazure.storage.blob.SharedAccessBlobPermissions; -import com.microsoft.windowsazure.storage.blob.SharedAccessBlobPolicy; -import com.microsoft.windowsazure.storage.core.Base64; +import com.microsoft.azure.storage.AccessCondition; +import com.microsoft.azure.storage.CloudStorageAccount; +import com.microsoft.azure.storage.StorageCredentials; +import com.microsoft.azure.storage.StorageCredentialsAccountAndKey; +import com.microsoft.azure.storage.StorageCredentialsAnonymous; +import com.microsoft.azure.storage.blob.BlobContainerPermissions; +import com.microsoft.azure.storage.blob.BlobContainerPublicAccessType; +import com.microsoft.azure.storage.blob.BlobOutputStream; +import com.microsoft.azure.storage.blob.CloudBlobClient; +import com.microsoft.azure.storage.blob.CloudBlobContainer; +import com.microsoft.azure.storage.blob.CloudBlockBlob; +import com.microsoft.azure.storage.blob.SharedAccessBlobPermissions; +import com.microsoft.azure.storage.blob.SharedAccessBlobPolicy; +import com.microsoft.azure.storage.core.Base64; /** * Helper class to create WASB file systems backed by either a mock in-memory diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java index 047ea1b8fac37..cde0e38ef6a44 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockStorageInterface.java @@ -22,10 +22,12 @@ import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; +import java.lang.reflect.Method; import java.net.URI; import java.net.URISyntaxException; import java.util.ArrayList; import java.util.Calendar; +import java.util.Date; import java.util.EnumSet; import java.util.HashMap; import java.util.HashSet; @@ -35,21 +37,21 @@ import org.apache.commons.httpclient.util.URIUtil; import org.apache.commons.lang.NotImplementedException; -import com.microsoft.windowsazure.storage.CloudStorageAccount; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.RetryPolicyFactory; -import com.microsoft.windowsazure.storage.StorageCredentials; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.StorageUri; -import com.microsoft.windowsazure.storage.blob.BlobListingDetails; -import com.microsoft.windowsazure.storage.blob.BlobProperties; -import com.microsoft.windowsazure.storage.blob.BlobRequestOptions; -import com.microsoft.windowsazure.storage.blob.CloudBlob; -import com.microsoft.windowsazure.storage.blob.CloudBlobContainer; -import com.microsoft.windowsazure.storage.blob.CloudBlobDirectory; -import com.microsoft.windowsazure.storage.blob.CopyState; -import com.microsoft.windowsazure.storage.blob.ListBlobItem; -import com.microsoft.windowsazure.storage.blob.PageRange; +import com.microsoft.azure.storage.CloudStorageAccount; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.RetryPolicyFactory; +import com.microsoft.azure.storage.StorageCredentials; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.StorageUri; +import com.microsoft.azure.storage.blob.BlobListingDetails; +import com.microsoft.azure.storage.blob.BlobProperties; +import com.microsoft.azure.storage.blob.BlobRequestOptions; +import com.microsoft.azure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.blob.CloudBlobContainer; +import com.microsoft.azure.storage.blob.CloudBlobDirectory; +import com.microsoft.azure.storage.blob.CopyState; +import com.microsoft.azure.storage.blob.ListBlobItem; +import com.microsoft.azure.storage.blob.PageRange; import javax.ws.rs.core.UriBuilder; import javax.ws.rs.core.UriBuilderException; @@ -357,18 +359,42 @@ protected MockCloudBlobWrapper(URI uri, HashMap metadata, this.uri = uri; this.metadata = metadata; this.properties = new BlobProperties(); - this.properties.setLength(length); - this.properties.setLastModified( - Calendar.getInstance(TimeZone.getTimeZone("UTC")).getTime()); + + this.properties=updateLastModifed(this.properties); + this.properties=updateLength(this.properties,length); + } + + protected BlobProperties updateLastModifed(BlobProperties properties){ + try{ + Method setLastModified =properties.getClass(). + getDeclaredMethod("setLastModified", Date.class); + setLastModified.setAccessible(true); + setLastModified.invoke(this.properties, + Calendar.getInstance(TimeZone.getTimeZone("UTC")).getTime()); + }catch(Exception e){ + throw new RuntimeException(e); + } + return properties; } - + + protected BlobProperties updateLength(BlobProperties properties,int length) { + try{ + Method setLength =properties.getClass(). + getDeclaredMethod("setLength", long.class); + setLength.setAccessible(true); + setLength.invoke(this.properties, length); + }catch (Exception e){ + throw new RuntimeException(e); + } + return properties; + } + protected void refreshProperties(boolean getMetadata) { if (backingStore.exists(convertUriToDecodedString(uri))) { byte[] content = backingStore.getContent(convertUriToDecodedString(uri)); properties = new BlobProperties(); - properties.setLength(content.length); - properties.setLastModified( - Calendar.getInstance(TimeZone.getTimeZone("UTC")).getTime()); + this.properties=updateLastModifed(this.properties); + this.properties=updateLength(this.properties, content.length); if (getMetadata) { metadata = backingStore.getMetadata(convertUriToDecodedString(uri)); } @@ -403,7 +429,7 @@ public void setMetadata(HashMap metadata) { } @Override - public void startCopyFromBlob(URI source, + public void startCopyFromBlob(URI source, BlobRequestOptions options, OperationContext opContext) throws StorageException, URISyntaxException { backingStore.copy(convertUriToDecodedString(source), convertUriToDecodedString(uri)); //TODO: set the backingStore.properties.CopyState and diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java index 01cf71399fa0b..9ce6cc9b81b33 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java @@ -58,9 +58,9 @@ import org.apache.hadoop.fs.azure.AzureException; import org.apache.hadoop.fs.azure.NativeAzureFileSystem.FolderRenamePending; -import com.microsoft.windowsazure.storage.AccessCondition; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.CloudBlob; +import com.microsoft.azure.storage.AccessCondition; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.CloudBlob; /* * Tests the Native Azure file system (WASB) against an actual blob store if diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureFileSystemErrorConditions.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureFileSystemErrorConditions.java index febb6053baa5b..ace57dc8785ca 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureFileSystemErrorConditions.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureFileSystemErrorConditions.java @@ -37,9 +37,9 @@ import org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.TestHookOperationContext; import org.junit.Test; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.SendingRequestEvent; -import com.microsoft.windowsazure.storage.StorageEvent; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.SendingRequestEvent; +import com.microsoft.azure.storage.StorageEvent; public class TestAzureFileSystemErrorConditions { private static final int ALL_THREE_FILE_SIZE = 1024; diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobDataValidation.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobDataValidation.java index 25bd338e2ccf1..9237ade981447 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobDataValidation.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobDataValidation.java @@ -39,16 +39,16 @@ import org.junit.After; import org.junit.Test; -import com.microsoft.windowsazure.storage.Constants; -import com.microsoft.windowsazure.storage.OperationContext; -import com.microsoft.windowsazure.storage.ResponseReceivedEvent; -import com.microsoft.windowsazure.storage.StorageErrorCodeStrings; -import com.microsoft.windowsazure.storage.StorageEvent; -import com.microsoft.windowsazure.storage.StorageException; -import com.microsoft.windowsazure.storage.blob.BlockEntry; -import com.microsoft.windowsazure.storage.blob.BlockSearchMode; -import com.microsoft.windowsazure.storage.blob.CloudBlockBlob; -import com.microsoft.windowsazure.storage.core.Base64; +import com.microsoft.azure.storage.Constants; +import com.microsoft.azure.storage.OperationContext; +import com.microsoft.azure.storage.ResponseReceivedEvent; +import com.microsoft.azure.storage.StorageErrorCodeStrings; +import com.microsoft.azure.storage.StorageEvent; +import com.microsoft.azure.storage.StorageException; +import com.microsoft.azure.storage.blob.BlockEntry; +import com.microsoft.azure.storage.blob.BlockSearchMode; +import com.microsoft.azure.storage.blob.CloudBlockBlob; +import com.microsoft.azure.storage.core.Base64; /** * Test that we do proper data integrity validation with MD5 checks as diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestContainerChecks.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestContainerChecks.java index 727f540efaf58..56ec881243047 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestContainerChecks.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestContainerChecks.java @@ -32,9 +32,9 @@ import org.junit.After; import org.junit.Test; -import com.microsoft.windowsazure.storage.blob.BlobOutputStream; -import com.microsoft.windowsazure.storage.blob.CloudBlobContainer; -import com.microsoft.windowsazure.storage.blob.CloudBlockBlob; +import com.microsoft.azure.storage.blob.BlobOutputStream; +import com.microsoft.azure.storage.blob.CloudBlobContainer; +import com.microsoft.azure.storage.blob.CloudBlockBlob; /** * Tests that WASB creates containers only if needed. diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestOutOfBandAzureBlobOperationsLive.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestOutOfBandAzureBlobOperationsLive.java index 9ac67e70f8033..60b01c616eb00 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestOutOfBandAzureBlobOperationsLive.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestOutOfBandAzureBlobOperationsLive.java @@ -29,8 +29,8 @@ import org.junit.Before; import org.junit.Test; -import com.microsoft.windowsazure.storage.blob.BlobOutputStream; -import com.microsoft.windowsazure.storage.blob.CloudBlockBlob; +import com.microsoft.azure.storage.blob.BlobOutputStream; +import com.microsoft.azure.storage.blob.CloudBlockBlob; public class TestOutOfBandAzureBlobOperationsLive { private FileSystem fs; diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbUriAndConfiguration.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbUriAndConfiguration.java index 0360e323170d4..a4ca6fdda4ea9 100644 --- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbUriAndConfiguration.java +++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbUriAndConfiguration.java @@ -42,8 +42,8 @@ import org.junit.Assert; import org.junit.Test; -import com.microsoft.windowsazure.storage.blob.CloudBlobContainer; -import com.microsoft.windowsazure.storage.blob.CloudBlockBlob; +import com.microsoft.azure.storage.blob.CloudBlobContainer; +import com.microsoft.azure.storage.blob.CloudBlockBlob; public class TestWasbUriAndConfiguration { diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java index ab5b8024d72dc..e3c58e9a226e6 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java @@ -38,10 +38,10 @@ /** * The CopyListing abstraction is responsible for how the list of * sources and targets is constructed, for DistCp's copy function. - * The copy-listing should be a SequenceFile, - * located at the path specified to buildListing(), - * each entry being a pair of (Source relative path, source file status), - * all the paths being fully qualified. + * The copy-listing should be a + * SequenceFile<Text, CopyListingFileStatus>, located at the path + * specified to buildListing(), each entry being a pair of (Source relative + * path, source file status), all the paths being fully qualified. */ public abstract class CopyListing extends Configured { @@ -95,8 +95,8 @@ public final void buildListing(Path pathToListFile, * Validate input and output paths * * @param options - Input options - * @throws InvalidInputException: If inputs are invalid - * @throws IOException: any Exception with FS + * @throws InvalidInputException If inputs are invalid + * @throws IOException any Exception with FS */ protected abstract void validatePaths(DistCpOptions options) throws IOException, InvalidInputException; @@ -105,7 +105,7 @@ protected abstract void validatePaths(DistCpOptions options) * The interface to be implemented by sub-classes, to create the source/target file listing. * @param pathToListFile Path on HDFS where the listing file is written. * @param options Input Options for DistCp (indicating source/target paths.) - * @throws IOException: Thrown on failure to create the listing file. + * @throws IOException Thrown on failure to create the listing file. */ protected abstract void doBuildListing(Path pathToListFile, DistCpOptions options) throws IOException; @@ -224,7 +224,9 @@ public static CopyListing getCopyListing(Configuration configuration, Credentials credentials, DistCpOptions options) throws IOException { - + if (options.shouldUseDiff()) { + return new GlobbedCopyListing(configuration, credentials); + } String copyListingClassName = configuration.get(DistCpConstants. CONF_LABEL_COPY_LISTING_CLASS, ""); Class copyListingClass; diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListingFileStatus.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListingFileStatus.java index 9b31dfbdd8f35..8af799a48fe65 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListingFileStatus.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListingFileStatus.java @@ -79,7 +79,7 @@ public CopyListingFileStatus(FileStatus fileStatus) throws IOException { /** * Returns the full logical ACL. * - * @return List containing full logical ACL + * @return List containing full logical ACL */ public List getAclEntries() { return AclUtil.getAclFromPermAndEntries(getPermission(), @@ -89,7 +89,7 @@ public List getAclEntries() { /** * Sets optional ACL entries. * - * @param aclEntries List containing all ACL entries + * @param aclEntries List containing all ACL entries */ public void setAclEntries(List aclEntries) { this.aclEntries = aclEntries; @@ -98,7 +98,7 @@ public void setAclEntries(List aclEntries) { /** * Returns all xAttrs. * - * @return Map containing all xAttrs + * @return Map containing all xAttrs */ public Map getXAttrs() { return xAttrs != null ? xAttrs : Collections.emptyMap(); @@ -107,7 +107,7 @@ public Map getXAttrs() { /** * Sets optional xAttrs. * - * @param xAttrs Map containing all xAttrs + * @param xAttrs Map containing all xAttrs */ public void setXAttrs(Map xAttrs) { this.xAttrs = xAttrs; diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DiffInfo.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DiffInfo.java new file mode 100644 index 0000000000000..b617de78803c3 --- /dev/null +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DiffInfo.java @@ -0,0 +1,90 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tools; + +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.DFSUtil; +import org.apache.hadoop.hdfs.protocol.SnapshotDiffReport; + +/** + * Information presenting a rename/delete op derived from a snapshot diff entry. + * This includes the source file/dir of the rename/delete op, and the target + * file/dir of a rename op. + */ +class DiffInfo { + static final Comparator sourceComparator = new Comparator() { + @Override + public int compare(DiffInfo d1, DiffInfo d2) { + return d2.source.compareTo(d1.source); + } + }; + + static final Comparator targetComparator = new Comparator() { + @Override + public int compare(DiffInfo d1, DiffInfo d2) { + return d1.target == null ? -1 : + (d2.target == null ? 1 : d1.target.compareTo(d2.target)); + } + }; + + /** The source file/dir of the rename or deletion op */ + final Path source; + /** + * The intermediate file/dir for the op. For a rename or a delete op, + * we first rename the source to this tmp file/dir. + */ + private Path tmp; + /** The target file/dir of the rename op. Null means the op is deletion. */ + final Path target; + + DiffInfo(Path source, Path target) { + assert source != null; + this.source = source; + this.target= target; + } + + void setTmp(Path tmp) { + this.tmp = tmp; + } + + Path getTmp() { + return tmp; + } + + static DiffInfo[] getDiffs(SnapshotDiffReport report, Path targetDir) { + List diffs = new ArrayList<>(); + for (SnapshotDiffReport.DiffReportEntry entry : report.getDiffList()) { + if (entry.getType() == SnapshotDiffReport.DiffType.DELETE) { + final Path source = new Path(targetDir, + DFSUtil.bytes2String(entry.getSourcePath())); + diffs.add(new DiffInfo(source, null)); + } else if (entry.getType() == SnapshotDiffReport.DiffType.RENAME) { + final Path source = new Path(targetDir, + DFSUtil.bytes2String(entry.getSourcePath())); + final Path target = new Path(targetDir, + DFSUtil.bytes2String(entry.getTargetPath())); + diffs.add(new DiffInfo(source, target)); + } + } + return diffs.toArray(new DiffInfo[diffs.size()]); + } +} diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java index d202f0a9bdf05..6921a1eea3285 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java @@ -18,17 +18,22 @@ package org.apache.hadoop.tools; +import java.io.IOException; +import java.util.Random; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapreduce.Cluster; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.JobContext; import org.apache.hadoop.mapreduce.JobSubmissionFiles; -import org.apache.hadoop.mapreduce.Cluster; import org.apache.hadoop.tools.CopyListing.*; import org.apache.hadoop.tools.mapred.CopyMapper; import org.apache.hadoop.tools.mapred.CopyOutputFormat; @@ -37,9 +42,6 @@ import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; -import java.io.IOException; -import java.util.Random; - import com.google.common.annotations.VisibleForTesting; /** @@ -51,14 +53,16 @@ * launch the copy-job. DistCp may alternatively be sub-classed to fine-tune * behaviour. */ +@InterfaceAudience.Public +@InterfaceStability.Evolving public class DistCp extends Configured implements Tool { /** - * Priority of the ResourceManager shutdown hook. + * Priority of the shutdown hook. */ - public static final int SHUTDOWN_HOOK_PRIORITY = 30; + static final int SHUTDOWN_HOOK_PRIORITY = 30; - private static final Log LOG = LogFactory.getLog(DistCp.class); + static final Log LOG = LogFactory.getLog(DistCp.class); private DistCpOptions inputOptions; private Path metaFolder; @@ -66,7 +70,7 @@ public class DistCp extends Configured implements Tool { private static final String PREFIX = "_distcp"; private static final String WIP_PREFIX = "._WIP_"; private static final String DISTCP_DEFAULT_XML = "distcp-default.xml"; - public static final Random rand = new Random(); + static final Random rand = new Random(); private boolean submitted; private FileSystem jobFS; @@ -76,7 +80,7 @@ public class DistCp extends Configured implements Tool { * (E.g. source-paths, target-location, etc.) * @param inputOptions Options (indicating source-paths, target-location.) * @param configuration The Hadoop configuration against which the Copy-mapper must run. - * @throws Exception, on failure. + * @throws Exception */ public DistCp(Configuration configuration, DistCpOptions inputOptions) throws Exception { Configuration config = new Configuration(configuration); @@ -90,7 +94,7 @@ public DistCp(Configuration configuration, DistCpOptions inputOptions) throws Ex * To be used with the ToolRunner. Not for public consumption. */ @VisibleForTesting - public DistCp() {} + DistCp() {} /** * Implementation of Tool::run(). Orchestrates the copy of source file(s) @@ -100,6 +104,7 @@ public DistCp() {} * @param argv List of arguments passed to DistCp, from the ToolRunner. * @return On success, it returns 0. Else, -1. */ + @Override public int run(String[] argv) { if (argv.length < 1) { OptionsParser.usage(); @@ -142,21 +147,37 @@ public int run(String[] argv) { * Implements the core-execution. Creates the file-list for copy, * and launches the Hadoop-job, to do the copy. * @return Job handle - * @throws Exception, on failure. + * @throws Exception */ public Job execute() throws Exception { + Job job = createAndSubmitJob(); + + if (inputOptions.shouldBlock()) { + waitForJobCompletion(job); + } + return job; + } + + /** + * Create and submit the mapreduce job. + * @return The mapreduce job object that has been submitted + */ + public Job createAndSubmitJob() throws Exception { assert inputOptions != null; assert getConf() != null; - Job job = null; try { synchronized(this) { //Don't cleanup while we are setting up. metaFolder = createMetaFolderPath(); jobFS = metaFolder.getFileSystem(getConf()); - job = createJob(); } + if (inputOptions.shouldUseDiff()) { + if (!DistCpSync.sync(inputOptions, getConf())) { + inputOptions.disableUsingDiff(); + } + } createInputFileListing(job); job.submit(); @@ -169,15 +190,23 @@ public Job execute() throws Exception { String jobID = job.getJobID().toString(); job.getConfiguration().set(DistCpConstants.CONF_LABEL_DISTCP_JOB_ID, jobID); - LOG.info("DistCp job-id: " + jobID); - if (inputOptions.shouldBlock() && !job.waitForCompletion(true)) { - throw new IOException("DistCp failure: Job " + jobID + " has failed: " - + job.getStatus().getFailureInfo()); - } + return job; } + /** + * Wait for the given job to complete. + * @param job the given mapreduce job that has already been submitted + */ + public void waitForJobCompletion(Job job) throws Exception { + assert job != null; + if (!job.waitForCompletion(true)) { + throw new IOException("DistCp failure: Job " + job.getJobID() + + " has failed: " + job.getStatus().getFailureInfo()); + } + } + /** * Set targetPathExists in both inputOptions and job config, * for the benefit of CopyCommitter @@ -372,7 +401,7 @@ protected Path getFileListingPath() throws IOException { * job staging directory * * @return Returns the working folder information - * @throws Exception - EXception if any + * @throws Exception - Exception if any */ private Path createMetaFolderPath() throws Exception { Configuration configuration = getConf(); @@ -436,7 +465,7 @@ private boolean isSubmitted() { private static class Cleanup implements Runnable { private final DistCp distCp; - public Cleanup(DistCp distCp) { + Cleanup(DistCp distCp) { this.distCp = distCp; } diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java index 7e71096952d34..a1af2af767f1e 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java @@ -53,6 +53,7 @@ public class DistCpConstants { public static final String CONF_LABEL_SKIP_CRC = "distcp.skip.crc"; public static final String CONF_LABEL_OVERWRITE = "distcp.copy.overwrite"; public static final String CONF_LABEL_APPEND = "distcp.copy.append"; + public static final String CONF_LABEL_DIFF = "distcp.copy.diff"; public static final String CONF_LABEL_BANDWIDTH_MB = "distcp.map.bandwidth.mb"; public static final String CONF_LABEL_MAX_CHUNKS_TOLERABLE = @@ -134,4 +135,6 @@ public class DistCpConstants { * Value of reserved raw HDFS directory when copying raw.* xattrs. */ static final String HDFS_RESERVED_RAW_DIRECTORY_NAME = "/.reserved/raw"; + + static final String HDFS_DISTCP_DIFF_DIRECTORY_NAME = ".distcp.diff.tmp"; } diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java index d263f8273dc91..e9c7d46152e9c 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java @@ -41,8 +41,6 @@ public enum DistCpOptionSwitch { * target file. Note that when preserving checksum type, block size is also * preserved. * - * @see PRESERVE_STATUS_DEFAULT - * * If any of the optional switches are present among rbugpcaxt, then * only the corresponding file attribute is preserved. */ @@ -105,7 +103,7 @@ public enum DistCpOptionSwitch { * Copy all the source files and commit them atomically to the target * This is typically useful in cases where there is a process * polling for availability of a file/dir. This option is incompatible - * with SYNC_FOLDERS & DELETE_MISSING + * with SYNC_FOLDERS and DELETE_MISSING */ ATOMIC_COMMIT(DistCpConstants.CONF_LABEL_ATOMIC_COPY, new Option("atomic", false, "Commit all changes or none")), @@ -149,6 +147,11 @@ public enum DistCpOptionSwitch { new Option("append", false, "Reuse existing data in target files and append new data to them if possible")), + DIFF(DistCpConstants.CONF_LABEL_DIFF, + new Option("diff", false, + "Use snapshot diff report to identify the difference between source and target"), + 2), + /** * Should DisctpExecution be blocking */ @@ -178,6 +181,11 @@ public enum DistCpOptionSwitch { this.option = option; } + DistCpOptionSwitch(String confLabel, Option option, int argNum) { + this(confLabel, option); + this.option.setArgs(argNum); + } + /** * Get Configuration label for the option * @return configuration label name diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java index 57d2fb7eb1cac..709e5832bab90 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java @@ -42,6 +42,7 @@ public class DistCpOptions { private boolean append = false; private boolean skipCRC = false; private boolean blocking = true; + private boolean useDiff = false; private int maxMaps = DistCpConstants.DEFAULT_MAPS; private int mapBandwidth = DistCpConstants.DEFAULT_BANDWIDTH_MB; @@ -61,6 +62,9 @@ public class DistCpOptions { private Path sourceFileListing; private List sourcePaths; + private String fromSnapshot; + private String toSnapshot; + private Path targetPath; // targetPathExist is a derived field, it's initialized in the @@ -264,6 +268,29 @@ public void setAppend(boolean append) { this.append = append; } + public boolean shouldUseDiff() { + return this.useDiff; + } + + public String getFromSnapshot() { + return this.fromSnapshot; + } + + public String getToSnapshot() { + return this.toSnapshot; + } + + public void setUseDiff(boolean useDiff, String fromSnapshot, String toSnapshot) { + validate(DistCpOptionSwitch.DIFF, useDiff); + this.useDiff = useDiff; + this.fromSnapshot = fromSnapshot; + this.toSnapshot = toSnapshot; + } + + public void disableUsingDiff() { + this.useDiff = false; + } + /** * Should CRC/checksum check be skipped while checking files are identical * @@ -508,6 +535,7 @@ public void validate(DistCpOptionSwitch option, boolean value) { boolean skipCRC = (option == DistCpOptionSwitch.SKIP_CRC ? value : this.skipCRC); boolean append = (option == DistCpOptionSwitch.APPEND ? value : this.append); + boolean useDiff = (option == DistCpOptionSwitch.DIFF ? value : this.useDiff); if (syncFolder && atomicCommit) { throw new IllegalArgumentException("Atomic commit can't be used with " + @@ -536,6 +564,10 @@ public void validate(DistCpOptionSwitch option, boolean value) { throw new IllegalArgumentException( "Append is disallowed when skipping CRC"); } + if ((!syncFolder || !deleteMissing) && useDiff) { + throw new IllegalArgumentException( + "Diff is valid only with update and delete options"); + } } /** @@ -556,6 +588,8 @@ public void appendToConf(Configuration conf) { String.valueOf(overwrite)); DistCpOptionSwitch.addToConf(conf, DistCpOptionSwitch.APPEND, String.valueOf(append)); + DistCpOptionSwitch.addToConf(conf, DistCpOptionSwitch.DIFF, + String.valueOf(useDiff)); DistCpOptionSwitch.addToConf(conf, DistCpOptionSwitch.SKIP_CRC, String.valueOf(skipCRC)); DistCpOptionSwitch.addToConf(conf, DistCpOptionSwitch.BANDWIDTH, diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java new file mode 100644 index 0000000000000..26d7eb416d376 --- /dev/null +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java @@ -0,0 +1,192 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tools; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.DistributedFileSystem; +import org.apache.hadoop.hdfs.protocol.SnapshotDiffReport; + +import java.io.IOException; +import java.util.Arrays; +import java.util.List; +import java.util.Random; + +/** + * This class provides the basic functionality to sync two FileSystems based on + * the snapshot diff report. More specifically, we have the following settings: + * 1. Both the source and target FileSystem must be DistributedFileSystem + * 2. Two snapshots (e.g., s1 and s2) have been created on the source FS. + * The diff between these two snapshots will be copied to the target FS. + * 3. The target has the same snapshot s1. No changes have been made on the + * target since s1. All the files/directories in the target are the same with + * source.s1 + */ +class DistCpSync { + + static boolean sync(DistCpOptions inputOptions, Configuration conf) + throws IOException { + List sourcePaths = inputOptions.getSourcePaths(); + if (sourcePaths.size() != 1) { + // we only support one source dir which must be a snapshottable directory + DistCp.LOG.warn(sourcePaths.size() + " source paths are provided"); + return false; + } + final Path sourceDir = sourcePaths.get(0); + final Path targetDir = inputOptions.getTargetPath(); + + final FileSystem sfs = sourceDir.getFileSystem(conf); + final FileSystem tfs = targetDir.getFileSystem(conf); + // currently we require both the source and the target file system are + // DistributedFileSystem. + if (!(sfs instanceof DistributedFileSystem) || + !(tfs instanceof DistributedFileSystem)) { + DistCp.LOG.warn("To use diff-based distcp, the FileSystems needs to" + + " be DistributedFileSystem"); + return false; + } + final DistributedFileSystem sourceFs = (DistributedFileSystem) sfs; + final DistributedFileSystem targetFs= (DistributedFileSystem) tfs; + + // make sure targetFS has no change between from and the current states + if (!checkNoChange(inputOptions, targetFs, targetDir)) { + return false; + } + + Path tmpDir = null; + try { + tmpDir = createTargetTmpDir(targetFs, targetDir); + DiffInfo[] diffs = getDiffs(inputOptions, sourceFs, sourceDir, targetDir); + if (diffs == null) { + return false; + } + // do the real sync work: deletion and rename + syncDiff(diffs, targetFs, tmpDir); + return true; + } catch (Exception e) { + DistCp.LOG.warn("Failed to use snapshot diff for distcp", e); + return false; + } finally { + deleteTargetTmpDir(targetFs, tmpDir); + // TODO: since we have tmp directory, we can support "undo" with failures + } + } + + private static Path createTargetTmpDir(DistributedFileSystem targetFs, + Path targetDir) throws IOException { + final Path tmp = new Path(targetDir, + DistCpConstants.HDFS_DISTCP_DIFF_DIRECTORY_NAME + DistCp.rand.nextInt()); + if (!targetFs.mkdirs(tmp)) { + throw new IOException("The tmp directory " + tmp + " already exists"); + } + return tmp; + } + + private static void deleteTargetTmpDir(DistributedFileSystem targetFs, + Path tmpDir) { + try { + if (tmpDir != null) { + targetFs.delete(tmpDir, true); + } + } catch (IOException e) { + DistCp.LOG.error("Unable to cleanup tmp dir: " + tmpDir, e); + } + } + + /** + * Compute the snapshot diff on the given file system. Return true if the diff + * is empty, i.e., no changes have happened in the FS. + */ + private static boolean checkNoChange(DistCpOptions inputOptions, + DistributedFileSystem fs, Path path) { + try { + SnapshotDiffReport targetDiff = + fs.getSnapshotDiffReport(path, inputOptions.getFromSnapshot(), ""); + if (!targetDiff.getDiffList().isEmpty()) { + DistCp.LOG.warn("The target has been modified since snapshot " + + inputOptions.getFromSnapshot()); + return false; + } else { + return true; + } + } catch (IOException e) { + DistCp.LOG.warn("Failed to compute snapshot diff on " + path, e); + } + return false; + } + + @VisibleForTesting + static DiffInfo[] getDiffs(DistCpOptions inputOptions, + DistributedFileSystem fs, Path sourceDir, Path targetDir) { + try { + SnapshotDiffReport sourceDiff = fs.getSnapshotDiffReport(sourceDir, + inputOptions.getFromSnapshot(), inputOptions.getToSnapshot()); + return DiffInfo.getDiffs(sourceDiff, targetDir); + } catch (IOException e) { + DistCp.LOG.warn("Failed to compute snapshot diff on " + sourceDir, e); + } + return null; + } + + private static void syncDiff(DiffInfo[] diffs, + DistributedFileSystem targetFs, Path tmpDir) throws IOException { + moveToTmpDir(diffs, targetFs, tmpDir); + moveToTarget(diffs, targetFs); + } + + /** + * Move all the source files that should be renamed or deleted to the tmp + * directory. + */ + private static void moveToTmpDir(DiffInfo[] diffs, + DistributedFileSystem targetFs, Path tmpDir) throws IOException { + // sort the diffs based on their source paths to make sure the files and + // subdirs are moved before moving their parents/ancestors. + Arrays.sort(diffs, DiffInfo.sourceComparator); + Random random = new Random(); + for (DiffInfo diff : diffs) { + Path tmpTarget = new Path(tmpDir, diff.source.getName()); + while (targetFs.exists(tmpTarget)) { + tmpTarget = new Path(tmpDir, diff.source.getName() + random.nextInt()); + } + diff.setTmp(tmpTarget); + targetFs.rename(diff.source, tmpTarget); + } + } + + /** + * Finish the rename operations: move all the intermediate files/directories + * from the tmp dir to the final targets. + */ + private static void moveToTarget(DiffInfo[] diffs, + DistributedFileSystem targetFs) throws IOException { + // sort the diffs based on their target paths to make sure the parent + // directories are created first. + Arrays.sort(diffs, DiffInfo.targetComparator); + for (DiffInfo diff : diffs) { + if (diff.target != null) { + if (!targetFs.exists(diff.target.getParent())) { + targetFs.mkdirs(diff.target.getParent()); + } + targetFs.rename(diff.getTmp(), diff.target); + } + } + } +} diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java index 4bbc30dea292c..a3a76ef585164 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java @@ -18,13 +18,22 @@ package org.apache.hadoop.tools; -import org.apache.commons.cli.*; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.CommandLineParser; +import org.apache.commons.cli.GnuParser; +import org.apache.commons.cli.HelpFormatter; +import org.apache.commons.cli.Options; +import org.apache.commons.cli.ParseException; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.fs.Path; import org.apache.hadoop.tools.DistCpOptions.FileAttribute; -import java.util.*; +import com.google.common.base.Preconditions; /** * The OptionsParser parses out the command-line options passed to DistCp, @@ -63,7 +72,7 @@ protected String[] flatten(Options options, String[] arguments, boolean stopAtNo * @param args Command-line arguments (excluding the options consumed * by the GenericOptionsParser). * @return The Options object, corresponding to the specified command-line. - * @throws IllegalArgumentException: Thrown if the parse fails. + * @throws IllegalArgumentException Thrown if the parse fails. */ public static DistCpOptions parse(String args[]) throws IllegalArgumentException { @@ -207,6 +216,13 @@ public static DistCpOptions parse(String args[]) throws IllegalArgumentException } } + if (command.hasOption(DistCpOptionSwitch.DIFF.getSwitch())) { + String[] snapshots = getVals(command, DistCpOptionSwitch.DIFF.getSwitch()); + Preconditions.checkArgument(snapshots != null && snapshots.length == 2, + "Must provide both the starting and ending snapshot names"); + option.setUseDiff(true, snapshots[0], snapshots[1]); + } + if (command.hasOption(DistCpOptionSwitch.FILE_LIMIT.getSwitch())) { String fileLimitString = getVal(command, DistCpOptionSwitch.FILE_LIMIT.getSwitch().trim()); @@ -247,6 +263,10 @@ private static String getVal(CommandLine command, String swtch) { } } + private static String[] getVals(CommandLine command, String option) { + return command.getOptionValues(option); + } + public static void usage() { HelpFormatter formatter = new HelpFormatter(); formatter.printHelp("distcp OPTIONS [source_path...] \n\nOPTIONS", cliOptions); diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java index f9cfc86d9ea41..6dc827a92e369 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java @@ -141,7 +141,7 @@ public void doBuildListing(Path pathToListingFile, DistCpOptions options) throws } /** * Collect the list of - * + * {@literal } * to be copied and write to the sequence file. In essence, any file or * directory that need to be copied or sync-ed is written as an entry to the * sequence file, with the possible exception of the source root: diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java index 197edd91c3266..9ec57f426aa13 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java @@ -70,7 +70,7 @@ public CopyCommitter(Path outputPath, TaskAttemptContext context) throws IOExcep this.taskAttemptContext = context; } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public void commitJob(JobContext jobContext) throws IOException { Configuration conf = jobContext.getConfiguration(); @@ -90,7 +90,8 @@ public void commitJob(JobContext jobContext) throws IOException { } try { - if (conf.getBoolean(DistCpConstants.CONF_LABEL_DELETE_MISSING, false)) { + if (conf.getBoolean(DistCpConstants.CONF_LABEL_DELETE_MISSING, false) + && !(conf.getBoolean(DistCpConstants.CONF_LABEL_DIFF, false))) { deleteMissing(conf); } else if (conf.getBoolean(DistCpConstants.CONF_LABEL_ATOMIC_COPY, false)) { commitData(conf); @@ -102,7 +103,7 @@ public void commitJob(JobContext jobContext) throws IOException { } } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public void abortJob(JobContext jobContext, JobStatus.State state) throws IOException { @@ -132,6 +133,9 @@ private void cleanupTempFiles(JobContext context) { private void deleteAttemptTempFiles(Path targetWorkPath, FileSystem targetFS, String jobId) throws IOException { + if (targetWorkPath == null) { + return; + } FileStatus[] tempFiles = targetFS.globStatus( new Path(targetWorkPath, ".distcp.tmp." + jobId.replaceAll("job","attempt") + "*")); diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java index ab57127ed7834..cca36df67f59b 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java @@ -45,7 +45,7 @@ /** * Mapper class that executes the DistCp copy operation. - * Implements the o.a.h.mapreduce.Mapper<> interface. + * Implements the o.a.h.mapreduce.Mapper interface. */ public class CopyMapper extends Mapper { @@ -182,10 +182,11 @@ private Path findCacheFile(Path[] cacheFiles, String fileName) { } /** - * Implementation of the Mapper<>::map(). Does the copy. + * Implementation of the Mapper::map(). Does the copy. * @param relPath The target path. * @param sourceFileStatus The source path. * @throws IOException + * @throws InterruptedException */ @Override public void map(Text relPath, CopyListingFileStatus sourceFileStatus, diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java index eb43aa3927b00..a5bd605a8ef14 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyOutputFormat.java @@ -97,13 +97,13 @@ private static Path getCommitDirectory(Configuration conf) { } } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException { return new CopyCommitter(getOutputPath(context), context); } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public void checkOutputSpecs(JobContext context) throws IOException { Configuration conf = context.getConfiguration(); diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java index 1d6115685e5dc..65d644bfefafa 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java @@ -84,8 +84,7 @@ public RetriableFileCopyCommand(boolean skipCrc, String description, * This is the actual copy-implementation. * @param arguments Argument-list to the command. * @return Number of bytes copied. - * @throws Exception: CopyReadException, if there are read-failures. All other - * failures are IOExceptions. + * @throws Exception */ @SuppressWarnings("unchecked") @Override diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/UniformSizeInputFormat.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/UniformSizeInputFormat.java index 4add0bb4690ea..8dc7a659856dc 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/UniformSizeInputFormat.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/UniformSizeInputFormat.java @@ -38,7 +38,7 @@ import java.util.ArrayList; /** - * UniformSizeInputFormat extends the InputFormat<> class, to produce + * UniformSizeInputFormat extends the InputFormat class, to produce * input-splits for DistCp. * It looks at the copy-listing and groups the contents into input-splits such * that the total-number of bytes to be copied for each input split is @@ -55,7 +55,7 @@ public class UniformSizeInputFormat * approximately equal. * @param context JobContext for the job. * @return The list of uniformly-distributed input-splits. - * @throws IOException: On failure. + * @throws IOException * @throws InterruptedException */ @Override diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java index f5303d54c5ceb..38269c755fecd 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicInputFormat.java @@ -64,7 +64,7 @@ public class DynamicInputFormat extends InputFormat { * tasks. * @param jobContext JobContext for the map job. * @return The list of (empty) dynamic input-splits. - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override @@ -343,7 +343,7 @@ static int getNumEntriesPerChunk(Configuration configuration) { * @param inputSplit The split for which the RecordReader is required. * @param taskAttemptContext TaskAttemptContext for the current attempt. * @return DynamicRecordReader instance. - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicRecordReader.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicRecordReader.java index 40d75f44713c3..00b3c69de3dda 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicRecordReader.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/lib/DynamicRecordReader.java @@ -57,7 +57,7 @@ public class DynamicRecordReader extends RecordReader { * RecordReader to read from chunks. * @param inputSplit The InputSplit for the map. Ignored entirely. * @param taskAttemptContext The AttemptContext. - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override @@ -88,7 +88,7 @@ private int getTotalNumRecords() { * been completely exhausted, an new chunk is acquired and read, * transparently. * @return True, if the nextValue() could be traversed to. False, otherwise. - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override @@ -130,7 +130,7 @@ public boolean nextKeyValue() /** * Implementation of RecordReader::getCurrentKey(). * @return The key of the current record. (i.e. the source-path.) - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override @@ -142,7 +142,7 @@ public K getCurrentKey() /** * Implementation of RecordReader::getCurrentValue(). * @return The value of the current record. (i.e. the target-path.) - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override @@ -154,7 +154,7 @@ public V getCurrentValue() /** * Implementation of RecordReader::getProgress(). * @return A fraction [0.0,1.0] indicating the progress of a DistCp mapper. - * @throws IOException, on failure. + * @throws IOException * @throws InterruptedException */ @Override @@ -192,7 +192,7 @@ private int getNumChunksLeft() throws IOException { /** * Implementation of RecordReader::close(). * Closes the RecordReader. - * @throws IOException, on failure. + * @throws IOException */ @Override public void close() diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java index ca7566b0f1492..d34faafba8744 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java @@ -65,7 +65,7 @@ public class DistCpUtils { * @param path The path of the file whose size is sought. * @param configuration Configuration, to retrieve the appropriate FileSystem. * @return The file-size, in number of bytes. - * @throws IOException, on failure. + * @throws IOException */ public static long getFileSize(Path path, Configuration configuration) throws IOException { @@ -270,7 +270,7 @@ public static void preserve(FileSystem targetFS, Path path, * * @param fileSystem FileSystem containing the file * @param fileStatus FileStatus of file - * @return List containing full logical ACL + * @return List containing full logical ACL * @throws IOException if there is an I/O error */ public static List getAcl(FileSystem fileSystem, @@ -285,7 +285,7 @@ public static List getAcl(FileSystem fileSystem, * * @param fileSystem FileSystem containing the file * @param path file path - * @return Map containing all xAttrs + * @return Map containing all xAttrs * @throws IOException if there is an I/O error */ public static Map getXAttrs(FileSystem fileSystem, diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java index 563372e0097e0..c27b2e1758edd 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java @@ -77,7 +77,7 @@ public RetriableCommand(String description, RetryPolicy retryPolicy) { * 2. the command may no longer be retried (e.g. runs out of retry-attempts). * @param arguments The list of arguments for the command. * @return Generic "Object" from doExecute(), on success. - * @throws IOException, IOException, on complete failure. + * @throws Exception */ public Object execute(Object... arguments) throws Exception { Exception latestException; diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java index d08a3012e27aa..9e435d99afd60 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java @@ -62,7 +62,7 @@ public void close() throws IOException { rawStream.close(); } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public int read() throws IOException { throttle(); @@ -73,7 +73,7 @@ public int read() throws IOException { return data; } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public int read(byte[] b) throws IOException { throttle(); @@ -84,7 +84,7 @@ public int read(byte[] b) throws IOException { return readLen; } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public int read(byte[] b, int off, int len) throws IOException { throttle(); @@ -155,7 +155,7 @@ public long getTotalSleepTime() { return totalSleepTime; } - /** @inheritDoc */ + /** {@inheritDoc} */ @Override public String toString() { return "ThrottledInputStream{" + diff --git a/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java b/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java new file mode 100644 index 0000000000000..7d5dad06c85ea --- /dev/null +++ b/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpSync.java @@ -0,0 +1,349 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.tools; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdfs.DFSTestUtil; +import org.apache.hadoop.hdfs.DistributedFileSystem; +import org.apache.hadoop.hdfs.HdfsConfiguration; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.apache.hadoop.hdfs.protocol.SnapshotDiffReport; +import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.io.SequenceFile; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapreduce.Mapper; +import org.apache.hadoop.security.Credentials; +import org.apache.hadoop.tools.mapred.CopyMapper; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; + +public class TestDistCpSync { + private MiniDFSCluster cluster; + private final Configuration conf = new HdfsConfiguration(); + private DistributedFileSystem dfs; + private DistCpOptions options; + private final Path source = new Path("/source"); + private final Path target = new Path("/target"); + private final long BLOCK_SIZE = 1024; + private final short DATA_NUM = 1; + + @Before + public void setUp() throws Exception { + cluster = new MiniDFSCluster.Builder(conf).numDataNodes(DATA_NUM).build(); + cluster.waitActive(); + + dfs = cluster.getFileSystem(); + dfs.mkdirs(source); + dfs.mkdirs(target); + + options = new DistCpOptions(Arrays.asList(source), target); + options.setSyncFolder(true); + options.setDeleteMissing(true); + options.setUseDiff(true, "s1", "s2"); + options.appendToConf(conf); + + conf.set(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH, target.toString()); + conf.set(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH, target.toString()); + } + + @After + public void tearDown() throws Exception { + IOUtils.cleanup(null, dfs); + if (cluster != null) { + cluster.shutdown(); + } + } + + /** + * Test the sync returns false in the following scenarios: + * 1. the source/target dir are not snapshottable dir + * 2. the source/target does not have the given snapshots + * 3. changes have been made in target + */ + @Test + public void testFallback() throws Exception { + // the source/target dir are not snapshottable dir + Assert.assertFalse(DistCpSync.sync(options, conf)); + + // the source/target does not have the given snapshots + dfs.allowSnapshot(source); + dfs.allowSnapshot(target); + Assert.assertFalse(DistCpSync.sync(options, conf)); + + dfs.createSnapshot(source, "s1"); + dfs.createSnapshot(source, "s2"); + dfs.createSnapshot(target, "s1"); + Assert.assertTrue(DistCpSync.sync(options, conf)); + + // changes have been made in target + final Path subTarget = new Path(target, "sub"); + dfs.mkdirs(subTarget); + Assert.assertFalse(DistCpSync.sync(options, conf)); + + dfs.delete(subTarget, true); + Assert.assertTrue(DistCpSync.sync(options, conf)); + } + + /** + * create some files and directories under the given directory. + * the final subtree looks like this: + * dir/ + * foo/ bar/ + * d1/ f1 d2/ f2 + * f3 f4 + */ + private void initData(Path dir) throws Exception { + final Path foo = new Path(dir, "foo"); + final Path bar = new Path(dir, "bar"); + final Path d1 = new Path(foo, "d1"); + final Path f1 = new Path(foo, "f1"); + final Path d2 = new Path(bar, "d2"); + final Path f2 = new Path(bar, "f2"); + final Path f3 = new Path(d1, "f3"); + final Path f4 = new Path(d2, "f4"); + + DFSTestUtil.createFile(dfs, f1, BLOCK_SIZE, DATA_NUM, 0); + DFSTestUtil.createFile(dfs, f2, BLOCK_SIZE, DATA_NUM, 0); + DFSTestUtil.createFile(dfs, f3, BLOCK_SIZE, DATA_NUM, 0); + DFSTestUtil.createFile(dfs, f4, BLOCK_SIZE, DATA_NUM, 0); + } + + /** + * make some changes under the given directory (created in the above way). + * 1. rename dir/foo/d1 to dir/bar/d1 + * 2. delete dir/bar/d1/f3 + * 3. rename dir/foo to /dir/bar/d1/foo + * 4. delete dir/bar/d1/foo/f1 + * 5. create file dir/bar/d1/foo/f1 whose size is 2*BLOCK_SIZE + * 6. append one BLOCK to file dir/bar/f2 + * 7. rename dir/bar to dir/foo + * + * Thus after all these ops the subtree looks like this: + * dir/ + * foo/ + * d1/ f2(A) d2/ + * foo/ f4 + * f1(new) + */ + private void changeData(Path dir) throws Exception { + final Path foo = new Path(dir, "foo"); + final Path bar = new Path(dir, "bar"); + final Path d1 = new Path(foo, "d1"); + final Path f2 = new Path(bar, "f2"); + + final Path bar_d1 = new Path(bar, "d1"); + dfs.rename(d1, bar_d1); + final Path f3 = new Path(bar_d1, "f3"); + dfs.delete(f3, true); + final Path newfoo = new Path(bar_d1, "foo"); + dfs.rename(foo, newfoo); + final Path f1 = new Path(newfoo, "f1"); + dfs.delete(f1, true); + DFSTestUtil.createFile(dfs, f1, 2 * BLOCK_SIZE, DATA_NUM, 0); + DFSTestUtil.appendFile(dfs, f2, (int) BLOCK_SIZE); + dfs.rename(bar, new Path(dir, "foo")); + } + + /** + * Test the basic functionality. + */ + @Test + public void testSync() throws Exception { + initData(source); + initData(target); + dfs.allowSnapshot(source); + dfs.allowSnapshot(target); + dfs.createSnapshot(source, "s1"); + dfs.createSnapshot(target, "s1"); + + // make changes under source + changeData(source); + dfs.createSnapshot(source, "s2"); + + // do the sync + Assert.assertTrue(DistCpSync.sync(options, conf)); + + // build copy listing + final Path listingPath = new Path("/tmp/META/fileList.seq"); + CopyListing listing = new GlobbedCopyListing(conf, new Credentials()); + listing.buildListing(listingPath, options); + + Map copyListing = getListing(listingPath); + CopyMapper copyMapper = new CopyMapper(); + StubContext stubContext = new StubContext(conf, null, 0); + Mapper.Context context = + stubContext.getContext(); + // Enable append + context.getConfiguration().setBoolean( + DistCpOptionSwitch.APPEND.getConfigLabel(), true); + copyMapper.setup(context); + for (Map.Entry entry : copyListing.entrySet()) { + copyMapper.map(entry.getKey(), entry.getValue(), context); + } + + // verify that we only copied new appended data of f2 and the new file f1 + Assert.assertEquals(BLOCK_SIZE * 3, stubContext.getReporter() + .getCounter(CopyMapper.Counter.BYTESCOPIED).getValue()); + + // verify the source and target now has the same structure + verifyCopy(dfs.getFileStatus(source), dfs.getFileStatus(target), false); + } + + private Map getListing(Path listingPath) + throws Exception { + SequenceFile.Reader reader = new SequenceFile.Reader(conf, + SequenceFile.Reader.file(listingPath)); + Text key = new Text(); + CopyListingFileStatus value = new CopyListingFileStatus(); + Map values = new HashMap<>(); + while (reader.next(key, value)) { + values.put(key, value); + key = new Text(); + value = new CopyListingFileStatus(); + } + return values; + } + + private void verifyCopy(FileStatus s, FileStatus t, boolean compareName) + throws Exception { + Assert.assertEquals(s.isDirectory(), t.isDirectory()); + if (compareName) { + Assert.assertEquals(s.getPath().getName(), t.getPath().getName()); + } + if (!s.isDirectory()) { + // verify the file content is the same + byte[] sbytes = DFSTestUtil.readFileBuffer(dfs, s.getPath()); + byte[] tbytes = DFSTestUtil.readFileBuffer(dfs, t.getPath()); + Assert.assertArrayEquals(sbytes, tbytes); + } else { + FileStatus[] slist = dfs.listStatus(s.getPath()); + FileStatus[] tlist = dfs.listStatus(t.getPath()); + Assert.assertEquals(slist.length, tlist.length); + for (int i = 0; i < slist.length; i++) { + verifyCopy(slist[i], tlist[i], true); + } + } + } + + private void initData2(Path dir) throws Exception { + final Path test = new Path(dir, "test"); + final Path foo = new Path(dir, "foo"); + final Path bar = new Path(dir, "bar"); + final Path f1 = new Path(test, "f1"); + final Path f2 = new Path(foo, "f2"); + final Path f3 = new Path(bar, "f3"); + + DFSTestUtil.createFile(dfs, f1, BLOCK_SIZE, DATA_NUM, 0L); + DFSTestUtil.createFile(dfs, f2, BLOCK_SIZE, DATA_NUM, 1L); + DFSTestUtil.createFile(dfs, f3, BLOCK_SIZE, DATA_NUM, 2L); + } + + private void changeData2(Path dir) throws Exception { + final Path tmpFoo = new Path(dir, "tmpFoo"); + final Path test = new Path(dir, "test"); + final Path foo = new Path(dir, "foo"); + final Path bar = new Path(dir, "bar"); + + dfs.rename(test, tmpFoo); + dfs.rename(foo, test); + dfs.rename(bar, foo); + dfs.rename(tmpFoo, bar); + } + + @Test + public void testSync2() throws Exception { + initData2(source); + initData2(target); + dfs.allowSnapshot(source); + dfs.allowSnapshot(target); + dfs.createSnapshot(source, "s1"); + dfs.createSnapshot(target, "s1"); + + // make changes under source + changeData2(source); + dfs.createSnapshot(source, "s2"); + + SnapshotDiffReport report = dfs.getSnapshotDiffReport(source, "s1", "s2"); + System.out.println(report); + + // do the sync + Assert.assertTrue(DistCpSync.sync(options, conf)); + verifyCopy(dfs.getFileStatus(source), dfs.getFileStatus(target), false); + } + + private void initData3(Path dir) throws Exception { + final Path test = new Path(dir, "test"); + final Path foo = new Path(dir, "foo"); + final Path bar = new Path(dir, "bar"); + final Path f1 = new Path(test, "file"); + final Path f2 = new Path(foo, "file"); + final Path f3 = new Path(bar, "file"); + + DFSTestUtil.createFile(dfs, f1, BLOCK_SIZE, DATA_NUM, 0L); + DFSTestUtil.createFile(dfs, f2, BLOCK_SIZE * 2, DATA_NUM, 1L); + DFSTestUtil.createFile(dfs, f3, BLOCK_SIZE * 3, DATA_NUM, 2L); + } + + private void changeData3(Path dir) throws Exception { + final Path test = new Path(dir, "test"); + final Path foo = new Path(dir, "foo"); + final Path bar = new Path(dir, "bar"); + final Path f1 = new Path(test, "file"); + final Path f2 = new Path(foo, "file"); + final Path f3 = new Path(bar, "file"); + final Path newf1 = new Path(test, "newfile"); + final Path newf2 = new Path(foo, "newfile"); + final Path newf3 = new Path(bar, "newfile"); + + dfs.rename(f1, newf1); + dfs.rename(f2, newf2); + dfs.rename(f3, newf3); + } + + /** + * Test a case where there are multiple source files with the same name + */ + @Test + public void testSync3() throws Exception { + initData3(source); + initData3(target); + dfs.allowSnapshot(source); + dfs.allowSnapshot(target); + dfs.createSnapshot(source, "s1"); + dfs.createSnapshot(target, "s1"); + + // make changes under source + changeData3(source); + dfs.createSnapshot(source, "s2"); + + SnapshotDiffReport report = dfs.getSnapshotDiffReport(source, "s1", "s2"); + System.out.println(report); + + // do the sync + Assert.assertTrue(DistCpSync.sync(options, conf)); + verifyCopy(dfs.getFileStatus(source), dfs.getFileStatus(target), false); + } +} diff --git a/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java b/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java index 30fb25b07c740..cc9da3351d244 100644 --- a/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java +++ b/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java @@ -584,7 +584,7 @@ public void testAppendOption() { // make sure -append is only valid when -update is specified try { - options = OptionsParser.parse(new String[] { "-append", + OptionsParser.parse(new String[] { "-append", "hdfs://localhost:8020/source/first", "hdfs://localhost:8020/target/" }); fail("Append should fail if update option is not specified"); @@ -595,7 +595,7 @@ public void testAppendOption() { // make sure -append is invalid when skipCrc is specified try { - options = OptionsParser.parse(new String[] { + OptionsParser.parse(new String[] { "-append", "-update", "-skipcrccheck", "hdfs://localhost:8020/source/first", "hdfs://localhost:8020/target/" }); @@ -605,4 +605,75 @@ public void testAppendOption() { "Append is disallowed when skipping CRC", e); } } + + @Test + public void testDiffOption() { + Configuration conf = new Configuration(); + Assert.assertFalse(conf.getBoolean(DistCpOptionSwitch.DIFF.getConfigLabel(), + false)); + + DistCpOptions options = OptionsParser.parse(new String[] { "-update", + "-delete", "-diff", "s1", "s2", + "hdfs://localhost:8020/source/first", + "hdfs://localhost:8020/target/" }); + options.appendToConf(conf); + Assert.assertTrue(conf.getBoolean(DistCpOptionSwitch.DIFF.getConfigLabel(), false)); + Assert.assertTrue(options.shouldUseDiff()); + Assert.assertEquals("s1", options.getFromSnapshot()); + Assert.assertEquals("s2", options.getToSnapshot()); + + options = OptionsParser.parse(new String[] { + "-delete", "-diff", "s1", ".", "-update", + "hdfs://localhost:8020/source/first", + "hdfs://localhost:8020/target/" }); + options.appendToConf(conf); + Assert.assertTrue(conf.getBoolean(DistCpOptionSwitch.DIFF.getConfigLabel(), + false)); + Assert.assertTrue(options.shouldUseDiff()); + Assert.assertEquals("s1", options.getFromSnapshot()); + Assert.assertEquals(".", options.getToSnapshot()); + + // -diff requires two option values + try { + OptionsParser.parse(new String[] {"-diff", "s1", "-delete", "-update", + "hdfs://localhost:8020/source/first", + "hdfs://localhost:8020/target/" }); + fail("-diff should fail with only one snapshot name"); + } catch (IllegalArgumentException e) { + GenericTestUtils.assertExceptionContains( + "Must provide both the starting and ending snapshot names", e); + } + + // make sure -diff is only valid when -update and -delete is specified + try { + OptionsParser.parse(new String[] { "-diff", "s1", "s2", + "hdfs://localhost:8020/source/first", + "hdfs://localhost:8020/target/" }); + fail("-diff should fail if -update or -delete option is not specified"); + } catch (IllegalArgumentException e) { + GenericTestUtils.assertExceptionContains( + "Diff is valid only with update and delete options", e); + } + + try { + OptionsParser.parse(new String[] { "-diff", "s1", "s2", "-update", + "hdfs://localhost:8020/source/first", + "hdfs://localhost:8020/target/" }); + fail("-diff should fail if -update or -delete option is not specified"); + } catch (IllegalArgumentException e) { + GenericTestUtils.assertExceptionContains( + "Diff is valid only with update and delete options", e); + } + + try { + OptionsParser.parse(new String[] { "-diff", "s1", "s2", + "-delete", "-overwrite", + "hdfs://localhost:8020/source/first", + "hdfs://localhost:8020/target/" }); + fail("-diff should fail if -update or -delete option is not specified"); + } catch (IllegalArgumentException e) { + GenericTestUtils.assertExceptionContains( + "Diff is valid only with update and delete options", e); + } + } } diff --git a/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java b/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java index 050bfbe2a2011..449ecbf91294e 100644 --- a/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java +++ b/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java @@ -60,7 +60,9 @@ * b) Directory on dfs to archive the logs. * c) The sort/grep patterns for analyzing the files and separator for boundaries. * Usage: - * Logalyzer -archive -archiveDir -analysis -logs -grep -sort -separator + * Logalyzer -archive -archiveDir <directory to archive logs> -analysis + * <directory> -logs <log-list uri> -grep <pattern> -sort + * <col1, col2> -separator <separator> *

      */ diff --git a/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java b/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java index 593c1a462bfb2..7a80e8df6c82b 100644 --- a/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java +++ b/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java @@ -35,7 +35,7 @@ * {@link ResourceUsageEmulatorPlugin} is also configured with a feedback module * i.e a {@link ResourceCalculatorPlugin}, to monitor the current resource * usage. {@link ResourceUsageMetrics} decides the final resource usage value to - * emulate. {@link Progressive} keeps track of the task's progress.

      + * emulate. {@link Progressive} keeps track of the task's progress. * *

      * diff --git a/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestRandomAlgorithm.java b/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestRandomAlgorithm.java index cd5548364212b..4e85ce28922ec 100644 --- a/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestRandomAlgorithm.java +++ b/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestRandomAlgorithm.java @@ -30,8 +30,6 @@ import org.junit.Test; -import com.sun.tools.javac.code.Attribute.Array; - public class TestRandomAlgorithm { private static final int[][] parameters = new int[][] { {5, 1, 1}, diff --git a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/RestClientBindings.java b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/RestClientBindings.java index 25a7e9373d451..d11c369bb2cf5 100644 --- a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/RestClientBindings.java +++ b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/RestClientBindings.java @@ -31,10 +31,10 @@ /** * This class implements the binding logic between Hadoop configurations * and the swift rest client. - *

      + *

      * The swift rest client takes a Properties instance containing * the string values it uses to bind to a swift endpoint. - *

      + *

      * This class extracts the values for a specific filesystem endpoint * and then builds an appropriate Properties file. */ @@ -188,7 +188,7 @@ private static void set(Properties props, String key, String optVal) { /** * Copy a (trimmed) property from the configuration file to the properties file. - *

      + *

      * If marked as required and not found in the configuration, an * exception is raised. * If not required -and missing- then the property will not be set. diff --git a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java index 28f8b47f4ac4f..55dad11124293 100644 --- a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java +++ b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java @@ -1061,10 +1061,9 @@ protected void setup(PutMethod method) throws * Authenticate to Openstack Keystone * As well as returning the access token, the member fields {@link #token}, * {@link #endpointURI} and {@link #objectLocationURI} are set up for re-use. - *

      + *

      * This method is re-entrant -if more than one thread attempts to authenticate * neither will block -but the field values with have those of the last caller. - *

      * * @return authenticated access token */ @@ -1575,6 +1574,7 @@ private static StringRequestEntity toJsonEntity(String data) throws * @param path path to object * @param endpointURI damain url e.g. http://domain.com * @return valid URI for object + * @throws SwiftException */ public static URI pathToURI(SwiftObjectPath path, URI endpointURI) throws SwiftException { @@ -1820,7 +1820,7 @@ public boolean isLocationAware() { /** * Get the blocksize of this filesystem - * @return a blocksize >0 + * @return a blocksize > 0 */ public long getBlocksizeKB() { return blocksizeKB; diff --git a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java index b70f7efef5817..27a572fd842a5 100644 --- a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java +++ b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystem.java @@ -225,10 +225,10 @@ public boolean isDirectory(Path f) throws IOException { * Return an array containing hostnames, offset and size of * portions of the given file. For a nonexistent * file or regions, null will be returned. - *

      + *

      * This call is most helpful with DFS, where it returns * hostnames of machines that contain the given file. - *

      + *

      * The FileSystem will simply return an elt containing 'localhost'. */ @Override @@ -645,7 +645,7 @@ public List getOperationStatistics() { /** * Low level method to do a deep listing of all entries, not stopping * at the next directory entry. This is to let tests be confident that - * recursive deletes &c really are working. + * recursive deletes really are working. * @param path path to recurse down * @param newest ask for the newest data, potentially slower than not. * @return a potentially empty array of file status diff --git a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java index 0138eae412d7d..6d812a0e6ea3a 100644 --- a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java +++ b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java @@ -518,7 +518,7 @@ public boolean objectExists(SwiftObjectPath path) throws IOException { * Rename through copy-and-delete. this is a consequence of the * Swift filesystem using the path as the hash * into the Distributed Hash Table, "the ring" of filenames. - *

      + *

      * Because of the nature of the operation, it is not atomic. * * @param src source file/dir @@ -847,7 +847,7 @@ public static List extractUris(String json, Path path) throws } /** - * Insert a throttled wait if the throttle delay >0 + * Insert a throttled wait if the throttle delay > 0 * @throws InterruptedIOException if interrupted during sleep */ public void throttle() throws InterruptedIOException { @@ -878,7 +878,7 @@ public List getOperationStatistics() { * raised. This lets the caller distinguish a file not found with * other reasons for failure, so handles race conditions in recursive * directory deletes better. - *

      + *

      * The problem being addressed is: caller A requests a recursive directory * of directory /dir ; caller B requests a delete of a file /dir/file, * between caller A enumerating the files contents, and requesting a delete diff --git a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/SwiftTestUtils.java b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/SwiftTestUtils.java index c9e26acf3d4e1..01ec739eda806 100644 --- a/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/SwiftTestUtils.java +++ b/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/util/SwiftTestUtils.java @@ -236,7 +236,7 @@ public static void compareByteArrays(byte[] src, /** * Convert a byte to a character for printing. If the - * byte value is < 32 -and hence unprintable- the byte is + * byte value is < 32 -and hence unprintable- the byte is * returned as a two digit hex value * @param b byte * @return the printable character string diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/InputDemuxer.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/InputDemuxer.java index cd99e1ce989c5..0927a771d3bd1 100644 --- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/InputDemuxer.java +++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/InputDemuxer.java @@ -45,12 +45,12 @@ public interface InputDemuxer extends Closeable { public void bindTo(Path path, Configuration conf) throws IOException; /** - * Get the next pair. The name should preserve the original job + * Get the next <name, input> pair. The name should preserve the original job * history file or job conf file name. The input object should be closed * before calling getNext() again. The old input object would be invalid after * calling getNext() again. * - * @return the next pair. + * @return the next <name, input> pair. */ public Pair getNext() throws IOException; } diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/datatypes/util/MapReduceJobPropertiesParser.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/datatypes/util/MapReduceJobPropertiesParser.java index c2537bec39878..7547eca23192c 100644 --- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/datatypes/util/MapReduceJobPropertiesParser.java +++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/datatypes/util/MapReduceJobPropertiesParser.java @@ -67,8 +67,9 @@ * ignoring user-specific and hard-to-parse keys but also provides a consistent * view for all possible inputs. So if users invoke the * {@link #parseJobProperty(String, String)} API with either - * <"mapreduce.job.user.name", "bob"> or <"user.name", "bob">, then the result - * would be a {@link UserName} {@link DataType} wrapping the user-name "bob". + * <"mapreduce.job.user.name", "bob"> or <"user.name", "bob">, + * then the result would be a {@link UserName} {@link DataType} wrapping + * the user-name "bob". */ @SuppressWarnings("deprecation") public class MapReduceJobPropertiesParser implements JobPropertyParser { diff --git a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/package-info.java b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/package-info.java index b88b37e2c2ed5..2253225234835 100644 --- a/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/package-info.java +++ b/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/package-info.java @@ -41,7 +41,7 @@ * String conf_filename = .. // assume the job configuration filename here * * // construct a list of interesting properties - * List interestedProperties = new ArrayList(); + * List<String> interestedProperties = new ArrayList<String>(); * interestedProperties.add("mapreduce.job.name"); * * JobConfigurationParser jcp = @@ -154,7 +154,7 @@ * TopologyBuilder tb = new TopologyBuilder(); * * // construct a list of interesting properties - * List interestingProperties = new ArrayList(); + * List<String> interestingProperties = new ArrayList%lt;String>(); * // add the interesting properties here * interestingProperties.add("mapreduce.job.name"); * @@ -207,7 +207,7 @@ * JobBuilder jb = new JobBuilder(jobID); * * // construct a list of interesting properties - * List interestingProperties = new ArrayList(); + * List<String> interestingProperties = new ArrayList%lt;String>(); * // add the interesting properties here * interestingProperties.add("mapreduce.job.name"); * @@ -269,7 +269,7 @@ * TopologyBuilder tb = new TopologyBuilder(); * * // construct a list of interesting properties - * List interestingProperties = new ArrayList(); + * List<String> interestingProperties = new ArrayList%lt;String>(); * // add the interesting properties here * interestingProperties.add("mapreduce.job.name"); * diff --git a/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md b/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md index ca179eeffee5f..2cffc86302695 100644 --- a/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md +++ b/hadoop-tools/hadoop-sls/src/site/markdown/SchedulerLoadSimulator.md @@ -43,7 +43,7 @@ The Yarn Scheduler Load Simulator (SLS) is such a tool, which can simulate large o The simulator will exercise the real Yarn `ResourceManager` removing the network factor by simulating `NodeManagers` and `ApplicationMasters` via handling and dispatching `NM`/`AMs` heartbeat events from within the same JVM. To keep tracking of scheduler behavior and performance, a scheduler wrapper will wrap the real scheduler. -The size of the cluster and the application load can be loaded from configuration files, which are generated from job history files directly by adopting [Apache Rumen](https://hadoop.apache.org/docs/stable/rumen.html). +The size of the cluster and the application load can be loaded from configuration files, which are generated from job history files directly by adopting [Apache Rumen](../hadoop-rumen/Rumen.html). The simulator will produce real time metrics while executing, including: diff --git a/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Buffer.java b/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Buffer.java index 50cc1a1912f0d..737d63d11d683 100644 --- a/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Buffer.java +++ b/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Buffer.java @@ -25,10 +25,10 @@ /** * A byte sequence that is used as a Java native type for buffer. - * It is resizable and distinguishes between the count of the seqeunce and + * It is resizable and distinguishes between the count of the sequence and * the current capacity. * - * @deprecated Replaced by Avro. + * @deprecated Replaced by Avro. */ @Deprecated @InterfaceAudience.Public @@ -124,7 +124,7 @@ public int getCapacity() { /** * Change the capacity of the backing storage. - * The data is preserved if newCapacity >= getCount(). + * The data is preserved if newCapacity {@literal >=} getCount(). * @param newCapacity The new capacity in bytes. */ public void setCapacity(int newCapacity) { @@ -162,7 +162,7 @@ public void reset() { public void truncate() { setCapacity(count); } - + /** * Append specified bytes to the buffer. * diff --git a/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java b/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java index d5be59c92c1e8..59e2080c3ed52 100644 --- a/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java +++ b/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java @@ -28,9 +28,9 @@ import org.apache.hadoop.io.WritableUtils; /** - * Various utility functions for Hadooop record I/O runtime. + * Various utility functions for Hadoop record I/O runtime. * - * @deprecated Replaced by Avro. + * @deprecated Replaced by Avro. */ @Deprecated @InterfaceAudience.Public @@ -462,8 +462,8 @@ public static int getVIntSize(long i) { /** * Serializes a long to a binary stream with zero-compressed encoding. - * For -112 <= i <= 127, only one byte is used with the actual value. - * For other values of i, the first byte value indicates whether the + * For {@literal -112 <= i <= 127}, only one byte is used with the actual + * value. For other values of i, the first byte value indicates whether the * long is positive or negative, and the number of bytes that follow. * If the first byte value v is between -113 and -120, the following long * is positive, with number of bytes that follow are -(v+112). diff --git a/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm b/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm index 0b64586591387..7f2412e100989 100644 --- a/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm +++ b/hadoop-tools/hadoop-streaming/src/site/markdown/HadoopStreaming.md.vm @@ -201,7 +201,7 @@ To specify additional local temp directories use: -D mapred.system.dir=/tmp/system -D mapred.temp.dir=/tmp/temp -**Note:** For more details on job configuration parameters see: [mapred-default.xml](./mapred-default.xml) +**Note:** For more details on job configuration parameters see: [mapred-default.xml](../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml) $H4 Specifying Map-Only Jobs @@ -322,7 +322,7 @@ More Usage Examples $H3 Hadoop Partitioner Class -Hadoop has a library class, [KeyFieldBasedPartitioner](../../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html), that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on certain key fields, not the whole keys. For example: +Hadoop has a library class, [KeyFieldBasedPartitioner](../api/org/apache/hadoop/mapred/lib/KeyFieldBasedPartitioner.html), that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on certain key fields, not the whole keys. For example: hadoop jar hadoop-streaming-${project.version}.jar \ -D stream.map.output.field.separator=. \ @@ -372,7 +372,7 @@ Sorting within each partition for the reducer(all 4 fields used for sorting) $H3 Hadoop Comparator Class -Hadoop has a library class, [KeyFieldBasedComparator](../../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html), that is useful for many applications. This class provides a subset of features provided by the Unix/GNU Sort. For example: +Hadoop has a library class, [KeyFieldBasedComparator](../api/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedComparator.html), that is useful for many applications. This class provides a subset of features provided by the Unix/GNU Sort. For example: hadoop jar hadoop-streaming-${project.version}.jar \ -D mapreduce.job.output.key.comparator.class=org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator \ @@ -406,7 +406,7 @@ Sorting output for the reducer (where second field used for sorting) $H3 Hadoop Aggregate Package -Hadoop has a library package called [Aggregate](../../org/apache/hadoop/mapred/lib/aggregate/package-summary.html). Aggregate provides a special reducer class and a special combiner class, and a list of simple aggregators that perform aggregations such as "sum", "max", "min" and so on over a sequence of values. Aggregate allows you to define a mapper plugin class that is expected to generate "aggregatable items" for each input key/value pair of the mappers. The combiner/reducer will aggregate those aggregatable items by invoking the appropriate aggregators. +Hadoop has a library package called [Aggregate](../api/org/apache/hadoop/mapred/lib/aggregate/package-summary.html). Aggregate provides a special reducer class and a special combiner class, and a list of simple aggregators that perform aggregations such as "sum", "max", "min" and so on over a sequence of values. Aggregate allows you to define a mapper plugin class that is expected to generate "aggregatable items" for each input key/value pair of the mappers. The combiner/reducer will aggregate those aggregatable items by invoking the appropriate aggregators. To use Aggregate, simply specify "-reducer aggregate": @@ -441,7 +441,7 @@ The python program myAggregatorForKeyCount.py looks like: $H3 Hadoop Field Selection Class -Hadoop has a library class, [FieldSelectionMapReduce](../../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html), that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example: +Hadoop has a library class, [FieldSelectionMapReduce](../api/org/apache/hadoop/mapred/lib/FieldSelectionMapReduce.html), that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example: hadoop jar hadoop-streaming-${project.version}.jar \ -D mapreduce.map.output.key.field.separator=. \ @@ -480,7 +480,7 @@ As an example, consider the problem of zipping (compressing) a set of files acro $H3 How many reducers should I use? -See MapReduce Tutorial for details: [Reducer](./MapReduceTutorial.html#Reducer) +See MapReduce Tutorial for details: [Reducer](../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Reducer) $H3 If I set up an alias in my shell script, will that work after -mapper? @@ -546,6 +546,13 @@ You can use the record reader StreamXmlRecordReader to process XML documents. Anything found between BEGIN\_STRING and END\_STRING would be treated as one record for map tasks. +The name-value properties that StreamXmlRecordReader understands are: + +* (strings) 'begin' - Characters marking beginning of record, and 'end' - Characters marking end of record. +* (boolean) 'slowmatch' - Toggle to look for begin and end characters, but within CDATA instead of regular tags. Defaults to false. +* (integer) 'lookahead' - Maximum lookahead bytes to sync CDATA when using 'slowmatch', should be larger than 'maxrec'. Defaults to 2*'maxrec'. +* (integer) 'maxrec' - Maximum record size to read between each match during 'slowmatch'. Defaults to 50000 bytes. + $H3 How do I update counters in streaming applications? A streaming process can use the stderr to emit counter information. `reporter:counter:,,` should be sent to stderr to update the counter. @@ -556,4 +563,4 @@ A streaming process can use the stderr to emit status information. To set a stat $H3 How do I get the Job variables in a streaming job's mapper/reducer? -See [Configured Parameters](./MapReduceTutorial.html#Configured_Parameters). During the execution of a streaming job, the names of the "mapred" parameters are transformed. The dots ( . ) become underscores ( \_ ). For example, mapreduce.job.id becomes mapreduce\_job\_id and mapreduce.job.jar becomes mapreduce\_job\_jar. In your code, use the parameter names with the underscores. +See [Configured Parameters](../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Configured_Parameters). During the execution of a streaming job, the names of the "mapred" parameters are transformed. The dots ( . ) become underscores ( \_ ). For example, mapreduce.job.id becomes mapreduce\_job\_id and mapreduce.job.jar becomes mapreduce\_job\_jar. In your code, use the parameter names with the underscores. diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 0850f0bef4716..f6093e2497f1f 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -26,6 +26,9 @@ Trunk - Unreleased YARN-3199. Fair Scheduler documentation improvements (Rohit Agarwal via aw) + YARN-2280. Resource manager web service fields are not accessible + (Krisztian Horvath via aw) + OPTIMIZATIONS BUG FIXES @@ -45,6 +48,81 @@ Trunk - Unreleased YARN-2428. LCE default banned user list should have yarn (Varun Saxena via aw) +Release 2.8.0 - UNRELEASED + + INCOMPATIBLE CHANGES + + NEW FEATURES + + YARN-3345. Add non-exclusive node label API. (Wangda Tan via jianhe) + + IMPROVEMENTS + + YARN-1880. Cleanup TestApplicationClientProtocolOnHA + (ozawa via harsh) + + YARN-3243. CapacityScheduler should pass headroom from parent to children + to make sure ParentQueue obey its capacity limits. (Wangda Tan via jianhe) + + YARN-3273. Improve scheduler UI to facilitate scheduling analysis and + debugging. (Rohith Sharmaks via jianhe) + + YARN-3357. Move TestFifoScheduler to FIFO package. (Rohith Sharmaks + via devaraj) + + YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to + track used-resources-by-label. (Wangda Tan via jianhe) + + YARN-3350. YARN RackResolver spams logs with messages at info level. + (Wilfred Spiegelenburg via junping_du) + + YARN-2868. FairScheduler: Metric for latency to allocate first container + for an application. (Ray Chiang via kasha) + + YARN-3397. yarn rmadmin should skip -failover. (J.Andreina via kasha) + + YARN-3288. Document and fix indentation in the DockerContainerExecutor code + + YARN-2495. Allow admin specify labels from each NM (Distributed + configuration for node label). (Naganarasimha G R via wangda) + + YARN-3258. FairScheduler: Need to add more logging to investigate + allocations. (Anubhav Dhoot via ozawa) + + YARN-3428. Debug log resources to be localized for a container. (kasha) + + OPTIMIZATIONS + + YARN-3339. TestDockerContainerExecutor should pull a single image and not + the entire centos repository. (Ravindra Kumar Naik via raviprak) + + BUG FIXES + + YARN-3197. Confusing log generated by CapacityScheduler. (Varun Saxena + via devaraj) + + YARN-3305. Normalize AM resource request on app submission. (Rohith Sharmaks + via jianhe) + + YARN-3205 FileSystemRMStateStore should disable FileSystem Cache to avoid + get a Filesystem with an old configuration. (Zhihai Xu via ozawa) + + YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) + + YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to + fully qualified path. (Xuan Gong via junping_du) + + YARN-3241. FairScheduler handles "invalid" queue names inconsistently. + (Zhihai Xu via kasha) + + YARN-3383. AdminService should use "warn" instead of "info" to log exception + when operation fails. (Li Lu via wangda) + + YARN-3400. [JDK 8] Build Failure due to unreported exceptions in + RPCUtil (rkanter) + + YARN-3412. RM tests should use MockRM where possible. (kasha) + Release 2.7.0 - UNRELEASED INCOMPATIBLE CHANGES @@ -89,11 +167,17 @@ Release 2.7.0 - UNRELEASED YARN-2217. [YARN-1492] Shared cache client side changes. (Chris Trezzo via kasha) - YARN-2616 [YARN-913] Add CLI client to the registry to list, view - and manipulate entries. (Akshay Radia via stevel) + YARN-2616 [YARN-913] Add CLI client to the registry to list, view + and manipulate entries. (Akshay Radia via stevel) YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) + YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. + (Ryu Kobayashi via ozawa) + + YARN-2786. Created a yarn cluster CLI and seeded with one command for listing + node-labels collection. (Wangda Tan via vinodkv) + IMPROVEMENTS YARN-3005. [JDK7] Use switch statement for String instead of if-else @@ -290,8 +374,6 @@ Release 2.7.0 - UNRELEASED YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager restart. (Jason Lowe via junping_du) - YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha) - YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. (Wangda Tan via jianhe) @@ -348,6 +430,30 @@ Release 2.7.0 - UNRELEASED YARN-3281. Added RMStateStore to StateMachine visualization list. (Chengbing Liu via jianhe) + YARN-3272. Surface container locality info in RM web UI. + (Jian He via wangda) + + YARN-3122. Metrics for container's actual CPU usage. + (Anubhav Dhoot via kasha) + + YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and + Xuan Gong via jianhe) + + YARN-2190. Added CPU and memory limit options to the default container + executor for Windows containers. (Chuan Liu via jianhe) + + YARN-3296. Mark ResourceCalculatorProcessTree class as Public for configurable + resource monitoring. (Hitesh Shah via junping_du) + + YARN-3187. Documentation of Capacity Scheduler Queue mapping based on user + or group. (Gururaj Shetty via jianhe) + + YARN-2854. Updated the documentation of the timeline service and the generic + history service. (Naganarasimha G R via zjshen) + + YARN-2777. Mark the end of individual log in aggregated log. + (Varun Saxena via xgong) + OPTIMIZATIONS YARN-2990. FairScheduler's delay-scheduling always waits for node-local and @@ -689,6 +795,72 @@ Release 2.7.0 - UNRELEASED YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. (Wangda Tan via vinodkv) + YARN-3131. YarnClientImpl should check FAILED and KILLED state in + submitApplication (Chang Li via jlowe) + + YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending + jobs. (Siqi Li via kasha) + + YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving + events for old client. (Zhihai Xu via kasha) + + YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired + (Zhijie Shen via xgong) + + YARN-3275. CapacityScheduler: Preemption happening on non-preemptable + queues (Eric Payne via jlowe) + + YARN-3287. Made TimelineClient put methods do as the correct login context. + (Daryn Sharp and Jonathan Eagles via zjshen) + + YARN-3300. Outstanding_resource_requests table should not be shown in AHS. + (Xuan Gong via jianhe) + + YARN-3295. Fix documentation nits found in markdown conversion. + (Masatake Iwasaki via ozawa) + + YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM + web page. (Xuan Gong via zjshen) + + YARN-3338. Exclude jline dependency from YARN. (Zhijie Shen via xgong) + + YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating + running logs of application when rolling is enabled. (Xuan Gong via vinodkv) + + YARN-3267. Timelineserver applies the ACL rules after applying the limit on + the number of records (Chang Li via jeagles) + + YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work + in ATS / RM web ui. (Naganarasimha G R via xgong) + + YARN-1453. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in + doc comments. (Akira AJISAKA, Andrew Purtell, and Allen Wittenauer via ozawa) + + YARN-3349. Treat all exceptions as failure in + TestFSRMStateStore#testFSRMStateStoreClientRetry. (Zhihai Xu via ozawa) + + YARN-3379. Fixed missing data in localityTable and ResourceRequests table + in RM WebUI. (Xuan Gong via jianhe) + + YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. + (Brahma Reddy Battula via wangda) + + YARN-3384. TestLogAggregationService.verifyContainerLogs fails after + YARN-2777. (Naganarasimha G R via ozawa) + + YARN-3336. FileSystem memory leak in DelegationTokenRenewer. + (Zhihai Xu via cnauroth) + + YARN-3393. Getting application(s) goes wrong when app finishes before + starting the attempt. (Zhijie Shen via xgong) + + YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. + (Varun Saxena via xgong) + + YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and + removing inconsistencies in the default values. (Junping Du and Karthik + Kambatla via vinodkv) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES diff --git a/hadoop-yarn-project/hadoop-yarn/bin/yarn b/hadoop-yarn-project/hadoop-yarn/bin/yarn index f1a06a6d3e23c..fddee46b13f1d 100644 --- a/hadoop-yarn-project/hadoop-yarn/bin/yarn +++ b/hadoop-yarn-project/hadoop-yarn/bin/yarn @@ -25,6 +25,7 @@ function hadoop_usage echo " applicationattempt prints applicationattempt(s) report" echo " classpath prints the class path needed to get the" echo " Hadoop jar and the required libraries" + echo " cluster prints cluster information" echo " container prints container(s) report" echo " daemonlog get/set the log level for each daemon" echo " jar run a jar file" @@ -81,7 +82,12 @@ case "${COMMAND}" in set -- "${COMMAND}" "$@" ;; classpath) - hadoop_do_classpath_subcommand "$@" + hadoop_do_classpath_subcommand CLASS "$@" + ;; + cluster) + CLASS=org.apache.hadoop.yarn.client.cli.ClusterCLI + hadoop_debug "Append YARN_CLIENT_OPTS onto YARN_OPTS" + YARN_OPTS="${YARN_OPTS} ${YARN_CLIENT_OPTS}" ;; daemonlog) CLASS=org.apache.hadoop.log.LogLevel diff --git a/hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh b/hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh index c6963d9883f87..75fb1f8205d32 100644 --- a/hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh +++ b/hadoop-yarn-project/hadoop-yarn/bin/yarn-daemons.sh @@ -55,9 +55,14 @@ hadoop_error "WARNING: Attempting to execute replacement \"yarn --slaves --daemo # we're going to turn this into # yarn --slaves --daemon (start|stop) (rest of options) # -argv=(${HADOOP_USER_PARAMS[@]/start}) -argv=(${argv[@]/stop}) -argv=(${argv[@]/status}) +for (( i = 0; i < ${#HADOOP_USER_PARAMS[@]}; i++ )) +do + if [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^start$ ]] || + [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^stop$ ]] || + [[ "${HADOOP_USER_PARAMS[$i]}" =~ ^status$ ]]; then + unset HADOOP_USER_PARAMS[$i] + fi +done -${yarnscript} --slaves --daemon "${daemonmode}" "${argv[@]}" +${yarnscript} --slaves --daemon "${daemonmode}" "${HADOOP_USER_PARAMS[@]}" diff --git a/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd b/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd index 3f68b16a07083..c29ee539f3745 100644 --- a/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd +++ b/hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd @@ -192,6 +192,11 @@ goto :eof set yarn-command-arguments=%yarn-command% %yarn-command-arguments% goto :eof +:cluster + set CLASS=org.apache.hadoop.yarn.client.cli.ClusterCLI + set YARN_OPTS=%YARN_OPTS% %YARN_CLIENT_OPTS% + goto :eof + :container set CLASS=org.apache.hadoop.yarn.client.cli.ApplicationCLI set YARN_OPTS=%YARN_OPTS% %YARN_CLIENT_OPTS% @@ -312,6 +317,7 @@ goto :eof @echo jar ^ run a jar file @echo application prints application(s) report/kill application @echo applicationattempt prints applicationattempt(s) report + @echo cluster prints cluster information @echo container prints container(s) report @echo node prints node report(s) @echo queue prints queue information diff --git a/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml b/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml index 70f1a71fbcb74..943ecb06285f6 100644 --- a/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml +++ b/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml @@ -62,6 +62,16 @@ + + + + + + + + + + @@ -142,12 +152,22 @@ + + + + + + + + + + @@ -205,6 +225,18 @@ + + + + + + + + + + + + @@ -404,6 +436,11 @@ + + + + + diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java new file mode 100644 index 0000000000000..8234c2fb80a01 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java @@ -0,0 +1,335 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.api; + +import java.io.IOException; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability.Stable; +import org.apache.hadoop.classification.InterfaceStability.Unstable; +import org.apache.hadoop.io.retry.Idempotent; +import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenRequest; +import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse; +import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse; +import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenRequest; +import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenResponse; +import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; +import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.ApplicationReport; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ContainerReport; +import org.apache.hadoop.yarn.api.records.Token; +import org.apache.hadoop.yarn.exceptions.YarnException; + +/** + *

      + * The protocol between clients and the ResourceManager or + * ApplicationHistoryServer to get information on applications, + * application attempts and containers. + *

      + * + */ +@Private +@Unstable +public interface ApplicationBaseProtocol { + + /** + * The interface used by clients to get a report of an Application from the + * ResourceManager or ApplicationHistoryServer. + *

      + * The client, via {@link GetApplicationReportRequest} provides the + * {@link ApplicationId} of the application. + *

      + * In secure mode,the ResourceManager or + * ApplicationHistoryServer verifies access to the application, + * queue etc. before accepting the request. + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with a {@link GetApplicationReportResponse} which includes the + * {@link ApplicationReport} for the application. + *

      + * If the user does not have VIEW_APP access then the following + * fields in the report will be set to stubbed values: + *

        + *
      • host - set to "N/A"
        • + *
      • RPC port - set to -1
        • + *
      • client token - set to "N/A"
        • + *
      • diagnostics - set to "N/A"
        • + *
      • tracking URL - set to "N/A"
        • + *
      • original tracking URL - set to "N/A"
        • + *
      • resource usage report - all values are -1
        • + *
      + * + * @param request + * request for an application report + * @return application report + * @throws YarnException + * @throws IOException + */ + @Public + @Stable + @Idempotent + public GetApplicationReportResponse getApplicationReport( + GetApplicationReportRequest request) throws YarnException, IOException; + + /** + *

      + * The interface used by clients to get a report of Applications matching the + * filters defined by {@link GetApplicationsRequest} in the cluster from the + * ResourceManager or ApplicationHistoryServer. + *

      + * + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with a {@link GetApplicationsResponse} which includes the + * {@link ApplicationReport} for the applications. + *

      + * + *

      + * If the user does not have VIEW_APP access for an application + * then the corresponding report will be filtered as described in + * {@link #getApplicationReport(GetApplicationReportRequest)}. + *

      + * + * @param request + * request for report on applications + * @return report on applications matching the given application types defined + * in the request + * @throws YarnException + * @throws IOException + * @see GetApplicationsRequest + */ + @Public + @Stable + @Idempotent + public GetApplicationsResponse + getApplications(GetApplicationsRequest request) throws YarnException, + IOException; + + /** + * The interface used by clients to get a report of an Application Attempt + * from the ResourceManager or + * ApplicationHistoryServer + *

      + * The client, via {@link GetApplicationAttemptReportRequest} provides the + * {@link ApplicationAttemptId} of the application attempt. + *

      + * In secure mode,the ResourceManager or + * ApplicationHistoryServer verifies access to the method before + * accepting the request. + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with a {@link GetApplicationAttemptReportResponse} which includes + * the {@link ApplicationAttemptReport} for the application attempt. + *

      + * If the user does not have VIEW_APP access then the following + * fields in the report will be set to stubbed values: + *

        + *
      • host
        • + *
      • RPC port
        • + *
      • client token
        • + *
      • diagnostics - set to "N/A"
        • + *
      • tracking URL
        • + *
      + * + * @param request + * request for an application attempt report + * @return application attempt report + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + @Idempotent + public GetApplicationAttemptReportResponse getApplicationAttemptReport( + GetApplicationAttemptReportRequest request) throws YarnException, + IOException; + + /** + *

      + * The interface used by clients to get a report of all Application attempts + * in the cluster from the ResourceManager or + * ApplicationHistoryServer + *

      + * + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with a {@link GetApplicationAttemptsRequest} which includes the + * {@link ApplicationAttemptReport} for all the applications attempts of a + * specified application attempt. + *

      + * + *

      + * If the user does not have VIEW_APP access for an application + * then the corresponding report will be filtered as described in + * {@link #getApplicationAttemptReport(GetApplicationAttemptReportRequest)}. + *

      + * + * @param request + * request for reports on all application attempts of an application + * @return reports on all application attempts of an application + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + @Idempotent + public GetApplicationAttemptsResponse getApplicationAttempts( + GetApplicationAttemptsRequest request) throws YarnException, IOException; + + /** + *

      + * The interface used by clients to get a report of an Container from the + * ResourceManager or ApplicationHistoryServer + *

      + * + *

      + * The client, via {@link GetContainerReportRequest} provides the + * {@link ContainerId} of the container. + *

      + * + *

      + * In secure mode,the ResourceManager or + * ApplicationHistoryServer verifies access to the method before + * accepting the request. + *

      + * + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with a {@link GetContainerReportResponse} which includes the + * {@link ContainerReport} for the container. + *

      + * + * @param request + * request for a container report + * @return container report + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + @Idempotent + public GetContainerReportResponse getContainerReport( + GetContainerReportRequest request) throws YarnException, IOException; + + /** + *

      + * The interface used by clients to get a report of Containers for an + * application attempt from the ResourceManager or + * ApplicationHistoryServer + *

      + * + *

      + * The client, via {@link GetContainersRequest} provides the + * {@link ApplicationAttemptId} of the application attempt. + *

      + * + *

      + * In secure mode,the ResourceManager or + * ApplicationHistoryServer verifies access to the method before + * accepting the request. + *

      + * + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with a {@link GetContainersResponse} which includes a list of + * {@link ContainerReport} for all the containers of a specific application + * attempt. + *

      + * + * @param request + * request for a list of container reports of an application attempt. + * @return reports on all containers of an application attempt + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + @Idempotent + public GetContainersResponse getContainers(GetContainersRequest request) + throws YarnException, IOException; + + /** + *

      + * The interface used by clients to get delegation token, enabling the + * containers to be able to talk to the service using those tokens. + * + *

      + * The ResourceManager or ApplicationHistoryServer + * responds with the delegation {@link Token} that can be used by the client + * to speak to this service. + * + * @param request + * request to get a delegation token for the client. + * @return delegation token that can be used to talk to this service + * @throws YarnException + * @throws IOException + */ + @Public + @Stable + @Idempotent + public GetDelegationTokenResponse getDelegationToken( + GetDelegationTokenRequest request) throws YarnException, IOException; + + /** + * Renew an existing delegation {@link Token}. + * + * @param request + * the delegation token to be renewed. + * @return the new expiry time for the delegation token. + * @throws YarnException + * @throws IOException + */ + @Private + @Unstable + @Idempotent + public RenewDelegationTokenResponse renewDelegationToken( + RenewDelegationTokenRequest request) throws YarnException, IOException; + + /** + * Cancel an existing delegation {@link Token}. + * + * @param request + * the delegation token to be cancelled. + * @return an empty response. + * @throws YarnException + * @throws IOException + */ + @Private + @Unstable + @Idempotent + public CancelDelegationTokenResponse cancelDelegationToken( + CancelDelegationTokenRequest request) throws YarnException, IOException; + +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java index b5f5cc0725776..8b9937b1ce701 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java @@ -20,33 +20,17 @@ import java.io.IOException; -import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceAudience.Public; import org.apache.hadoop.classification.InterfaceStability.Stable; import org.apache.hadoop.classification.InterfaceStability.Unstable; import org.apache.hadoop.io.retry.Idempotent; -import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenRequest; -import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsResponse; import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsResponse; import org.apache.hadoop.yarn.api.protocolrecords.GetClusterMetricsRequest; import org.apache.hadoop.yarn.api.protocolrecords.GetClusterMetricsResponse; import org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodeLabelsRequest; import org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodeLabelsResponse; import org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodesRequest; import org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodesResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse; import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesRequest; import org.apache.hadoop.yarn.api.protocolrecords.GetLabelsToNodesResponse; import org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationRequest; @@ -61,8 +45,6 @@ import org.apache.hadoop.yarn.api.protocolrecords.KillApplicationResponse; import org.apache.hadoop.yarn.api.protocolrecords.MoveApplicationAcrossQueuesRequest; import org.apache.hadoop.yarn.api.protocolrecords.MoveApplicationAcrossQueuesResponse; -import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenRequest; -import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenResponse; import org.apache.hadoop.yarn.api.protocolrecords.ReservationDeleteRequest; import org.apache.hadoop.yarn.api.protocolrecords.ReservationDeleteResponse; import org.apache.hadoop.yarn.api.protocolrecords.ReservationSubmissionRequest; @@ -71,19 +53,13 @@ import org.apache.hadoop.yarn.api.protocolrecords.ReservationUpdateResponse; import org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationRequest; import org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationResponse; -import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; -import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.ApplicationReport; import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext; -import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerLaunchContext; -import org.apache.hadoop.yarn.api.records.ContainerReport; import org.apache.hadoop.yarn.api.records.NodeReport; import org.apache.hadoop.yarn.api.records.ReservationId; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.api.records.ResourceRequest; -import org.apache.hadoop.yarn.api.records.Token; import org.apache.hadoop.yarn.api.records.YarnClusterMetrics; import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException; import org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException; @@ -96,7 +72,7 @@ */ @Public @Stable -public interface ApplicationClientProtocol { +public interface ApplicationClientProtocol extends ApplicationBaseProtocol { /** *

      The interface used by clients to obtain a new {@link ApplicationId} for * submitting new applications.

      @@ -159,9 +135,6 @@ public GetNewApplicationResponse getNewApplication( * @return (empty) response on accepting the submission * @throws YarnException * @throws IOException - * @throws InvalidResourceRequestException - * The exception is thrown when a {@link ResourceRequest} is out of - * the range of the configured lower and upper resource boundaries. * @see #getNewApplication(GetNewApplicationRequest) */ @Public @@ -198,44 +171,6 @@ public KillApplicationResponse forceKillApplication( KillApplicationRequest request) throws YarnException, IOException; - /** - *

      The interface used by clients to get a report of an Application from - * the ResourceManager.

      - * - *

      The client, via {@link GetApplicationReportRequest} provides the - * {@link ApplicationId} of the application.

      - * - *

      In secure mode,the ResourceManager verifies access to the - * application, queue etc. before accepting the request.

      - * - *

      The ResourceManager responds with a - * {@link GetApplicationReportResponse} which includes the - * {@link ApplicationReport} for the application.

      - * - *

      If the user does not have VIEW_APP access then the - * following fields in the report will be set to stubbed values: - *

        - *
      • host - set to "N/A"
        • - *
      • RPC port - set to -1
        • - *
      • client token - set to "N/A"
        • - *
      • diagnostics - set to "N/A"
        • - *
      • tracking URL - set to "N/A"
        • - *
      • original tracking URL - set to "N/A"
        • - *
      • resource usage report - all values are -1
        • - *

      - * - * @param request request for an application report - * @return application report - * @throws YarnException - * @throws IOException - */ - @Public - @Stable - @Idempotent - public GetApplicationReportResponse getApplicationReport( - GetApplicationReportRequest request) - throws YarnException, IOException; - /** *

      The interface used by clients to get metrics about the cluster from * the ResourceManager.

      @@ -256,35 +191,7 @@ public GetApplicationReportResponse getApplicationReport( public GetClusterMetricsResponse getClusterMetrics( GetClusterMetricsRequest request) throws YarnException, IOException; - - /** - *

      The interface used by clients to get a report of Applications - * matching the filters defined by {@link GetApplicationsRequest} - * in the cluster from the ResourceManager.

      - * - *

      The ResourceManager responds with a - * {@link GetApplicationsResponse} which includes the - * {@link ApplicationReport} for the applications.

      - * - *

      If the user does not have VIEW_APP access for an - * application then the corresponding report will be filtered as - * described in {@link #getApplicationReport(GetApplicationReportRequest)}. - *

      - * - * @param request request for report on applications - * @return report on applications matching the given application types - * defined in the request - * @throws YarnException - * @throws IOException - * @see GetApplicationsRequest - */ - @Public - @Stable - @Idempotent - public GetApplicationsResponse getApplications( - GetApplicationsRequest request) - throws YarnException, IOException; - + /** *

      The interface used by clients to get a report of all nodes * in the cluster from the ResourceManager.

      @@ -346,56 +253,7 @@ public GetQueueInfoResponse getQueueInfo( public GetQueueUserAclsInfoResponse getQueueUserAcls( GetQueueUserAclsInfoRequest request) throws YarnException, IOException; - - /** - *

      The interface used by clients to get delegation token, enabling the - * containers to be able to talk to the service using those tokens. - * - *

      The ResourceManager responds with the delegation - * {@link Token} that can be used by the client to speak to this - * service. - * @param request request to get a delegation token for the client. - * @return delegation token that can be used to talk to this service - * @throws YarnException - * @throws IOException - */ - @Public - @Stable - @Idempotent - public GetDelegationTokenResponse getDelegationToken( - GetDelegationTokenRequest request) - throws YarnException, IOException; - - /** - * Renew an existing delegation {@link Token}. - * - * @param request the delegation token to be renewed. - * @return the new expiry time for the delegation token. - * @throws YarnException - * @throws IOException - */ - @Private - @Unstable - @Idempotent - public RenewDelegationTokenResponse renewDelegationToken( - RenewDelegationTokenRequest request) throws YarnException, - IOException; - /** - * Cancel an existing delegation {@link Token}. - * - * @param request the delegation token to be cancelled. - * @return an empty response. - * @throws YarnException - * @throws IOException - */ - @Private - @Unstable - @Idempotent - public CancelDelegationTokenResponse cancelDelegationToken( - CancelDelegationTokenRequest request) throws YarnException, - IOException; - /** * Move an application to a new queue. * @@ -410,153 +268,6 @@ public CancelDelegationTokenResponse cancelDelegationToken( public MoveApplicationAcrossQueuesResponse moveApplicationAcrossQueues( MoveApplicationAcrossQueuesRequest request) throws YarnException, IOException; - /** - *

      - * The interface used by clients to get a report of an Application Attempt - * from the ResourceManager - *

      - * - *

      - * The client, via {@link GetApplicationAttemptReportRequest} provides the - * {@link ApplicationAttemptId} of the application attempt. - *

      - * - *

      - * In secure mode,the ResourceManager verifies access to - * the method before accepting the request. - *

      - * - *

      - * The ResourceManager responds with a - * {@link GetApplicationAttemptReportResponse} which includes the - * {@link ApplicationAttemptReport} for the application attempt. - *

      - * - *

      - * If the user does not have VIEW_APP access then the following - * fields in the report will be set to stubbed values: - *

        - *
      • host
        • - *
      • RPC port
        • - *
      • client token
        • - *
      • diagnostics - set to "N/A"
        • - *
      • tracking URL
        • - *
      - *

      - * - * @param request - * request for an application attempt report - * @return application attempt report - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - @Idempotent - public GetApplicationAttemptReportResponse getApplicationAttemptReport( - GetApplicationAttemptReportRequest request) throws YarnException, - IOException; - - /** - *

      - * The interface used by clients to get a report of all Application attempts - * in the cluster from the ResourceManager - *

      - * - *

      - * The ResourceManager responds with a - * {@link GetApplicationAttemptsRequest} which includes the - * {@link ApplicationAttemptReport} for all the applications attempts of a - * specified application attempt. - *

      - * - *

      - * If the user does not have VIEW_APP access for an application - * then the corresponding report will be filtered as described in - * {@link #getApplicationAttemptReport(GetApplicationAttemptReportRequest)}. - *

      - * - * @param request - * request for reports on all application attempts of an application - * @return reports on all application attempts of an application - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - @Idempotent - public GetApplicationAttemptsResponse getApplicationAttempts( - GetApplicationAttemptsRequest request) throws YarnException, IOException; - - /** - *

      - * The interface used by clients to get a report of an Container from the - * ResourceManager - *

      - * - *

      - * The client, via {@link GetContainerReportRequest} provides the - * {@link ContainerId} of the container. - *

      - * - *

      - * In secure mode,the ResourceManager verifies access to the - * method before accepting the request. - *

      - * - *

      - * The ResourceManager responds with a - * {@link GetContainerReportResponse} which includes the - * {@link ContainerReport} for the container. - *

      - * - * @param request - * request for a container report - * @return container report - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - @Idempotent - public GetContainerReportResponse getContainerReport( - GetContainerReportRequest request) throws YarnException, IOException; - - /** - *

      - * The interface used by clients to get a report of Containers for an - * application attempt from the ResourceManager - *

      - * - *

      - * The client, via {@link GetContainersRequest} provides the - * {@link ApplicationAttemptId} of the application attempt. - *

      - * - *

      - * In secure mode,the ResourceManager verifies access to the - * method before accepting the request. - *

      - * - *

      - * The ResourceManager responds with a - * {@link GetContainersResponse} which includes a list of - * {@link ContainerReport} for all the containers of a specific application - * attempt. - *

      - * - * @param request - * request for a list of container reports of an application attempt. - * @return reports on all containers of an application attempt - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - @Idempotent - public GetContainersResponse getContainers(GetContainersRequest request) - throws YarnException, IOException; - /** *

      * The interface used by clients to submit a new reservation to the diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationHistoryProtocol.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationHistoryProtocol.java index 0bfd2eda47edc..fc8e8850f6856 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationHistoryProtocol.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationHistoryProtocol.java @@ -18,37 +18,8 @@ package org.apache.hadoop.yarn.api; -import java.io.IOException; - -import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceAudience.Public; import org.apache.hadoop.classification.InterfaceStability.Unstable; -import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenRequest; -import org.apache.hadoop.yarn.api.protocolrecords.CancelDelegationTokenResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetContainersResponse; -import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenRequest; -import org.apache.hadoop.yarn.api.protocolrecords.GetDelegationTokenResponse; -import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenRequest; -import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenResponse; -import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; -import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; -import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.ApplicationReport; -import org.apache.hadoop.yarn.api.records.ContainerId; -import org.apache.hadoop.yarn.api.records.ContainerReport; -import org.apache.hadoop.yarn.api.records.Token; -import org.apache.hadoop.yarn.exceptions.YarnException; /** *

      @@ -58,277 +29,5 @@ */ @Public @Unstable -public interface ApplicationHistoryProtocol { - - /** - *

      - * The interface used by clients to get a report of an Application from the - * ResourceManager. - *

      - * - *

      - * The client, via {@link GetApplicationReportRequest} provides the - * {@link ApplicationId} of the application. - *

      - * - *

      - * In secure mode,the ApplicationHistoryServer verifies access to - * the application, queue etc. before accepting the request. - *

      - * - *

      - * The ApplicationHistoryServer responds with a - * {@link GetApplicationReportResponse} which includes the - * {@link ApplicationReport} for the application. - *

      - * - *

      - * If the user does not have VIEW_APP access then the following - * fields in the report will be set to stubbed values: - *

        - *
      • host - set to "N/A"
        • - *
      • RPC port - set to -1
        • - *
      • client token - set to "N/A"
        • - *
      • diagnostics - set to "N/A"
        • - *
      • tracking URL - set to "N/A"
        • - *
      • original tracking URL - set to "N/A"
        • - *
      • resource usage report - all values are -1
        • - *
      - *

      - * - * @param request - * request for an application report - * @return application report - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetApplicationReportResponse getApplicationReport( - GetApplicationReportRequest request) throws YarnException, IOException; - - /** - *

      - * The interface used by clients to get a report of all Applications in the - * cluster from the ApplicationHistoryServer. - *

      - * - *

      - * The ApplicationHistoryServer responds with a - * {@link GetApplicationsResponse} which includes a list of - * {@link ApplicationReport} for all the applications. - *

      - * - *

      - * If the user does not have VIEW_APP access for an application - * then the corresponding report will be filtered as described in - * {@link #getApplicationReport(GetApplicationReportRequest)}. - *

      - * - * @param request - * request for reports on all the applications - * @return report on applications matching the given application types defined - * in the request - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetApplicationsResponse - getApplications(GetApplicationsRequest request) throws YarnException, - IOException; - - /** - *

      - * The interface used by clients to get a report of an Application Attempt - * from the ApplicationHistoryServer. - *

      - * - *

      - * The client, via {@link GetApplicationAttemptReportRequest} provides the - * {@link ApplicationAttemptId} of the application attempt. - *

      - * - *

      - * In secure mode,the ApplicationHistoryServer verifies access to - * the method before accepting the request. - *

      - * - *

      - * The ApplicationHistoryServer responds with a - * {@link GetApplicationAttemptReportResponse} which includes the - * {@link ApplicationAttemptReport} for the application attempt. - *

      - * - *

      - * If the user does not have VIEW_APP access then the following - * fields in the report will be set to stubbed values: - *

        - *
      • host
        • - *
      • RPC port
        • - *
      • client token
        • - *
      • diagnostics - set to "N/A"
        • - *
      • tracking URL
        • - *
      - *

      - * - * @param request - * request for an application attempt report - * @return application attempt report - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetApplicationAttemptReportResponse getApplicationAttemptReport( - GetApplicationAttemptReportRequest request) throws YarnException, - IOException; - - /** - *

      - * The interface used by clients to get a report of all Application attempts - * in the cluster from the ApplicationHistoryServer. - *

      - * - *

      - * The ApplicationHistoryServer responds with a - * {@link GetApplicationAttemptsRequest} which includes the - * {@link ApplicationAttemptReport} for all the applications attempts of a - * specified application attempt. - *

      - * - *

      - * If the user does not have VIEW_APP access for an application - * then the corresponding report will be filtered as described in - * {@link #getApplicationAttemptReport(GetApplicationAttemptReportRequest)}. - *

      - * - * @param request - * request for reports on all application attempts of an application - * @return reports on all application attempts of an application - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetApplicationAttemptsResponse getApplicationAttempts( - GetApplicationAttemptsRequest request) throws YarnException, IOException; - - /** - *

      - * The interface used by clients to get a report of an Container from the - * ApplicationHistoryServer. - *

      - * - *

      - * The client, via {@link GetContainerReportRequest} provides the - * {@link ContainerId} of the container. - *

      - * - *

      - * In secure mode,the ApplicationHistoryServer verifies access to - * the method before accepting the request. - *

      - * - *

      - * The ApplicationHistoryServer responds with a - * {@link GetContainerReportResponse} which includes the - * {@link ContainerReport} for the container. - *

      - * - * @param request - * request for a container report - * @return container report - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetContainerReportResponse getContainerReport( - GetContainerReportRequest request) throws YarnException, IOException; - - /** - *

      - * The interface used by clients to get a report of Containers for an - * application attempt from the ApplciationHistoryServer. - *

      - * - *

      - * The client, via {@link GetContainersRequest} provides the - * {@link ApplicationAttemptId} of the application attempt. - *

      - * - *

      - * In secure mode,the ApplicationHistoryServer verifies access to - * the method before accepting the request. - *

      - * - *

      - * The ApplicationHistoryServer responds with a - * {@link GetContainersResponse} which includes a list of - * {@link ContainerReport} for all the containers of a specific application - * attempt. - *

      - * - * @param request - * request for a list of container reports of an application attempt. - * @return reports on all containers of an application attempt - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetContainersResponse getContainers(GetContainersRequest request) - throws YarnException, IOException; - - /** - *

      - * The interface used by clients to get delegation token, enabling the - * containers to be able to talk to the service using those tokens. - *

      - * - *

      - * The ApplicationHistoryServer responds with the delegation - * token {@link Token} that can be used by the client to speak to this - * service. - *

      - * - * @param request - * request to get a delegation token for the client. - * @return delegation token that can be used to talk to this service - * @throws YarnException - * @throws IOException - */ - @Public - @Unstable - public GetDelegationTokenResponse getDelegationToken( - GetDelegationTokenRequest request) throws YarnException, IOException; - - /** - * Renew an existing delegation token. - * - * @param request - * the delegation token to be renewed. - * @return the new expiry time for the delegation token. - * @throws YarnException - * @throws IOException - */ - @Private - @Unstable - public RenewDelegationTokenResponse renewDelegationToken( - RenewDelegationTokenRequest request) throws YarnException, IOException; - - /** - * Cancel an existing delegation token. - * - * @param request - * the delegation token to be cancelled. - * @return an empty response. - * @throws YarnException - * @throws IOException - */ - @Private - @Unstable - public CancelDelegationTokenResponse cancelDelegationToken( - CancelDelegationTokenRequest request) throws YarnException, IOException; +public interface ApplicationHistoryProtocol extends ApplicationBaseProtocol { } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java index 62316a64b692e..2458d9ba43173 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java @@ -35,19 +35,18 @@ * ResourceManager to obtain resources in the cluster.

      * *

      The request includes: - *

        - *
      • A response id to track duplicate responses.
        • - *
      • Progress information.
        • - *
      • - * A list of {@link ResourceRequest} to inform the - * ResourceManager about the application's - * resource requirements. - *
        • - *
      • - * A list of unused {@link Container} which are being returned. - *
        • - *
      - *

      + *
        + *
      • A response id to track duplicate responses.
        • + *
      • Progress information.
        • + *
      • + * A list of {@link ResourceRequest} to inform the + * ResourceManager about the application's + * resource requirements. + *
        • + *
      • + * A list of unused {@link Container} which are being returned. + *
        • + *
      * * @see ApplicationMasterProtocol#allocate(AllocateRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java index 46ac6428a04c2..c4fdb79f4e181 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java @@ -39,27 +39,27 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      The response sent by the ResourceManager the - * ApplicationMaster during resource negotiation.

      - * - *

      The response, includes: - *

        - *
      • Response ID to track duplicate responses.
        • - *
      • - * An AMCommand sent by ResourceManager to let the ApplicationMaster - * take some actions (resync, shutdown etc.). - *
      • A list of newly allocated {@link Container}.
        • - *
      • A list of completed {@link Container}s' statuses.
        • - *
      • - * The available headroom for resources in the cluster for the - * application. - *
        • - *
      • A list of nodes whose status has been updated.
        • - *
      • The number of available nodes in a cluster.
        • - *
      • A description of resources requested back by the cluster
        • - *
      • AMRMToken, if AMRMToken has been rolled over
        • - *
      - *

      + * The response sent by the ResourceManager the + * ApplicationMaster during resource negotiation. + *

      + * The response, includes: + *

        + *
      • Response ID to track duplicate responses.
        • + *
      • + * An AMCommand sent by ResourceManager to let the + * {@code ApplicationMaster} take some actions (resync, shutdown etc.). + *
        • + *
      • A list of newly allocated {@link Container}.
        • + *
      • A list of completed {@link Container}s' statuses.
        • + *
      • + * The available headroom for resources in the cluster for the + * application. + *
        • + *
      • A list of nodes whose status has been updated.
        • + *
      • The number of available nodes in a cluster.
        • + *
      • A description of resources requested back by the cluster
        • + *
      • AMRMToken, if AMRMToken has been rolled over
        • + *
      * * @see ApplicationMasterProtocol#allocate(AllocateRequest) */ @@ -220,16 +220,16 @@ public static AllocateResponse newInstance(int responseId, public abstract void setNumClusterNodes(int numNodes); /** - *

      Get the description of containers owned by the AM, but requested back by + * Get the description of containers owned by the AM, but requested back by * the cluster. Note that the RM may have an inconsistent view of the * resources owned by the AM. These messages are advisory, and the AM may - * elect to ignore them.

      - * - *

      The message is a snapshot of the resources the RM wants back from the AM. + * elect to ignore them. + *

      + * The message is a snapshot of the resources the RM wants back from the AM. * While demand persists, the RM will repeat its request; applications should - * not interpret each message as a request for additional + * not interpret each message as a request for additional * resources on top of previous messages. Resources requested consistently - * over some duration may be forcibly killed by the RM.

      + * over some duration may be forcibly killed by the RM. * * @return A specification of the resources to reclaim from this AM. */ @@ -242,15 +242,17 @@ public static AllocateResponse newInstance(int responseId, public abstract void setPreemptionMessage(PreemptionMessage request); /** - *

      Get the list of NMTokens required for communicating with NM. New NMTokens - * issued only if

      - *

      1) AM is receiving first container on underlying NodeManager.
      + * Get the list of NMTokens required for communicating with NM. New NMTokens + * issued only if + *

      + * 1) AM is receiving first container on underlying NodeManager.
      * OR
      * 2) NMToken master key rolled over in ResourceManager and AM is getting new - * container on the same underlying NodeManager.

      - *

      AM will receive one NMToken per NM irrespective of the number of containers + * container on the same underlying NodeManager. + *

      + * AM will receive one NMToken per NM irrespective of the number of containers * issued on same NM. AM is expected to store these tokens until issued a - * new token for the same NM.

      + * new token for the same NM. */ @Public @Stable diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterRequest.java index 15c36802bbe7d..cbbe9c5ccece1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterRequest.java @@ -25,19 +25,18 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      The finalization request sent by the ApplicationMaster to - * inform the ResourceManager about its completion.

      - * - *

      The final request includes details such: - *

        - *
      • Final state of the ApplicationMaster
        • - *
      • - * Diagnostic information in case of failure of the - * ApplicationMaster - *
        • - *
      • Tracking URL
        • - *
      - *

      + * The finalization request sent by the {@code ApplicationMaster} to + * inform the {@code ResourceManager} about its completion. + *

      + * The final request includes details such: + *

        + *
      • Final state of the {@code ApplicationMaster}
        • + *
      • + * Diagnostic information in case of failure of the + * {@code ApplicationMaster} + *
        • + *
      • Tracking URL
        • + *
      * * @see ApplicationMasterProtocol#finishApplicationMaster(FinishApplicationMasterRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterResponse.java index 8de2c73d88a24..6647a10bb040c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterResponse.java @@ -26,22 +26,19 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      * The response sent by the ResourceManager to a * ApplicationMaster on it's completion. - *

      - * *

      * The response, includes: *

        *
      • A flag which indicates that the application has successfully unregistered * with the RM and the application can safely stop.
        • *
      - *

      + *

      * Note: The flag indicates whether the application has successfully * unregistered and is safe to stop. The application may stop after the flag is * true. If the application stops before the flag is true then the RM may retry - * the application . + * the application. * * @see ApplicationMasterProtocol#finishApplicationMaster(FinishApplicationMasterRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java index 7fc58d67aefd2..35392e44d5233 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java @@ -34,7 +34,6 @@ *

      The request from clients to get a report of Applications * in the cluster from the ResourceManager.

      * - * * @see ApplicationClientProtocol#getApplications(GetApplicationsRequest) */ @Public diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterMetricsResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterMetricsResponse.java index 6329aac21b405..18a08077de381 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterMetricsResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterMetricsResponse.java @@ -27,8 +27,8 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      The response sent by the ResourceManager to a client - * requesting cluster metrics.

      + * The response sent by the ResourceManager to a client + * requesting cluster metrics. * * @see YarnClusterMetrics * @see ApplicationClientProtocol#getClusterMetrics(GetClusterMetricsRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesRequest.java index f9f77a3af2fbb..60c63ca033279 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesRequest.java @@ -28,11 +28,9 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      * The request sent by the ApplicationMaster to the * NodeManager to get {@link ContainerStatus} of requested * containers. - *

      * * @see ContainerManagementProtocol#getContainerStatuses(GetContainerStatusesRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesResponse.java index b0a0f0e48826d..68e6a8c2e3d55 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusesResponse.java @@ -32,11 +32,9 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      * The response sent by the NodeManager to the * ApplicationMaster when asked to obtain the * ContainerStatus of requested containers. - *

      * * @see ContainerManagementProtocol#getContainerStatuses(GetContainerStatusesRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoRequest.java index df3342fedcb71..0e33e21d211c4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoRequest.java @@ -63,7 +63,7 @@ public abstract class GetQueueInfoRequest { public abstract void setQueueName(String queueName); /** - * Is information about active applications required? + * Is information about active applications required? * @return true if applications' information is to be included, * else false */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoResponse.java index ea6cb7b6c47c4..b97a5a3ecc1c6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoResponse.java @@ -27,12 +27,11 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      The response sent by the ResourceManager to a client - * requesting information about queues in the system.

      - * - *

      The response includes a {@link QueueInfo} which has details such as - * queue name, used/total capacities, running applications, child queues etc - * .

      + * The response sent by the {@code ResourceManager} to a client + * requesting information about queues in the system. + *

      + * The response includes a {@link QueueInfo} which has details such as + * queue name, used/total capacities, running applications, child queues etc. * * @see QueueInfo * @see ApplicationClientProtocol#getQueueInfo(GetQueueInfoRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/KillApplicationResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/KillApplicationResponse.java index 77bb71d67969c..7225bf59fbe3f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/KillApplicationResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/KillApplicationResponse.java @@ -26,21 +26,20 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      * The response sent by the ResourceManager to the client aborting * a submitted application. - *

      *

      * The response, includes: *

        - *
      • A flag which indicates that the process of killing the application is - * completed or not.
        • + *
      • + * A flag which indicates that the process of killing the application is + * completed or not. + *
        • *
      * Note: user is recommended to wait until this flag becomes true, otherwise if * the ResourceManager crashes before the process of killing the * application is completed, the ResourceManager may retry this * application on recovery. - *

      * * @see ApplicationClientProtocol#forceKillApplication(KillApplicationRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterRequest.java index 6b0185461fd18..395e190ff5916 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterRequest.java @@ -24,16 +24,15 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      The request sent by the ApplicationMaster to - * ResourceManager on registration.

      - * - *

      The registration includes details such as: - *

        - *
      • Hostname on which the AM is running.
        • - *
      • RPC Port
        • - *
      • Tracking URL
        • - *
      - *

      + * The request sent by the {@code ApplicationMaster} to {@code ResourceManager} + * on registration. + *

      + * The registration includes details such as: + *

        + *
      • Hostname on which the AM is running.
        • + *
      • RPC Port
        • + *
      • Tracking URL
        • + *
      * * @see ApplicationMasterProtocol#registerApplicationMaster(RegisterApplicationMasterRequest) */ @@ -83,20 +82,20 @@ public static RegisterApplicationMasterRequest newInstance(String host, public abstract void setHost(String host); /** - * Get the RPC port on which the ApplicationMaster - * is responding. - * @return the RPC port on which the ApplicationMaster is - * responding + * Get the RPC port on which the {@code ApplicationMaster} is + * responding. + * @return the RPC port on which the {@code ApplicationMaster} + * is responding */ @Public @Stable public abstract int getRpcPort(); /** - * Set the RPC port on which the ApplicationMaster is + * Set the RPC port on which the {@code ApplicationMaster} is * responding. - * @param port RPC port on which the ApplicationMaster is - * responding + * @param port RPC port on which the {@code ApplicationMaster} + * is responding */ @Public @Stable diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java index 33daf2812308a..1a51ba6c0086e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java @@ -36,16 +36,15 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      The response sent by the ResourceManager to a new - * ApplicationMaster on registration.

      - * - *

      The response contains critical details such as: + * The response sent by the {@code ResourceManager} to a new + * {@code ApplicationMaster} on registration. + *

      + * The response contains critical details such as: *

        *
      • Maximum capability for allocated resources in the cluster.
        • - *
      • ApplicationACLs for the application.
        • + *
      • {@code ApplicationACL}s for the application.
        • *
      • ClientToAMToken master key.
        • *
      - *

      * * @see ApplicationMasterProtocol#registerApplicationMaster(RegisterApplicationMasterRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerRequest.java index 1dcefb2e3ea01..50179a962bad2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerRequest.java @@ -74,10 +74,11 @@ public static StartContainerRequest newInstance( public abstract void setContainerLaunchContext(ContainerLaunchContext context); /** - *

      Get the container token to be used for authorization during starting - * container.

      - *

      Note: {@link NMToken} will be used for authenticating communication with - * NodeManager.

      + * Get the container token to be used for authorization during starting + * container. + *

      + * Note: {@link NMToken} will be used for authenticating communication with + * {@code NodeManager}. * @return the container token to be used for authorization during starting * container. * @see NMToken diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptReport.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptReport.java index 53c18ae3ceff7..b7f9c1b3a04f6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptReport.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptReport.java @@ -24,24 +24,19 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      - * ApplicationAttemptReport is a report of an application attempt. - *

      - * + * {@code ApplicationAttemptReport} is a report of an application attempt. *

      * It includes details such as: *

        - *
      • {@link ApplicationAttemptId} of the application.
        • - *
      • Host on which the ApplicationMaster of this attempt is - * running.
        • - *
      • RPC port of the ApplicationMaster of this attempt.
        • - *
      • Tracking URL.
        • - *
      • Diagnostic information in case of errors.
        • - *
      • {@link YarnApplicationAttemptState} of the application attempt.
        • - *
      • {@link ContainerId} of the master Container.
        • + *
      • {@link ApplicationAttemptId} of the application.
        • + *
      • Host on which the ApplicationMaster of this attempt is + * running.
        • + *
      • RPC port of the ApplicationMaster of this attempt.
        • + *
      • Tracking URL.
        • + *
      • Diagnostic information in case of errors.
        • + *
      • {@link YarnApplicationAttemptState} of the application attempt.
        • + *
      • {@link ContainerId} of the master Container.
        • *
      - *

      - * */ @Public @Unstable diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java index 412c22b13dbdb..ff4fb52a555d9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java @@ -28,23 +28,22 @@ import java.util.Set; /** - *

      ApplicationReport is a report of an application.

      - * - *

      It includes details such as: - *

        - *
      • {@link ApplicationId} of the application.
        • - *
      • Applications user.
        • - *
      • Application queue.
        • - *
      • Application name.
        • - *
      • Host on which the ApplicationMaster is running.
        • - *
      • RPC port of the ApplicationMaster.
        • - *
      • Tracking URL.
        • - *
      • {@link YarnApplicationState} of the application.
        • - *
      • Diagnostic information in case of errors.
        • - *
      • Start time of the application.
        • - *
      • Client {@link Token} of the application (if security is enabled).
        • - *
      - *

      + * {@code ApplicationReport} is a report of an application. + *

      + * It includes details such as: + *

        + *
      • {@link ApplicationId} of the application.
        • + *
      • Applications user.
        • + *
      • Application queue.
        • + *
      • Application name.
        • + *
      • Host on which the ApplicationMaster is running.
        • + *
      • RPC port of the ApplicationMaster.
        • + *
      • Tracking URL.
        • + *
      • {@link YarnApplicationState} of the application.
        • + *
      • Diagnostic information in case of errors.
        • + *
      • Start time of the application.
        • + *
      • Client {@link Token} of the application (if security is enabled).
        • + *
      * * @see ApplicationClientProtocol#getApplicationReport(org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest) */ @@ -341,20 +340,20 @@ public static ApplicationReport newInstance(ApplicationId applicationId, /** * Get the AMRM token of the application. - *

      + *

      * The AMRM token is required for AM to RM scheduling operations. For * managed Application Masters Yarn takes care of injecting it. For unmanaged * Applications Masters, the token must be obtained via this method and set * in the {@link org.apache.hadoop.security.UserGroupInformation} of the * current user. - *

      + *

      * The AMRM token will be returned only if all the following conditions are * met: - *

    7. - *
        the requester is the owner of the ApplicationMaster
      - *
        the application master is an unmanaged ApplicationMaster
      - *
        the application master is in ACCEPTED state
      - *
      8. + *
        + *
      • the requester is the owner of the ApplicationMaster
        • + *
      • the application master is an unmanaged ApplicationMaster
        • + *
      • the application master is in ACCEPTED state
        • + *
      * Else this method returns NULL. * * @return the AM to RM token if available. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java index c4014fc6eec26..21cd1bbe88547 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java @@ -33,32 +33,34 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      ApplicationSubmissionContext represents all of the - * information needed by the ResourceManager to launch - * the ApplicationMaster for an application.

      - * - *

      It includes details such as: - *

        - *
      • {@link ApplicationId} of the application.
        • - *
      • Application user.
        • - *
      • Application name.
        • - *
      • {@link Priority} of the application.
        • - *
      • - * {@link ContainerLaunchContext} of the container in which the - * ApplicationMaster is executed. - *
        • - *
      • maxAppAttempts. The maximum number of application attempts. + * {@code ApplicationSubmissionContext} represents all of the + * information needed by the {@code ResourceManager} to launch + * the {@code ApplicationMaster} for an application. + *

        + * It includes details such as: + *

          + *
        • {@link ApplicationId} of the application.
          • + *
        • Application user.
          • + *
        • Application name.
          • + *
        • {@link Priority} of the application.
          • + *
        • + * {@link ContainerLaunchContext} of the container in which the + * ApplicationMaster is executed. + *
          • + *
        • + * maxAppAttempts. The maximum number of application attempts. * It should be no larger than the global number of max attempts in the - * Yarn configuration.
          • - *
        • attemptFailuresValidityInterval. The default value is -1. - * when attemptFailuresValidityInterval in milliseconds is set to > 0, - * the failure number will no take failures which happen out of the - * validityInterval into failure count. If failure count reaches to - * maxAppAttempts, the application will be failed. - *
          • + * Yarn configuration. + * + *
        • + * attemptFailuresValidityInterval. The default value is -1. + * when attemptFailuresValidityInterval in milliseconds is set to + * {@literal >} 0, the failure number will no take failures which happen + * out of the validityInterval into failure count. If failure count + * reaches to maxAppAttempts, the application will be failed. + *
          • *
        • Optional, application-specific {@link LogAggregationContext}
          • - *
        - *

        + *
      * * @see ContainerLaunchContext * @see ApplicationClientProtocol#submitApplication(org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java index 279f12759a13f..38fa8b90e78f5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java @@ -27,34 +27,31 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      Container represents an allocated resource in the cluster. - *

      - * - *

      The ResourceManager is the sole authority to allocate any - * Container to applications. The allocated Container + * {@code Container} represents an allocated resource in the cluster. + *

      + * The {@code ResourceManager} is the sole authority to allocate any + * {@code Container} to applications. The allocated {@code Container} * is always on a single node and has a unique {@link ContainerId}. It has - * a specific amount of {@link Resource} allocated.

      - * - *

      It includes details such as: - *

        - *
      • {@link ContainerId} for the container, which is globally unique.
        • - *
      • - * {@link NodeId} of the node on which it is allocated. - *
        • - *
      • HTTP uri of the node.
        • - *
      • {@link Resource} allocated to the container.
        • - *
      • {@link Priority} at which the container was allocated.
        • - *
      • - * Container {@link Token} of the container, used to securely verify - * authenticity of the allocation. - *
        • - *
      - *

      + * a specific amount of {@link Resource} allocated. + *

      + * It includes details such as: + *

        + *
      • {@link ContainerId} for the container, which is globally unique.
        • + *
      • + * {@link NodeId} of the node on which it is allocated. + *
        • + *
      • HTTP uri of the node.
        • + *
      • {@link Resource} allocated to the container.
        • + *
      • {@link Priority} at which the container was allocated.
        • + *
      • + * Container {@link Token} of the container, used to securely verify + * authenticity of the allocation. + *
        • + *
      * - *

      Typically, an ApplicationMaster receives the - * Container from the ResourceManager during - * resource-negotiation and then talks to the NodeManager to - * start/stop containers.

      + * Typically, an {@code ApplicationMaster} receives the {@code Container} + * from the {@code ResourceManager} during resource-negotiation and then + * talks to the {@code NodeManager} to start/stop containers. * * @see ApplicationMasterProtocol#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest) * @see ContainerManagementProtocol#startContainers(org.apache.hadoop.yarn.api.protocolrecords.StartContainersRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerLaunchContext.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerLaunchContext.java index a648fef0fc57f..932945bdd10f6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerLaunchContext.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerLaunchContext.java @@ -30,24 +30,23 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      ContainerLaunchContext represents all of the information - * needed by the NodeManager to launch a container.

      - * - *

      It includes details such as: - *

        - *
      • {@link ContainerId} of the container.
        • - *
      • {@link Resource} allocated to the container.
        • - *
      • User to whom the container is allocated.
        • - *
      • Security tokens (if security is enabled).
        • - *
      • - * {@link LocalResource} necessary for running the container such - * as binaries, jar, shared-objects, side-files etc. - *
        • - *
      • Optional, application-specific binary service data.
        • - *
      • Environment variables for the launched process.
        • - *
      • Command to launch the container.
        • - *
      - *

      + * {@code ContainerLaunchContext} represents all of the information + * needed by the {@code NodeManager} to launch a container. + *

      + * It includes details such as: + *

        + *
      • {@link ContainerId} of the container.
        • + *
      • {@link Resource} allocated to the container.
        • + *
      • User to whom the container is allocated.
        • + *
      • Security tokens (if security is enabled).
        • + *
      • + * {@link LocalResource} necessary for running the container such + * as binaries, jar, shared-objects, side-files etc. + *
        • + *
      • Optional, application-specific binary service data.
        • + *
      • Environment variables for the launched process.
        • + *
      • Command to launch the container.
        • + *
      * * @see ContainerManagementProtocol#startContainers(org.apache.hadoop.yarn.api.protocolrecords.StartContainersRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerReport.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerReport.java index 4cce77f9b22ef..11d7bca680300 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerReport.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerReport.java @@ -24,26 +24,22 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      - * ContainerReport is a report of an container. - *

      - * + * {@code ContainerReport} is a report of an container. *

      * It includes details such as: *

        - *
      • {@link ContainerId} of the container.
        • - *
      • Allocated Resources to the container.
        • - *
      • Assigned Node id.
        • - *
      • Assigned Priority.
        • - *
      • Creation Time.
        • - *
      • Finish Time.
        • - *
      • Container Exit Status.
        • - *
      • {@link ContainerState} of the container.
        • - *
      • Diagnostic information in case of errors.
        • - *
      • Log URL.
        • + *
      • {@link ContainerId} of the container.
        • + *
      • Allocated Resources to the container.
        • + *
      • Assigned Node id.
        • + *
      • Assigned Priority.
        • + *
      • Creation Time.
        • + *
      • Finish Time.
        • + *
      • Container Exit Status.
        • + *
      • {@link ContainerState} of the container.
        • + *
      • Diagnostic information in case of errors.
        • + *
      • Log URL.
        • + *
      • nodeHttpAddress
        • *
      - *

      - * */ @Public @@ -54,7 +50,8 @@ public abstract class ContainerReport { public static ContainerReport newInstance(ContainerId containerId, Resource allocatedResource, NodeId assignedNode, Priority priority, long creationTime, long finishTime, String diagnosticInfo, String logUrl, - int containerExitStatus, ContainerState containerState) { + int containerExitStatus, ContainerState containerState, + String nodeHttpAddress) { ContainerReport report = Records.newRecord(ContainerReport.class); report.setContainerId(containerId); report.setAllocatedResource(allocatedResource); @@ -66,6 +63,7 @@ public static ContainerReport newInstance(ContainerId containerId, report.setLogUrl(logUrl); report.setContainerExitStatus(containerExitStatus); report.setContainerState(containerState); + report.setNodeHttpAddress(nodeHttpAddress); return report; } @@ -199,4 +197,16 @@ public static ContainerReport newInstance(ContainerId containerId, @Unstable public abstract void setContainerExitStatus(int containerExitStatus); + /** + * Get the Node Http address of the container + * + * @return the node http address of the container + */ + @Public + @Unstable + public abstract String getNodeHttpAddress(); + + @Private + @Unstable + public abstract void setNodeHttpAddress(String nodeHttpAddress); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java index 574373c2e78dc..5ccf6dceb6ba2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerStatus.java @@ -25,17 +25,16 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      ContainerStatus represents the current status of a - * Container.

      - * - *

      It provides details such as: - *

        - *
      • ContainerId of the container.
        • - *
      • ContainerState of the container.
        • - *
      • Exit status of a completed container.
        • - *
      • Diagnostic message for a failed container.
        • - *
      - *

      + * {@code ContainerStatus} represents the current status of a + * {@code Container}. + *

      + * It provides details such as: + *

        + *
      • {@code ContainerId} of the container.
        • + *
      • {@code ContainerState} of the container.
        • + *
      • Exit status of a completed container.
        • + *
      • Diagnostic message for a failed container.
        • + *
      */ @Public @Stable diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceType.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceType.java index d1aa45b23a329..1552cdf8cc37d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceType.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceType.java @@ -23,22 +23,22 @@ import org.apache.hadoop.yarn.api.ContainerManagementProtocol; /** - *

      LocalResourceType specifies the type - * of a resource localized by the NodeManager.

      - * - *

      The type can be one of: - *

        - *
      • - * {@link #FILE} - Regular file i.e. uninterpreted bytes. - *
        • - *
      • - * {@link #ARCHIVE} - Archive, which is automatically unarchived by the - * NodeManager. - *
        • - *
      • - * {@link #PATTERN} - A hybrid between {@link #ARCHIVE} and {@link #FILE}. - *
      - *

      + * {@code LocalResourceType} specifies the type + * of a resource localized by the {@code NodeManager}. + *

      + * The type can be one of: + *

        + *
      • + * {@link #FILE} - Regular file i.e. uninterpreted bytes. + *
        • + *
      • + * {@link #ARCHIVE} - Archive, which is automatically unarchived by the + * NodeManager. + *
        • + *
      • + * {@link #PATTERN} - A hybrid between {@link #ARCHIVE} and {@link #FILE}. + *
        • + *
      * * @see LocalResource * @see ContainerLaunchContext diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceVisibility.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceVisibility.java index d368bfb8139a3..2b71991c526ff 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceVisibility.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LocalResourceVisibility.java @@ -23,22 +23,21 @@ import org.apache.hadoop.yarn.api.ContainerManagementProtocol; /** - *

      LocalResourceVisibility specifies the visibility - * of a resource localized by the NodeManager.

      - * - *

      The visibility can be one of: - *

        - *
      • {@link #PUBLIC} - Shared by all users on the node.
        • - *
      • - * {@link #PRIVATE} - Shared among all applications of the - * same user on the node. - *
        • - *
      • - * {@link #APPLICATION} - Shared only among containers of the - * same application on the node. - *
        • - *
      - *

      + * {@code LocalResourceVisibility} specifies the visibility + * of a resource localized by the {@code NodeManager}. + *

      + * The visibility can be one of: + *

        + *
      • {@link #PUBLIC} - Shared by all users on the node.
        • + *
      • + * {@link #PRIVATE} - Shared among all applications of the + * same user on the node. + *
        • + *
      • + * {@link #APPLICATION} - Shared only among containers of the + * same application on the node. + *
        • + *
      * * @see LocalResource * @see ContainerLaunchContext diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LogAggregationContext.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LogAggregationContext.java index 46c1809b417fd..938300474f367 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LogAggregationContext.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/LogAggregationContext.java @@ -24,21 +24,37 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      LogAggregationContext represents all of the - * information needed by the NodeManager to handle - * the logs for an application.

      - * - *

      It includes details such as: - *

        - *
      • includePattern. It uses Java Regex to filter the log files + * {@code LogAggregationContext} represents all of the + * information needed by the {@code NodeManager} to handle + * the logs for an application. + *

        + * It includes details such as: + *

          + *
        • + * includePattern. It uses Java Regex to filter the log files + * which match the defined include pattern and those log files + * will be uploaded when the application finishes. + *
          • + *
        • + * excludePattern. It uses Java Regex to filter the log files + * which match the defined exclude pattern and those log files + * will not be uploaded when application finishes. If the log file + * name matches both the include and the exclude pattern, this file + * will be excluded eventually. + *
          • + *
        • + * rolledLogsIncludePattern. It uses Java Regex to filter the log files * which match the defined include pattern and those log files - * will be uploaded.
          • - *
        • excludePattern. It uses Java Regex to filter the log files + * will be aggregated in a rolling fashion. + *
          • + *
        • + * rolledLogsExcludePattern. It uses Java Regex to filter the log files * which match the defined exclude pattern and those log files - * will not be uploaded. If the log file name matches both the - * include and the exclude pattern, this file will be excluded eventually
          • - *
        - *

        + * will not be aggregated in a rolling fashion. If the log file + * name matches both the include and the exclude pattern, this file + * will be excluded eventually. + *
        • + *
      * * @see ApplicationSubmissionContext */ @@ -57,8 +73,23 @@ public static LogAggregationContext newInstance(String includePattern, return context; } + @Public + @Unstable + public static LogAggregationContext newInstance(String includePattern, + String excludePattern, String rolledLogsIncludePattern, + String rolledLogsExcludePattern) { + LogAggregationContext context = + Records.newRecord(LogAggregationContext.class); + context.setIncludePattern(includePattern); + context.setExcludePattern(excludePattern); + context.setRolledLogsIncludePattern(rolledLogsIncludePattern); + context.setRolledLogsExcludePattern(rolledLogsExcludePattern); + return context; + } + /** - * Get include pattern + * Get include pattern. This includePattern only takes affect + * on logs that exist at the time of application finish. * * @return include pattern */ @@ -67,7 +98,8 @@ public static LogAggregationContext newInstance(String includePattern, public abstract String getIncludePattern(); /** - * Set include pattern + * Set include pattern. This includePattern only takes affect + * on logs that exist at the time of application finish. * * @param includePattern */ @@ -76,7 +108,8 @@ public static LogAggregationContext newInstance(String includePattern, public abstract void setIncludePattern(String includePattern); /** - * Get exclude pattern + * Get exclude pattern. This excludePattern only takes affect + * on logs that exist at the time of application finish. * * @return exclude pattern */ @@ -85,11 +118,50 @@ public static LogAggregationContext newInstance(String includePattern, public abstract String getExcludePattern(); /** - * Set exclude pattern + * Set exclude pattern. This excludePattern only takes affect + * on logs that exist at the time of application finish. * * @param excludePattern */ @Public @Unstable public abstract void setExcludePattern(String excludePattern); + + /** + * Get include pattern in a rolling fashion. + * + * @return include pattern + */ + @Public + @Unstable + public abstract String getRolledLogsIncludePattern(); + + /** + * Set include pattern in a rolling fashion. + * + * @param rolledLogsIncludePattern + */ + @Public + @Unstable + public abstract void setRolledLogsIncludePattern( + String rolledLogsIncludePattern); + + /** + * Get exclude pattern for aggregation in a rolling fashion. + * + * @return exclude pattern + */ + @Public + @Unstable + public abstract String getRolledLogsExcludePattern(); + + /** + * Set exclude pattern for in a rolling fashion. + * + * @param rolledLogsExcludePattern + */ + @Public + @Unstable + public abstract void setRolledLogsExcludePattern( + String rolledLogsExcludePattern); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java new file mode 100644 index 0000000000000..23da1f40b7bb8 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeLabel.java @@ -0,0 +1,55 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.api.records; + +import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability.Stable; +import org.apache.hadoop.classification.InterfaceStability.Unstable; +import org.apache.hadoop.yarn.util.Records; + +@Public +@Unstable +public abstract class NodeLabel { + @Public + @Unstable + public static NodeLabel newInstance(String nodeLabel, + boolean isExclusive) { + NodeLabel request = + Records.newRecord(NodeLabel.class); + request.setNodeLabel(nodeLabel); + request.setIsExclusive(isExclusive); + return request; + } + + @Public + @Stable + public abstract String getNodeLabel(); + + @Public + @Unstable + public abstract void setNodeLabel(String nodeLabel); + + @Public + @Stable + public abstract boolean getIsExclusive(); + + @Public + @Unstable + public abstract void setIsExclusive(boolean isExclusive); +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeReport.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeReport.java index 7ba866d1d1a14..aac19bd7d9b57 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeReport.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeReport.java @@ -28,19 +28,18 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      NodeReport is a summary of runtime information of a - * node in the cluster.

      - * - *

      It includes details such as: - *

        - *
      • {@link NodeId} of the node.
        • - *
      • HTTP Tracking URL of the node.
        • - *
      • Rack name for the node.
        • - *
      • Used {@link Resource} on the node.
        • - *
      • Total available {@link Resource} of the node.
        • - *
      • Number of running containers on the node.
        • - *
      - *

      + * {@code NodeReport} is a summary of runtime information of a node + * in the cluster. + *

      + * It includes details such as: + *

        + *
      • {@link NodeId} of the node.
        • + *
      • HTTP Tracking URL of the node.
        • + *
      • Rack name for the node.
        • + *
      • Used {@link Resource} on the node.
        • + *
      • Total available {@link Resource} of the node.
        • + *
      • Number of running containers on the node.
        • + *
      * * @see ApplicationClientProtocol#getClusterNodes(org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodesRequest) */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java index 976d18125cf24..cdf34f8fc70aa 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java @@ -24,36 +24,36 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      A {@link PreemptionMessage} is part of the RM-AM protocol, and it is used by + * A {@link PreemptionMessage} is part of the RM-AM protocol, and it is used by * the RM to specify resources that the RM wants to reclaim from this - * ApplicationMaster (AM). The AM receives a {@link + * {@code ApplicationMaster} (AM). The AM receives a {@link * StrictPreemptionContract} message encoding which containers the platform may * forcibly kill, granting it an opportunity to checkpoint state or adjust its * execution plan. The message may also include a {@link PreemptionContract} * granting the AM more latitude in selecting which resources to return to the - * cluster.

      - * - *

      The AM should decode both parts of the message. The {@link + * cluster. + *

      + * The AM should decode both parts of the message. The {@link * StrictPreemptionContract} specifies particular allocations that the RM * requires back. The AM can checkpoint containers' state, adjust its execution * plan to move the computation, or take no action and hope that conditions that - * caused the RM to ask for the container will change.

      - * - *

      In contrast, the {@link PreemptionContract} also includes a description of + * caused the RM to ask for the container will change. + *

      + * In contrast, the {@link PreemptionContract} also includes a description of * resources with a set of containers. If the AM releases containers matching * that profile, then the containers enumerated in {@link - * PreemptionContract#getContainers()} may not be killed.

      - * - *

      Each preemption message reflects the RM's current understanding of the - * cluster state, so a request to return N containers may not + * PreemptionContract#getContainers()} may not be killed. + *

      + * Each preemption message reflects the RM's current understanding of the + * cluster state, so a request to return N containers may not * reflect containers the AM is releasing, recently exited containers the RM has * yet to learn about, or new containers allocated before the message was * generated. Conversely, an RM may request a different profile of containers in - * subsequent requests.

      - * - *

      The policy enforced by the RM is part of the scheduler. Generally, only + * subsequent requests. + *

      + * The policy enforced by the RM is part of the scheduler. Generally, only * containers that have been requested consistently should be killed, but the - * details are not specified.

      + * details are not specified. */ @Public @Evolving diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueACL.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueACL.java index c6777db03eedb..585faf86d5f5e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueACL.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueACL.java @@ -23,18 +23,15 @@ import org.apache.hadoop.yarn.api.ApplicationClientProtocol; /** - *

      - * QueueACL enumerates the various ACLs for queues. - *

      - * + * {@code QueueACL} enumerates the various ACLs for queues. *

      * The ACL is one of: *

        - *
      • {@link #SUBMIT_APPLICATIONS} - ACL to submit applications to the - * queue.
        • - *
      • {@link #ADMINISTER_QUEUE} - ACL to administer the queue.
        • + *
      • + * {@link #SUBMIT_APPLICATIONS} - ACL to submit applications to the queue. + *
        • + *
      • {@link #ADMINISTER_QUEUE} - ACL to administer the queue.
        • *
      - *

      * * @see QueueInfo * @see ApplicationClientProtocol#getQueueUserAcls(org.apache.hadoop.yarn.api.protocolrecords.GetQueueUserAclsInfoRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java index d762b41dfdf4c..bee5275a98a8a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java @@ -29,19 +29,18 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      QueueInfo is a report of the runtime information of the queue.

      - * - *

      It includes information such as: - *

        - *
      • Queue name.
        • - *
      • Capacity of the queue.
        • - *
      • Maximum capacity of the queue.
        • - *
      • Current capacity of the queue.
        • - *
      • Child queues.
        • - *
      • Running applications.
        • - *
      • {@link QueueState} of the queue.
        • - *
      - *

      + * QueueInfo is a report of the runtime information of the queue. + *

      + * It includes information such as: + *

        + *
      • Queue name.
        • + *
      • Capacity of the queue.
        • + *
      • Maximum capacity of the queue.
        • + *
      • Current capacity of the queue.
        • + *
      • Child queues.
        • + *
      • Running applications.
        • + *
      • {@link QueueState} of the queue.
        • + *
      * * @see QueueState * @see ApplicationClientProtocol#getQueueInfo(org.apache.hadoop.yarn.api.protocolrecords.GetQueueInfoRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueState.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueState.java index 01698de654f2f..2bc0407f44031 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueState.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueState.java @@ -23,14 +23,13 @@ import org.apache.hadoop.yarn.api.ApplicationClientProtocol; /** - *

      State of a Queue.

      - * - *

      A queue is in one of: - *

        - *
      • {@link #RUNNING} - normal state.
        • - *
      • {@link #STOPPED} - not accepting new application submissions. - *
      - *

      + * State of a Queue. + *

      + * A queue is in one of: + *

        + *
      • {@link #RUNNING} - normal state.
        • + *
      • {@link #STOPPED} - not accepting new application submissions.
        • + *
      * * @see QueueInfo * @see ApplicationClientProtocol#getQueueInfo(org.apache.hadoop.yarn.api.protocolrecords.GetQueueInfoRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequest.java index a4f058510265d..4ebe1c29d7696 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequest.java @@ -25,23 +25,18 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      * {@link ReservationRequest} represents the request made by an application to * the {@code ResourceManager} to reserve {@link Resource}s. - *

      - * *

      * It includes: *

        - *
      • {@link Resource} required for each request.
        • - *
      • - * Number of containers, of above specifications, which are required by the - * application.
        • - *
      • - * Concurrency that indicates the gang size of the request.
        • + *
      • {@link Resource} required for each request.
        • + *
      • + * Number of containers, of above specifications, which are required by the + * application. + *
        • + *
      • Concurrency that indicates the gang size of the request.
        • *
      - *

      - * */ @Public @Unstable diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequestInterpreter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequestInterpreter.java index 1ee96c2d498be..3b826263c770d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequestInterpreter.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ReservationRequestInterpreter.java @@ -33,14 +33,13 @@ public enum ReservationRequestInterpreter { * Requires that exactly ONE among the {@link ReservationRequest} submitted as * of a {@link ReservationDefinition} is satisfied to satisfy the overall * {@link ReservationDefinition}. - * + *

      * WHEN TO USE THIS: This is useful when the user have multiple equivalent * ways to run an application, and wants to expose to the ReservationAgent - * such flexibility. For example an application could use one <32GB,16core> - * container for 10min, or 16 <2GB,1core> containers for 15min, the - * ReservationAgent will decide which one of the two it is best for the system - * to place. - * + * such flexibility. For example an application could use one + * {@literal <32GB,16core>} container for 10min, or 16 {@literal <2GB,1core>} + * containers for 15min, the ReservationAgent will decide which one of the + * two it is best for the system to place. */ R_ANY, @@ -49,16 +48,16 @@ public enum ReservationRequestInterpreter { * {@link ReservationDefinition} are satisfied for the overall * {@link ReservationDefinition} to be satisfied. No constraints are imposed * on the temporal ordering of the allocation used to satisfy the - * ResourceRequeusts. - * + * ResourceRequests. + *

      * WHEN TO USE THIS: This is useful to capture a scenario in which the user * cares for multiple ReservationDefinition to be all accepted, or none. For - * example, a user might want a reservation R1: with 10 x <8GB,4core> for - * 10min, and a reservation R2: with 2 <1GB,1core> for 1h, and only if both - * are satisfied the workflow run in this reservation succeeds. The key - * differentiator from ALL and ORDER, ORDER_NO_GAP, is that ALL imposes no - * restrictions on the relative allocations used to place R1 and R2 above. - * + * example, a user might want a reservation R1: with 10 x + * {@literal <8GB,4core>} for 10min, and a reservation R2: + * with 2 {@literal <1GB,1core>} for 1h, and only if both are satisfied + * the workflow run in this reservation succeeds. The key differentiator + * from ALL and ORDER, ORDER_NO_GAP, is that ALL imposes no restrictions + * on the relative allocations used to place R1 and R2 above. */ R_ALL, @@ -73,15 +72,16 @@ public enum ReservationRequestInterpreter { * constraints are imposed on temporal gaps between subsequent allocations * (the last instant of the previous allocation can be an arbitrary long * period of time before the first instant of the subsequent allocation). - * + *

      * WHEN TO USE THIS: Like ALL this requires all ReservationDefinitions to be * placed, but it also imposes a time ordering on the allocations used. This * is important if the ReservationDefinition(s) are used to describe a * workflow with inherent inter-stage dependencies. For example, a first job - * runs in a ReservaitonDefinition R1 (10 x <1GB,1core> for 20min), and its - * output is consumed by a second job described by a ReservationDefinition R2 - * (5 x <1GB,1core>) for 50min). R2 allocation cannot overlap R1, as R2 models - * a job depending on the output of the job modeled by R1. + * runs in a ReservaitonDefinition R1 (10 x {@literal <1GB,1core>} + * for 20min), and its output is consumed by a second job described by + * a ReservationDefinition R2 (5 x {@literal <1GB,1core>}) for 50min). + * R2 allocation cannot overlap R1, as R2 models a job depending on + * the output of the job modeled by R1. */ R_ORDER, diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java index 2f17ac96f5ad5..790120a0eb687 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java @@ -27,31 +27,30 @@ import org.apache.hadoop.yarn.util.Records; /** - *

      ResourceRequest represents the request made by an - * application to the ResourceManager to obtain various - * Container allocations.

      - * - *

      It includes: - *

        - *
      • {@link Priority} of the request.
        • - *
      • - * The name of the machine or rack on which the allocation is - * desired. A special value of * signifies that - * any host/rack is acceptable to the application. - *
        • - *
      • {@link Resource} required for each request.
        • - *
      • - * Number of containers, of above specifications, which are required - * by the application. - *
        • - *
      • - * A boolean relaxLocality flag, defaulting to true, - * which tells the ResourceManager if the application wants - * locality to be loose (i.e. allows fall-through to rack or any) - * or strict (i.e. specify hard constraint on resource allocation). - *
        • - *
      - *

      + * {@code ResourceRequest} represents the request made + * by an application to the {@code ResourceManager} + * to obtain various {@code Container} allocations. + *

      + * It includes: + *

        + *
      • {@link Priority} of the request.
        • + *
      • + * The name of the machine or rack on which the allocation is + * desired. A special value of * signifies that + * any host/rack is acceptable to the application. + *
        • + *
      • {@link Resource} required for each request.
        • + *
      • + * Number of containers, of above specifications, which are required + * by the application. + *
        • + *
      • + * A boolean relaxLocality flag, defaulting to {@code true}, + * which tells the {@code ResourceManager} if the application wants + * locality to be loose (i.e. allows fall-through to rack or any) + * or strict (i.e. specify hard constraint on resource allocation). + *
        • + *
      * * @see Resource * @see ApplicationMasterProtocol#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest) @@ -267,7 +266,7 @@ public static boolean isAnyLocation(String hostName) { /** * Set node label expression of this resource request. Now only support * specifying a single node label. In the future we will support more complex - * node label expression specification like AND(&&), OR(||), etc. + * node label expression specification like {@code AND(&&), OR(||)}, etc. * * Any please note that node label expression now can only take effect when * the resource request has resourceName = ANY diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java index ff06eeada1778..13e9a10340fc0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java @@ -184,6 +184,12 @@ private static void addDeprecatedKeys() { public static final boolean DEFAULT_RM_SCHEDULER_USE_PORT_FOR_NODE_NAME = false; + /** Enable Resource Manager webapp ui actions */ + public static final String RM_WEBAPP_UI_ACTIONS_ENABLED = + RM_PREFIX + "webapp.ui-actions.enabled"; + public static final boolean DEFAULT_RM_WEBAPP_UI_ACTIONS_ENABLED = + true; + /** Whether the RM should enable Reservation System */ public static final String RM_RESERVATION_SYSTEM_ENABLE = RM_PREFIX + "reservation-system.enable"; @@ -727,8 +733,9 @@ private static void addDeprecatedKeys() { /** * How long to wait between aggregated log retention checks. If set to - * a value <= 0 then the value is computed as one-tenth of the log retention - * setting. Be careful set this too small and you will spam the name node. + * a value {@literal <=} 0 then the value is computed as one-tenth of the + * log retention setting. Be careful set this too small and you will spam + * the name node. */ public static final String LOG_AGGREGATION_RETAIN_CHECK_INTERVAL_SECONDS = YARN_PREFIX + "log-aggregation.retain-check-interval-seconds"; @@ -1021,6 +1028,18 @@ private static void addDeprecatedKeys() { public static final long DEFAULT_NM_LINUX_CONTAINER_CGROUPS_DELETE_DELAY = 20; + /** + * Indicates if memory and CPU limits will be set for the Windows Job + * Object for the containers launched by the default container executor. + */ + public static final String NM_WINDOWS_CONTAINER_MEMORY_LIMIT_ENABLED = + NM_PREFIX + "windows-container.memory-limit.enabled"; + public static final boolean DEFAULT_NM_WINDOWS_CONTAINER_MEMORY_LIMIT_ENABLED = false; + + public static final String NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED = + NM_PREFIX + "windows-container.cpu-limit.enabled"; + public static final boolean DEFAULT_NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED = false; + /** /* The Windows group that the windows-secure-container-executor should run as. */ @@ -1700,6 +1719,18 @@ private static void addDeprecatedKeys() { public static final String NODE_LABELS_ENABLED = NODE_LABELS_PREFIX + "enabled"; public static final boolean DEFAULT_NODE_LABELS_ENABLED = false; + + public static final String NODELABEL_CONFIGURATION_TYPE = + NODE_LABELS_PREFIX + "configuration-type"; + + public static final String CENTALIZED_NODELABEL_CONFIGURATION_TYPE = + "centralized"; + + public static final String DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE = + "distributed"; + + public static final String DEFAULT_NODELABEL_CONFIGURATION_TYPE = + CENTALIZED_NODELABEL_CONFIGURATION_TYPE; public YarnConfiguration() { super(); @@ -1789,4 +1820,9 @@ public static String getClusterId(Configuration conf) { } return clusterId; } + + /* For debugging. mp configurations to system output as XML format. */ + public static void main(String[] args) throws Exception { + new YarnConfiguration(new Configuration()).writeXml(System.out); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java index 2061aeffda5b7..8777e00607894 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java @@ -48,6 +48,8 @@ import org.apache.hadoop.yarn.server.api.protocolrecords.RemoveFromClusterNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeResponse; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceResponse; @@ -120,8 +122,8 @@ public UpdateNodeResourceResponse updateNodeResource( @Public @Evolving @Idempotent - public AddToClusterNodeLabelsResponse addToClusterNodeLabels(AddToClusterNodeLabelsRequest request) - throws YarnException, IOException; + public AddToClusterNodeLabelsResponse addToClusterNodeLabels( + AddToClusterNodeLabelsRequest request) throws YarnException, IOException; @Public @Evolving @@ -134,4 +136,10 @@ public RemoveFromClusterNodeLabelsResponse removeFromClusterNodeLabels( @Idempotent public ReplaceLabelsOnNodeResponse replaceLabelsOnNode( ReplaceLabelsOnNodeRequest request) throws YarnException, IOException; + + @Public + @Evolving + @Idempotent + public UpdateNodeLabelsResponse updateNodeLabels( + UpdateNodeLabelsRequest request) throws YarnException, IOException; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeLabelsRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeLabelsRequest.java new file mode 100644 index 0000000000000..44bdc6573272f --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeLabelsRequest.java @@ -0,0 +1,49 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.api.protocolrecords; + +import java.util.List; + +import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability.Unstable; +import org.apache.hadoop.yarn.api.records.NodeLabel; +import org.apache.hadoop.yarn.util.Records; + +@Public +@Unstable +public abstract class UpdateNodeLabelsRequest { + @Public + @Unstable + public static UpdateNodeLabelsRequest newInstance( + List NodeLabels) { + UpdateNodeLabelsRequest request = + Records.newRecord(UpdateNodeLabelsRequest.class); + request.setNodeLabels(NodeLabels); + return request; + } + + @Public + @Unstable + public abstract void setNodeLabels( + List NodeLabels); + + @Public + @Unstable + public abstract List getNodeLabels(); +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeLabelsResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeLabelsResponse.java new file mode 100644 index 0000000000000..eb704c0c0bac4 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeLabelsResponse.java @@ -0,0 +1,37 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.api.protocolrecords; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability.Unstable; +import org.apache.hadoop.yarn.util.Records; + +@Public +@Unstable +public class UpdateNodeLabelsResponse { + + @Private + @Unstable + public static UpdateNodeLabelsResponse newInstance() { + UpdateNodeLabelsResponse response = + Records.newRecord(UpdateNodeLabelsResponse.class); + return response; + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeResourceRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeResourceRequest.java index d1ab781cda8ae..d540ccebb4629 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeResourceRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/UpdateNodeResourceRequest.java @@ -54,7 +54,7 @@ public static UpdateNodeResourceRequest newInstance( /** * Get the map from NodeId to ResourceOption. - * @return the map of + * @return the map of {@code } */ @Public @Evolving @@ -62,7 +62,7 @@ public static UpdateNodeResourceRequest newInstance( /** * Set the map from NodeId to ResourceOption. - * @param nodeResourceMap the map of + * @param nodeResourceMap the map of {@code } */ @Public @Evolving diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto index 7f54b8e3eb627..6646718a49af8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto @@ -42,4 +42,5 @@ service ResourceManagerAdministrationProtocolService { rpc addToClusterNodeLabels(AddToClusterNodeLabelsRequestProto) returns (AddToClusterNodeLabelsResponseProto); rpc removeFromClusterNodeLabels(RemoveFromClusterNodeLabelsRequestProto) returns (RemoveFromClusterNodeLabelsResponseProto); rpc replaceLabelsOnNodes(ReplaceLabelsOnNodeRequestProto) returns (ReplaceLabelsOnNodeResponseProto); + rpc updateNodeLabels(UpdateNodeLabelsRequestProto) returns (UpdateNodeLabelsResponseProto); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto index 900e349dff857..0d5b5c75650eb 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto @@ -97,6 +97,14 @@ message ReplaceLabelsOnNodeResponseProto { } +message UpdateNodeLabelsRequestProto { + repeated NodeLabelProto nodeLabels = 1; +} + + +message UpdateNodeLabelsResponseProto { +} + ////////////////////////////////////////////////////////////////// ///////////// RM Failover related records //////////////////////// ////////////////////////////////////////////////////////////////// diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto index 4e29d2fc82b57..b396f4daa8b29 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto @@ -98,6 +98,7 @@ message ContainerReportProto { optional string log_url = 8; optional int32 container_exit_status = 9; optional ContainerStateProto container_state = 10; + optional string node_http_address = 11; } enum YarnApplicationStateProto { @@ -238,11 +239,20 @@ message NodeIdToLabelsProto { repeated string nodeLabels = 2; } +message StringArrayProto { + repeated string elements = 1; +} + message LabelsToNodeIdsProto { optional string nodeLabels = 1; repeated NodeIdProto nodeId = 2; } +message NodeLabelProto { + optional string nodeLabel = 1; + optional bool isExclusive = 2 [default = true]; +} + //////////////////////////////////////////////////////////////////////// ////// From AM_RM_Protocol ///////////////////////////////////////////// //////////////////////////////////////////////////////////////////////// @@ -313,6 +323,8 @@ message ApplicationSubmissionContextProto { message LogAggregationContextProto { optional string include_pattern = 1 [default = ".*"]; optional string exclude_pattern = 2 [default = ""]; + optional string rolled_logs_include_pattern = 3 [default = ""]; + optional string rolled_logs_exclude_pattern = 4 [default = ".*"]; } enum ApplicationAccessTypeProto { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java index 46b5850ebb1ae..5e6fa464730c2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java @@ -243,7 +243,7 @@ public void run() { .getApplicationHistoryServer() .getTimelineStore() .getEntities(ApplicationMaster.DSEntity.DS_APP_ATTEMPT.toString(), - null, null, null, null, null, null, null, null); + null, null, null, null, null, null, null, null, null); Assert.assertNotNull(entitiesAttempts); Assert.assertEquals(1, entitiesAttempts.getEntities().size()); Assert.assertEquals(2, entitiesAttempts.getEntities().get(0).getEvents() @@ -261,7 +261,7 @@ public void run() { .getApplicationHistoryServer() .getTimelineStore() .getEntities(ApplicationMaster.DSEntity.DS_CONTAINER.toString(), null, - null, null, null, null, null, null, null); + null, null, null, null, null, null, null, null); Assert.assertNotNull(entities); Assert.assertEquals(2, entities.getEntities().size()); Assert.assertEquals(entities.getEntities().get(0).getEntityType() diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java index b3c5308240a89..b590a51f10802 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java @@ -56,28 +56,22 @@ public AHSClient(String name) { } /** - *

      * Get a report of the given Application. - *

      - * *

      * In secure mode, YARN verifies access to the application, queue * etc. before accepting the request. - *

      - * *

      * If the user does not have VIEW_APP access then the following * fields in the report will be set to stubbed values: *

        - *
      • host - set to "N/A"
        • - *
      • RPC port - set to -1
        • - *
      • client token - set to "N/A"
        • - *
      • diagnostics - set to "N/A"
        • - *
      • tracking URL - set to "N/A"
        • - *
      • original tracking URL - set to "N/A"
        • - *
      • resource usage report - all values are -1
        • + *
      • host - set to "N/A"
        • + *
      • RPC port - set to -1
        • + *
      • client token - set to "N/A"
        • + *
      • diagnostics - set to "N/A"
        • + *
      • tracking URL - set to "N/A"
        • + *
      • original tracking URL - set to "N/A"
        • + *
      • resource usage report - all values are -1
        • *
      - *

      * * @param appId * {@link ApplicationId} of the application that needs a report @@ -121,7 +115,7 @@ public abstract List getApplications() * a report * @return application attempt report * @throws YarnException - * @throws {@link ApplicationAttemptNotFoundException} if application attempt + * @throws ApplicationAttemptNotFoundException if application attempt * not found * @throws IOException */ @@ -157,7 +151,7 @@ public abstract List getApplicationAttempts( * {@link ContainerId} of the container that needs a report * @return container report * @throws YarnException - * @throws {@link ContainerNotFoundException} if container not found + * @throws ContainerNotFoundException if container not found * @throws IOException */ public abstract ContainerReport getContainerReport(ContainerId containerId) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java index 9923806367833..bfe10d60fd027 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java @@ -349,7 +349,7 @@ public abstract void updateBlacklist(List blacklistAdditions, * Set the NM token cache for the AMRMClient. This cache must * be shared with the {@link NMClient} used to manage containers for the * AMRMClient - *

      + *

      * If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} * singleton instance will be used. * @@ -363,7 +363,7 @@ public void setNMTokenCache(NMTokenCache nmTokenCache) { * Get the NM token cache of the AMRMClient. This cache must be * shared with the {@link NMClient} used to manage containers for the * AMRMClient. - *

      + *

      * If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} * singleton instance will be used. * diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java index 721728e06f00c..08b911b1175c2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMClient.java @@ -125,7 +125,7 @@ public abstract ContainerStatus getContainerStatus(ContainerId containerId, * Set the NM Token cache of the NMClient. This cache must be * shared with the {@link AMRMClient} that requested the containers managed * by this NMClient - *

      + *

      * If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} * singleton instance will be used. * @@ -139,7 +139,7 @@ public void setNMTokenCache(NMTokenCache nmTokenCache) { * Get the NM token cache of the NMClient. This cache must be * shared with the {@link AMRMClient} that requested the containers managed * by this NMClient - *

      + *

      * If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} * singleton instance will be used. * diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMTokenCache.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMTokenCache.java index 0e7356fe1f978..0c349cc429bd2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMTokenCache.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/NMTokenCache.java @@ -34,26 +34,26 @@ /** * NMTokenCache manages NMTokens required for an Application Master * communicating with individual NodeManagers. - *

      + *

      * By default Yarn client libraries {@link AMRMClient} and {@link NMClient} use * {@link #getSingleton()} instance of the cache. *

        - *
      • Using the singleton instance of the cache is appropriate when running a - * single ApplicationMaster in the same JVM.
        • - *
      • When using the singleton, users don't need to do anything special, - * {@link AMRMClient} and {@link NMClient} are already set up to use the default - * singleton {@link NMTokenCache}
        • + *
      • + * Using the singleton instance of the cache is appropriate when running a + * single ApplicationMaster in the same JVM. + *
        • + *
      • + * When using the singleton, users don't need to do anything special, + * {@link AMRMClient} and {@link NMClient} are already set up to use the + * default singleton {@link NMTokenCache} + *
        • *
      - *

      * If running multiple Application Masters in the same JVM, a different cache * instance should be used for each Application Master. - *

      *

        - *
      • - * If using the {@link AMRMClient} and the {@link NMClient}, setting up and using - * an instance cache is as follows: - *

        - * + *

      • + * If using the {@link AMRMClient} and the {@link NMClient}, setting up + * and using an instance cache is as follows: * 
          *   NMTokenCache nmTokenCache = new NMTokenCache();
  *   AMRMClient rmClient = AMRMClient.createAMRMClient();
@@ -61,12 +61,10 @@
  *   nmClient.setNMTokenCache(nmTokenCache);
  *   ...
  *
        - *
        • - *
      • - * If using the {@link AMRMClientAsync} and the {@link NMClientAsync}, setting up - * and using an instance cache is as follows: - *

        - * + *

        • + *
      • + * If using the {@link AMRMClientAsync} and the {@link NMClientAsync}, + * setting up and using an instance cache is as follows: * 
          *   NMTokenCache nmTokenCache = new NMTokenCache();
  *   AMRMClient rmClient = AMRMClient.createAMRMClient();
@@ -76,13 +74,11 @@
  *   NMClientAsync nmClientAsync = new NMClientAsync("nmClient", nmClient, [NM_CALLBACK]);
  *   ...
  *
        - *
        • - *
      • - * If using {@link ApplicationMasterProtocol} and - * {@link ContainerManagementProtocol} directly, setting up and using an - * instance cache is as follows: - *

        - * + *

        • + *
      • + * If using {@link ApplicationMasterProtocol} and + * {@link ContainerManagementProtocol} directly, setting up and using an + * instance cache is as follows: * 
          *   NMTokenCache nmTokenCache = new NMTokenCache();
  *   ...
@@ -100,12 +96,12 @@
  *   nmPro.startContainer(container, containerContext);
  *   ...
  *
        - *
        • + * *
      - * It is also possible to mix the usage of a client (AMRMClient or - * NMClient, or the async versions of them) with a protocol proxy ( - * ContainerManagementProtocolProxy or - * ApplicationMasterProtocol). + * It is also possible to mix the usage of a client ({@code AMRMClient} or + * {@code NMClient}, or the async versions of them) with a protocol proxy + * ({@code ContainerManagementProtocolProxy} or + * {@code ApplicationMasterProtocol}). */ @Public @Evolving diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java index d96761a9dd051..f61773110a304 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java @@ -32,14 +32,12 @@ import org.apache.hadoop.io.Text; import org.apache.hadoop.service.AbstractService; import org.apache.hadoop.yarn.api.ApplicationClientProtocol; -import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest; import org.apache.hadoop.yarn.api.protocolrecords.ReservationDeleteRequest; import org.apache.hadoop.yarn.api.protocolrecords.ReservationDeleteResponse; import org.apache.hadoop.yarn.api.protocolrecords.ReservationSubmissionRequest; import org.apache.hadoop.yarn.api.protocolrecords.ReservationSubmissionResponse; import org.apache.hadoop.yarn.api.protocolrecords.ReservationUpdateRequest; import org.apache.hadoop.yarn.api.protocolrecords.ReservationUpdateResponse; -import org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationRequest; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; import org.apache.hadoop.yarn.api.records.ApplicationId; @@ -58,8 +56,10 @@ import org.apache.hadoop.yarn.api.records.YarnApplicationState; import org.apache.hadoop.yarn.api.records.YarnClusterMetrics; import org.apache.hadoop.yarn.client.api.impl.YarnClientImpl; +import org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException; import org.apache.hadoop.yarn.exceptions.ApplicationIdNotProvidedException; import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException; +import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException; import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.security.AMRMTokenIdentifier; @@ -171,7 +171,6 @@ public abstract void killApplication(ApplicationId applicationId) throws YarnExc *
    8. original tracking URL - set to "N/A"
      9. *
    9. resource usage report - all values are -1
      10. * - *

      * * @param appId * {@link ApplicationId} of the application that needs a report @@ -184,20 +183,20 @@ public abstract ApplicationReport getApplicationReport(ApplicationId appId) /** * Get the AMRM token of the application. - *

      + *

      * The AMRM token is required for AM to RM scheduling operations. For * managed Application Masters Yarn takes care of injecting it. For unmanaged * Applications Masters, the token must be obtained via this method and set * in the {@link org.apache.hadoop.security.UserGroupInformation} of the * current user. - *

      + *

      * The AMRM token will be returned only if all the following conditions are * met: - *

    10. - *
        the requester is the owner of the ApplicationMaster
      - *
        the application master is an unmanaged ApplicationMaster
      - *
        the application master is in ACCEPTED state
      - *
      11. + *
        + *
      • the requester is the owner of the ApplicationMaster
        • + *
      • the application master is an unmanaged ApplicationMaster
        • + *
      • the application master is in ACCEPTED state
        • + *
      * Else this method returns NULL. * * @param appId {@link ApplicationId} of the application to get the AMRM token @@ -415,7 +414,7 @@ public abstract List getQueueAclsInfo() throws YarnException, * a report * @return application attempt report * @throws YarnException - * @throws {@link ApplicationAttemptNotFoundException} if application attempt + * @throws ApplicationAttemptNotFoundException if application attempt * not found * @throws IOException */ @@ -450,7 +449,7 @@ public abstract List getApplicationAttempts( * {@link ContainerId} of the container that needs a report * @return container report * @throws YarnException - * @throws {@link ContainerNotFoundException} if container not found. + * @throws ContainerNotFoundException if container not found. * @throws IOException */ public abstract ContainerReport getContainerReport(ContainerId containerId) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java index 6acf7d808578b..d6b36bbf8ddb9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java @@ -254,13 +254,22 @@ public YarnClientApplication createApplication() int pollCount = 0; long startTime = System.currentTimeMillis(); - + EnumSet waitingStates = + EnumSet.of(YarnApplicationState.NEW, + YarnApplicationState.NEW_SAVING, + YarnApplicationState.SUBMITTED); + EnumSet failToSubmitStates = + EnumSet.of(YarnApplicationState.FAILED, + YarnApplicationState.KILLED); while (true) { try { - YarnApplicationState state = - getApplicationReport(applicationId).getYarnApplicationState(); - if (!state.equals(YarnApplicationState.NEW) && - !state.equals(YarnApplicationState.NEW_SAVING)) { + ApplicationReport appReport = getApplicationReport(applicationId); + YarnApplicationState state = appReport.getYarnApplicationState(); + if (!waitingStates.contains(state)) { + if(failToSubmitStates.contains(state)) { + throw new YarnException("Failed to submit " + applicationId + + " to YARN : " + appReport.getDiagnostics()); + } LOG.info("Submitted application " + applicationId); break; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java index 108ad0b722003..dd4a949ed4f27 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java @@ -63,7 +63,7 @@ public class ApplicationCLI extends YarnCLI { "%30s\t%20s\t%35s\t%35s" + System.getProperty("line.separator"); private static final String CONTAINER_PATTERN = - "%30s\t%20s\t%20s\t%20s\t%20s\t%35s" + "%30s\t%20s\t%20s\t%20s\t%20s\t%20s\t%35s" + System.getProperty("line.separator"); private static final String APP_TYPE_CMD = "appTypes"; @@ -355,6 +355,9 @@ private int printContainerReport(String containerId) throws YarnException, containerReportStr.println(containerReport.getLogUrl()); containerReportStr.print("\tHost : "); containerReportStr.println(containerReport.getAssignedNode()); + containerReportStr.print("\tNodeHttpAddress : "); + containerReportStr.println(containerReport.getNodeHttpAddress() == null + ? "N/A" : containerReport.getNodeHttpAddress()); containerReportStr.print("\tDiagnostics : "); containerReportStr.print(containerReport.getDiagnosticsInfo()); } else { @@ -595,7 +598,7 @@ private void listContainers(String appAttemptId) throws YarnException, .getContainers(ConverterUtils.toApplicationAttemptId(appAttemptId)); writer.println("Total number of containers " + ":" + appsReport.size()); writer.printf(CONTAINER_PATTERN, "Container-Id", "Start Time", - "Finish Time", "State", "Host", "LOG-URL"); + "Finish Time", "State", "Host", "Node Http Address", "LOG-URL"); for (ContainerReport containerReport : appsReport) { writer.printf( CONTAINER_PATTERN, @@ -603,7 +606,9 @@ private void listContainers(String appAttemptId) throws YarnException, Times.format(containerReport.getCreationTime()), Times.format(containerReport.getFinishTime()), containerReport.getContainerState(), containerReport - .getAssignedNode(), containerReport.getLogUrl()); + .getAssignedNode(), containerReport.getNodeHttpAddress() == null + ? "N/A" : containerReport.getNodeHttpAddress(), + containerReport.getLogUrl()); } writer.flush(); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ClusterCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ClusterCLI.java new file mode 100644 index 0000000000000..39248039ed197 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ClusterCLI.java @@ -0,0 +1,157 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.client.cli; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.io.PrintWriter; +import java.io.UnsupportedEncodingException; +import java.nio.charset.Charset; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Set; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.GnuParser; +import org.apache.commons.cli.HelpFormatter; +import org.apache.commons.cli.MissingArgumentException; +import org.apache.commons.cli.Options; +import org.apache.commons.lang.StringUtils; +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.util.ToolRunner; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.exceptions.YarnException; +import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; + +import com.google.common.annotations.VisibleForTesting; + +/** + * Cluster CLI used to get over all information of the cluster + */ +@Private +public class ClusterCLI extends YarnCLI { + private static final String TITLE = "yarn cluster"; + public static final String LIST_LABELS_CMD = "list-node-labels"; + public static final String DIRECTLY_ACCESS_NODE_LABEL_STORE = + "directly-access-node-label-store"; + public static final String CMD = "cluster"; + private boolean accessLocal = false; + static CommonNodeLabelsManager localNodeLabelsManager = null; + + public static void main(String[] args) throws Exception { + ClusterCLI cli = new ClusterCLI(); + cli.setSysOutPrintStream(System.out); + cli.setSysErrPrintStream(System.err); + int res = ToolRunner.run(cli, args); + cli.stop(); + System.exit(res); + } + + @Override + public int run(String[] args) throws Exception { + Options opts = new Options(); + + opts.addOption("lnl", LIST_LABELS_CMD, false, + "List cluster node-label collection"); + opts.addOption("h", HELP_CMD, false, "Displays help for all commands."); + opts.addOption("dnl", DIRECTLY_ACCESS_NODE_LABEL_STORE, false, + "Directly access node label store, " + + "with this option, all node label related operations" + + " will NOT connect RM. Instead, they will" + + " access/modify stored node labels directly." + + " By default, it is false (access via RM)." + + " AND PLEASE NOTE: if you configured " + + YarnConfiguration.FS_NODE_LABELS_STORE_ROOT_DIR + + " to a local directory" + + " (instead of NFS or HDFS), this option will only work" + + " when the command run on the machine where RM is running." + + " Also, this option is UNSTABLE, could be removed in future" + + " releases."); + + int exitCode = -1; + CommandLine parsedCli = null; + try { + parsedCli = new GnuParser().parse(opts, args); + } catch (MissingArgumentException ex) { + sysout.println("Missing argument for options"); + printUsage(opts); + return exitCode; + } + + if (parsedCli.hasOption(DIRECTLY_ACCESS_NODE_LABEL_STORE)) { + accessLocal = true; + } + + if (parsedCli.hasOption(LIST_LABELS_CMD)) { + printClusterNodeLabels(); + } else if (parsedCli.hasOption(HELP_CMD)) { + printUsage(opts); + return 0; + } else { + syserr.println("Invalid Command Usage : "); + printUsage(opts); + } + return 0; + } + + private List sortStrSet(Set labels) { + List list = new ArrayList(); + list.addAll(labels); + Collections.sort(list); + return list; + } + + void printClusterNodeLabels() throws YarnException, IOException { + Set nodeLabels = null; + if (accessLocal) { + nodeLabels = + getNodeLabelManagerInstance(getConf()).getClusterNodeLabels(); + } else { + nodeLabels = client.getClusterNodeLabels(); + } + sysout.println(String.format("Node Labels: %s", + StringUtils.join(sortStrSet(nodeLabels).iterator(), ","))); + } + + @VisibleForTesting + static synchronized CommonNodeLabelsManager + getNodeLabelManagerInstance(Configuration conf) { + if (localNodeLabelsManager == null) { + localNodeLabelsManager = new CommonNodeLabelsManager(); + localNodeLabelsManager.init(conf); + localNodeLabelsManager.start(); + } + return localNodeLabelsManager; + } + + @VisibleForTesting + void printUsage(Options opts) throws UnsupportedEncodingException { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + PrintWriter pw = + new PrintWriter(new OutputStreamWriter(baos, Charset.forName("UTF-8"))); + new HelpFormatter().printHelp(pw, HelpFormatter.DEFAULT_WIDTH, TITLE, null, + opts, HelpFormatter.DEFAULT_LEFT_PAD, HelpFormatter.DEFAULT_DESC_PAD, + null); + pw.close(); + sysout.println(baos.toString("UTF-8")); + } +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java index 4642add12e33d..420eeb0ff12d5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java @@ -134,7 +134,8 @@ protected void setErrOut(PrintStream errOut) { private static void appendHAUsage(final StringBuilder usageBuilder) { for (Map.Entry cmdEntry : USAGE.entrySet()) { - if (cmdEntry.getKey().equals("-help")) { + if (cmdEntry.getKey().equals("-help") + || cmdEntry.getKey().equals("-failover")) { continue; } UsageInfo usageInfo = cmdEntry.getValue(); @@ -225,7 +226,7 @@ private static void printHelp(String cmd, boolean isHAEnabled) { } if (isHAEnabled) { for (String cmdKey : USAGE.keySet()) { - if (!cmdKey.equals("-help")) { + if (!cmdKey.equals("-help") && !cmdKey.equals("-failover")) { buildHelpMsg(cmdKey, helpBuilder); helpBuilder.append("\n"); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java index da7d50529ac9f..f468bc115a966 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java @@ -77,7 +77,6 @@ import org.apache.hadoop.yarn.api.protocolrecords.RenewDelegationTokenResponse; import org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationRequest; import org.apache.hadoop.yarn.api.protocolrecords.SubmitApplicationResponse; -import org.apache.hadoop.yarn.api.records.AMCommand; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; import org.apache.hadoop.yarn.api.records.ApplicationId; @@ -646,7 +645,7 @@ public ApplicationReport createFakeAppReport() { ApplicationReport report = ApplicationReport.newInstance(appId, attemptId, "fakeUser", "fakeQueue", "fakeApplicationName", "localhost", 0, null, - YarnApplicationState.FAILED, "fake an application report", "", + YarnApplicationState.FINISHED, "fake an application report", "", 1000l, 1200l, FinalApplicationStatus.FAILED, null, "", 50f, "fakeApplicationType", null); return report; @@ -715,7 +714,8 @@ public ApplicationAttemptReport createFakeApplicationAttemptReport() { public ContainerReport createFakeContainerReport() { return ContainerReport.newInstance(createFakeContainerId(), null, NodeId.newInstance("localhost", 0), null, 1000l, 1200l, "", "", 0, - ContainerState.COMPLETE); + ContainerState.COMPLETE, + "http://" + NodeId.newInstance("localhost", 0).toString()); } public List createFakeContainerReports() { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java index bfc6656c2d46d..8e00554ad6c24 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java @@ -93,7 +93,8 @@ public void testGetClusterMetricsOnHA() throws Exception { public void testGetApplicationsOnHA() throws Exception { List reports = client.getApplications(); - Assert.assertTrue(reports != null && !reports.isEmpty()); + Assert.assertTrue(reports != null); + Assert.assertFalse(reports.isEmpty()); Assert.assertEquals(cluster.createFakeAppReports(), reports); } @@ -101,7 +102,8 @@ public void testGetApplicationsOnHA() throws Exception { @Test(timeout = 15000) public void testGetClusterNodesOnHA() throws Exception { List reports = client.getNodeReports(NodeState.RUNNING); - Assert.assertTrue(reports != null && !reports.isEmpty()); + Assert.assertTrue(reports != null); + Assert.assertFalse(reports.isEmpty()); Assert.assertEquals(cluster.createFakeNodeReports(), reports); } @@ -117,8 +119,8 @@ public void testGetQueueInfoOnHA() throws Exception { @Test(timeout = 15000) public void testGetQueueUserAclsOnHA() throws Exception { List queueUserAclsList = client.getQueueAclsInfo(); - Assert.assertTrue(queueUserAclsList != null - && !queueUserAclsList.isEmpty()); + Assert.assertTrue(queueUserAclsList != null); + Assert.assertFalse(queueUserAclsList.isEmpty()); Assert.assertEquals(cluster.createFakeQueueUserACLInfoList(), queueUserAclsList); } @@ -136,7 +138,8 @@ public void testGetApplicationAttemptReportOnHA() throws Exception { public void testGetApplicationAttemptsOnHA() throws Exception { List reports = client.getApplicationAttempts(cluster.createFakeAppId()); - Assert.assertTrue(reports != null && !reports.isEmpty()); + Assert.assertTrue(reports != null); + Assert.assertFalse(reports.isEmpty()); Assert.assertEquals(cluster.createFakeApplicationAttemptReports(), reports); } @@ -153,7 +156,8 @@ public void testGetContainerReportOnHA() throws Exception { public void testGetContainersOnHA() throws Exception { List reports = client.getContainers(cluster.createFakeApplicationAttemptId()); - Assert.assertTrue(reports != null && !reports.isEmpty()); + Assert.assertTrue(reports != null); + Assert.assertFalse(reports.isEmpty()); Assert.assertEquals(cluster.createFakeContainerReports(), reports); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java index 8885769df2d18..8167a58f8e669 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java @@ -70,7 +70,7 @@ public void testResourceTrackerOnHA() throws Exception { NodeStatus.newInstance(NodeId.newInstance("localhost", 0), 0, null, null, null); NodeHeartbeatRequest request2 = - NodeHeartbeatRequest.newInstance(status, null, null); + NodeHeartbeatRequest.newInstance(status, null, null,null); resourceTracker.nodeHeartbeat(request2); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java index a88189e5c0d84..c3e3c414771db 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSClient.java @@ -371,14 +371,16 @@ private void createAppReports() { ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(), 1), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, - 5678, "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE); + 5678, "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReports.add(container); ContainerReport container1 = ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(), 2), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, - 5678, "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE); + 5678, "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReports.add(container1); containers.put(attempt.getApplicationAttemptId(), containerReports); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java index 7e97134437c73..de669f2098a6d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java @@ -117,7 +117,6 @@ import org.apache.hadoop.yarn.util.Clock; import org.apache.hadoop.yarn.util.Records; import org.apache.hadoop.yarn.util.UTCClock; -import org.apache.hadoop.yarn.util.timeline.TimelineUtils; import org.apache.log4j.Level; import org.apache.log4j.LogManager; import org.apache.log4j.Logger; @@ -157,12 +156,9 @@ public void testSubmitApplication() { YarnApplicationState[] exitStates = new YarnApplicationState[] { - YarnApplicationState.SUBMITTED, YarnApplicationState.ACCEPTED, YarnApplicationState.RUNNING, - YarnApplicationState.FINISHED, - YarnApplicationState.FAILED, - YarnApplicationState.KILLED + YarnApplicationState.FINISHED }; // Submit an application without ApplicationId provided @@ -203,6 +199,54 @@ public void testSubmitApplication() { client.stop(); } + @Test (timeout = 30000) + public void testSubmitIncorrectQueue() throws IOException { + MiniYARNCluster cluster = new MiniYARNCluster("testMRAMTokens", 1, 1, 1); + YarnClient rmClient = null; + try { + cluster.init(new YarnConfiguration()); + cluster.start(); + final Configuration yarnConf = cluster.getConfig(); + rmClient = YarnClient.createYarnClient(); + rmClient.init(yarnConf); + rmClient.start(); + YarnClientApplication newApp = rmClient.createApplication(); + + ApplicationId appId = newApp.getNewApplicationResponse().getApplicationId(); + + // Create launch context for app master + ApplicationSubmissionContext appContext + = Records.newRecord(ApplicationSubmissionContext.class); + + // set the application id + appContext.setApplicationId(appId); + + // set the application name + appContext.setApplicationName("test"); + + // Set the queue to which this application is to be submitted in the RM + appContext.setQueue("nonexist"); + + // Set up the container launch context for the application master + ContainerLaunchContext amContainer + = Records.newRecord(ContainerLaunchContext.class); + appContext.setAMContainerSpec(amContainer); + appContext.setResource(Resource.newInstance(1024, 1)); + // appContext.setUnmanagedAM(unmanaged); + + // Submit the application to the applications manager + rmClient.submitApplication(appContext); + Assert.fail("Job submission should have thrown an exception"); + } catch (YarnException e) { + Assert.assertTrue(e.getMessage().contains("Failed to submit")); + } finally { + if (rmClient != null) { + rmClient.stop(); + } + cluster.stop(); + } + } + @Test public void testKillApplication() throws Exception { MockRM rm = new MockRM(); @@ -568,13 +612,15 @@ private List createAppReports() { ContainerReport container = ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(), 1), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, 5678, - "diagnosticInfo", "logURL", 0, ContainerState.RUNNING); + "diagnosticInfo", "logURL", 0, ContainerState.RUNNING, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReports.add(container); ContainerReport container1 = ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(), 2), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, 5678, - "diagnosticInfo", "logURL", 0, ContainerState.RUNNING); + "diagnosticInfo", "logURL", 0, ContainerState.RUNNING, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReports.add(container1); containers.put(attempt.getApplicationAttemptId(), containerReports); @@ -585,18 +631,21 @@ private List createAppReports() { container = ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(), 1), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, 5678, - "diagnosticInfo", "logURL", 0, null); + "diagnosticInfo", "logURL", 0, null, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReportsForAHS.add(container); container1 = ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(), 2), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, 5678, - "diagnosticInfo", "HSlogURL", 0, null); + "diagnosticInfo", "HSlogURL", 0, null, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReportsForAHS.add(container1); ContainerReport container2 = ContainerReport.newInstance( ContainerId.newContainerId(attempt.getApplicationAttemptId(),3), null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, 5678, - "diagnosticInfo", "HSlogURL", 0, ContainerState.COMPLETE); + "diagnosticInfo", "HSlogURL", 0, ContainerState.COMPLETE, + "http://" + NodeId.newInstance("host", 2345).toString()); containerReportsForAHS.add(container2); containersFromAHS.put(attempt.getApplicationAttemptId(), containerReportsForAHS); @@ -998,7 +1047,7 @@ protected void serviceStop() throws Exception { public ApplicationReport getApplicationReport(ApplicationId appId) { ApplicationReport report = mock(ApplicationReport.class); when(report.getYarnApplicationState()) - .thenReturn(YarnApplicationState.SUBMITTED); + .thenReturn(YarnApplicationState.RUNNING); return report; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestClusterCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestClusterCLI.java new file mode 100644 index 0000000000000..f9ccf877048c4 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestClusterCLI.java @@ -0,0 +1,158 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.client.cli; + +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.ByteArrayOutputStream; +import java.io.PrintStream; +import java.io.PrintWriter; +import java.util.HashSet; + +import org.apache.hadoop.yarn.client.api.YarnClient; +import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; +import org.junit.Before; +import org.junit.Test; + +import com.google.common.collect.ImmutableSet; + +public class TestClusterCLI { + ByteArrayOutputStream sysOutStream; + private PrintStream sysOut; + ByteArrayOutputStream sysErrStream; + private PrintStream sysErr; + + @Before + public void setup() { + sysOutStream = new ByteArrayOutputStream(); + sysOut = spy(new PrintStream(sysOutStream)); + sysErrStream = new ByteArrayOutputStream(); + sysErr = spy(new PrintStream(sysErrStream)); + System.setOut(sysOut); + } + + @Test + public void testGetClusterNodeLabels() throws Exception { + YarnClient client = mock(YarnClient.class); + when(client.getClusterNodeLabels()).thenReturn( + ImmutableSet.of("label1", "label2")); + ClusterCLI cli = new ClusterCLI(); + cli.setClient(client); + cli.setSysOutPrintStream(sysOut); + cli.setSysErrPrintStream(sysErr); + + int rc = + cli.run(new String[] { ClusterCLI.CMD, "-" + ClusterCLI.LIST_LABELS_CMD }); + assertEquals(0, rc); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + PrintWriter pw = new PrintWriter(baos); + pw.print("Node Labels: label1,label2"); + pw.close(); + verify(sysOut).println(baos.toString("UTF-8")); + } + + @Test + public void testGetClusterNodeLabelsWithLocalAccess() throws Exception { + YarnClient client = mock(YarnClient.class); + when(client.getClusterNodeLabels()).thenReturn( + ImmutableSet.of("remote1", "remote2")); + ClusterCLI cli = new ClusterCLI(); + cli.setClient(client); + cli.setSysOutPrintStream(sysOut); + cli.setSysErrPrintStream(sysErr); + ClusterCLI.localNodeLabelsManager = mock(CommonNodeLabelsManager.class); + when(ClusterCLI.localNodeLabelsManager.getClusterNodeLabels()) + .thenReturn(ImmutableSet.of("local1", "local2")); + + int rc = + cli.run(new String[] { ClusterCLI.CMD, + "-" + ClusterCLI.LIST_LABELS_CMD, + "-" + ClusterCLI.DIRECTLY_ACCESS_NODE_LABEL_STORE }); + assertEquals(0, rc); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + PrintWriter pw = new PrintWriter(baos); + // it should return local* instead of remote* + pw.print("Node Labels: local1,local2"); + pw.close(); + verify(sysOut).println(baos.toString("UTF-8")); + } + + @Test + public void testGetEmptyClusterNodeLabels() throws Exception { + YarnClient client = mock(YarnClient.class); + when(client.getClusterNodeLabels()).thenReturn(new HashSet()); + ClusterCLI cli = new ClusterCLI(); + cli.setClient(client); + cli.setSysOutPrintStream(sysOut); + cli.setSysErrPrintStream(sysErr); + + int rc = + cli.run(new String[] { ClusterCLI.CMD, "-" + ClusterCLI.LIST_LABELS_CMD }); + assertEquals(0, rc); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + PrintWriter pw = new PrintWriter(baos); + pw.print("Node Labels: "); + pw.close(); + verify(sysOut).println(baos.toString("UTF-8")); + } + + @Test + public void testHelp() throws Exception { + ClusterCLI cli = new ClusterCLI(); + cli.setSysOutPrintStream(sysOut); + cli.setSysErrPrintStream(sysErr); + + int rc = + cli.run(new String[] { "cluster", "--help" }); + assertEquals(0, rc); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + PrintWriter pw = new PrintWriter(baos); + pw.println("usage: yarn cluster"); + pw.println(" -dnl,--directly-access-node-label-store Directly access node label"); + pw.println(" store, with this option, all"); + pw.println(" node label related operations"); + pw.println(" will NOT connect RM. Instead,"); + pw.println(" they will access/modify stored"); + pw.println(" node labels directly. By"); + pw.println(" default, it is false (access"); + pw.println(" via RM). AND PLEASE NOTE: if"); + pw.println(" you configured"); + pw.println(" yarn.node-labels.fs-store.root-"); + pw.println(" dir to a local directory"); + pw.println(" (instead of NFS or HDFS), this"); + pw.println(" option will only work when the"); + pw.println(" command run on the machine"); + pw.println(" where RM is running. Also, this"); + pw.println(" option is UNSTABLE, could be"); + pw.println(" removed in future releases."); + pw.println(" -h,--help Displays help for all commands."); + pw.println(" -lnl,--list-node-labels List cluster node-label"); + pw.println(" collection"); + pw.close(); + verify(sysOut).println(baos.toString("UTF-8")); + } +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java index c22494c9eb6fe..606711008cba7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java @@ -364,16 +364,15 @@ public void testHelp() throws Exception { assertEquals(0, rmAdminCLIWithHAEnabled.run(args)); oldOutPrintStream.println(dataOut); String expectedHelpMsg = - "yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshSuper" + - "UserGroupsConfiguration] [-refreshUserToGroupsMappings] " + - "[-refreshAdminAcls] [-refreshServiceAcl] [-getGroup" + - " [username]] [[-addToClusterNodeLabels [label1,label2,label3]]" + - " [-removeFromClusterNodeLabels [label1,label2,label3]] [-replaceLabelsOnNode " + - "[node1[:port]=label1,label2 node2[:port]=label1] [-directlyAccessNodeLabelStore]] " + - "[-transitionToActive [--forceactive] ] " + - "[-transitionToStandby ] [-failover" + - " [--forcefence] [--forceactive] ] " + - "[-getServiceState ] [-checkHealth ] [-help [cmd]]"; + "yarn rmadmin [-refreshQueues] [-refreshNodes] [-refreshSuper" + + "UserGroupsConfiguration] [-refreshUserToGroupsMappings] " + + "[-refreshAdminAcls] [-refreshServiceAcl] [-getGroup" + + " [username]] [[-addToClusterNodeLabels [label1,label2,label3]]" + + " [-removeFromClusterNodeLabels [label1,label2,label3]] [-replaceLabelsOnNode " + + "[node1[:port]=label1,label2 node2[:port]=label1] [-directlyAccessNodeLabelStore]] " + + "[-transitionToActive [--forceactive] ] " + + "[-transitionToStandby ] " + + "[-getServiceState ] [-checkHealth ] [-help [cmd]]"; String actualHelpMsg = dataOut.toString(); assertTrue(String.format("Help messages: %n " + actualHelpMsg + " %n doesn't include expected " + "messages: %n" + expectedHelpMsg), actualHelpMsg.contains(expectedHelpMsg diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java index 088969f6b32b7..4b60c521673f3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java @@ -224,7 +224,8 @@ public void testGetContainerReport() throws Exception { ContainerId containerId = ContainerId.newContainerId(attemptId, 1); ContainerReport container = ContainerReport.newInstance(containerId, null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, 1234, 5678, - "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE); + "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE, + "http://" + NodeId.newInstance("host", 2345).toString()); when(client.getContainerReport(any(ContainerId.class))).thenReturn( container); int result = cli.run(new String[] { "container", "-status", @@ -240,6 +241,7 @@ public void testGetContainerReport() throws Exception { pw.println("\tState : COMPLETE"); pw.println("\tLOG-URL : logURL"); pw.println("\tHost : host:1234"); + pw.println("\tNodeHttpAddress : http://host:2345"); pw.println("\tDiagnostics : diagnosticInfo"); pw.close(); String appReportStr = baos.toString("UTF-8"); @@ -259,13 +261,16 @@ public void testGetContainers() throws Exception { long time1=1234,time2=5678; ContainerReport container = ContainerReport.newInstance(containerId, null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, time1, time2, - "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE); + "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE, + "http://" + NodeId.newInstance("host", 2345).toString()); ContainerReport container1 = ContainerReport.newInstance(containerId1, null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, time1, time2, - "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE); + "diagnosticInfo", "logURL", 0, ContainerState.COMPLETE, + "http://" + NodeId.newInstance("host", 2345).toString()); ContainerReport container2 = ContainerReport.newInstance(containerId2, null, NodeId.newInstance("host", 1234), Priority.UNDEFINED, time1,0, - "diagnosticInfo", "", 0, ContainerState.RUNNING); + "diagnosticInfo", "", 0, ContainerState.RUNNING, + "http://" + NodeId.newInstance("host", 2345).toString()); List reports = new ArrayList(); reports.add(container); reports.add(container1); @@ -273,6 +278,7 @@ public void testGetContainers() throws Exception { DateFormat dateFormat=new SimpleDateFormat("EEE MMM dd HH:mm:ss Z yyyy"); when(client.getContainers(any(ApplicationAttemptId.class))).thenReturn( reports); + sysOutStream.reset(); int result = cli.run(new String[] { "container", "-list", attemptId.toString() }); assertEquals(0, result); @@ -285,24 +291,28 @@ public void testGetContainers() throws Exception { pw.print("\t Finish Time"); pw.print("\t State"); pw.print("\t Host"); + pw.print("\t Node Http Address"); pw.println("\t LOG-URL"); pw.print(" container_1234_0005_01_000001"); pw.print("\t"+dateFormat.format(new Date(time1))); pw.print("\t"+dateFormat.format(new Date(time2))); pw.print("\t COMPLETE"); pw.print("\t host:1234"); + pw.print("\t http://host:2345"); pw.println("\t logURL"); pw.print(" container_1234_0005_01_000002"); pw.print("\t"+dateFormat.format(new Date(time1))); pw.print("\t"+dateFormat.format(new Date(time2))); pw.print("\t COMPLETE"); pw.print("\t host:1234"); + pw.print("\t http://host:2345"); pw.println("\t logURL"); pw.print(" container_1234_0005_01_000003"); pw.print("\t"+dateFormat.format(new Date(time1))); pw.print("\t N/A"); pw.print("\t RUNNING"); pw.print("\t host:1234"); + pw.print("\t http://host:2345"); pw.println("\t "); pw.close(); String appReportStr = baos.toString("UTF-8"); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerReportPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerReportPBImpl.java index 18c452f499934..1f0405f7274a5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerReportPBImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerReportPBImpl.java @@ -338,4 +338,23 @@ private ContainerState convertFromProtoFormat( ContainerStateProto containerState) { return ProtoUtils.convertFromProtoFormat(containerState); } + + @Override + public String getNodeHttpAddress() { + ContainerReportProtoOrBuilder p = viaProto ? proto : builder; + if (!p.hasNodeHttpAddress()) { + return null; + } + return (p.getNodeHttpAddress()); + } + + @Override + public void setNodeHttpAddress(String nodeHttpAddress) { + maybeInitBuilder(); + if (nodeHttpAddress == null) { + builder.clearNodeHttpAddress(); + return; + } + builder.setNodeHttpAddress(nodeHttpAddress); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LogAggregationContextPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LogAggregationContextPBImpl.java index dc7a21d2a5b4d..f6409bb582537 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LogAggregationContextPBImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/LogAggregationContextPBImpl.java @@ -20,6 +20,7 @@ import org.apache.hadoop.yarn.api.records.LogAggregationContext; import org.apache.hadoop.yarn.proto.YarnProtos.LogAggregationContextProto; import org.apache.hadoop.yarn.proto.YarnProtos.LogAggregationContextProtoOrBuilder; + import com.google.protobuf.TextFormat; public class LogAggregationContextPBImpl extends LogAggregationContext{ @@ -116,4 +117,42 @@ public void setExcludePattern(String excludePattern) { } builder.setExcludePattern(excludePattern); } + + @Override + public String getRolledLogsIncludePattern() { + LogAggregationContextProtoOrBuilder p = viaProto ? proto : builder; + if (! p.hasRolledLogsIncludePattern()) { + return null; + } + return p.getRolledLogsIncludePattern(); + } + + @Override + public void setRolledLogsIncludePattern(String rolledLogsIncludePattern) { + maybeInitBuilder(); + if (rolledLogsIncludePattern == null) { + builder.clearRolledLogsIncludePattern(); + return; + } + builder.setRolledLogsIncludePattern(rolledLogsIncludePattern); + } + + @Override + public String getRolledLogsExcludePattern() { + LogAggregationContextProtoOrBuilder p = viaProto ? proto : builder; + if (! p.hasRolledLogsExcludePattern()) { + return null; + } + return p.getRolledLogsExcludePattern(); + } + + @Override + public void setRolledLogsExcludePattern(String rolledLogsExcludePattern) { + maybeInitBuilder(); + if (rolledLogsExcludePattern == null) { + builder.clearRolledLogsExcludePattern(); + return; + } + builder.setRolledLogsExcludePattern(rolledLogsExcludePattern); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeLabelPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeLabelPBImpl.java new file mode 100644 index 0000000000000..661359eaf7956 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NodeLabelPBImpl.java @@ -0,0 +1,106 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.yarn.api.records.impl.pb; + +import org.apache.hadoop.yarn.api.records.NodeLabel; +import org.apache.hadoop.yarn.proto.YarnProtos.NodeLabelProto; +import org.apache.hadoop.yarn.proto.YarnProtos.NodeLabelProtoOrBuilder; + +public class NodeLabelPBImpl extends NodeLabel { + NodeLabelProto proto = + NodeLabelProto.getDefaultInstance(); + NodeLabelProto.Builder builder = null; + boolean viaProto = false; + + public NodeLabelPBImpl() { + builder = NodeLabelProto.newBuilder(); + } + + public NodeLabelPBImpl(NodeLabelProto proto) { + this.proto = proto; + viaProto = true; + } + + public NodeLabelProto getProto() { + mergeLocalToProto(); + proto = viaProto ? proto : builder.build(); + viaProto = true; + return proto; + } + + private void mergeLocalToProto() { + if (viaProto) + maybeInitBuilder(); + proto = builder.build(); + viaProto = true; + } + + @Override + public boolean equals(Object other) { + if (other == null) + return false; + if (other.getClass().isAssignableFrom(this.getClass())) { + return this.getProto().equals(this.getClass().cast(other).getProto()); + } + return false; + } + + private void maybeInitBuilder() { + if (viaProto || builder == null) { + builder = NodeLabelProto.newBuilder(proto); + } + viaProto = false; + } + + @Override + public int hashCode() { + return getProto().hashCode(); + } + + @Override + public String getNodeLabel() { + NodeLabelProtoOrBuilder p = viaProto ? proto : builder; + if (!p.hasNodeLabel()) { + return null; + } + return (p.getNodeLabel()); + } + + @Override + public void setNodeLabel(String nodeLabel) { + maybeInitBuilder(); + if (nodeLabel == null) { + builder.clearNodeLabel(); + return; + } + builder.setNodeLabel(nodeLabel); + } + + @Override + public boolean getIsExclusive() { + NodeLabelProtoOrBuilder p = viaProto ? proto : builder; + return p.getIsExclusive(); + } + + @Override + public void setIsExclusive(boolean isExclusive) { + maybeInitBuilder(); + builder.setIsExclusive(isExclusive); + } + +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java index af684929e1dc0..df6c7a46aad96 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java @@ -47,6 +47,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.SecurityUtil; +import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.security.authentication.client.ConnectionConfigurator; import org.apache.hadoop.security.ssl.SSLFactory; import org.apache.hadoop.security.token.Token; @@ -108,6 +109,8 @@ public class TimelineClientImpl extends TimelineClient { private DelegationTokenAuthenticator authenticator; private DelegationTokenAuthenticatedURL.Token token; private URI resURI; + private UserGroupInformation authUgi; + private String doAsUser; @Private @VisibleForTesting @@ -252,6 +255,15 @@ public TimelineClientImpl() { } protected void serviceInit(Configuration conf) throws Exception { + UserGroupInformation ugi = UserGroupInformation.getCurrentUser(); + UserGroupInformation realUgi = ugi.getRealUser(); + if (realUgi != null) { + authUgi = realUgi; + doAsUser = ugi.getShortUserName(); + } else { + authUgi = ugi; + doAsUser = null; + } ClientConfig cc = new DefaultClientConfig(); cc.getClasses().add(YarnJacksonJaxbJsonProvider.class); connConfigurator = newConnConfigurator(conf); @@ -301,16 +313,20 @@ public void putDomain(TimelineDomain domain) throws IOException, doPosting(domain, "domain"); } - private ClientResponse doPosting(Object obj, String path) throws IOException, YarnException { + private ClientResponse doPosting(final Object obj, final String path) + throws IOException, YarnException { ClientResponse resp; try { - resp = doPostingObject(obj, path); - } catch (RuntimeException re) { - // runtime exception is expected if the client cannot connect the server - String msg = - "Failed to get the response from the timeline server."; - LOG.error(msg, re); - throw re; + resp = authUgi.doAs(new PrivilegedExceptionAction() { + @Override + public ClientResponse run() throws Exception { + return doPostingObject(obj, path); + } + }); + } catch (UndeclaredThrowableException e) { + throw new IOException(e.getCause()); + } catch (InterruptedException ie) { + throw new IOException(ie); } if (resp == null || resp.getClientResponseStatus() != ClientResponse.Status.OK) { @@ -331,11 +347,6 @@ private ClientResponse doPosting(Object obj, String path) throws IOException, Ya @Override public Token getDelegationToken( final String renewer) throws IOException, YarnException { - boolean isProxyAccess = - UserGroupInformation.getCurrentUser().getAuthenticationMethod() - == UserGroupInformation.AuthenticationMethod.PROXY; - final String doAsUser = isProxyAccess ? - UserGroupInformation.getCurrentUser().getShortUserName() : null; PrivilegedExceptionAction> getDTAction = new PrivilegedExceptionAction>() { @@ -357,11 +368,6 @@ public Token run() public long renewDelegationToken( final Token timelineDT) throws IOException, YarnException { - boolean isProxyAccess = - UserGroupInformation.getCurrentUser().getAuthenticationMethod() - == UserGroupInformation.AuthenticationMethod.PROXY; - final String doAsUser = isProxyAccess ? - UserGroupInformation.getCurrentUser().getShortUserName() : null; boolean useHttps = YarnConfiguration.useHttps(this.getConfig()); final String scheme = useHttps ? "https" : "http"; final InetSocketAddress address = SecurityUtil.getTokenServiceAddr(timelineDT); @@ -393,11 +399,6 @@ public Long run() throws Exception { public void cancelDelegationToken( final Token timelineDT) throws IOException, YarnException { - boolean isProxyAccess = - UserGroupInformation.getCurrentUser().getAuthenticationMethod() - == UserGroupInformation.AuthenticationMethod.PROXY; - final String doAsUser = isProxyAccess ? - UserGroupInformation.getCurrentUser().getShortUserName() : null; boolean useHttps = YarnConfiguration.useHttps(this.getConfig()); final String scheme = useHttps ? "https" : "http"; final InetSocketAddress address = SecurityUtil.getTokenServiceAddr(timelineDT); @@ -433,14 +434,9 @@ private Object operateDelegationToken( @Override public Object run() throws IOException { // Try pass the request, if fail, keep retrying - boolean isProxyAccess = - UserGroupInformation.getCurrentUser().getAuthenticationMethod() - == UserGroupInformation.AuthenticationMethod.PROXY; - UserGroupInformation callerUGI = isProxyAccess ? - UserGroupInformation.getCurrentUser().getRealUser() - : UserGroupInformation.getCurrentUser(); + authUgi.checkTGTAndReloginFromKeytab(); try { - return callerUGI.doAs(action); + return authUgi.doAs(action); } catch (UndeclaredThrowableException e) { throw new IOException(e.getCause()); } catch (InterruptedException e) { @@ -480,27 +476,15 @@ private class TimelineURLConnectionFactory @Override public HttpURLConnection getHttpURLConnection(final URL url) throws IOException { - boolean isProxyAccess = - UserGroupInformation.getCurrentUser().getAuthenticationMethod() - == UserGroupInformation.AuthenticationMethod.PROXY; - UserGroupInformation callerUGI = isProxyAccess ? - UserGroupInformation.getCurrentUser().getRealUser() - : UserGroupInformation.getCurrentUser(); - final String doAsUser = isProxyAccess ? - UserGroupInformation.getCurrentUser().getShortUserName() : null; + authUgi.checkTGTAndReloginFromKeytab(); try { - return callerUGI.doAs(new PrivilegedExceptionAction() { - @Override - public HttpURLConnection run() throws Exception { - return new DelegationTokenAuthenticatedURL( - authenticator, connConfigurator).openConnection(url, token, - doAsUser); - } - }); + return new DelegationTokenAuthenticatedURL( + authenticator, connConfigurator).openConnection(url, token, + doAsUser); } catch (UndeclaredThrowableException e) { throw new IOException(e.getCause()); - } catch (InterruptedException e) { - throw new IOException(e); + } catch (AuthenticationException ae) { + throw new IOException(ae); } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java index ada066948aa5e..1086ff03f528f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java @@ -70,6 +70,21 @@ private static T instantiateException( } } + private static T instantiateYarnException( + Class cls, RemoteException re) throws RemoteException { + return instantiateException(cls, re); + } + + private static T instantiateIOException( + Class cls, RemoteException re) throws RemoteException { + return instantiateException(cls, re); + } + + private static T instantiateRuntimeException( + Class cls, RemoteException re) throws RemoteException { + return instantiateException(cls, re); + } + /** * Utility method that unwraps and returns appropriate exceptions. * @@ -94,17 +109,17 @@ public static Void unwrapAndThrowException(ServiceException se) // Assume this to be a new exception type added to YARN. This isn't // absolutely correct since the RPC layer could add an exception as // well. - throw instantiateException(YarnException.class, re); + throw instantiateYarnException(YarnException.class, re); } if (YarnException.class.isAssignableFrom(realClass)) { - throw instantiateException( + throw instantiateYarnException( realClass.asSubclass(YarnException.class), re); } else if (IOException.class.isAssignableFrom(realClass)) { - throw instantiateException(realClass.asSubclass(IOException.class), + throw instantiateIOException(realClass.asSubclass(IOException.class), re); } else if (RuntimeException.class.isAssignableFrom(realClass)) { - throw instantiateException( + throw instantiateRuntimeException( realClass.asSubclass(RuntimeException.class), re); } else { throw re; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java index b6693326eaef6..57f655b497ad0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java @@ -167,17 +167,18 @@ public static class LogValue { private Set uploadedFiles = new HashSet(); private final Set alreadyUploadedLogFiles; private Set allExistingFileMeta = new HashSet(); + private final boolean appFinished; // TODO Maybe add a version string here. Instead of changing the version of // the entire k-v format public LogValue(List rootLogDirs, ContainerId containerId, String user) { - this(rootLogDirs, containerId, user, null, new HashSet()); + this(rootLogDirs, containerId, user, null, new HashSet(), true); } public LogValue(List rootLogDirs, ContainerId containerId, String user, LogAggregationContext logAggregationContext, - Set alreadyUploadedLogFiles) { + Set alreadyUploadedLogFiles, boolean appFinished) { this.rootLogDirs = new ArrayList(rootLogDirs); this.containerId = containerId; this.user = user; @@ -186,6 +187,7 @@ public LogValue(List rootLogDirs, ContainerId containerId, Collections.sort(this.rootLogDirs); this.logAggregationContext = logAggregationContext; this.alreadyUploadedLogFiles = alreadyUploadedLogFiles; + this.appFinished = appFinished; } private Set getPendingLogFilesToUploadForThisContainer() { @@ -296,17 +298,15 @@ private Set getPendingLogFilesToUpload(File containerLogDir) { } if (this.logAggregationContext != null && candidates.size() > 0) { - if (this.logAggregationContext.getIncludePattern() != null - && !this.logAggregationContext.getIncludePattern().isEmpty()) { - filterFiles(this.logAggregationContext.getIncludePattern(), - candidates, false); - } + filterFiles( + this.appFinished ? this.logAggregationContext.getIncludePattern() + : this.logAggregationContext.getRolledLogsIncludePattern(), + candidates, false); - if (this.logAggregationContext.getExcludePattern() != null - && !this.logAggregationContext.getExcludePattern().isEmpty()) { - filterFiles(this.logAggregationContext.getExcludePattern(), - candidates, true); - } + filterFiles( + this.appFinished ? this.logAggregationContext.getExcludePattern() + : this.logAggregationContext.getRolledLogsExcludePattern(), + candidates, true); Iterable mask = Iterables.filter(candidates, new Predicate() { @@ -323,14 +323,15 @@ public boolean apply(File next) { private void filterFiles(String pattern, Set candidates, boolean exclusion) { - Pattern filterPattern = - Pattern.compile(pattern); - for (Iterator candidatesItr = candidates.iterator(); candidatesItr + if (pattern != null && !pattern.isEmpty()) { + Pattern filterPattern = Pattern.compile(pattern); + for (Iterator candidatesItr = candidates.iterator(); candidatesItr .hasNext();) { - File candidate = candidatesItr.next(); - boolean match = filterPattern.matcher(candidate.getName()).find(); - if ((!match && !exclusion) || (match && exclusion)) { - candidatesItr.remove(); + File candidate = candidatesItr.next(); + boolean match = filterPattern.matcher(candidate.getName()).find(); + if ((!match && !exclusion) || (match && exclusion)) { + candidatesItr.remove(); + } } } } @@ -378,7 +379,7 @@ public LogWriter(final Configuration conf, final Path remoteAppLogFile, userUgi.doAs(new PrivilegedExceptionAction() { @Override public FSDataOutputStream run() throws Exception { - fc = FileContext.getFileContext(conf); + fc = FileContext.getFileContext(remoteAppLogFile.toUri(), conf); fc.setUMask(APP_LOG_FILE_UMASK); return fc.create( remoteAppLogFile, @@ -470,7 +471,8 @@ public static class LogReader { public LogReader(Configuration conf, Path remoteAppLogFile) throws IOException { - FileContext fileContext = FileContext.getFileContext(conf); + FileContext fileContext = + FileContext.getFileContext(remoteAppLogFile.toUri(), conf); this.fsDataIStream = fileContext.open(remoteAppLogFile); reader = new TFile.Reader(this.fsDataIStream, fileContext.getFileStatus( @@ -710,6 +712,7 @@ private static void readContainerLogs(DataInputStream valueStream, pendingRead > buf.length ? buf.length : (int) pendingRead; len = valueStream.read(buf, 0, toRead); } + out.println("End of LogType:" + fileType); out.println(""); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java index e2da664a1829f..a5e2756aee7a6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/CommonNodeLabelsManager.java @@ -24,6 +24,7 @@ import java.util.EnumSet; import java.util.HashMap; import java.util.HashSet; +import java.util.List; import java.util.Map; import java.util.Map.Entry; import java.util.Set; @@ -40,16 +41,19 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.service.AbstractService; import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.api.records.NodeLabel; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.event.AsyncDispatcher; import org.apache.hadoop.yarn.event.Dispatcher; import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; import org.apache.hadoop.yarn.nodelabels.event.NodeLabelsStoreEvent; import org.apache.hadoop.yarn.nodelabels.event.NodeLabelsStoreEventType; import org.apache.hadoop.yarn.nodelabels.event.RemoveClusterNodeLabels; import org.apache.hadoop.yarn.nodelabels.event.StoreNewClusterNodeLabels; +import org.apache.hadoop.yarn.nodelabels.event.StoreUpdateNodeLabelsEvent; import org.apache.hadoop.yarn.nodelabels.event.UpdateNodeToLabelsMappingsEvent; import org.apache.hadoop.yarn.util.resource.Resources; @@ -83,8 +87,8 @@ public class CommonNodeLabelsManager extends AbstractService { protected Dispatcher dispatcher; - protected ConcurrentMap labelCollections = - new ConcurrentHashMap(); + protected ConcurrentMap labelCollections = + new ConcurrentHashMap(); protected ConcurrentMap nodeCollections = new ConcurrentHashMap(); @@ -181,6 +185,13 @@ protected void handleStoreEvent(NodeLabelsStoreEvent event) { store.updateNodeToLabelsMappings(updateNodeToLabelsMappingsEvent .getNodeToLabels()); break; + case UPDATE_NODE_LABELS: + StoreUpdateNodeLabelsEvent + storeSetNodeLabelsEventEvent = + (StoreUpdateNodeLabelsEvent) event; + store.updateNodeLabels(storeSetNodeLabelsEventEvent + .getUpdatedNodeLabels()); + break; } } catch (IOException e) { LOG.error("Failed to store label modification to storage"); @@ -214,7 +225,7 @@ protected void serviceInit(Configuration conf) throws Exception { initNodeLabelStore(conf); } - labelCollections.put(NO_LABEL, new NodeLabel(NO_LABEL)); + labelCollections.put(NO_LABEL, new RMNodeLabel(NO_LABEL)); } protected void initNodeLabelStore(Configuration conf) throws Exception { @@ -288,7 +299,7 @@ public void addToCluserNodeLabels(Set labels) throws IOException { for (String label : labels) { // shouldn't overwrite it to avoid changing the Label.resource if (this.labelCollections.get(label) == null) { - this.labelCollections.put(label, new NodeLabel(label)); + this.labelCollections.put(label, new RMNodeLabel(label)); newLabels.add(label); } } @@ -344,7 +355,7 @@ protected void checkAddLabelsToNode( /** * add more labels to nodes * - * @param addedLabelsToNode node -> labels map + * @param addedLabelsToNode node {@literal ->} labels map */ public void addLabelsToNode(Map> addedLabelsToNode) throws IOException { @@ -614,7 +625,7 @@ protected void internalUpdateLabelsOnNodes( * remove labels from nodes, labels being removed most be contained by these * nodes * - * @param removeLabelsFromNode node -> labels map + * @param removeLabelsFromNode node {@literal ->} labels map */ public void removeLabelsFromNode(Map> removeLabelsFromNode) @@ -668,7 +679,7 @@ protected void checkReplaceLabelsOnNode( /** * replace labels to nodes * - * @param replaceLabelsToNode node -> labels map + * @param replaceLabelsToNode node {@literal ->} labels map */ public void replaceLabelsOnNode(Map> replaceLabelsToNode) throws IOException { @@ -746,7 +757,7 @@ public Map> getLabelsToNodes(Set labels) { if(label.equals(NO_LABEL)) { continue; } - NodeLabel nodeLabelInfo = labelCollections.get(label); + RMNodeLabel nodeLabelInfo = labelCollections.get(label); if(nodeLabelInfo != null) { Set nodeIds = nodeLabelInfo.getAssociatedNodeIds(); if (!nodeIds.isEmpty()) { @@ -777,6 +788,60 @@ public Set getClusterNodeLabels() { readLock.unlock(); } } + + private void checkUpdateNodeLabels( + List updatedNodeLabels) throws YarnException { + // pre-check + for (NodeLabel label : updatedNodeLabels) { + if (!labelCollections.containsKey(label.getNodeLabel())) { + String message = + String.format( + "Trying to update a non-existing node-label=%s", + label.getNodeLabel()); + LOG.error(message); + throw new YarnException(message); + } + } + } + + @SuppressWarnings("unchecked") + public void updateNodeLabels( + List updatedNodeLabels) throws YarnException { + try { + writeLock.lock(); + checkUpdateNodeLabels(updatedNodeLabels); + + for (NodeLabel label : updatedNodeLabels) { + RMNodeLabel rmLabel = labelCollections.get(label.getNodeLabel()); + rmLabel.setIsExclusive(label.getIsExclusive()); + } + + if (null != dispatcher && !updatedNodeLabels.isEmpty()) { + dispatcher.getEventHandler().handle( + new StoreUpdateNodeLabelsEvent(updatedNodeLabels)); + } + } finally { + writeLock.unlock(); + } + } + + public boolean isExclusiveNodeLabel(String nodeLabel) throws IOException { + try { + readLock.lock(); + RMNodeLabel label = labelCollections.get(nodeLabel); + if (label == null) { + String message = + "Getting is-exclusive-node-label, node-label = " + nodeLabel + + ", is not existed."; + LOG.error(message); + throw new IOException(message); + } + return label.getIsExclusive(); + } finally { + readLock.unlock(); + } + } + private void checkAndThrowLabelName(String label) throws IOException { if (label == null || label.isEmpty() || label.length() > MAX_LABEL_LENGTH) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java index 6e685ee3301d6..0c7192ff78d39 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java @@ -21,6 +21,7 @@ import java.io.EOFException; import java.io.IOException; import java.util.Collection; +import java.util.List; import java.util.Map; import java.util.Set; @@ -34,16 +35,21 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.api.records.NodeLabel; import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.AddToClusterNodeLabelsRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RemoveFromClusterNodeLabelsRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.ReplaceLabelsOnNodeRequestProto; +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsRequestProto; import org.apache.hadoop.yarn.server.api.protocolrecords.AddToClusterNodeLabelsRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.RemoveFromClusterNodeLabelsRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.AddToClusterNodeLabelsRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RemoveFromClusterNodeLabelsRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeRequestPBImpl; +import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsRequestPBImpl; import com.google.common.collect.Sets; @@ -60,7 +66,7 @@ public FileSystemNodeLabelsStore(CommonNodeLabelsManager mgr) { protected static final String EDITLOG_FILENAME = "nodelabel.editlog"; protected enum SerializedLogType { - ADD_LABELS, NODE_TO_LABELS, REMOVE_LABELS + ADD_LABELS, NODE_TO_LABELS, REMOVE_LABELS, UPDATE_NODE_LABELS } Path fsWorkingPath; @@ -150,9 +156,19 @@ public void removeClusterNodeLabels(Collection labels) .newHashSet(labels.iterator()))).getProto().writeDelimitedTo(editlogOs); ensureCloseEditlogFile(); } + + @Override + public void updateNodeLabels(List updatedNodeLabels) + throws IOException { + ensureAppendEditlogFile(); + editlogOs.writeInt(SerializedLogType.UPDATE_NODE_LABELS.ordinal()); + ((UpdateNodeLabelsRequestPBImpl) UpdateNodeLabelsRequest + .newInstance(updatedNodeLabels)).getProto().writeDelimitedTo(editlogOs); + ensureCloseEditlogFile(); + } @Override - public void recover() throws IOException { + public void recover() throws YarnException, IOException { /* * Steps of recover * 1) Read from last mirror (from mirror or mirror.old) @@ -221,6 +237,14 @@ public void recover() throws IOException { mgr.replaceLabelsOnNode(map); break; } + case UPDATE_NODE_LABELS: { + List attributes = + new UpdateNodeLabelsRequestPBImpl( + UpdateNodeLabelsRequestProto.parseDelimitedFrom(is)) + .getNodeLabels(); + mgr.updateNodeLabels(attributes); + break; + } } } catch (EOFException e) { // EOF hit, break diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelsStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelsStore.java index 857d81b115b85..6bd90db62ca9a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelsStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabelsStore.java @@ -21,11 +21,14 @@ import java.io.Closeable; import java.io.IOException; import java.util.Collection; +import java.util.List; import java.util.Map; import java.util.Set; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.api.records.NodeLabel; +import org.apache.hadoop.yarn.exceptions.YarnException; public abstract class NodeLabelsStore implements Closeable { protected final CommonNodeLabelsManager mgr; @@ -35,7 +38,7 @@ public NodeLabelsStore(CommonNodeLabelsManager mgr) { } /** - * Store node -> label + * Store node {@literal ->} label */ public abstract void updateNodeToLabelsMappings( Map> nodeToLabels) throws IOException; @@ -52,11 +55,16 @@ public abstract void storeNewClusterNodeLabels(Set label) public abstract void removeClusterNodeLabels(Collection labels) throws IOException; + /** + * Update node labels + */ + public abstract void updateNodeLabels( + List updatedNodeLabels) throws IOException; + /** * Recover labels and node to labels mappings from store - * @param conf */ - public abstract void recover() throws IOException; + public abstract void recover() throws IOException, YarnException; public void init(Configuration conf) throws Exception {} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabel.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeLabel.java similarity index 82% rename from hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabel.java rename to hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeLabel.java index 1765a65b26c30..7638e21ed31bc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/NodeLabel.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/RMNodeLabel.java @@ -25,17 +25,18 @@ import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.util.resource.Resources; -public class NodeLabel implements Comparable { +public class RMNodeLabel implements Comparable { private Resource resource; private int numActiveNMs; private String labelName; private Set nodeIds; + private boolean exclusive = true; - public NodeLabel(String labelName) { + public RMNodeLabel(String labelName) { this(labelName, Resource.newInstance(0, 0), 0); } - protected NodeLabel(String labelName, Resource res, int activeNMs) { + protected RMNodeLabel(String labelName, Resource res, int activeNMs) { this.labelName = labelName; this.resource = res; this.numActiveNMs = activeNMs; @@ -76,12 +77,20 @@ public String getLabelName() { return labelName; } - public NodeLabel getCopy() { - return new NodeLabel(labelName, resource, numActiveNMs); + public void setIsExclusive(boolean exclusive) { + this.exclusive = exclusive; + } + + public boolean getIsExclusive() { + return this.exclusive; + } + + public RMNodeLabel getCopy() { + return new RMNodeLabel(labelName, resource, numActiveNMs); } @Override - public int compareTo(NodeLabel o) { + public int compareTo(RMNodeLabel o) { // We should always put empty label entry first after sorting if (labelName.isEmpty() != o.getLabelName().isEmpty()) { if (labelName.isEmpty()) { @@ -95,8 +104,8 @@ public int compareTo(NodeLabel o) { @Override public boolean equals(Object obj) { - if (obj instanceof NodeLabel) { - NodeLabel other = (NodeLabel) obj; + if (obj instanceof RMNodeLabel) { + RMNodeLabel other = (RMNodeLabel) obj; return Resources.equals(resource, other.getResource()) && StringUtils.equals(labelName, other.getLabelName()) && (other.getNumActiveNMs() == numActiveNMs); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/NodeLabelsStoreEventType.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/NodeLabelsStoreEventType.java index efa2dbebac81f..ce2917609c334 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/NodeLabelsStoreEventType.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/NodeLabelsStoreEventType.java @@ -21,5 +21,6 @@ public enum NodeLabelsStoreEventType { REMOVE_LABELS, ADD_LABELS, - STORE_NODE_TO_LABELS + STORE_NODE_TO_LABELS, + UPDATE_NODE_LABELS } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/StoreUpdateNodeLabelsEvent.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/StoreUpdateNodeLabelsEvent.java new file mode 100644 index 0000000000000..7c050123616ef --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/event/StoreUpdateNodeLabelsEvent.java @@ -0,0 +1,36 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.nodelabels.event; + +import java.util.List; + +import org.apache.hadoop.yarn.api.records.NodeLabel; + +public class StoreUpdateNodeLabelsEvent extends NodeLabelsStoreEvent { + private List updatedNodeLabels; + + public StoreUpdateNodeLabelsEvent(List updateNodeLabels) { + super(NodeLabelsStoreEventType.UPDATE_NODE_LABELS); + this.updatedNodeLabels = updateNodeLabels; + } + + public List getUpdatedNodeLabels() { + return updatedNodeLabels; + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java index c2d813b8b5745..8cb225f9f734d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceManagerAdministrationProtocolPBClientImpl.java @@ -40,6 +40,7 @@ import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshUserToGroupsMappingsRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RemoveFromClusterNodeLabelsRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.ReplaceLabelsOnNodeRequestProto; +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeResourceRequestProto; import org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocol; import org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB; @@ -61,6 +62,8 @@ import org.apache.hadoop.yarn.server.api.protocolrecords.RemoveFromClusterNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeResponse; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.AddToClusterNodeLabelsRequestPBImpl; @@ -81,6 +84,8 @@ import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RemoveFromClusterNodeLabelsResponsePBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeResponsePBImpl; +import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsRequestPBImpl; +import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsResponsePBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeResourceRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeResourceResponsePBImpl; @@ -263,4 +268,18 @@ public ReplaceLabelsOnNodeResponse replaceLabelsOnNode( return null; } } + + @Override + public UpdateNodeLabelsResponse updateNodeLabels( + UpdateNodeLabelsRequest request) throws YarnException, IOException { + UpdateNodeLabelsRequestProto requestProto = + ((UpdateNodeLabelsRequestPBImpl) request).getProto(); + try { + return new UpdateNodeLabelsResponsePBImpl( + proxy.updateNodeLabels(null, requestProto)); + } catch (ServiceException e) { + RPCUtil.unwrapAndThrowException(e); + return null; + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java index 0eaf5811e6011..fe4c812c22584 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceManagerAdministrationProtocolPBServiceImpl.java @@ -42,6 +42,8 @@ import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RemoveFromClusterNodeLabelsResponseProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.ReplaceLabelsOnNodeRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.ReplaceLabelsOnNodeResponseProto; +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsRequestProto; +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsResponseProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeResourceRequestProto; import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeResourceResponseProto; import org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocol; @@ -55,6 +57,8 @@ import org.apache.hadoop.yarn.server.api.protocolrecords.RefreshUserToGroupsMappingsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.RemoveFromClusterNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeResponse; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.AddToClusterNodeLabelsRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.AddToClusterNodeLabelsResponsePBImpl; @@ -74,6 +78,8 @@ import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RemoveFromClusterNodeLabelsResponsePBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeResponsePBImpl; +import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsRequestPBImpl; +import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsResponsePBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeResourceRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeResourceResponsePBImpl; @@ -268,4 +274,21 @@ public ReplaceLabelsOnNodeResponseProto replaceLabelsOnNodes( throw new ServiceException(e); } } + + @Override + public UpdateNodeLabelsResponseProto updateNodeLabels( + RpcController controller, UpdateNodeLabelsRequestProto proto) + throws ServiceException { + UpdateNodeLabelsRequest request = + new UpdateNodeLabelsRequestPBImpl(proto); + try { + UpdateNodeLabelsResponse response = + real.updateNodeLabels(request); + return ((UpdateNodeLabelsResponsePBImpl) response).getProto(); + } catch (YarnException e) { + throw new ServiceException(e); + } catch (IOException e) { + throw new ServiceException(e); + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/UpdateNodeLabelsRequestPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/UpdateNodeLabelsRequestPBImpl.java new file mode 100644 index 0000000000000..f18a8db048521 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/UpdateNodeLabelsRequestPBImpl.java @@ -0,0 +1,145 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.yarn.api.records.NodeLabel; +import org.apache.hadoop.yarn.api.records.impl.pb.NodeLabelPBImpl; +import org.apache.hadoop.yarn.proto.YarnProtos.NodeLabelProto; +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsRequestProto; +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsRequestProtoOrBuilder; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsRequest; + +public class UpdateNodeLabelsRequestPBImpl extends + UpdateNodeLabelsRequest { + UpdateNodeLabelsRequestProto proto = + UpdateNodeLabelsRequestProto.getDefaultInstance(); + UpdateNodeLabelsRequestProto.Builder builder = null; + private List updatedNodeLabels; + boolean viaProto = false; + + public UpdateNodeLabelsRequestPBImpl() { + builder = UpdateNodeLabelsRequestProto.newBuilder(); + } + + public UpdateNodeLabelsRequestPBImpl( + UpdateNodeLabelsRequestProto proto) { + this.proto = proto; + viaProto = true; + } + + public UpdateNodeLabelsRequestProto getProto() { + mergeLocalToProto(); + proto = viaProto ? proto : builder.build(); + viaProto = true; + return proto; + } + + private void mergeLocalToProto() { + if (viaProto) + maybeInitBuilder(); + mergeLocalToBuilder(); + proto = builder.build(); + viaProto = true; + } + + private void mergeLocalToBuilder() { + if (this.updatedNodeLabels != null) { + addNodeLabelsToProto(); + } + } + + private void addNodeLabelsToProto() { + maybeInitBuilder(); + builder.clearNodeLabels(); + List protoList = + new ArrayList(); + for (NodeLabel r : this.updatedNodeLabels) { + protoList.add(convertToProtoFormat(r)); + } + builder.addAllNodeLabels(protoList); + } + + @Override + public boolean equals(Object other) { + if (other == null) + return false; + if (other.getClass().isAssignableFrom(this.getClass())) { + return this.getProto().equals(this.getClass().cast(other).getProto()); + } + return false; + } + + @Override + public int hashCode() { + assert false : "hashCode not designed"; + return 0; + } + + private void maybeInitBuilder() { + if (viaProto || builder == null) { + builder = UpdateNodeLabelsRequestProto.newBuilder(proto); + } + viaProto = false; + } + + @Override + public void setNodeLabels(List updatedNodeLabels) { + maybeInitBuilder(); + if (updatedNodeLabels == null) { + builder.clearNodeLabels(); + } + this.updatedNodeLabels = updatedNodeLabels; + } + + private void initLocalNodeLabels() { + UpdateNodeLabelsRequestProtoOrBuilder p = viaProto ? proto : builder; + List attributesProtoList = + p.getNodeLabelsList(); + this.updatedNodeLabels = new ArrayList(); + for (NodeLabelProto r : attributesProtoList) { + this.updatedNodeLabels.add(convertFromProtoFormat(r)); + } + } + + @Override + public List getNodeLabels() { + if (this.updatedNodeLabels != null) { + return this.updatedNodeLabels; + } + initLocalNodeLabels(); + return this.updatedNodeLabels; + } + + private NodeLabel + convertFromProtoFormat(NodeLabelProto p) { + return new NodeLabelPBImpl(p); + } + + private NodeLabelProto convertToProtoFormat(NodeLabel t) { + return ((NodeLabelPBImpl) t).getProto(); + } + + @Override + public String toString() { + return getProto().toString(); + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/UpdateNodeLabelsResponsePBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/UpdateNodeLabelsResponsePBImpl.java new file mode 100644 index 0000000000000..217b8fc22bde3 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/UpdateNodeLabelsResponsePBImpl.java @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb; + +import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsResponseProto; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsResponse; + +public class UpdateNodeLabelsResponsePBImpl extends + UpdateNodeLabelsResponse { + UpdateNodeLabelsResponseProto proto = + UpdateNodeLabelsResponseProto.getDefaultInstance(); + UpdateNodeLabelsResponseProto.Builder builder = null; + boolean viaProto = false; + + public UpdateNodeLabelsResponsePBImpl() { + builder = UpdateNodeLabelsResponseProto.newBuilder(); + } + + public UpdateNodeLabelsResponsePBImpl( + UpdateNodeLabelsResponseProto proto) { + this.proto = proto; + viaProto = true; + } + + public UpdateNodeLabelsResponseProto getProto() { + proto = viaProto ? proto : builder.build(); + viaProto = true; + return proto; + } + + @Override + public int hashCode() { + return getProto().hashCode(); + } + + @Override + public boolean equals(Object other) { + if (other == null) + return false; + if (other.getClass().isAssignableFrom(this.getClass())) { + return this.getProto().equals(this.getClass().cast(other).getProto()); + } + return false; + } + + @Override + public String toString() { + return getProto().toString().replaceAll("\\n", ", ") + .replaceAll("\\s+", " "); + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/security/ApplicationACLsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/security/ApplicationACLsManager.java index 4cacfca8e2e65..4daaa68312208 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/security/ApplicationACLsManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/security/ApplicationACLsManager.java @@ -93,7 +93,6 @@ public void removeApplication(ApplicationId appId) { * @param applicationAccessType * @param applicationOwner * @param applicationId - * @throws AccessControlException */ public boolean checkAccess(UserGroupInformation callerUGI, ApplicationAccessType applicationAccessType, String applicationOwner, diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/CpuTimeTracker.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/CpuTimeTracker.java new file mode 100644 index 0000000000000..b09a4b6840060 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/CpuTimeTracker.java @@ -0,0 +1,100 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.util; + +import org.apache.hadoop.classification.InterfaceAudience; +import org.apache.hadoop.classification.InterfaceStability; + +import java.math.BigInteger; + +@InterfaceAudience.Private +@InterfaceStability.Unstable +public class CpuTimeTracker { + public static final int UNAVAILABLE = + ResourceCalculatorProcessTree.UNAVAILABLE; + final long MINIMUM_UPDATE_INTERVAL; + + // CPU used time since system is on (ms) + BigInteger cumulativeCpuTime = BigInteger.ZERO; + + // CPU used time read last time (ms) + BigInteger lastCumulativeCpuTime = BigInteger.ZERO; + + // Unix timestamp while reading the CPU time (ms) + long sampleTime; + long lastSampleTime; + float cpuUsage; + BigInteger jiffyLengthInMillis; + + public CpuTimeTracker(long jiffyLengthInMillis) { + this.jiffyLengthInMillis = BigInteger.valueOf(jiffyLengthInMillis); + this.cpuUsage = UNAVAILABLE; + this.sampleTime = UNAVAILABLE; + this.lastSampleTime = UNAVAILABLE; + MINIMUM_UPDATE_INTERVAL = 10 * jiffyLengthInMillis; + } + + /** + * Return percentage of cpu time spent over the time since last update. + * CPU time spent is based on elapsed jiffies multiplied by amount of + * time for 1 core. Thus, if you use 2 cores completely you would have spent + * twice the actual time between updates and this will return 200%. + * + * @return Return percentage of cpu usage since last update, {@link + * CpuTimeTracker#UNAVAILABLE} if there haven't been 2 updates more than + * {@link CpuTimeTracker#MINIMUM_UPDATE_INTERVAL} apart + */ + public float getCpuTrackerUsagePercent() { + if (lastSampleTime == UNAVAILABLE || + lastSampleTime > sampleTime) { + // lastSampleTime > sampleTime may happen when the system time is changed + lastSampleTime = sampleTime; + lastCumulativeCpuTime = cumulativeCpuTime; + return cpuUsage; + } + // When lastSampleTime is sufficiently old, update cpuUsage. + // Also take a sample of the current time and cumulative CPU time for the + // use of the next calculation. + if (sampleTime > lastSampleTime + MINIMUM_UPDATE_INTERVAL) { + cpuUsage = + ((cumulativeCpuTime.subtract(lastCumulativeCpuTime)).floatValue()) + * 100F / ((float) (sampleTime - lastSampleTime)); + lastSampleTime = sampleTime; + lastCumulativeCpuTime = cumulativeCpuTime; + } + return cpuUsage; + } + + public void updateElapsedJiffies(BigInteger elapedJiffies, long sampleTime) { + this.cumulativeCpuTime = elapedJiffies.multiply(jiffyLengthInMillis); + this.sampleTime = sampleTime; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + sb.append("SampleTime " + this.sampleTime); + sb.append(" CummulativeCpuTime " + this.cumulativeCpuTime); + sb.append(" LastSampleTime " + this.lastSampleTime); + sb.append(" LastCummulativeCpuTime " + this.lastCumulativeCpuTime); + sb.append(" CpuUsage " + this.cpuUsage); + sb.append(" JiffyLengthMillisec " + this.jiffyLengthInMillis); + return sb.toString(); + } +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java index 2347f4041cc92..ab1511a7e7525 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java @@ -23,6 +23,7 @@ import java.io.FileNotFoundException; import java.io.InputStreamReader; import java.io.IOException; +import java.math.BigInteger; import java.nio.charset.Charset; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -41,8 +42,6 @@ public class LinuxResourceCalculatorPlugin extends ResourceCalculatorPlugin { private static final Log LOG = LogFactory.getLog(LinuxResourceCalculatorPlugin.class); - public static final int UNAVAILABLE = -1; - /** * proc's meminfo virtual file has keys-values in the format * "key:[ \t]*value[ \t]kB". @@ -74,6 +73,7 @@ public class LinuxResourceCalculatorPlugin extends ResourceCalculatorPlugin { private static final Pattern CPU_TIME_FORMAT = Pattern.compile("^cpu[ \t]*([0-9]*)" + "[ \t]*([0-9]*)[ \t]*([0-9]*)[ \t].*"); + private CpuTimeTracker cpuTimeTracker; private String procfsMemFile; private String procfsCpuFile; @@ -87,12 +87,6 @@ public class LinuxResourceCalculatorPlugin extends ResourceCalculatorPlugin { private long inactiveSize = 0; // inactive cache memory (kB) private int numProcessors = 0; // number of processors on the system private long cpuFrequency = 0L; // CPU frequency on the system (kHz) - private long cumulativeCpuTime = 0L; // CPU used time since system is on (ms) - private long lastCumulativeCpuTime = 0L; // CPU used time read last time (ms) - // Unix timestamp while reading the CPU time (ms) - private float cpuUsage = UNAVAILABLE; - private long sampleTime = UNAVAILABLE; - private long lastSampleTime = UNAVAILABLE; boolean readMemInfoFile = false; boolean readCpuInfoFile = false; @@ -106,10 +100,8 @@ long getCurrentTime() { } public LinuxResourceCalculatorPlugin() { - procfsMemFile = PROCFS_MEMFILE; - procfsCpuFile = PROCFS_CPUINFO; - procfsStatFile = PROCFS_STAT; - jiffyLengthInMillis = ProcfsBasedProcessTree.JIFFY_LENGTH_IN_MILLIS; + this(PROCFS_MEMFILE, PROCFS_CPUINFO, PROCFS_STAT, + ProcfsBasedProcessTree.JIFFY_LENGTH_IN_MILLIS); } /** @@ -128,6 +120,7 @@ public LinuxResourceCalculatorPlugin(String procfsMemFile, this.procfsCpuFile = procfsCpuFile; this.procfsStatFile = procfsStatFile; this.jiffyLengthInMillis = jiffyLengthInMillis; + this.cpuTimeTracker = new CpuTimeTracker(jiffyLengthInMillis); } /** @@ -276,12 +269,13 @@ private void readProcStatFile() { long uTime = Long.parseLong(mat.group(1)); long nTime = Long.parseLong(mat.group(2)); long sTime = Long.parseLong(mat.group(3)); - cumulativeCpuTime = uTime + nTime + sTime; // milliseconds + cpuTimeTracker.updateElapsedJiffies( + BigInteger.valueOf(uTime + nTime + sTime), + getCurrentTime()); break; } str = in.readLine(); } - cumulativeCpuTime *= jiffyLengthInMillis; } catch (IOException io) { LOG.warn("Error reading the stream " + io); } finally { @@ -345,32 +339,18 @@ public long getCpuFrequency() { @Override public long getCumulativeCpuTime() { readProcStatFile(); - return cumulativeCpuTime; + return cpuTimeTracker.cumulativeCpuTime.longValue(); } /** {@inheritDoc} */ @Override public float getCpuUsage() { readProcStatFile(); - sampleTime = getCurrentTime(); - if (lastSampleTime == UNAVAILABLE || - lastSampleTime > sampleTime) { - // lastSampleTime > sampleTime may happen when the system time is changed - lastSampleTime = sampleTime; - lastCumulativeCpuTime = cumulativeCpuTime; - return cpuUsage; - } - // When lastSampleTime is sufficiently old, update cpuUsage. - // Also take a sample of the current time and cumulative CPU time for the - // use of the next calculation. - final long MINIMUM_UPDATE_INTERVAL = 10 * jiffyLengthInMillis; - if (sampleTime > lastSampleTime + MINIMUM_UPDATE_INTERVAL) { - cpuUsage = (float)(cumulativeCpuTime - lastCumulativeCpuTime) * 100F / - ((float)(sampleTime - lastSampleTime) * getNumProcessors()); - lastSampleTime = sampleTime; - lastCumulativeCpuTime = cumulativeCpuTime; + float overallCpuUsage = cpuTimeTracker.getCpuTrackerUsagePercent(); + if (overallCpuUsage != CpuTimeTracker.UNAVAILABLE) { + overallCpuUsage = overallCpuUsage / getNumProcessors(); } - return cpuUsage; + return overallCpuUsage; } /** diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java index 69aa96dafa705..df9d28a61eeea 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java @@ -66,6 +66,8 @@ public class ProcfsBasedProcessTree extends ResourceCalculatorProcessTree { public static final String PROCFS_CMDLINE_FILE = "cmdline"; public static final long PAGE_SIZE; public static final long JIFFY_LENGTH_IN_MILLIS; // in millisecond + private final CpuTimeTracker cpuTimeTracker; + private Clock clock; enum MemInfo { SIZE("Size"), RSS("Rss"), PSS("Pss"), SHARED_CLEAN("Shared_Clean"), @@ -138,13 +140,13 @@ public static MemInfo getMemInfoByName(String name) { static private String deadPid = "-1"; private String pid = deadPid; static private Pattern numberPattern = Pattern.compile("[1-9][0-9]*"); - private Long cpuTime = 0L; + private long cpuTime = UNAVAILABLE; protected Map processTree = new HashMap(); public ProcfsBasedProcessTree(String pid) { - this(pid, PROCFS); + this(pid, PROCFS, new SystemClock()); } @Override @@ -157,6 +159,10 @@ public void setConf(Configuration conf) { } } + public ProcfsBasedProcessTree(String pid, String procfsDir) { + this(pid, procfsDir, new SystemClock()); + } + /** * Build a new process tree rooted at the pid. * @@ -165,11 +171,14 @@ public void setConf(Configuration conf) { * * @param pid root of the process tree * @param procfsDir the root of a proc file system - only used for testing. + * @param clock clock for controlling time for testing */ - public ProcfsBasedProcessTree(String pid, String procfsDir) { + public ProcfsBasedProcessTree(String pid, String procfsDir, Clock clock) { super(pid); + this.clock = clock; this.pid = getValidPID(pid); this.procfsDir = procfsDir; + this.cpuTimeTracker = new CpuTimeTracker(JIFFY_LENGTH_IN_MILLIS); } /** @@ -331,122 +340,158 @@ public String getProcessTreeDump() { return ret.toString(); } - /** - * Get the cumulative virtual memory used by all the processes in the - * process-tree that are older than the passed in age. - * - * @param olderThanAge processes above this age are included in the - * memory addition - * @return cumulative virtual memory used by the process-tree in bytes, - * for processes older than this age. - */ @Override - public long getCumulativeVmem(int olderThanAge) { - long total = 0; + public long getVirtualMemorySize(int olderThanAge) { + long total = UNAVAILABLE; for (ProcessInfo p : processTree.values()) { - if ((p != null) && (p.getAge() > olderThanAge)) { - total += p.getVmem(); + if (p != null) { + if (total == UNAVAILABLE ) { + total = 0; + } + if (p.getAge() > olderThanAge) { + total += p.getVmem(); + } } } return total; } + + @Override + @SuppressWarnings("deprecation") + public long getCumulativeVmem(int olderThanAge) { + return getVirtualMemorySize(olderThanAge); + } - /** - * Get the cumulative resident set size (rss) memory used by all the processes - * in the process-tree that are older than the passed in age. - * - * @param olderThanAge processes above this age are included in the - * memory addition - * @return cumulative rss memory used by the process-tree in bytes, - * for processes older than this age. return 0 if it cannot be - * calculated - */ @Override - public long getCumulativeRssmem(int olderThanAge) { + public long getRssMemorySize(int olderThanAge) { if (PAGE_SIZE < 0) { - return 0; + return UNAVAILABLE; } if (smapsEnabled) { - return getSmapBasedCumulativeRssmem(olderThanAge); + return getSmapBasedRssMemorySize(olderThanAge); } + boolean isAvailable = false; long totalPages = 0; for (ProcessInfo p : processTree.values()) { - if ((p != null) && (p.getAge() > olderThanAge)) { - totalPages += p.getRssmemPage(); + if ((p != null) ) { + if (p.getAge() > olderThanAge) { + totalPages += p.getRssmemPage(); + } + isAvailable = true; } } - return totalPages * PAGE_SIZE; // convert # pages to byte + return isAvailable ? totalPages * PAGE_SIZE : UNAVAILABLE; // convert # pages to byte + } + + @Override + @SuppressWarnings("deprecation") + public long getCumulativeRssmem(int olderThanAge) { + return getRssMemorySize(olderThanAge); } /** - * Get the cumulative resident set size (RSS) memory used by all the processes + * Get the resident set size (RSS) memory used by all the processes * in the process-tree that are older than the passed in age. RSS is * calculated based on SMAP information. Skip mappings with "r--s", "r-xs" * permissions to get real RSS usage of the process. * * @param olderThanAge * processes above this age are included in the memory addition - * @return cumulative rss memory used by the process-tree in bytes, for - * processes older than this age. return 0 if it cannot be calculated + * @return rss memory used by the process-tree in bytes, for + * processes older than this age. return {@link #UNAVAILABLE} if it cannot + * be calculated. */ - private long getSmapBasedCumulativeRssmem(int olderThanAge) { - long total = 0; + private long getSmapBasedRssMemorySize(int olderThanAge) { + long total = UNAVAILABLE; for (ProcessInfo p : processTree.values()) { - if ((p != null) && (p.getAge() > olderThanAge)) { - ProcessTreeSmapMemInfo procMemInfo = processSMAPTree.get(p.getPid()); - if (procMemInfo != null) { - for (ProcessSmapMemoryInfo info : procMemInfo.getMemoryInfoList()) { - // Do not account for r--s or r-xs mappings - if (info.getPermission().trim() - .equalsIgnoreCase(READ_ONLY_WITH_SHARED_PERMISSION) - || info.getPermission().trim() - .equalsIgnoreCase(READ_EXECUTE_WITH_SHARED_PERMISSION)) { - continue; - } - total += - Math.min(info.sharedDirty, info.pss) + info.privateDirty - + info.privateClean; - if (LOG.isDebugEnabled()) { - LOG.debug(" total(" + olderThanAge + "): PID : " + p.getPid() - + ", SharedDirty : " + info.sharedDirty + ", PSS : " - + info.pss + ", Private_Dirty : " + info.privateDirty - + ", Private_Clean : " + info.privateClean + ", total : " - + (total * KB_TO_BYTES)); + if (p != null) { + // set resource to 0 instead of UNAVAILABLE + if (total == UNAVAILABLE){ + total = 0; + } + if (p.getAge() > olderThanAge) { + ProcessTreeSmapMemInfo procMemInfo = processSMAPTree.get(p.getPid()); + if (procMemInfo != null) { + for (ProcessSmapMemoryInfo info : procMemInfo.getMemoryInfoList()) { + // Do not account for r--s or r-xs mappings + if (info.getPermission().trim() + .equalsIgnoreCase(READ_ONLY_WITH_SHARED_PERMISSION) + || info.getPermission().trim() + .equalsIgnoreCase(READ_EXECUTE_WITH_SHARED_PERMISSION)) { + continue; + } + + total += + Math.min(info.sharedDirty, info.pss) + info.privateDirty + + info.privateClean; + if (LOG.isDebugEnabled()) { + LOG.debug(" total(" + olderThanAge + "): PID : " + p.getPid() + + ", SharedDirty : " + info.sharedDirty + ", PSS : " + + info.pss + ", Private_Dirty : " + info.privateDirty + + ", Private_Clean : " + info.privateClean + ", total : " + + (total * KB_TO_BYTES)); + } } } - } - if (LOG.isDebugEnabled()) { - LOG.debug(procMemInfo.toString()); + + if (LOG.isDebugEnabled()) { + LOG.debug(procMemInfo.toString()); + } } } + + } + if (total > 0) { + total *= KB_TO_BYTES; // convert to bytes } - total = (total * KB_TO_BYTES); // convert to bytes LOG.info("SmapBasedCumulativeRssmem (bytes) : " + total); return total; // size } - /** - * Get the CPU time in millisecond used by all the processes in the - * process-tree since the process-tree created - * - * @return cumulative CPU time in millisecond since the process-tree created - * return 0 if it cannot be calculated - */ @Override public long getCumulativeCpuTime() { if (JIFFY_LENGTH_IN_MILLIS < 0) { - return 0; + return UNAVAILABLE; } long incJiffies = 0; + boolean isAvailable = false; for (ProcessInfo p : processTree.values()) { if (p != null) { incJiffies += p.getDtime(); + // data is available + isAvailable = true; } } - cpuTime += incJiffies * JIFFY_LENGTH_IN_MILLIS; + if (isAvailable) { + // reset cpuTime to 0 instead of UNAVAILABLE + if (cpuTime == UNAVAILABLE) { + cpuTime = 0L; + } + cpuTime += incJiffies * JIFFY_LENGTH_IN_MILLIS; + } return cpuTime; } + private BigInteger getTotalProcessJiffies() { + BigInteger totalStime = BigInteger.ZERO; + long totalUtime = 0; + for (ProcessInfo p : processTree.values()) { + if (p != null) { + totalUtime += p.getUtime(); + totalStime = totalStime.add(p.getStime()); + } + } + return totalStime.add(BigInteger.valueOf(totalUtime)); + } + + @Override + public float getCpuUsagePercent() { + BigInteger processTotalJiffies = getTotalProcessJiffies(); + cpuTimeTracker.updateElapsedJiffies(processTotalJiffies, + clock.getTime()); + return cpuTimeTracker.getCpuTrackerUsagePercent(); + } + private static String getValidPID(String pid) { if (pid == null) return deadPid; Matcher m = numberPattern.matcher(pid); @@ -962,4 +1007,48 @@ public String toString() { return sb.toString(); } } + + /** + * Test the {@link ProcfsBasedProcessTree} + * + * @param args + */ + public static void main(String[] args) { + if (args.length != 1) { + System.out.println("Provide "); + return; + } + + int numprocessors = + ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, null) + .getNumProcessors(); + System.out.println("Number of processors " + numprocessors); + + System.out.println("Creating ProcfsBasedProcessTree for process " + + args[0]); + ProcfsBasedProcessTree procfsBasedProcessTree = new + ProcfsBasedProcessTree(args[0]); + procfsBasedProcessTree.updateProcessTree(); + + System.out.println(procfsBasedProcessTree.getProcessTreeDump()); + System.out.println("Get cpu usage " + procfsBasedProcessTree + .getCpuUsagePercent()); + + try { + // Sleep so we can compute the CPU usage + Thread.sleep(500L); + } catch (InterruptedException e) { + // do nothing + } + + procfsBasedProcessTree.updateProcessTree(); + + System.out.println(procfsBasedProcessTree.getProcessTreeDump()); + System.out.println("Cpu usage " + procfsBasedProcessTree + .getCpuUsagePercent()); + System.out.println("Vmem usage in bytes " + procfsBasedProcessTree + .getVirtualMemorySize()); + System.out.println("Rss mem usage in bytes " + procfsBasedProcessTree + .getRssMemorySize()); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java index cc2a56c3be681..c44c2cf6b2cbb 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java @@ -102,11 +102,15 @@ private static Node coreResolve(String hostName) { String rName = null; if (rNameList == null || rNameList.get(0) == null) { rName = NetworkTopology.DEFAULT_RACK; - LOG.info("Couldn't resolve " + hostName + ". Falling back to " - + NetworkTopology.DEFAULT_RACK); + if (LOG.isDebugEnabled()) { + LOG.debug("Couldn't resolve " + hostName + ". Falling back to " + + NetworkTopology.DEFAULT_RACK); + } } else { rName = rNameList.get(0); - LOG.info("Resolved " + hostName + " to " + rName); + if (LOG.isDebugEnabled()) { + LOG.debug("Resolved " + hostName + " to " + rName); + } } return new NodeBase(hostName, rName); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java index 85f6f1af7bf51..01d79d9f49bfd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java @@ -22,18 +22,24 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.classification.InterfaceAudience.Public; import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; /** * Interface class to obtain process resource usage - * + * NOTE: This class should not be used by external users, but only by external + * developers to extend and include their own process-tree implementation, + * especially for platforms other than Linux and Windows. */ -@Private +@Public +@Evolving public abstract class ResourceCalculatorProcessTree extends Configured { static final Log LOG = LogFactory .getLog(ResourceCalculatorProcessTree.class); + public static final int UNAVAILABLE = -1; /** * Create process-tree instance with specified root process. @@ -63,57 +69,131 @@ public ResourceCalculatorProcessTree(String root) { public abstract String getProcessTreeDump(); /** - * Get the cumulative virtual memory used by all the processes in the + * Get the virtual memory used by all the processes in the + * process-tree. + * + * @return virtual memory used by the process-tree in bytes, + * {@link #UNAVAILABLE} if it cannot be calculated. + */ + public long getVirtualMemorySize() { + return getVirtualMemorySize(0); + } + + /** + * Get the virtual memory used by all the processes in the * process-tree. * - * @return cumulative virtual memory used by the process-tree in bytes. + * @return virtual memory used by the process-tree in bytes, + * {@link #UNAVAILABLE} if it cannot be calculated. */ + @Deprecated public long getCumulativeVmem() { return getCumulativeVmem(0); } /** - * Get the cumulative resident set size (rss) memory used by all the processes + * Get the resident set size (rss) memory used by all the processes + * in the process-tree. + * + * @return rss memory used by the process-tree in bytes, + * {@link #UNAVAILABLE} if it cannot be calculated. + */ + public long getRssMemorySize() { + return getRssMemorySize(0); + } + + /** + * Get the resident set size (rss) memory used by all the processes * in the process-tree. * - * @return cumulative rss memory used by the process-tree in bytes. return 0 - * if it cannot be calculated + * @return rss memory used by the process-tree in bytes, + * {@link #UNAVAILABLE} if it cannot be calculated. */ + @Deprecated public long getCumulativeRssmem() { return getCumulativeRssmem(0); } /** - * Get the cumulative virtual memory used by all the processes in the + * Get the virtual memory used by all the processes in the * process-tree that are older than the passed in age. * * @param olderThanAge processes above this age are included in the - * memory addition - * @return cumulative virtual memory used by the process-tree in bytes, - * for processes older than this age. + * memory addition + * @return virtual memory used by the process-tree in bytes for + * processes older than the specified age, {@link #UNAVAILABLE} if it + * cannot be calculated. */ - public abstract long getCumulativeVmem(int olderThanAge); + public long getVirtualMemorySize(int olderThanAge) { + return UNAVAILABLE; + } + + /** + * Get the virtual memory used by all the processes in the + * process-tree that are older than the passed in age. + * + * @param olderThanAge processes above this age are included in the + * memory addition + * @return virtual memory used by the process-tree in bytes for + * processes older than the specified age, {@link #UNAVAILABLE} if it + * cannot be calculated. + */ + @Deprecated + public long getCumulativeVmem(int olderThanAge) { + return UNAVAILABLE; + } /** - * Get the cumulative resident set size (rss) memory used by all the processes + * Get the resident set size (rss) memory used by all the processes + * in the process-tree that are older than the passed in age. + * + * @param olderThanAge processes above this age are included in the + * memory addition + * @return rss memory used by the process-tree in bytes for + * processes older than specified age, {@link #UNAVAILABLE} if it cannot be + * calculated. + */ + public long getRssMemorySize(int olderThanAge) { + return UNAVAILABLE; + } + + /** + * Get the resident set size (rss) memory used by all the processes * in the process-tree that are older than the passed in age. * * @param olderThanAge processes above this age are included in the - * memory addition - * @return cumulative rss memory used by the process-tree in bytes, - * for processes older than this age. return 0 if it cannot be - * calculated + * memory addition + * @return rss memory used by the process-tree in bytes for + * processes older than specified age, {@link #UNAVAILABLE} if it cannot be + * calculated. */ - public abstract long getCumulativeRssmem(int olderThanAge); + @Deprecated + public long getCumulativeRssmem(int olderThanAge) { + return UNAVAILABLE; + } /** * Get the CPU time in millisecond used by all the processes in the - * process-tree since the process-tree created + * process-tree since the process-tree was created * - * @return cumulative CPU time in millisecond since the process-tree created - * return 0 if it cannot be calculated + * @return cumulative CPU time in millisecond since the process-tree + * created, {@link #UNAVAILABLE} if it cannot be calculated. */ - public abstract long getCumulativeCpuTime(); + public long getCumulativeCpuTime() { + return UNAVAILABLE; + } + + /** + * Get the CPU usage by all the processes in the process-tree based on + * average between samples as a ratio of overall CPU cycles similar to top. + * Thus, if 2 out of 4 cores are used this should return 200.0. + * + * @return percentage CPU usage since the process-tree was created, + * {@link #UNAVAILABLE} if it cannot be calculated. + */ + public float getCpuUsagePercent() { + return UNAVAILABLE; + } /** Verify that the tree process id is same as its process group id. * @return true if the process id matches else return false. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java index e6f23f0e52363..3343a1713e66f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java @@ -88,7 +88,7 @@ public static String pjoin(Object... args) { } /** - * Join on slash & colon (e.g., path args in routing spec) + * Join on slash and colon (e.g., path args in routing spec) * @param args to join * @return args joined with /: */ @@ -116,7 +116,7 @@ public static String joins(String sep, Object...args) { } /** - * Split on space & trim results. + * Split on space and trim results. * @param s the string to split * @return an iterable of strings */ @@ -125,7 +125,7 @@ public static Iterable split(CharSequence s) { } /** - * Split on _ & trim results + * Split on _ and trim results * @param s the string to split * @return an iterable of strings */ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsBasedProcessTree.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsBasedProcessTree.java index 143d236f01ac6..7d9c7d3249252 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsBasedProcessTree.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsBasedProcessTree.java @@ -34,7 +34,7 @@ public class WindowsBasedProcessTree extends ResourceCalculatorProcessTree { static final Log LOG = LogFactory .getLog(WindowsBasedProcessTree.class); - + static class ProcessInfo { String pid; // process pid long vmem; // virtual memory @@ -45,8 +45,8 @@ static class ProcessInfo { } private String taskProcessId = null; - private long cpuTimeMs = 0; - private Map processTree = + private long cpuTimeMs = UNAVAILABLE; + private Map processTree = new HashMap(); public static boolean isAvailable() { @@ -173,33 +173,63 @@ public String getProcessTreeDump() { } @Override - public long getCumulativeVmem(int olderThanAge) { - long total = 0; + public long getVirtualMemorySize(int olderThanAge) { + long total = UNAVAILABLE; for (ProcessInfo p : processTree.values()) { - if ((p != null) && (p.age > olderThanAge)) { - total += p.vmem; + if (p != null) { + if (total == UNAVAILABLE) { + total = 0; + } + if (p.age > olderThanAge) { + total += p.vmem; + } } } return total; } + + @Override + @SuppressWarnings("deprecation") + public long getCumulativeVmem(int olderThanAge) { + return getVirtualMemorySize(olderThanAge); + } @Override - public long getCumulativeRssmem(int olderThanAge) { - long total = 0; + public long getRssMemorySize(int olderThanAge) { + long total = UNAVAILABLE; for (ProcessInfo p : processTree.values()) { - if ((p != null) && (p.age > olderThanAge)) { - total += p.workingSet; + if (p != null) { + if (total == UNAVAILABLE) { + total = 0; + } + if (p.age > olderThanAge) { + total += p.workingSet; + } } } return total; } + + @Override + @SuppressWarnings("deprecation") + public long getCumulativeRssmem(int olderThanAge) { + return getRssMemorySize(olderThanAge); + } @Override public long getCumulativeCpuTime() { for (ProcessInfo p : processTree.values()) { + if (cpuTimeMs == UNAVAILABLE) { + cpuTimeMs = 0; + } cpuTimeMs += p.cpuTimeMsDelta; } return cpuTimeMs; } + @Override + public float getCpuUsagePercent() { + return CpuTimeTracker.UNAVAILABLE; + } + } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java index a205bd1f57443..bcb0421d0ab66 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java @@ -260,4 +260,9 @@ public static Resource componentwiseMin(Resource lhs, Resource rhs) { return createResource(Math.min(lhs.getMemory(), rhs.getMemory()), Math.min(lhs.getVirtualCores(), rhs.getVirtualCores())); } + + public static Resource componentwiseMax(Resource lhs, Resource rhs) { + return createResource(Math.max(lhs.getMemory(), rhs.getMemory()), + Math.max(lhs.getVirtualCores(), rhs.getVirtualCores())); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java index 7e836b5bac4f0..b04bc5dd1be3d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java @@ -81,7 +81,11 @@ public ResponseInfo _(String key, Object value) { } public ResponseInfo _(String key, String url, Object anchor) { - items.add(Item.of(key, url, anchor)); + if (url == null) { + items.add(Item.of(key, anchor, false)); + } else { + items.add(Item.of(key, url, anchor)); + } return this; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java index 1b5840f5db42b..bda24aacbbe23 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java @@ -52,13 +52,13 @@ /** * Helpers to create an embedded webapp. * - *

      Quick start:

      + * Quick start: * 
        *   WebApp wa = WebApps.$for(myApp).start();
      * Starts a webapp with default routes binds to 0.0.0.0 (all network interfaces) * on an ephemeral port, which can be obtained with:
        *   int port = wa.port();
      - *

      With more options:

      + * With more options: * 
        *   WebApp wa = WebApps.$for(myApp).at(address, port).
  *                        with(configuration).
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java
index 62c3c7a0ef3d0..1200690b82513 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java
@@ -22,6 +22,9 @@
 
 @InterfaceAudience.LimitedPrivate({"YARN", "MapReduce"})
 public interface YarnWebParams {
+  static final String RM_WEB_UI = "ResourceManager";
+  static final String APP_HISTORY_WEB_UI = "ApplicationHistoryServer";
+  
   String NM_NODENAME = "nm.id";
   String APPLICATION_ID = "app.id";
   String APPLICATION_ATTEMPT_ID = "appattempt.id";
@@ -33,4 +36,5 @@ public interface YarnWebParams {
   String QUEUE_NAME = "queue.name";
   String NODE_STATE = "node.state";
   String NODE_LABEL = "node.label";
+  String WEB_UI_TYPE = "web.ui.type";
 }
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java
index 3aeb33e38a3be..459c1106988f8 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java
@@ -130,11 +130,23 @@ public static String getProxyHostAndPort(Configuration conf) {
     return addr;
   }
 
+  public static String getResolvedRemoteRMWebAppURLWithScheme(
+      Configuration conf) {
+    return getHttpSchemePrefix(conf)
+        + getResolvedRemoteRMWebAppURLWithoutScheme(conf);
+  }
+
   public static String getResolvedRMWebAppURLWithScheme(Configuration conf) {
     return getHttpSchemePrefix(conf)
         + getResolvedRMWebAppURLWithoutScheme(conf);
   }
   
+  public static String getResolvedRemoteRMWebAppURLWithoutScheme(
+      Configuration conf) {
+    return getResolvedRemoteRMWebAppURLWithoutScheme(conf,
+        YarnConfiguration.useHttps(conf) ? Policy.HTTPS_ONLY : Policy.HTTP_ONLY);
+  }
+
   public static String getResolvedRMWebAppURLWithoutScheme(Configuration conf) {
     return getResolvedRMWebAppURLWithoutScheme(conf,
         YarnConfiguration.useHttps(conf) ? Policy.HTTPS_ONLY : Policy.HTTP_ONLY);
@@ -143,6 +155,23 @@ public static String getResolvedRMWebAppURLWithoutScheme(Configuration conf) {
   public static String getResolvedRMWebAppURLWithoutScheme(Configuration conf,
       Policy httpPolicy) {
     InetSocketAddress address = null;
+    if (httpPolicy == Policy.HTTPS_ONLY) {
+      address =
+          conf.getSocketAddr(YarnConfiguration.RM_WEBAPP_HTTPS_ADDRESS,
+              YarnConfiguration.DEFAULT_RM_WEBAPP_HTTPS_ADDRESS,
+              YarnConfiguration.DEFAULT_RM_WEBAPP_HTTPS_PORT);
+    } else {
+      address =
+          conf.getSocketAddr(YarnConfiguration.RM_WEBAPP_ADDRESS,
+              YarnConfiguration.DEFAULT_RM_WEBAPP_ADDRESS,
+              YarnConfiguration.DEFAULT_RM_WEBAPP_PORT);      
+    }
+    return getResolvedAddress(address);
+  }
+
+  public static String getResolvedRemoteRMWebAppURLWithoutScheme(Configuration conf,
+      Policy httpPolicy) {
+    InetSocketAddress address = null;
     String rmId = null;
     if (HAUtil.isHAEnabled(conf)) {
       // If HA enabled, pick one of the RM-IDs and rely on redirect to go to
@@ -167,7 +196,7 @@ public static String getResolvedRMWebAppURLWithoutScheme(Configuration conf,
                   : HAUtil.addSuffix(
                   YarnConfiguration.RM_WEBAPP_ADDRESS, rmId),
               YarnConfiguration.DEFAULT_RM_WEBAPP_ADDRESS,
-              YarnConfiguration.DEFAULT_RM_WEBAPP_PORT);      
+              YarnConfiguration.DEFAULT_RM_WEBAPP_PORT);
     }
     return getResolvedAddress(address);
   }
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/HtmlBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/HtmlBlock.java
index 6ee0d1c9c32e9..a785c0c73d0bb 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/HtmlBlock.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/HtmlBlock.java
@@ -30,6 +30,8 @@
 @InterfaceAudience.LimitedPrivate({"YARN", "MapReduce"})
 public abstract class HtmlBlock extends TextView implements SubView {
 
+  protected static final String UNAVAILABLE = "N/A";
+
   public class Block extends Hamlet {
     Block(PrintWriter out, int level, boolean wasInline) {
       super(out, level, wasInline);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index df730d565c512..66400c8831f5f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -1074,6 +1074,20 @@
     false
   
 
+  
+    This flag determines whether memory limit will be set for the Windows Job
+    Object of the containers launched by the default container executor.
+    yarn.nodemanager.windows-container.memory-limit.enabled
+    false
+  
+
+  
+    This flag determines whether CPU limit will be set for the Windows Job
+    Object of the containers launched by the default container executor.
+    yarn.nodemanager.windows-container.cpu-limit.enabled
+    false
+  
+
   
     T-file compression types used to compress aggregated logs.
     yarn.nodemanager.log-aggregation.compression-type
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
index 8b4879855da2b..80299c0b63892 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
@@ -36,15 +36,280 @@
 import org.apache.commons.lang.math.LongRange;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.security.proto.SecurityProtos.*;
-import org.apache.hadoop.yarn.api.protocolrecords.*;
-import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.*;
-import org.apache.hadoop.yarn.api.records.*;
-import org.apache.hadoop.yarn.api.records.impl.pb.*;
-import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.*;
-import org.apache.hadoop.yarn.proto.YarnProtos.*;
-import org.apache.hadoop.yarn.proto.YarnServiceProtos.*;
-import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.*;
+import org.apache.hadoop.security.proto.SecurityProtos.CancelDelegationTokenRequestProto;
+import org.apache.hadoop.security.proto.SecurityProtos.CancelDelegationTokenResponseProto;
+import org.apache.hadoop.security.proto.SecurityProtos.GetDelegationTokenRequestProto;
+import org.apache.hadoop.security.proto.SecurityProtos.GetDelegationTokenResponseProto;
+import org.apache.hadoop.security.proto.SecurityProtos.RenewDelegationTokenRequestProto;
+import org.apache.hadoop.security.proto.SecurityProtos.RenewDelegationTokenResponseProto;
+import org.apache.hadoop.security.proto.SecurityProtos.TokenProto;
+import org.apache.hadoop.yarn.api.protocolrecords.StartContainerRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.CancelDelegationTokenRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.CancelDelegationTokenResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.FinishApplicationMasterRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.FinishApplicationMasterResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationAttemptReportRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationAttemptReportResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationAttemptsRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationAttemptsResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationReportRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationReportResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationsRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetApplicationsResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetClusterMetricsRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetClusterMetricsResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetClusterNodeLabelsRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetClusterNodeLabelsResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetClusterNodesRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetClusterNodesResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainerReportRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainerReportResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainerStatusesRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainerStatusesResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainersRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetContainersResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetDelegationTokenRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetDelegationTokenResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNewApplicationResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNodesToLabelsRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetNodesToLabelsResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueUserAclsInfoRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueUserAclsInfoResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.KillApplicationRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.KillApplicationResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.MoveApplicationAcrossQueuesRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.MoveApplicationAcrossQueuesResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.RegisterApplicationMasterRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.RegisterApplicationMasterResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.RenewDelegationTokenRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.RenewDelegationTokenResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.ReservationDeleteRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.ReservationDeleteResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.ReservationSubmissionRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.ReservationSubmissionResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.ReservationUpdateRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.ReservationUpdateResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainerRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StartContainersResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StopContainersRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.StopContainersResponsePBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.SubmitApplicationRequestPBImpl;
+import org.apache.hadoop.yarn.api.protocolrecords.impl.pb.SubmitApplicationResponsePBImpl;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ApplicationReport;
+import org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport;
+import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
+import org.apache.hadoop.yarn.api.records.Container;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
+import org.apache.hadoop.yarn.api.records.ContainerReport;
+import org.apache.hadoop.yarn.api.records.ContainerResourceDecrease;
+import org.apache.hadoop.yarn.api.records.ContainerResourceIncrease;
+import org.apache.hadoop.yarn.api.records.ContainerResourceIncreaseRequest;
+import org.apache.hadoop.yarn.api.records.ContainerStatus;
+import org.apache.hadoop.yarn.api.records.LocalResource;
+import org.apache.hadoop.yarn.api.records.LogAggregationContext;
+import org.apache.hadoop.yarn.api.records.NMToken;
+import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.api.records.NodeLabel;
+import org.apache.hadoop.yarn.api.records.NodeReport;
+import org.apache.hadoop.yarn.api.records.PreemptionContainer;
+import org.apache.hadoop.yarn.api.records.PreemptionContract;
+import org.apache.hadoop.yarn.api.records.PreemptionMessage;
+import org.apache.hadoop.yarn.api.records.PreemptionResourceRequest;
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.QueueInfo;
+import org.apache.hadoop.yarn.api.records.QueueState;
+import org.apache.hadoop.yarn.api.records.QueueUserACLInfo;
+import org.apache.hadoop.yarn.api.records.ReservationDefinition;
+import org.apache.hadoop.yarn.api.records.ReservationId;
+import org.apache.hadoop.yarn.api.records.ReservationRequest;
+import org.apache.hadoop.yarn.api.records.ReservationRequests;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.api.records.ResourceBlacklistRequest;
+import org.apache.hadoop.yarn.api.records.ResourceOption;
+import org.apache.hadoop.yarn.api.records.ResourceRequest;
+import org.apache.hadoop.yarn.api.records.SerializedException;
+import org.apache.hadoop.yarn.api.records.StrictPreemptionContract;
+import org.apache.hadoop.yarn.api.records.Token;
+import org.apache.hadoop.yarn.api.records.URL;
+import org.apache.hadoop.yarn.api.records.YarnClusterMetrics;
+import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptReportPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationReportPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationResourceUsageReportPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationSubmissionContextPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerLaunchContextPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerReportPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerResourceDecreasePBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerResourceIncreasePBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerResourceIncreaseRequestPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ContainerStatusPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.LocalResourcePBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.NMTokenPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.NodeIdPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.NodeLabelPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.PreemptionContainerPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.PreemptionContractPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.PreemptionMessagePBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.PreemptionResourceRequestPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.PriorityPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.QueueUserACLInfoPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ResourceBlacklistRequestPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ResourceOptionPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.ResourceRequestPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.StrictPreemptionContractPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.TokenPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.URLPBImpl;
+import org.apache.hadoop.yarn.api.records.impl.pb.YarnClusterMetricsPBImpl;
+import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationAttemptIdProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationAttemptReportProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationIdProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationResourceUsageReportProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationSubmissionContextProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerIdProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerLaunchContextProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerReportProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerResourceDecreaseProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerResourceIncreaseProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerResourceIncreaseRequestProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ContainerStatusProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.LocalResourceProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.NodeIdProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.NodeLabelProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.NodeReportProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.PreemptionContainerProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.PreemptionContractProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.PreemptionMessageProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.PreemptionResourceRequestProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.PriorityProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.QueueInfoProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.QueueUserACLInfoProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ResourceBlacklistRequestProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ResourceOptionProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ResourceProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ResourceRequestProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.SerializedExceptionProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.StrictPreemptionContractProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.URLProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.YarnClusterMetricsProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.AddToClusterNodeLabelsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.AddToClusterNodeLabelsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshAdminAclsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshAdminAclsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshNodesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshNodesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshQueuesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshQueuesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshServiceAclsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshServiceAclsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshSuperUserGroupsConfigurationRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshSuperUserGroupsConfigurationResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshUserToGroupsMappingsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RefreshUserToGroupsMappingsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RemoveFromClusterNodeLabelsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.RemoveFromClusterNodeLabelsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.ReplaceLabelsOnNodeRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.ReplaceLabelsOnNodeResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeLabelsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeResourceRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos.UpdateNodeResourceResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.AllocateRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.AllocateResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.FinishApplicationMasterRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.FinishApplicationMasterResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationAttemptReportRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationAttemptReportResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationAttemptsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationAttemptsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationReportRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationReportResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetClusterMetricsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetClusterMetricsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetClusterNodeLabelsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetClusterNodeLabelsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetClusterNodesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetClusterNodesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainerReportRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainerReportResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainerStatusesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainerStatusesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainersRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetContainersResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetLabelsToNodesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNewApplicationRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNewApplicationResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNodesToLabelsRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetNodesToLabelsResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetQueueInfoRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetQueueInfoResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetQueueUserAclsInfoRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.GetQueueUserAclsInfoResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.KillApplicationRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.KillApplicationResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.MoveApplicationAcrossQueuesRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.MoveApplicationAcrossQueuesResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.NMTokenProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.RegisterApplicationMasterRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.RegisterApplicationMasterResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.ReservationDeleteRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.ReservationDeleteResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.ReservationSubmissionRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.ReservationSubmissionResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.ReservationUpdateRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.ReservationUpdateResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.StartContainerRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.StartContainersRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.StartContainersResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.StopContainersRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.StopContainersResponseProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.SubmitApplicationRequestProto;
+import org.apache.hadoop.yarn.proto.YarnServiceProtos.SubmitApplicationResponseProto;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.AddToClusterNodeLabelsRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.AddToClusterNodeLabelsResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshAdminAclsRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshAdminAclsResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshNodesRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshNodesResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshQueuesRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshQueuesResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshServiceAclsRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshServiceAclsResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshSuperUserGroupsConfigurationRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshSuperUserGroupsConfigurationResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshUserToGroupsMappingsRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RefreshUserToGroupsMappingsResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RemoveFromClusterNodeLabelsRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RemoveFromClusterNodeLabelsResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.ReplaceLabelsOnNodeResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeLabelsResponsePBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeResourceRequestPBImpl;
+import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.UpdateNodeResourceResponsePBImpl;
 import org.apache.hadoop.yarn.util.resource.Resources;
 import org.junit.Assert;
 import org.junit.BeforeClass;
@@ -212,6 +477,7 @@ public static void setup() throws Exception {
     generateByNewInstance(StrictPreemptionContract.class);
     generateByNewInstance(PreemptionMessage.class);
     generateByNewInstance(StartContainerRequest.class);
+    generateByNewInstance(NodeLabel.class);
     // genByNewInstance does not apply to QueueInfo, cause
     // it is recursive(has sub queues)
     typeValueCache.put(QueueInfo.class, QueueInfo.newInstance("root", 1.0f,
@@ -1015,4 +1281,22 @@ public void testGetLabelsToNodesResponsePBImpl() throws Exception {
     validatePBImplRecord(GetLabelsToNodesResponsePBImpl.class,
         GetLabelsToNodesResponseProto.class);
   }
+  
+  @Test
+  public void testNodeLabelAttributesPBImpl() throws Exception {
+    validatePBImplRecord(NodeLabelPBImpl.class,
+        NodeLabelProto.class);
+  }
+  
+  @Test
+  public void testUpdateNodeLabelsRequestPBImpl() throws Exception {
+    validatePBImplRecord(UpdateNodeLabelsRequestPBImpl.class,
+        UpdateNodeLabelsRequestProto.class);
+  }
+  
+  @Test
+  public void testUpdateNodeLabelsResponsePBImpl() throws Exception {
+    validatePBImplRecord(UpdateNodeLabelsResponsePBImpl.class,
+        UpdateNodeLabelsResponseProto.class);
+  }
 }
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
index 4301bc9eee7d0..21e1655cceab8 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
@@ -257,7 +257,8 @@ private void testReadAcontainerLog(boolean logUploadedTime) throws Exception {
             + (logUploadedTime ? ("\nLog Upload Time:" + Times.format(System
               .currentTimeMillis())).length() : 0)
             + ("\nLogLength:" + numChars).length()
-            + "\nLog Contents:\n".length() + numChars + "\n".length();
+            + "\nLog Contents:\n".length() + numChars + "\n".length()
+            + "End of LogType:stdout\n".length();
     Assert.assertTrue("LogType not matched", s.contains("LogType:stdout"));
     Assert.assertTrue("log file:stderr should not be aggregated.", !s.contains("LogType:stderr"));
     Assert.assertTrue("log file:logs should not be aggregated.", !s.contains("LogType:logs"));
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/DummyCommonNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/DummyCommonNodeLabelsManager.java
index 65ea79f9c157b..67e611996ef11 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/DummyCommonNodeLabelsManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/DummyCommonNodeLabelsManager.java
@@ -20,17 +20,20 @@
 
 import java.io.IOException;
 import java.util.Collection;
+import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.api.records.NodeLabel;
 import org.apache.hadoop.yarn.event.InlineDispatcher;
 
 public class DummyCommonNodeLabelsManager extends CommonNodeLabelsManager {
   Map> lastNodeToLabels = null;
   Collection lastAddedlabels = null;
   Collection lastRemovedlabels = null;
+  List lastUpdatedNodeLabels = null;
 
   @Override
   public void initNodeLabelStore(Configuration conf) {
@@ -57,6 +60,12 @@ public void storeNewClusterNodeLabels(Set label) throws IOException {
         lastAddedlabels = label;
       }
 
+      @Override
+      public void updateNodeLabels(List updatedNodeLabels)
+        throws IOException {
+        lastUpdatedNodeLabels = updatedNodeLabels;
+      }
+
       @Override
       public void close() throws IOException {
         // do nothing 
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java
index d05c75ce63bd0..1e2326b8f0993 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestCommonNodeLabelsManager.java
@@ -29,7 +29,9 @@
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.api.records.NodeLabel;
 import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
 import org.junit.After;
 import org.junit.Assert;
 import org.junit.Before;
@@ -536,4 +538,30 @@ public void testNoMoreThanOneLabelExistedInOneHost() throws IOException {
     Assert.assertTrue("Should failed when #labels > 1 on a host after add",
         failed);
   }
+
+  @Test (timeout = 5000)
+  public void testUpdateNodeLabels() throws Exception {
+    boolean failed = false;
+
+    // should fail: label isn't exist
+    try {
+      mgr.updateNodeLabels(Arrays.asList(NodeLabel.newInstance(
+        "p1", false)));
+    } catch (YarnException e) {
+      failed = true;
+    }
+    Assert.assertTrue("Should fail since the node label doesn't exist", failed);
+
+    mgr.addToCluserNodeLabels(toSet("p1", "p2", "p3"));
+
+    mgr.updateNodeLabels(Arrays.asList(
+        NodeLabel.newInstance("p1", false), NodeLabel.newInstance("p2", true)));
+    Assert.assertEquals("p1", mgr.lastUpdatedNodeLabels.get(0).getNodeLabel());
+    Assert.assertFalse(mgr.lastUpdatedNodeLabels.get(0).getIsExclusive());
+    Assert.assertTrue(mgr.lastUpdatedNodeLabels.get(1).getIsExclusive());
+
+    // Check exclusive for p1/p2
+    Assert.assertFalse(mgr.isExclusiveNodeLabel("p1"));
+    Assert.assertTrue(mgr.isExclusiveNodeLabel("p2"));
+  }
 }
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
index 5cc026a74ccc1..6694290ce8a02 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
@@ -24,6 +24,7 @@
 import java.util.Map;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.NodeLabel;
 import org.apache.hadoop.yarn.conf.YarnConfiguration;
 import org.apache.hadoop.yarn.event.InlineDispatcher;
 import org.junit.After;
@@ -188,7 +189,7 @@ public void testEditlogRecover() throws Exception {
   }
   
   @SuppressWarnings({ "unchecked", "rawtypes" })
-  @Test//(timeout = 10000)
+  @Test (timeout = 10000)
   public void testSerilizationAfterRecovery() throws Exception {
     mgr.addToCluserNodeLabels(toSet("p1", "p2", "p3"));
     mgr.addToCluserNodeLabels(toSet("p4"));
@@ -218,6 +219,14 @@ public void testSerilizationAfterRecovery() throws Exception {
      * p4: n4 
      * p6: n6, n7
      */
+
+    mgr.updateNodeLabels(Arrays.asList(NodeLabel.newInstance("p2", false)));
+    mgr.updateNodeLabels(Arrays.asList(NodeLabel.newInstance("p6", false)));
+
+    /*
+     * Set p2/p6 to be exclusive
+     */
+
     // shutdown mgr and start a new mgr
     mgr.stop();
 
@@ -239,6 +248,10 @@ public void testSerilizationAfterRecovery() throws Exception {
         "p4", toSet(toNodeId("n4")),
         "p2", toSet(toNodeId("n2"))));
 
+    Assert.assertFalse(mgr.isExclusiveNodeLabel("p2"));
+    Assert.assertTrue(mgr.isExclusiveNodeLabel("p4"));
+    Assert.assertFalse(mgr.isExclusiveNodeLabel("p6"));
+
     /*
      * Add label p7,p8 then shutdown
      */
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
index c9a33d0f29080..ad09fdfbad409 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
@@ -171,8 +171,8 @@ public void parsingProcStatAndCpuFile() throws IOException {
     updateStatFile(uTime, nTime, sTime);
     assertEquals(plugin.getCumulativeCpuTime(),
                  FAKE_JIFFY_LENGTH * (uTime + nTime + sTime));
-    assertEquals(plugin.getCpuUsage(), (float)(LinuxResourceCalculatorPlugin.UNAVAILABLE),0.0);
-    
+    assertEquals(plugin.getCpuUsage(), (float)(CpuTimeTracker.UNAVAILABLE),0.0);
+
     // Advance the time and sample again to test the CPU usage calculation
     uTime += 100L;
     plugin.advanceTime(200L);
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
index 771925574cd8e..59cef07d28cc1 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
@@ -19,6 +19,7 @@
 package org.apache.hadoop.yarn.util;
 
 import static org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.KB_TO_BYTES;
+import static org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.UNAVAILABLE;
 import static org.junit.Assert.fail;
 import static org.junit.Assume.assumeTrue;
 
@@ -116,6 +117,7 @@ public void setup() throws IOException {
   }
 
   @Test(timeout = 30000)
+  @SuppressWarnings("deprecation")
   public void testProcessTree() throws Exception {
     try {
       Assert.assertTrue(ProcfsBasedProcessTree.isAvailable());
@@ -225,8 +227,12 @@ public void testProcessTree() throws Exception {
     // ProcessTree is gone now. Any further calls should be sane.
     p.updateProcessTree();
     Assert.assertFalse("ProcessTree must have been gone", isAlive(pid));
+    
     Assert.assertTrue(
-      "Cumulative vmem for the gone-process is " + p.getCumulativeVmem()
+      "vmem for the gone-process is " + p.getVirtualMemorySize()
+          + " . It should be zero.", p.getVirtualMemorySize() == 0);
+    Assert.assertTrue(
+      "vmem (old API) for the gone-process is " + p.getCumulativeVmem()
           + " . It should be zero.", p.getCumulativeVmem() == 0);
     Assert.assertTrue(p.toString().equals("[ ]"));
   }
@@ -236,8 +242,8 @@ protected ProcfsBasedProcessTree createProcessTree(String pid) {
   }
 
   protected ProcfsBasedProcessTree createProcessTree(String pid,
-      String procfsRootDir) {
-    return new ProcfsBasedProcessTree(pid, procfsRootDir);
+      String procfsRootDir, Clock clock) {
+    return new ProcfsBasedProcessTree(pid, procfsRootDir, clock);
   }
 
   protected void destroyProcessTree(String pid) throws IOException {
@@ -384,10 +390,13 @@ public void createMemoryMappingInfo(ProcessTreeSmapMemInfo[] procMemInfo) {
    *           files.
    */
   @Test(timeout = 30000)
+  @SuppressWarnings("deprecation")
   public void testCpuAndMemoryForProcessTree() throws IOException {
 
     // test processes
     String[] pids = { "100", "200", "300", "400" };
+    ControlledClock testClock = new ControlledClock(new SystemClock());
+    testClock.setTime(0);
     // create the fake procfs root directory.
     File procfsRootDir = new File(TEST_ROOT_DIR, "proc");
 
@@ -422,20 +431,24 @@ public void testCpuAndMemoryForProcessTree() throws IOException {
       // crank up the process tree class.
       Configuration conf = new Configuration();
       ProcfsBasedProcessTree processTree =
-          createProcessTree("100", procfsRootDir.getAbsolutePath());
+          createProcessTree("100", procfsRootDir.getAbsolutePath(), testClock);
       processTree.setConf(conf);
       // build the process tree.
       processTree.updateProcessTree();
 
-      // verify cumulative memory
-      Assert.assertEquals("Cumulative virtual memory does not match", 600000L,
-        processTree.getCumulativeVmem());
+      // verify virtual memory
+      Assert.assertEquals("Virtual memory does not match", 600000L,
+        processTree.getVirtualMemorySize());
 
       // verify rss memory
       long cumuRssMem =
           ProcfsBasedProcessTree.PAGE_SIZE > 0
-              ? 600L * ProcfsBasedProcessTree.PAGE_SIZE : 0L;
-      Assert.assertEquals("Cumulative rss memory does not match", cumuRssMem,
+              ? 600L * ProcfsBasedProcessTree.PAGE_SIZE : 
+                  ResourceCalculatorProcessTree.UNAVAILABLE;
+      Assert.assertEquals("rss memory does not match", cumuRssMem,
+        processTree.getRssMemorySize());
+      // verify old API
+      Assert.assertEquals("rss memory (old API) does not match", cumuRssMem,
         processTree.getCumulativeRssmem());
 
       // verify cumulative cpu time
@@ -444,11 +457,20 @@ public void testCpuAndMemoryForProcessTree() throws IOException {
               ? 7200L * ProcfsBasedProcessTree.JIFFY_LENGTH_IN_MILLIS : 0L;
       Assert.assertEquals("Cumulative cpu time does not match", cumuCpuTime,
         processTree.getCumulativeCpuTime());
+
+      // verify CPU usage
+      Assert.assertEquals("Percent CPU time should be set to -1 initially",
+          -1.0, processTree.getCpuUsagePercent(),
+          0.01);
+
       // Check by enabling smaps
       setSmapsInProceTree(processTree, true);
       // RSS=Min(shared_dirty,PSS)+PrivateClean+PrivateDirty (exclude r-xs,
       // r--s)
-      Assert.assertEquals("Cumulative rss memory does not match",
+      Assert.assertEquals("rss memory does not match",
+        (100 * KB_TO_BYTES * 3), processTree.getRssMemorySize());
+      // verify old API
+      Assert.assertEquals("rss memory (old API) does not match",
         (100 * KB_TO_BYTES * 3), processTree.getCumulativeRssmem());
 
       // test the cpu time again to see if it cumulates
@@ -460,15 +482,31 @@ public void testCpuAndMemoryForProcessTree() throws IOException {
               "100", "200000", "200", "3000", "500" });
       writeStatFiles(procfsRootDir, pids, procInfos, memInfo);
 
+      long elapsedTimeBetweenUpdatesMsec = 200000;
+      testClock.setTime(elapsedTimeBetweenUpdatesMsec);
       // build the process tree.
       processTree.updateProcessTree();
 
       // verify cumulative cpu time again
+      long prevCumuCpuTime = cumuCpuTime;
       cumuCpuTime =
           ProcfsBasedProcessTree.JIFFY_LENGTH_IN_MILLIS > 0
               ? 9400L * ProcfsBasedProcessTree.JIFFY_LENGTH_IN_MILLIS : 0L;
       Assert.assertEquals("Cumulative cpu time does not match", cumuCpuTime,
         processTree.getCumulativeCpuTime());
+
+      double expectedCpuUsagePercent =
+          (ProcfsBasedProcessTree.JIFFY_LENGTH_IN_MILLIS > 0) ?
+              (cumuCpuTime - prevCumuCpuTime) * 100.0 /
+                  elapsedTimeBetweenUpdatesMsec : 0;
+      // expectedCpuUsagePercent is given by (94000L - 72000) * 100/
+      //    200000;
+      // which in this case is 11. Lets verify that first
+      Assert.assertEquals(11, expectedCpuUsagePercent, 0.001);
+      Assert.assertEquals("Percent CPU time is not correct expected " +
+              expectedCpuUsagePercent, expectedCpuUsagePercent,
+          processTree.getCpuUsagePercent(),
+          0.01);
     } finally {
       FileUtil.fullyDelete(procfsRootDir);
     }
@@ -499,6 +537,7 @@ public void testMemForOlderProcesses() throws IOException {
     testMemForOlderProcesses(true);
   }
 
+  @SuppressWarnings("deprecation")
   private void testMemForOlderProcesses(boolean smapEnabled) throws IOException {
     // initial list of processes
     String[] pids = { "100", "200", "300", "400" };
@@ -535,12 +574,17 @@ private void testMemForOlderProcesses(boolean smapEnabled) throws IOException {
 
       // crank up the process tree class.
       ProcfsBasedProcessTree processTree =
-          createProcessTree("100", procfsRootDir.getAbsolutePath());
+          createProcessTree("100", procfsRootDir.getAbsolutePath(),
+              new SystemClock());
       setSmapsInProceTree(processTree, smapEnabled);
 
-      // verify cumulative memory
-      Assert.assertEquals("Cumulative memory does not match", 700000L,
+      // verify virtual memory
+      Assert.assertEquals("Virtual memory does not match", 700000L,
+        processTree.getVirtualMemorySize());
+      
+      Assert.assertEquals("Virtual memory (old API) does not match", 700000L,
         processTree.getCumulativeVmem());
+
       // write one more process as child of 100.
       String[] newPids = { "500" };
       setupPidDirs(procfsRootDir, newPids);
@@ -556,33 +600,59 @@ private void testMemForOlderProcesses(boolean smapEnabled) throws IOException {
 
       // check memory includes the new process.
       processTree.updateProcessTree();
-      Assert.assertEquals("Cumulative vmem does not include new process",
+      Assert.assertEquals("vmem does not include new process",
+        1200000L, processTree.getVirtualMemorySize());
+      
+      Assert.assertEquals("vmem (old API) does not include new process",
         1200000L, processTree.getCumulativeVmem());
+      
       if (!smapEnabled) {
         long cumuRssMem =
             ProcfsBasedProcessTree.PAGE_SIZE > 0
-                ? 1200L * ProcfsBasedProcessTree.PAGE_SIZE : 0L;
-        Assert.assertEquals("Cumulative rssmem does not include new process",
+                ? 1200L * ProcfsBasedProcessTree.PAGE_SIZE : 
+                    ResourceCalculatorProcessTree.UNAVAILABLE;
+        Assert.assertEquals("rssmem does not include new process",
+          cumuRssMem, processTree.getRssMemorySize());
+        // verify old API
+        Assert.assertEquals("rssmem (old API) does not include new process",
           cumuRssMem, processTree.getCumulativeRssmem());
       } else {
-        Assert.assertEquals("Cumulative rssmem does not include new process",
+        Assert.assertEquals("rssmem does not include new process",
+          100 * KB_TO_BYTES * 4, processTree.getRssMemorySize());
+        // verify old API
+        Assert.assertEquals("rssmem (old API) does not include new process",
           100 * KB_TO_BYTES * 4, processTree.getCumulativeRssmem());
       }
 
       // however processes older than 1 iteration will retain the older value
       Assert.assertEquals(
-        "Cumulative vmem shouldn't have included new process", 700000L,
-        processTree.getCumulativeVmem(1));
+        "vmem shouldn't have included new process", 700000L,
+        processTree.getVirtualMemorySize(1));
+      // verify old API
+      Assert.assertEquals(
+          "vmem (old API) shouldn't have included new process", 700000L,
+          processTree.getCumulativeVmem(1));
+      
       if (!smapEnabled) {
         long cumuRssMem =
             ProcfsBasedProcessTree.PAGE_SIZE > 0
-                ? 700L * ProcfsBasedProcessTree.PAGE_SIZE : 0L;
+                ? 700L * ProcfsBasedProcessTree.PAGE_SIZE : 
+                    ResourceCalculatorProcessTree.UNAVAILABLE;
+        Assert.assertEquals(
+          "rssmem shouldn't have included new process", cumuRssMem,
+          processTree.getRssMemorySize(1));
+        // Verify old API
         Assert.assertEquals(
-          "Cumulative rssmem shouldn't have included new process", cumuRssMem,
+          "rssmem (old API) shouldn't have included new process", cumuRssMem,
           processTree.getCumulativeRssmem(1));
+        
       } else {
         Assert.assertEquals(
-          "Cumulative rssmem shouldn't have included new process",
+          "rssmem shouldn't have included new process",
+          100 * KB_TO_BYTES * 3, processTree.getRssMemorySize(1));
+        // Verify old API
+        Assert.assertEquals(
+          "rssmem (old API) shouldn't have included new process",
           100 * KB_TO_BYTES * 3, processTree.getCumulativeRssmem(1));
       }
 
@@ -604,49 +674,81 @@ private void testMemForOlderProcesses(boolean smapEnabled) throws IOException {
 
       // processes older than 2 iterations should be same as before.
       Assert.assertEquals(
-        "Cumulative vmem shouldn't have included new processes", 700000L,
+        "vmem shouldn't have included new processes", 700000L,
+        processTree.getVirtualMemorySize(2));
+      
+      // verify old API
+      Assert.assertEquals(
+        "vmem (old API) shouldn't have included new processes", 700000L,
         processTree.getCumulativeVmem(2));
+      
       if (!smapEnabled) {
         long cumuRssMem =
             ProcfsBasedProcessTree.PAGE_SIZE > 0
-                ? 700L * ProcfsBasedProcessTree.PAGE_SIZE : 0L;
+                ? 700L * ProcfsBasedProcessTree.PAGE_SIZE : 
+                    ResourceCalculatorProcessTree.UNAVAILABLE;
         Assert.assertEquals(
-          "Cumulative rssmem shouldn't have included new processes",
+          "rssmem shouldn't have included new processes",
+          cumuRssMem, processTree.getRssMemorySize(2));
+        // Verify old API
+        Assert.assertEquals(
+          "rssmem (old API) shouldn't have included new processes",
           cumuRssMem, processTree.getCumulativeRssmem(2));
       } else {
         Assert.assertEquals(
-          "Cumulative rssmem shouldn't have included new processes",
+          "rssmem shouldn't have included new processes",
+          100 * KB_TO_BYTES * 3, processTree.getRssMemorySize(2));
+        // Verify old API
+        Assert.assertEquals(
+          "rssmem (old API) shouldn't have included new processes",
           100 * KB_TO_BYTES * 3, processTree.getCumulativeRssmem(2));
       }
 
       // processes older than 1 iteration should not include new process,
       // but include process 500
       Assert.assertEquals(
-        "Cumulative vmem shouldn't have included new processes", 1200000L,
+        "vmem shouldn't have included new processes", 1200000L,
+        processTree.getVirtualMemorySize(1));
+      // verify old API
+      Assert.assertEquals(
+        "vmem (old API) shouldn't have included new processes", 1200000L,
         processTree.getCumulativeVmem(1));
       if (!smapEnabled) {
         long cumuRssMem =
             ProcfsBasedProcessTree.PAGE_SIZE > 0
-                ? 1200L * ProcfsBasedProcessTree.PAGE_SIZE : 0L;
+                ? 1200L * ProcfsBasedProcessTree.PAGE_SIZE : 
+                    ResourceCalculatorProcessTree.UNAVAILABLE;
+        Assert.assertEquals(
+          "rssmem shouldn't have included new processes",
+          cumuRssMem, processTree.getRssMemorySize(1));
+        // verify old API
         Assert.assertEquals(
-          "Cumulative rssmem shouldn't have included new processes",
+          "rssmem (old API) shouldn't have included new processes",
           cumuRssMem, processTree.getCumulativeRssmem(1));
       } else {
         Assert.assertEquals(
-          "Cumulative rssmem shouldn't have included new processes",
+          "rssmem shouldn't have included new processes",
+          100 * KB_TO_BYTES * 4, processTree.getRssMemorySize(1));
+        Assert.assertEquals(
+          "rssmem (old API) shouldn't have included new processes",
           100 * KB_TO_BYTES * 4, processTree.getCumulativeRssmem(1));
       }
 
-      // no processes older than 3 iterations, this should be 0
+      // no processes older than 3 iterations
+      Assert.assertEquals(
+          "Getting non-zero vmem for processes older than 3 iterations",
+          0, processTree.getVirtualMemorySize(3));
+      // verify old API
       Assert.assertEquals(
-        "Getting non-zero vmem for processes older than 3 iterations", 0L,
-        processTree.getCumulativeVmem(3));
+          "Getting non-zero vmem (old API) for processes older than 3 iterations",
+          0, processTree.getCumulativeVmem(3));
       Assert.assertEquals(
-        "Getting non-zero rssmem for processes older than 3 iterations", 0L,
-        processTree.getCumulativeRssmem(3));
+          "Getting non-zero rssmem for processes older than 3 iterations",
+          0, processTree.getRssMemorySize(3));
+      // verify old API
       Assert.assertEquals(
-        "Getting non-zero rssmem for processes older than 3 iterations", 0L,
-        processTree.getCumulativeRssmem(3));
+          "Getting non-zero rssmem (old API) for processes older than 3 iterations",
+          0, processTree.getCumulativeRssmem(3));
     } finally {
       FileUtil.fullyDelete(procfsRootDir);
     }
@@ -672,7 +774,7 @@ public void testDestroyProcessTree() throws IOException {
       setupProcfsRootDir(procfsRootDir);
 
       // crank up the process tree class.
-      createProcessTree(pid, procfsRootDir.getAbsolutePath());
+      createProcessTree(pid, procfsRootDir.getAbsolutePath(), new SystemClock());
 
       // Let us not create stat file for pid 100.
       Assert.assertTrue(ProcfsBasedProcessTree.checkPidPgrpidForMatch(pid,
@@ -741,7 +843,8 @@ public void testProcessTreeDump() throws IOException {
       writeCmdLineFiles(procfsRootDir, pids, cmdLines);
 
       ProcfsBasedProcessTree processTree =
-          createProcessTree("100", procfsRootDir.getAbsolutePath());
+          createProcessTree("100", procfsRootDir.getAbsolutePath(),
+              new SystemClock());
       // build the process tree.
       processTree.updateProcessTree();
 
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
index 32ceb2378f6c7..777ea9f596e94 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
@@ -41,10 +41,20 @@ public String getProcessTreeDump() {
       return "Empty tree for testing";
     }
 
+    public long getRssMemorySize(int age) {
+      return 0;
+    }
+    
+    @SuppressWarnings("deprecation")
     public long getCumulativeRssmem(int age) {
       return 0;
     }
 
+    public long getVirtualMemorySize(int age) {
+      return 0;
+    }
+    
+    @SuppressWarnings("deprecation")
     public long getCumulativeVmem(int age) {
       return 0;
     }
@@ -53,6 +63,11 @@ public long getCumulativeCpuTime() {
       return 0;
     }
 
+    @Override
+    public float getCpuUsagePercent() {
+      return CpuTimeTracker.UNAVAILABLE;
+    }
+
     public boolean checkPidPgrpidForMatch() {
       return false;
     }
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWebAppUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWebAppUtils.java
new file mode 100644
index 0000000000000..169787ffc3b6c
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWebAppUtils.java
@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.util;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.net.UnknownHostException;
+import java.util.HashMap;
+import java.util.Map;
+
+public class TestWebAppUtils {
+
+  private static final String RM1_NODE_ID = "rm1";
+  private static final String RM2_NODE_ID = "rm2";
+
+  // Because WebAppUtils#getResolvedAddress tries to resolve the hostname, we add a static mapping for dummy hostnames
+  // to make this test run anywhere without having to give some resolvable hostnames
+  private static String dummyHostNames[] = {"host1", "host2", "host3"};
+  private static final String anyIpAddress = "1.2.3.4";
+  private static Map savedStaticResolution = new HashMap<>();
+
+  @BeforeClass
+  public static void initializeDummyHostnameResolution() throws Exception {
+    String previousIpAddress;
+    for (String hostName : dummyHostNames) {
+      if (null != (previousIpAddress = NetUtils.getStaticResolution(hostName))) {
+        savedStaticResolution.put(hostName, previousIpAddress);
+      }
+      NetUtils.addStaticResolution(hostName, anyIpAddress);
+    }
+  }
+
+  @AfterClass
+  public static void restoreDummyHostnameResolution() throws Exception {
+    for (Map.Entry hostnameToIpEntry : savedStaticResolution.entrySet()) {
+      NetUtils.addStaticResolution(hostnameToIpEntry.getKey(), hostnameToIpEntry.getValue());
+    }
+  }
+
+  @Test
+  public void TestRMWebAppURLRemoteAndLocal() throws UnknownHostException {
+    Configuration configuration = new Configuration();
+    final String rmAddress = "host1:8088";
+    configuration.set(YarnConfiguration.RM_WEBAPP_ADDRESS, rmAddress);
+    final String rm1Address = "host2:8088";
+    final String rm2Address = "host3:8088";
+    configuration.set(YarnConfiguration.RM_WEBAPP_ADDRESS + "." + RM1_NODE_ID, rm1Address);
+    configuration.set(YarnConfiguration.RM_WEBAPP_ADDRESS + "." + RM2_NODE_ID, rm2Address);
+    configuration.setBoolean(YarnConfiguration.RM_HA_ENABLED, true);
+    configuration.set(YarnConfiguration.RM_HA_IDS, RM1_NODE_ID + "," + RM2_NODE_ID);
+
+    String rmRemoteUrl = WebAppUtils.getResolvedRemoteRMWebAppURLWithoutScheme(configuration);
+    Assert.assertEquals("ResolvedRemoteRMWebAppUrl should resolve to the first HA RM address", rm1Address, rmRemoteUrl);
+
+    String rmLocalUrl = WebAppUtils.getResolvedRMWebAppURLWithoutScheme(configuration);
+    Assert.assertEquals("ResolvedRMWebAppUrl should resolve to the default RM webapp address", rmAddress, rmLocalUrl);
+  }
+}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java
index d5b5c37830215..80c5b02d7c118 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java
@@ -41,6 +41,7 @@ String getAllProcessInfoFromShell() {
   }
 
   @Test (timeout = 30000)
+  @SuppressWarnings("deprecation")
   public void tree() {
     if( !Shell.WINDOWS) {
       LOG.info("Platform not Windows. Not testing");
@@ -49,30 +50,42 @@ public void tree() {
     assertTrue("WindowsBasedProcessTree should be available on Windows", 
                WindowsBasedProcessTree.isAvailable());
     
-    
     WindowsBasedProcessTreeTester pTree = new WindowsBasedProcessTreeTester("-1");
     pTree.infoStr = "3524,1024,1024,500\r\n2844,1024,1024,500\r\n";
     pTree.updateProcessTree();
+    assertTrue(pTree.getVirtualMemorySize() == 2048);
     assertTrue(pTree.getCumulativeVmem() == 2048);
+    assertTrue(pTree.getVirtualMemorySize(0) == 2048);
     assertTrue(pTree.getCumulativeVmem(0) == 2048);
+    
+    assertTrue(pTree.getRssMemorySize() == 2048);
     assertTrue(pTree.getCumulativeRssmem() == 2048);
+    assertTrue(pTree.getRssMemorySize(0) == 2048);
     assertTrue(pTree.getCumulativeRssmem(0) == 2048);
     assertTrue(pTree.getCumulativeCpuTime() == 1000);
 
     pTree.infoStr = "3524,1024,1024,1000\r\n2844,1024,1024,1000\r\n1234,1024,1024,1000\r\n";
     pTree.updateProcessTree();
+    assertTrue(pTree.getVirtualMemorySize() == 3072);
     assertTrue(pTree.getCumulativeVmem() == 3072);
+    assertTrue(pTree.getVirtualMemorySize(1) == 2048);
     assertTrue(pTree.getCumulativeVmem(1) == 2048);
+    assertTrue(pTree.getRssMemorySize() == 3072);
     assertTrue(pTree.getCumulativeRssmem() == 3072);
+    assertTrue(pTree.getRssMemorySize(1) == 2048);
     assertTrue(pTree.getCumulativeRssmem(1) == 2048);
-    assertTrue(pTree.getCumulativeCpuTime() == 3000);    
+    assertTrue(pTree.getCumulativeCpuTime() == 3000);
 
     pTree.infoStr = "3524,1024,1024,1500\r\n2844,1024,1024,1500\r\n";
     pTree.updateProcessTree();
+    assertTrue(pTree.getVirtualMemorySize() == 2048);
     assertTrue(pTree.getCumulativeVmem() == 2048);
+    assertTrue(pTree.getVirtualMemorySize(2) == 2048);
     assertTrue(pTree.getCumulativeVmem(2) == 2048);
+    assertTrue(pTree.getRssMemorySize() == 2048);
     assertTrue(pTree.getCumulativeRssmem() == 2048);
+    assertTrue(pTree.getRssMemorySize(2) == 2048);
     assertTrue(pTree.getCumulativeRssmem(2) == 2048);
-    assertTrue(pTree.getCumulativeCpuTime() == 4000);    
+    assertTrue(pTree.getCumulativeCpuTime() == 4000);
   }
 }
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java
index 06a56d85d2982..858b6b123570e 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/binding/RegistryUtils.java
@@ -116,10 +116,10 @@ public static String serviceclassPath(String user,
   }
 
   /**
-   * Create a path to a service under a user & service class
+   * Create a path to a service under a user and service class
    * @param user username or ""
    * @param serviceClass service name
-   * @param serviceName service name unique for that user & service class
+   * @param serviceName service name unique for that user and service class
    * @return a full path
    */
   public static String servicePath(String user,
@@ -135,7 +135,7 @@ public static String servicePath(String user,
    * Create a path for listing components under a service
    * @param user username or ""
    * @param serviceClass service name
-   * @param serviceName service name unique for that user & service class
+   * @param serviceName service name unique for that user and service class
    * @return a full path
    */
   public static String componentListPath(String user,
@@ -149,7 +149,7 @@ public static String componentListPath(String user,
    * Create the path to a service record for a component
    * @param user username or ""
    * @param serviceClass service name
-   * @param serviceName service name unique for that user & service class
+   * @param serviceName service name unique for that user and service class
    * @param componentName unique name/ID of the component
    * @return a full path
    */
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/RegistryOperationsClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/RegistryOperationsClient.java
index db0393678678d..44cefed06ac6f 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/RegistryOperationsClient.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/RegistryOperationsClient.java
@@ -32,7 +32,7 @@
  *
  * For SASL, the client must be operating in the context of an authed user.
  *
- * For id:pass the client must have the relevant id & password, SASL is
+ * For id:pass the client must have the relevant id and password, SASL is
  * not used even if the client has credentials.
  *
  * For anonymous, nothing is used.
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java
index d81f24b7d64c4..edcf0859fc345 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/ZookeeperConfigOptions.java
@@ -71,14 +71,13 @@ public interface ZookeeperConfigOptions {
    * The SASL client username: {@value}.
    * 
      
    * Set this to the short name of the client, e.g, "user",
-   * not user/host, or user/host@REALM
+   * not {@code user/host}, or {@code user/host@REALM}
    */
   String PROP_ZK_SASL_CLIENT_USERNAME = "zookeeper.sasl.client.username";
 
   /**
    * The SASL Server context, referring to a context in the JVM's
    * JAAS context file: {@value}
-   * 
      
    */
   String PROP_ZK_SERVER_SASL_CONTEXT =
       ZooKeeperSaslServer.LOGIN_CONTEXT_NAME_KEY;
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
index 3fa0c1920dd15..88e9d67b79d72 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
@@ -47,16 +47,16 @@
 /**
  * This is a small, localhost Zookeeper service instance that is contained
  * in a YARN service...it's been derived from Apache Twill.
- *
+ * 
      
  * It implements {@link RegistryBindingSource} and provides binding information,
- * once started. Until start() is called, the hostname &
+ * once started. Until {@link #start()} is called, the hostname and
  * port may be undefined. Accordingly, the service raises an exception in this
  * condition.
- *
+ * 
      
  * If you wish to chain together a registry service with this one under
- * the same CompositeService, this service must be added
+ * the same {@code CompositeService}, this service must be added
  * as a child first.
- *
+ * 
      
  * It also sets the configuration parameter
  * {@link RegistryConstants#KEY_REGISTRY_ZK_QUORUM}
  * to its connection string. Any code with access to the service configuration
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/package-info.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/package-info.java
index 85d24b3a0250f..fe2a0a89f2c5a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/package-info.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/package-info.java
@@ -19,9 +19,10 @@
 /**
  * Basic services for the YARN registry
  * 
        
- *   
      • The {@link org.apache.hadoop.registry.server.services.RegistryAdminService}
    - * extends the shared Yarn Registry client with registry setup and - * (potentially asynchronous) administrative actions. + *
  • + * The {@link org.apache.hadoop.registry.server.services.RegistryAdminService} + * extends the shared Yarn Registry client with registry setup and + * (potentially asynchronous) administrative actions. *
    • *
  • * The {@link org.apache.hadoop.registry.server.services.MicroZookeeperService} @@ -33,8 +34,6 @@ * extends the standard YARN composite service by making its add and remove * methods public. It is a utility service used in parts of the codebase *
    • - * * - * */ package org.apache.hadoop.registry.server.services; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java index 8da1ea1aea6fc..e64ca14ce89da 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java @@ -56,27 +56,23 @@ import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerReport; import org.apache.hadoop.yarn.conf.YarnConfiguration; -import org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException; -import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException; -import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException; import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.ipc.YarnRPC; import org.apache.hadoop.yarn.server.timeline.security.authorize.TimelinePolicyProvider; import com.google.common.base.Preconditions; -public class ApplicationHistoryClientService extends AbstractService { +public class ApplicationHistoryClientService extends AbstractService implements + ApplicationHistoryProtocol { private static final Log LOG = LogFactory .getLog(ApplicationHistoryClientService.class); private ApplicationHistoryManager history; - private ApplicationHistoryProtocol protocolHandler; private Server server; private InetSocketAddress bindAddress; public ApplicationHistoryClientService(ApplicationHistoryManager history) { super("ApplicationHistoryClientService"); this.history = history; - this.protocolHandler = new ApplicationHSClientProtocolHandler(); } protected void serviceStart() throws Exception { @@ -95,7 +91,7 @@ protected void serviceStart() throws Exception { YarnConfiguration.TIMELINE_SERVICE_HANDLER_THREAD_COUNT); server = - rpc.getServer(ApplicationHistoryProtocol.class, protocolHandler, + rpc.getServer(ApplicationHistoryProtocol.class, this, address, conf, null, conf.getInt( YarnConfiguration.TIMELINE_SERVICE_HANDLER_THREAD_COUNT, YarnConfiguration.DEFAULT_TIMELINE_SERVICE_CLIENT_THREAD_COUNT)); @@ -126,11 +122,6 @@ protected void serviceStop() throws Exception { super.serviceStop(); } - @Private - public ApplicationHistoryProtocol getClientHandler() { - return this.protocolHandler; - } - @Private public InetSocketAddress getBindAddress() { return this.bindAddress; @@ -141,109 +132,97 @@ private void refreshServiceAcls(Configuration configuration, this.server.refreshServiceAcl(configuration, policyProvider); } - private class ApplicationHSClientProtocolHandler implements - ApplicationHistoryProtocol { - - @Override - public CancelDelegationTokenResponse cancelDelegationToken( - CancelDelegationTokenRequest request) throws YarnException, IOException { - // TODO Auto-generated method stub - return null; - } - - @Override - public GetApplicationAttemptReportResponse getApplicationAttemptReport( - GetApplicationAttemptReportRequest request) throws YarnException, - IOException { - ApplicationAttemptId appAttemptId = request.getApplicationAttemptId(); - try { - GetApplicationAttemptReportResponse response = - GetApplicationAttemptReportResponse.newInstance(history - .getApplicationAttempt(appAttemptId)); - return response; - } catch (IOException e) { - String msg = "ApplicationAttempt with id '" + appAttemptId + - "' doesn't exist in the history store."; - LOG.error(msg, e); - throw new ApplicationAttemptNotFoundException(msg); - } - } + @Override + public CancelDelegationTokenResponse cancelDelegationToken( + CancelDelegationTokenRequest request) throws YarnException, IOException { + // TODO Auto-generated method stub + return null; + } - @Override - public GetApplicationAttemptsResponse getApplicationAttempts( - GetApplicationAttemptsRequest request) throws YarnException, - IOException { - GetApplicationAttemptsResponse response = - GetApplicationAttemptsResponse - .newInstance(new ArrayList(history - .getApplicationAttempts(request.getApplicationId()).values())); + @Override + public GetApplicationAttemptReportResponse getApplicationAttemptReport( + GetApplicationAttemptReportRequest request) throws YarnException, + IOException { + ApplicationAttemptId appAttemptId = request.getApplicationAttemptId(); + try { + GetApplicationAttemptReportResponse response = + GetApplicationAttemptReportResponse.newInstance(history + .getApplicationAttempt(appAttemptId)); return response; + } catch (IOException e) { + LOG.error(e.getMessage(), e); + throw e; } + } - @Override - public GetApplicationReportResponse getApplicationReport( - GetApplicationReportRequest request) throws YarnException, IOException { - ApplicationId applicationId = request.getApplicationId(); - try { - GetApplicationReportResponse response = - GetApplicationReportResponse.newInstance(history - .getApplication(applicationId)); - return response; - } catch (IOException e) { - String msg = "Application with id '" + applicationId + - "' doesn't exist in the history store."; - LOG.error(msg, e); - throw new ApplicationNotFoundException(msg); - } - } + @Override + public GetApplicationAttemptsResponse getApplicationAttempts( + GetApplicationAttemptsRequest request) throws YarnException, IOException { + GetApplicationAttemptsResponse response = + GetApplicationAttemptsResponse + .newInstance(new ArrayList(history + .getApplicationAttempts(request.getApplicationId()).values())); + return response; + } - @Override - public GetApplicationsResponse getApplications( - GetApplicationsRequest request) throws YarnException, IOException { - GetApplicationsResponse response = - GetApplicationsResponse.newInstance(new ArrayList( - history.getAllApplications().values())); + @Override + public GetApplicationReportResponse getApplicationReport( + GetApplicationReportRequest request) throws YarnException, IOException { + ApplicationId applicationId = request.getApplicationId(); + try { + GetApplicationReportResponse response = + GetApplicationReportResponse.newInstance(history + .getApplication(applicationId)); return response; + } catch (IOException e) { + LOG.error(e.getMessage(), e); + throw e; } + } - @Override - public GetContainerReportResponse getContainerReport( - GetContainerReportRequest request) throws YarnException, IOException { - ContainerId containerId = request.getContainerId(); - try { - GetContainerReportResponse response = - GetContainerReportResponse.newInstance( - history.getContainer(containerId)); - return response; - } catch (IOException e) { - String msg = "Container with id '" + containerId + - "' doesn't exist in the history store."; - LOG.error(msg, e); - throw new ContainerNotFoundException(msg); - } - } + @Override + public GetApplicationsResponse + getApplications(GetApplicationsRequest request) throws YarnException, + IOException { + GetApplicationsResponse response = + GetApplicationsResponse.newInstance(new ArrayList( + history.getAllApplications().values())); + return response; + } - @Override - public GetContainersResponse getContainers(GetContainersRequest request) - throws YarnException, IOException { - GetContainersResponse response = - GetContainersResponse.newInstance(new ArrayList( - history.getContainers(request.getApplicationAttemptId()).values())); + @Override + public GetContainerReportResponse getContainerReport( + GetContainerReportRequest request) throws YarnException, IOException { + ContainerId containerId = request.getContainerId(); + try { + GetContainerReportResponse response = + GetContainerReportResponse.newInstance(history + .getContainer(containerId)); return response; + } catch (IOException e) { + LOG.error(e.getMessage(), e); + throw e; } + } - @Override - public GetDelegationTokenResponse getDelegationToken( - GetDelegationTokenRequest request) throws YarnException, IOException { - // TODO Auto-generated method stub - return null; - } + @Override + public GetContainersResponse getContainers(GetContainersRequest request) + throws YarnException, IOException { + GetContainersResponse response = + GetContainersResponse.newInstance(new ArrayList( + history.getContainers(request.getApplicationAttemptId()).values())); + return response; + } - @Override - public RenewDelegationTokenResponse renewDelegationToken( - RenewDelegationTokenRequest request) throws YarnException, IOException { - // TODO Auto-generated method stub - return null; - } + @Override + public GetDelegationTokenResponse getDelegationToken( + GetDelegationTokenRequest request) throws YarnException, IOException { + return null; + } + + @Override + public RenewDelegationTokenResponse renewDelegationToken( + RenewDelegationTokenRequest request) throws YarnException, IOException { + return null; } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManager.java index db25d298b3cd3..041c31bc37686 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManager.java @@ -18,11 +18,125 @@ package org.apache.hadoop.yarn.server.applicationhistoryservice; -import org.apache.hadoop.classification.InterfaceAudience; -import org.apache.hadoop.classification.InterfaceStability; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import java.io.IOException; +import java.util.Map; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceAudience.Public; +import org.apache.hadoop.classification.InterfaceStability.Unstable; +import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; +import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.ApplicationReport; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ContainerReport; +import org.apache.hadoop.yarn.exceptions.YarnException; + +@Private +@Unstable +public interface ApplicationHistoryManager { + /** + * This method returns Application {@link ApplicationReport} for the specified + * {@link ApplicationId}. + * + * @param appId + * + * @return {@link ApplicationReport} for the ApplicationId. + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + ApplicationReport getApplication(ApplicationId appId) throws YarnException, + IOException; + + /** + * This method returns all Application {@link ApplicationReport}s + * + * @return map of {@link ApplicationId} to {@link ApplicationReport}s. + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + Map getAllApplications() + throws YarnException, IOException; + + /** + * Application can have multiple application attempts + * {@link ApplicationAttemptReport}. This method returns the all + * {@link ApplicationAttemptReport}s for the Application. + * + * @param appId + * + * @return all {@link ApplicationAttemptReport}s for the Application. + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + Map getApplicationAttempts( + ApplicationId appId) throws YarnException, IOException; + + /** + * This method returns {@link ApplicationAttemptReport} for specified + * {@link ApplicationId}. + * + * @param appAttemptId + * {@link ApplicationAttemptId} + * @return {@link ApplicationAttemptReport} for ApplicationAttemptId + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + ApplicationAttemptReport getApplicationAttempt( + ApplicationAttemptId appAttemptId) throws YarnException, IOException; + + /** + * This method returns {@link ContainerReport} for specified + * {@link ContainerId}. + * + * @param containerId + * {@link ContainerId} + * @return {@link ContainerReport} for ContainerId + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + ContainerReport getContainer(ContainerId containerId) throws YarnException, + IOException; + + /** + * This method returns {@link ContainerReport} for specified + * {@link ApplicationAttemptId}. + * + * @param appAttemptId + * {@link ApplicationAttemptId} + * @return {@link ContainerReport} for ApplicationAttemptId + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + ContainerReport getAMContainer(ApplicationAttemptId appAttemptId) + throws YarnException, IOException; + + /** + * This method returns Map of {@link ContainerId} to {@link ContainerReport} + * for specified {@link ApplicationAttemptId}. + * + * @param appAttemptId + * {@link ApplicationAttemptId} + * @return Map of {@link ContainerId} to {@link ContainerReport} for + * ApplicationAttemptId + * @throws YarnException + * @throws IOException + */ + @Public + @Unstable + Map getContainers( + ApplicationAttemptId appAttemptId) throws YarnException, IOException; -@InterfaceAudience.Public -@InterfaceStability.Unstable -public interface ApplicationHistoryManager extends ApplicationContext { } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java index 803dc01d1d20b..c7cf07b1a88a7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java @@ -215,7 +215,7 @@ private ContainerReport convertToContainerReport( containerHistory.getStartTime(), containerHistory.getFinishTime(), containerHistory.getDiagnosticsInfo(), logUrl, containerHistory.getContainerExitStatus(), - containerHistory.getContainerState()); + containerHistory.getContainerState(), null); } @Override diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java index 22418a8535c06..49041c75cf519 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java @@ -415,6 +415,7 @@ private static ContainerReport convertToContainerReport( String diagnosticsInfo = null; int exitStatus = ContainerExitStatus.INVALID; ContainerState state = null; + String nodeHttpAddress = null; Map entityInfo = entity.getOtherInfo(); if (entityInfo != null) { if (entityInfo @@ -444,6 +445,12 @@ private static ContainerReport convertToContainerReport( allocatedPriority = (Integer) entityInfo.get( ContainerMetricsConstants.ALLOCATED_PRIORITY_ENTITY_INFO); } + if (entityInfo.containsKey( + ContainerMetricsConstants.ALLOCATED_HOST_HTTP_ADDRESS_ENTITY_INFO)) { + nodeHttpAddress = + (String) entityInfo + .get(ContainerMetricsConstants.ALLOCATED_HOST_HTTP_ADDRESS_ENTITY_INFO); + } } List events = entity.getEvents(); if (events != null) { @@ -493,7 +500,8 @@ private static ContainerReport convertToContainerReport( Resource.newInstance(allocatedMem, allocatedVcore), NodeId.newInstance(allocatedHost, allocatedPort), Priority.newInstance(allocatedPriority), - createdTime, finishedTime, diagnosticsInfo, logUrl, exitStatus, state); + createdTime, finishedTime, diagnosticsInfo, logUrl, exitStatus, state, + nodeHttpAddress); } private ApplicationReportExt generateApplicationReport(TimelineEntity entity, @@ -509,15 +517,14 @@ private ApplicationReportExt generateApplicationReport(TimelineEntity entity, if (app.appReport.getCurrentApplicationAttemptId() != null) { ApplicationAttemptReport appAttempt = getApplicationAttempt(app.appReport.getCurrentApplicationAttemptId()); - if (appAttempt != null) { - app.appReport.setHost(appAttempt.getHost()); - app.appReport.setRpcPort(appAttempt.getRpcPort()); - app.appReport.setTrackingUrl(appAttempt.getTrackingUrl()); - app.appReport.setOriginalTrackingUrl(appAttempt.getOriginalTrackingUrl()); - } + app.appReport.setHost(appAttempt.getHost()); + app.appReport.setRpcPort(appAttempt.getRpcPort()); + app.appReport.setTrackingUrl(appAttempt.getTrackingUrl()); + app.appReport.setOriginalTrackingUrl(appAttempt.getOriginalTrackingUrl()); } - } catch (AuthorizationException e) { + } catch (AuthorizationException | ApplicationAttemptNotFoundException e) { // AuthorizationException is thrown because the user doesn't have access + // It's possible that the app is finished before the first attempt is created. app.appReport.setDiagnostics(null); app.appReport.setCurrentApplicationAttemptId(null); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java index 0bafd3695eedd..88cd153344320 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java @@ -272,7 +272,7 @@ private void startWebApp() { .$for("applicationhistory", ApplicationHistoryClientService.class, ahsClientService, "ws") .with(conf).at(bindAddress).start( - new AHSWebApp(timelineDataManager, historyManager)); + new AHSWebApp(timelineDataManager, ahsClientService)); } catch (Exception e) { String msg = "AHSWebApp failed to start."; LOG.error(msg, e); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSView.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSView.java index 4baa75d1bddd5..152364e8722e4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSView.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSView.java @@ -25,9 +25,8 @@ import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES_ID; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.initID; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI.tableInit; - import org.apache.hadoop.yarn.server.webapp.AppsBlock; +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.view.TwoColumnLayout; @@ -41,7 +40,7 @@ public class AHSView extends TwoColumnLayout { protected void preHead(Page.HTML<_> html) { commonPreHead(html); set(DATATABLES_ID, "apps"); - set(initID(DATATABLES, "apps"), appsTableInit()); + set(initID(DATATABLES, "apps"), WebPageUtils.appsTableInit()); setTableStyles(html, "apps", ".queue {width:6em}", ".ui {width:8em}"); // Set the correct title. @@ -64,27 +63,4 @@ protected Class nav() { protected Class content() { return AppsBlock.class; } - - private String appsTableInit() { - // id, user, name, queue, starttime, finishtime, state, status, progress, ui - return tableInit().append(", 'aaData': appsTableData") - .append(", bDeferRender: true").append(", bProcessing: true") - - .append("\n, aoColumnDefs: ").append(getAppsTableColumnDefs()) - - // Sort by id upon page load - .append(", aaSorting: [[0, 'desc']]}").toString(); - } - - protected String getAppsTableColumnDefs() { - StringBuilder sb = new StringBuilder(); - return sb.append("[\n").append("{'sType':'numeric', 'aTargets': [0]") - .append(", 'mRender': parseHadoopID }") - - .append("\n, {'sType':'numeric', 'aTargets': [5, 6]") - .append(", 'mRender': renderHadoopDate }") - - .append("\n, {'sType':'numeric', bSearchable:false, 'aTargets': [9]") - .append(", 'mRender': parseHadoopProgress }]").toString(); - } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebApp.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebApp.java index 814752bcf5496..4b579c60e569a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebApp.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebApp.java @@ -19,8 +19,8 @@ import static org.apache.hadoop.yarn.util.StringHelper.pajoin; -import org.apache.hadoop.yarn.server.api.ApplicationContext; -import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManager; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService; import org.apache.hadoop.yarn.server.timeline.TimelineDataManager; import org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices; import org.apache.hadoop.yarn.webapp.GenericExceptionHandler; @@ -30,17 +30,17 @@ public class AHSWebApp extends WebApp implements YarnWebParams { - private ApplicationHistoryManager applicationHistoryManager; + private final ApplicationHistoryClientService historyClientService; private TimelineDataManager timelineDataManager; public AHSWebApp(TimelineDataManager timelineDataManager, - ApplicationHistoryManager applicationHistoryManager) { + ApplicationHistoryClientService historyClientService) { this.timelineDataManager = timelineDataManager; - this.applicationHistoryManager = applicationHistoryManager; + this.historyClientService = historyClientService; } - public ApplicationHistoryManager getApplicationHistoryManager() { - return applicationHistoryManager; + public ApplicationHistoryClientService getApplicationHistoryClientService() { + return historyClientService; } public TimelineDataManager getTimelineDataManager() { @@ -53,7 +53,7 @@ public void setup() { bind(AHSWebServices.class); bind(TimelineWebServices.class); bind(GenericExceptionHandler.class); - bind(ApplicationContext.class).toInstance(applicationHistoryManager); + bind(ApplicationBaseProtocol.class).toInstance(historyClientService); bind(TimelineDataManager.class).toInstance(timelineDataManager); route("/", AHSController.class); route(pajoin("/apps", APP_STATE), AHSController.class); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java index 2af4027cc386c..9edc9ab1973ba 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java @@ -33,7 +33,7 @@ import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.api.records.YarnApplicationState; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; import org.apache.hadoop.yarn.server.webapp.WebServices; import org.apache.hadoop.yarn.server.webapp.dao.AppAttemptInfo; import org.apache.hadoop.yarn.server.webapp.dao.AppAttemptsInfo; @@ -51,8 +51,8 @@ public class AHSWebServices extends WebServices { @Inject - public AHSWebServices(ApplicationContext appContext) { - super(appContext); + public AHSWebServices(ApplicationBaseProtocol appBaseProt) { + super(appBaseProt); } @GET diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppAttemptPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppAttemptPage.java index 63b44bde663d2..540f6e6bde9a9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppAttemptPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppAttemptPage.java @@ -21,9 +21,8 @@ import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES_ID; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.initID; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI.tableInit; - import org.apache.hadoop.yarn.server.webapp.AppAttemptBlock; +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.YarnWebParams; @@ -41,8 +40,10 @@ protected void preHead(Page.HTML<_> html) { $(YarnWebParams.APPLICATION_ATTEMPT_ID))); set(DATATABLES_ID, "containers"); - set(initID(DATATABLES, "containers"), containersTableInit()); + set(initID(DATATABLES, "containers"), WebPageUtils.containersTableInit()); setTableStyles(html, "containers", ".queue {width:6em}", ".ui {width:8em}"); + + set(YarnWebParams.WEB_UI_TYPE, YarnWebParams.APP_HISTORY_WEB_UI); } @Override @@ -50,16 +51,6 @@ protected Class content() { return AppAttemptBlock.class; } - private String containersTableInit() { - return tableInit().append(", 'aaData': containersTableData") - .append(", bDeferRender: true").append(", bProcessing: true") - - .append("\n, aoColumnDefs: ").append(getContainersTableColumnDefs()) - - // Sort by id upon page load - .append(", aaSorting: [[0, 'desc']]}").toString(); - } - protected String getContainersTableColumnDefs() { StringBuilder sb = new StringBuilder(); return sb.append("[\n").append("{'sType':'numeric', 'aTargets': [0]") diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppPage.java index 96ca65918b01e..cf92c1db9afe8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AppPage.java @@ -22,9 +22,8 @@ import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES_ID; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.initID; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI.tableInit; - import org.apache.hadoop.yarn.server.webapp.AppBlock; +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.YarnWebParams; @@ -40,9 +39,13 @@ protected void preHead(Page.HTML<_> html) { appId.isEmpty() ? "Bad request: missing application ID" : join( "Application ", $(YarnWebParams.APPLICATION_ID))); - set(DATATABLES_ID, "attempts"); - set(initID(DATATABLES, "attempts"), attemptsTableInit()); + set(DATATABLES_ID, "attempts ResourceRequests"); + set(initID(DATATABLES, "attempts"), WebPageUtils.attemptsTableInit()); setTableStyles(html, "attempts", ".queue {width:6em}", ".ui {width:8em}"); + + setTableStyles(html, "ResourceRequests"); + + set(YarnWebParams.WEB_UI_TYPE, YarnWebParams.APP_HISTORY_WEB_UI); } @Override @@ -50,16 +53,6 @@ protected Class content() { return AppBlock.class; } - private String attemptsTableInit() { - return tableInit().append(", 'aaData': attemptsTableData") - .append(", bDeferRender: true").append(", bProcessing: true") - - .append("\n, aoColumnDefs: ").append(getAttemptsTableColumnDefs()) - - // Sort by id upon page load - .append(", aaSorting: [[0, 'desc']]}").toString(); - } - protected String getAttemptsTableColumnDefs() { StringBuilder sb = new StringBuilder(); return sb.append("[\n").append("{'sType':'numeric', 'aTargets': [0]") diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java index 9fd2cfc95b391..d521f7012c6b1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/LeveldbTimelineStore.java @@ -40,6 +40,7 @@ import org.apache.hadoop.yarn.proto.YarnServerCommonProtos.VersionProto; import org.apache.hadoop.yarn.server.records.Version; import org.apache.hadoop.yarn.server.records.impl.pb.VersionPBImpl; +import org.apache.hadoop.yarn.server.timeline.TimelineDataManager.CheckAcl; import org.apache.hadoop.yarn.server.timeline.util.LeveldbUtils.KeyBuilder; import org.apache.hadoop.yarn.server.timeline.util.LeveldbUtils.KeyParser; import org.apache.hadoop.yarn.server.utils.LeveldbIterator; @@ -56,6 +57,7 @@ import static org.apache.hadoop.yarn.server.timeline.GenericObjectMapper.readReverseOrderedLong; import static org.apache.hadoop.yarn.server.timeline.GenericObjectMapper.writeReverseOrderedLong; +import static org.apache.hadoop.yarn.server.timeline.TimelineDataManager.DEFAULT_DOMAIN_ID; import static org.apache.hadoop.yarn.server.timeline.util.LeveldbUtils.prefixMatches; import static org.fusesource.leveldbjni.JniDBFactory.bytes; @@ -549,12 +551,13 @@ public int compare(byte[] o1, byte[] o2) { public TimelineEntities getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, String fromId, Long fromTs, NameValuePair primaryFilter, Collection secondaryFilters, - EnumSet fields) throws IOException { + EnumSet fields, CheckAcl checkAcl) throws IOException { if (primaryFilter == null) { // if no primary filter is specified, prefix the lookup with // ENTITY_ENTRY_PREFIX return getEntityByTime(ENTITY_ENTRY_PREFIX, entityType, limit, - windowStart, windowEnd, fromId, fromTs, secondaryFilters, fields); + windowStart, windowEnd, fromId, fromTs, secondaryFilters, + fields, checkAcl); } else { // if a primary filter is specified, prefix the lookup with // INDEXED_ENTRY_PREFIX + primaryFilterName + primaryFilterValue + @@ -564,7 +567,7 @@ public TimelineEntities getEntities(String entityType, .add(GenericObjectMapper.write(primaryFilter.getValue()), true) .add(ENTITY_ENTRY_PREFIX).getBytesForLookup(); return getEntityByTime(base, entityType, limit, windowStart, windowEnd, - fromId, fromTs, secondaryFilters, fields); + fromId, fromTs, secondaryFilters, fields, checkAcl); } } @@ -586,7 +589,7 @@ public TimelineEntities getEntities(String entityType, private TimelineEntities getEntityByTime(byte[] base, String entityType, Long limit, Long starttime, Long endtime, String fromId, Long fromTs, Collection secondaryFilters, - EnumSet fields) throws IOException { + EnumSet fields, CheckAcl checkAcl) throws IOException { LeveldbIterator iterator = null; try { KeyBuilder kb = KeyBuilder.newInstance().add(base).add(entityType); @@ -683,7 +686,12 @@ private TimelineEntities getEntityByTime(byte[] base, } } if (filterPassed) { - entities.addEntity(entity); + if (entity.getDomainId() == null) { + entity.setDomainId(DEFAULT_DOMAIN_ID); + } + if (checkAcl == null || checkAcl.check(entity)) { + entities.addEntity(entity); + } } } return entities; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/MemoryTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/MemoryTimelineStore.java index 9c5419e44c1cf..3489114233d06 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/MemoryTimelineStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/MemoryTimelineStore.java @@ -47,6 +47,9 @@ import org.apache.hadoop.yarn.api.records.timeline.TimelineDomains; import org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse; import org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse.TimelinePutError; +import org.apache.hadoop.yarn.server.timeline.TimelineDataManager.CheckAcl; + +import static org.apache.hadoop.yarn.server.timeline.TimelineDataManager.DEFAULT_DOMAIN_ID; /** * In-memory implementation of {@link TimelineStore}. This @@ -79,7 +82,7 @@ public MemoryTimelineStore() { public synchronized TimelineEntities getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, String fromId, Long fromTs, NameValuePair primaryFilter, Collection secondaryFilters, - EnumSet fields) { + EnumSet fields, CheckAcl checkAcl) throws IOException { if (limit == null) { limit = DEFAULT_LIMIT; } @@ -146,7 +149,12 @@ public synchronized TimelineEntities getEntities(String entityType, Long limit, continue; } } - entitiesSelected.add(entity); + if (entity.getDomainId() == null) { + entity.setDomainId(DEFAULT_DOMAIN_ID); + } + if (checkAcl == null || checkAcl.check(entity)) { + entitiesSelected.add(entity); + } } List entitiesToReturn = new ArrayList(); for (TimelineEntity entitySelected : entitiesSelected) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java index 888c28311579e..8c6b83a57eb2e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java @@ -90,6 +90,31 @@ protected void serviceInit(Configuration conf) throws Exception { super.serviceInit(conf); } + public interface CheckAcl { + boolean check(TimelineEntity entity) throws IOException; + } + + class CheckAclImpl implements CheckAcl { + final UserGroupInformation ugi; + + public CheckAclImpl(UserGroupInformation callerUGI) { + ugi = callerUGI; + } + + public boolean check(TimelineEntity entity) throws IOException { + try{ + return timelineACLsManager.checkAccess( + ugi, ApplicationAccessType.VIEW_APP, entity); + } catch (YarnException e) { + LOG.info("Error when verifying access for user " + ugi + + " on the events of the timeline entity " + + new EntityIdentifier(entity.getEntityId(), + entity.getEntityType()), e); + return false; + } + } + } + /** * Get the timeline entities that the given user have access to. The meaning * of each argument has been documented with @@ -118,28 +143,9 @@ public TimelineEntities getEntities( fromTs, primaryFilter, secondaryFilter, - fields); - if (entities != null) { - Iterator entitiesItr = - entities.getEntities().iterator(); - while (entitiesItr.hasNext()) { - TimelineEntity entity = entitiesItr.next(); - addDefaultDomainIdIfAbsent(entity); - try { - // check ACLs - if (!timelineACLsManager.checkAccess( - callerUGI, ApplicationAccessType.VIEW_APP, entity)) { - entitiesItr.remove(); - } - } catch (YarnException e) { - LOG.error("Error when verifying access for user " + callerUGI - + " on the events of the timeline entity " - + new EntityIdentifier(entity.getEntityId(), - entity.getEntityType()), e); - entitiesItr.remove(); - } - } - } + fields, + new CheckAclImpl(callerUGI)); + if (entities == null) { return new TimelineEntities(); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineReader.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineReader.java index aba1ba27c34b9..012f4fa7ef65b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineReader.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineReader.java @@ -31,6 +31,7 @@ import org.apache.hadoop.yarn.api.records.timeline.TimelineEvents; import org.apache.hadoop.yarn.api.records.timeline.TimelineDomain; import org.apache.hadoop.yarn.api.records.timeline.TimelineDomains; +import org.apache.hadoop.yarn.server.timeline.TimelineDataManager.CheckAcl; /** * This interface is for retrieving timeline information. @@ -106,7 +107,7 @@ enum Field { TimelineEntities getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, String fromId, Long fromTs, NameValuePair primaryFilter, Collection secondaryFilters, - EnumSet fieldsToRetrieve) throws IOException; + EnumSet fieldsToRetrieve, CheckAcl checkAcl) throws IOException; /** * This method retrieves the entity information for a given entity. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java index 39c10fb1fdf5d..a3c136c214f96 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java @@ -18,20 +18,11 @@ package org.apache.hadoop.yarn.server.timeline.security; -import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStreamReader; -import java.io.Reader; -import java.nio.charset.Charset; -import java.util.HashMap; -import java.util.Map; - +import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.http.FilterContainer; import org.apache.hadoop.http.FilterInitializer; import org.apache.hadoop.http.HttpServer2; -import org.apache.hadoop.io.IOUtils; import org.apache.hadoop.security.SecurityUtil; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; import org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler; @@ -42,20 +33,20 @@ import org.apache.hadoop.security.token.delegation.web.PseudoDelegationTokenAuthenticationHandler; import org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier; -import com.google.common.annotations.VisibleForTesting; +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; /** - *

    * Initializes {@link TimelineAuthenticationFilter} which provides support for * Kerberos HTTP SPNEGO authentication. - *

    *

    * It enables Kerberos HTTP SPNEGO plus delegation token authentication for the * timeline server. - *

    - * Refer to the core-default.xml file, after the comment 'HTTP + *

    + * Refer to the {@code core-default.xml} file, after the comment 'HTTP * Authentication' for details on the configuration options. All related - * configuration properties have 'hadoop.http.authentication.' as prefix. + * configuration properties have {@code hadoop.http.authentication.} as prefix. */ public class TimelineAuthenticationFilterInitializer extends FilterInitializer { @@ -64,21 +55,15 @@ public class TimelineAuthenticationFilterInitializer extends FilterInitializer { */ public static final String PREFIX = "yarn.timeline-service.http-authentication."; - private static final String SIGNATURE_SECRET_FILE = - TimelineAuthenticationFilter.SIGNATURE_SECRET + ".file"; - @VisibleForTesting Map filterConfig; /** - *

    * Initializes {@link TimelineAuthenticationFilter} - *

    *

    * Propagates to {@link TimelineAuthenticationFilter} configuration all YARN * configuration properties prefixed with - * "yarn.timeline-service.authentication." - *

    + * {@code yarn.timeline-service.authentication.} * * @param container * The filter container @@ -111,31 +96,6 @@ public void initFilter(FilterContainer container, Configuration conf) { } } - String signatureSecretFile = filterConfig.get(SIGNATURE_SECRET_FILE); - if (signatureSecretFile != null) { - Reader reader = null; - try { - StringBuilder secret = new StringBuilder(); - reader = new InputStreamReader(new FileInputStream(new File(signatureSecretFile)), - Charset.forName("UTF-8")); - - int c = reader.read(); - while (c > -1) { - secret.append((char) c); - c = reader.read(); - } - filterConfig - .put(TimelineAuthenticationFilter.SIGNATURE_SECRET, - secret.toString()); - } catch (IOException ex) { - throw new RuntimeException( - "Could not read HTTP signature secret file: " - + signatureSecretFile); - } finally { - IOUtils.closeStream(reader); - } - } - String authType = filterConfig.get(AuthenticationFilter.AUTH_TYPE); if (authType.equals(PseudoAuthenticationHandler.TYPE)) { filterConfig.put(AuthenticationFilter.AUTH_TYPE, diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java index 32d011eaa68a4..d03b26dc083d1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryClientService.java @@ -77,7 +77,7 @@ public void testApplicationReport() throws IOException, YarnException { GetApplicationReportRequest request = GetApplicationReportRequest.newInstance(appId); GetApplicationReportResponse response = - clientService.getClientHandler().getApplicationReport(request); + clientService.getApplicationReport(request); ApplicationReport appReport = response.getApplicationReport(); Assert.assertNotNull(appReport); Assert.assertEquals(123, appReport.getApplicationResourceUsageReport() @@ -98,7 +98,7 @@ public void testApplications() throws IOException, YarnException { ApplicationId appId1 = ApplicationId.newInstance(0, 2); GetApplicationsRequest request = GetApplicationsRequest.newInstance(); GetApplicationsResponse response = - clientService.getClientHandler().getApplications(request); + clientService.getApplications(request); List appReport = response.getApplicationList(); Assert.assertNotNull(appReport); Assert.assertEquals(appId, appReport.get(0).getApplicationId()); @@ -113,7 +113,7 @@ public void testApplicationAttemptReport() throws IOException, YarnException { GetApplicationAttemptReportRequest request = GetApplicationAttemptReportRequest.newInstance(appAttemptId); GetApplicationAttemptReportResponse response = - clientService.getClientHandler().getApplicationAttemptReport(request); + clientService.getApplicationAttemptReport(request); ApplicationAttemptReport attemptReport = response.getApplicationAttemptReport(); Assert.assertNotNull(attemptReport); @@ -131,7 +131,7 @@ public void testApplicationAttempts() throws IOException, YarnException { GetApplicationAttemptsRequest request = GetApplicationAttemptsRequest.newInstance(appId); GetApplicationAttemptsResponse response = - clientService.getClientHandler().getApplicationAttempts(request); + clientService.getApplicationAttempts(request); List attemptReports = response.getApplicationAttemptList(); Assert.assertNotNull(attemptReports); @@ -150,7 +150,7 @@ public void testContainerReport() throws IOException, YarnException { GetContainerReportRequest request = GetContainerReportRequest.newInstance(containerId); GetContainerReportResponse response = - clientService.getClientHandler().getContainerReport(request); + clientService.getContainerReport(request); ContainerReport container = response.getContainerReport(); Assert.assertNotNull(container); Assert.assertEquals(containerId, container.getContainerId()); @@ -169,7 +169,7 @@ public void testContainers() throws IOException, YarnException { GetContainersRequest request = GetContainersRequest.newInstance(appAttemptId); GetContainersResponse response = - clientService.getClientHandler().getContainers(request); + clientService.getContainers(request); List containers = response.getContainerList(); Assert.assertNotNull(containers); Assert.assertEquals(containerId, containers.get(0).getContainerId()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java index 50a15f1bfd357..8cf1240340596 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java @@ -76,6 +76,10 @@ public class TestApplicationHistoryManagerOnTimelineStore { @BeforeClass public static void prepareStore() throws Exception { store = createStore(SCALE); + TimelineEntities entities = new TimelineEntities(); + entities.addEntity(createApplicationTimelineEntity( + ApplicationId.newInstance(0, SCALE + 1), true, false)); + store.put(entities); } public static TimelineStore createStore(int scale) throws Exception { @@ -129,9 +133,9 @@ private static void prepareTimelineStore(TimelineStore store, int scale) TimelineEntities entities = new TimelineEntities(); ApplicationId appId = ApplicationId.newInstance(0, i); if (i == 2) { - entities.addEntity(createApplicationTimelineEntity(appId, true)); + entities.addEntity(createApplicationTimelineEntity(appId, true, true)); } else { - entities.addEntity(createApplicationTimelineEntity(appId, false)); + entities.addEntity(createApplicationTimelineEntity(appId, false, true)); } store.put(entities); for (int j = 1; j <= scale; ++j) { @@ -215,6 +219,27 @@ public ApplicationReport run() throws Exception { } } + @Test + public void testGetApplicationReportWithNotAttempt() throws Exception { + final ApplicationId appId = ApplicationId.newInstance(0, SCALE + 1); + ApplicationReport app; + if (callerUGI == null) { + app = historyManager.getApplication(appId); + } else { + app = + callerUGI.doAs(new PrivilegedExceptionAction () { + @Override + public ApplicationReport run() throws Exception { + return historyManager.getApplication(appId); + } + }); + } + Assert.assertNotNull(app); + Assert.assertEquals(appId, app.getApplicationId()); + Assert.assertEquals(ApplicationAttemptId.newInstance(appId, -1), + app.getCurrentApplicationAttemptId()); + } + @Test public void testGetApplicationAttemptReport() throws Exception { final ApplicationAttemptId appAttemptId = @@ -308,7 +333,7 @@ public void testGetApplications() throws Exception { Collection apps = historyManager.getAllApplications().values(); Assert.assertNotNull(apps); - Assert.assertEquals(SCALE, apps.size()); + Assert.assertEquals(SCALE + 1, apps.size()); } @Test @@ -408,7 +433,7 @@ public ContainerReport run() throws Exception { } private static TimelineEntity createApplicationTimelineEntity( - ApplicationId appId, boolean emptyACLs) { + ApplicationId appId, boolean emptyACLs, boolean noAttempt) { TimelineEntity entity = new TimelineEntity(); entity.setEntityType(ApplicationMetricsConstants.ENTITY_TYPE); entity.setEntityId(appId.toString()); @@ -447,8 +472,10 @@ private static TimelineEntity createApplicationTimelineEntity( FinalApplicationStatus.UNDEFINED.toString()); eventInfo.put(ApplicationMetricsConstants.STATE_EVENT_INFO, YarnApplicationState.FINISHED.toString()); - eventInfo.put(ApplicationMetricsConstants.LATEST_APP_ATTEMPT_EVENT_INFO, - ApplicationAttemptId.newInstance(appId, 1)); + if (noAttempt) { + eventInfo.put(ApplicationMetricsConstants.LATEST_APP_ATTEMPT_EVENT_INFO, + ApplicationAttemptId.newInstance(appId, 1)); + } tEvent.setEventInfo(eventInfo); entity.addEvent(tEvent); return entity; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebApp.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebApp.java index 7bac6f265c256..2cd75800068f6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebApp.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebApp.java @@ -20,15 +20,16 @@ import static org.apache.hadoop.yarn.webapp.Params.TITLE; import static org.mockito.Mockito.mock; -import org.junit.Assert; +import org.junit.Assert; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.YarnApplicationState; import org.apache.hadoop.yarn.conf.YarnConfiguration; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService; import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManager; import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl; import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryStore; @@ -68,8 +69,8 @@ public void testAppControllerIndex() throws Exception { @Test public void testView() throws Exception { Injector injector = - WebAppTests.createMockInjector(ApplicationContext.class, - mockApplicationHistoryManager(5, 1, 1)); + WebAppTests.createMockInjector(ApplicationBaseProtocol.class, + mockApplicationHistoryClientService(5, 1, 1)); AHSView ahsViewInstance = injector.getInstance(AHSView.class); ahsViewInstance.render(); @@ -89,8 +90,8 @@ public void testView() throws Exception { @Test public void testAppPage() throws Exception { Injector injector = - WebAppTests.createMockInjector(ApplicationContext.class, - mockApplicationHistoryManager(1, 5, 1)); + WebAppTests.createMockInjector(ApplicationBaseProtocol.class, + mockApplicationHistoryClientService(1, 5, 1)); AppPage appPageInstance = injector.getInstance(AppPage.class); appPageInstance.render(); @@ -105,8 +106,8 @@ public void testAppPage() throws Exception { @Test public void testAppAttemptPage() throws Exception { Injector injector = - WebAppTests.createMockInjector(ApplicationContext.class, - mockApplicationHistoryManager(1, 1, 5)); + WebAppTests.createMockInjector(ApplicationBaseProtocol.class, + mockApplicationHistoryClientService(1, 1, 5)); AppAttemptPage appAttemptPageInstance = injector.getInstance(AppAttemptPage.class); @@ -123,8 +124,8 @@ public void testAppAttemptPage() throws Exception { @Test public void testContainerPage() throws Exception { Injector injector = - WebAppTests.createMockInjector(ApplicationContext.class, - mockApplicationHistoryManager(1, 1, 1)); + WebAppTests.createMockInjector(ApplicationBaseProtocol.class, + mockApplicationHistoryClientService(1, 1, 1)); ContainerPage containerPageInstance = injector.getInstance(ContainerPage.class); @@ -141,10 +142,12 @@ public void testContainerPage() throws Exception { WebAppTests.flushOutput(injector); } - ApplicationHistoryManager mockApplicationHistoryManager(int numApps, + ApplicationHistoryClientService mockApplicationHistoryClientService(int numApps, int numAppAttempts, int numContainers) throws Exception { ApplicationHistoryManager ahManager = new MockApplicationHistoryManagerImpl(store); + ApplicationHistoryClientService historyClientService = + new ApplicationHistoryClientService(ahManager); for (int i = 1; i <= numApps; ++i) { ApplicationId appId = ApplicationId.newInstance(0, i); writeApplicationStartData(appId); @@ -161,7 +164,7 @@ ApplicationHistoryManager mockApplicationHistoryManager(int numApps, } writeApplicationFinishData(appId); } - return ahManager; + return historyClientService; } class MockApplicationHistoryManagerImpl extends ApplicationHistoryManagerImpl { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java index 41dda91479901..913b80dd062c5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TestAHSWebServices.java @@ -28,13 +28,11 @@ import javax.servlet.FilterConfig; import javax.servlet.ServletException; import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.http.lib.StaticUserWebFilter.StaticUserFilter; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; -import org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler; import org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ContainerId; @@ -44,7 +42,7 @@ import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState; import org.apache.hadoop.yarn.api.records.YarnApplicationState; import org.apache.hadoop.yarn.conf.YarnConfiguration; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService; import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore; import org.apache.hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryManagerOnTimelineStore; import org.apache.hadoop.yarn.server.security.ApplicationACLsManager; @@ -79,7 +77,7 @@ @RunWith(Parameterized.class) public class TestAHSWebServices extends JerseyTestBase { - private static ApplicationHistoryManagerOnTimelineStore historyManager; + private static ApplicationHistoryClientService historyClientService; private static final String[] USERS = new String[] { "foo" , "bar" }; @BeforeClass @@ -93,16 +91,23 @@ public static void setupClass() throws Exception { conf.setBoolean(YarnConfiguration.YARN_ACL_ENABLE, true); conf.set(YarnConfiguration.YARN_ADMIN_ACL, "foo"); ApplicationACLsManager appAclsManager = new ApplicationACLsManager(conf); - historyManager = + ApplicationHistoryManagerOnTimelineStore historyManager = new ApplicationHistoryManagerOnTimelineStore(dataManager, appAclsManager); historyManager.init(conf); - historyManager.start(); + historyClientService = new ApplicationHistoryClientService(historyManager) { + @Override + protected void serviceStart() throws Exception { + // Do Nothing + } + }; + historyClientService.init(conf); + historyClientService.start(); } @AfterClass public static void tearDownClass() throws Exception { - if (historyManager != null) { - historyManager.stop(); + if (historyClientService != null) { + historyClientService.stop(); } } @@ -118,7 +123,7 @@ protected void configureServlets() { bind(JAXBContextResolver.class); bind(AHSWebServices.class); bind(GenericExceptionHandler.class); - bind(ApplicationContext.class).toInstance(historyManager); + bind(ApplicationBaseProtocol.class).toInstance(historyClientService); serve("/*").with(GuiceContainer.class); filter("/*").through(TestSimpleAuthFilter.class); } @@ -372,5 +377,4 @@ public void testSingleContainer() throws Exception { assertEquals(ContainerState.COMPLETE.toString(), container.getString("containerState")); } - } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java index 15edecdeb24d8..c5c0f93a811e0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestLeveldbTimelineStore.java @@ -164,13 +164,13 @@ private boolean deleteNextEntity(String entityType, byte[] ts) @Test public void testGetEntityTypes() throws IOException { List entityTypes = ((LeveldbTimelineStore)store).getEntityTypes(); - assertEquals(6, entityTypes.size()); - assertEquals("OLD_ENTITY_TYPE_1", entityTypes.get(0)); - assertEquals(entityType1, entityTypes.get(1)); - assertEquals(entityType2, entityTypes.get(2)); - assertEquals(entityType4, entityTypes.get(3)); - assertEquals(entityType5, entityTypes.get(4)); - assertEquals(entityType7, entityTypes.get(5)); + assertEquals(7, entityTypes.size()); + assertEquals("ACL_ENTITY_TYPE_1", entityTypes.get(0)); + assertEquals("OLD_ENTITY_TYPE_1", entityTypes.get(1)); + assertEquals(entityType1, entityTypes.get(2)); + assertEquals(entityType2, entityTypes.get(3)); + assertEquals(entityType4, entityTypes.get(4)); + assertEquals(entityType5, entityTypes.get(5)); } @Test @@ -201,7 +201,7 @@ public void testDeleteEntities() throws IOException, InterruptedException { ((LeveldbTimelineStore)store).discardOldEntities(-123l); assertEquals(2, getEntities("type_1").size()); assertEquals(0, getEntities("type_2").size()); - assertEquals(5, ((LeveldbTimelineStore)store).getEntityTypes().size()); + assertEquals(6, ((LeveldbTimelineStore)store).getEntityTypes().size()); ((LeveldbTimelineStore)store).discardOldEntities(123l); assertEquals(0, getEntities("type_1").size()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java index f74956735a34b..87c3b24a002f0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestTimelineDataManager.java @@ -28,6 +28,7 @@ import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity; import org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse; import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.security.AdminACLsManager; import org.apache.hadoop.yarn.server.timeline.security.TimelineACLsManager; import org.junit.After; import org.junit.Assert; @@ -40,7 +41,8 @@ public class TestTimelineDataManager extends TimelineStoreTestUtils { private FileContext fsContext; private File fsPath; private TimelineDataManager dataManaer; - + private static TimelineACLsManager aclsManager; + private static AdminACLsManager adminACLsManager; @Before public void setup() throws Exception { fsPath = new File("target", this.getClass().getSimpleName() + @@ -58,8 +60,12 @@ public void setup() throws Exception { loadVerificationEntityData(); loadTestDomainData(); - TimelineACLsManager aclsManager = new TimelineACLsManager(conf); + conf.setBoolean(YarnConfiguration.YARN_ACL_ENABLE, false); + aclsManager = new TimelineACLsManager(conf); dataManaer = new TimelineDataManager(store, aclsManager); + conf.setBoolean(YarnConfiguration.YARN_ACL_ENABLE, true); + conf.set(YarnConfiguration.YARN_ADMIN_ACL, "admin"); + adminACLsManager = new AdminACLsManager(conf); } @After @@ -84,6 +90,22 @@ public void testGetOldEntityWithOutDomainId() throws Exception { TimelineDataManager.DEFAULT_DOMAIN_ID, entity.getDomainId()); } + @Test + public void testGetEntitiesAclEnabled() throws Exception { + AdminACLsManager oldAdminACLsManager = + aclsManager.setAdminACLsManager(adminACLsManager); + try { + TimelineEntities entities = dataManaer.getEntities( + "ACL_ENTITY_TYPE_1", null, null, null, null, null, null, 1l, null, + UserGroupInformation.createUserForTesting("owner_1", new String[] {"group1"})); + Assert.assertEquals(1, entities.getEntities().size()); + Assert.assertEquals("ACL_ENTITY_ID_11", + entities.getEntities().get(0).getEntityId()); + } finally { + aclsManager.setAdminACLsManager(oldAdminACLsManager); + } + } + @Test public void testGetOldEntitiesWithOutDomainId() throws Exception { TimelineEntities entities = dataManaer.getEntities( diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java index c99786df3ed89..da71f46f161f4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TimelineStoreTestUtils.java @@ -353,6 +353,19 @@ protected void loadTestDomainData() throws IOException { domain3.setReaders("reader_user_4 reader_group_4"); domain3.setWriters("writer_user_4 writer_group_4"); store.put(domain3); + + TimelineEntities entities = new TimelineEntities(); + if (store instanceof LeveldbTimelineStore) { + LeveldbTimelineStore leveldb = (LeveldbTimelineStore) store; + entities.setEntities(Collections.singletonList(createEntity( + "ACL_ENTITY_ID_11", "ACL_ENTITY_TYPE_1", 63l, null, null, null, null, + "domain_id_4"))); + leveldb.put(entities); + entities.setEntities(Collections.singletonList(createEntity( + "ACL_ENTITY_ID_22", "ACL_ENTITY_TYPE_1", 64l, null, null, null, null, + "domain_id_2"))); + leveldb.put(entities); + } } public void testGetSingleEntity() throws IOException { @@ -419,66 +432,66 @@ public void testGetSingleEntity() throws IOException { protected List getEntities(String entityType) throws IOException { return store.getEntities(entityType, null, null, null, null, null, - null, null, null).getEntities(); + null, null, null, null).getEntities(); } protected List getEntitiesWithPrimaryFilter( String entityType, NameValuePair primaryFilter) throws IOException { return store.getEntities(entityType, null, null, null, null, null, - primaryFilter, null, null).getEntities(); + primaryFilter, null, null, null).getEntities(); } protected List getEntitiesFromId(String entityType, String fromId) throws IOException { return store.getEntities(entityType, null, null, null, fromId, null, - null, null, null).getEntities(); + null, null, null, null).getEntities(); } protected List getEntitiesFromTs(String entityType, long fromTs) throws IOException { return store.getEntities(entityType, null, null, null, null, fromTs, - null, null, null).getEntities(); + null, null, null, null).getEntities(); } protected List getEntitiesFromIdWithPrimaryFilter( String entityType, NameValuePair primaryFilter, String fromId) throws IOException { return store.getEntities(entityType, null, null, null, fromId, null, - primaryFilter, null, null).getEntities(); + primaryFilter, null, null, null).getEntities(); } protected List getEntitiesFromTsWithPrimaryFilter( String entityType, NameValuePair primaryFilter, long fromTs) throws IOException { return store.getEntities(entityType, null, null, null, null, fromTs, - primaryFilter, null, null).getEntities(); + primaryFilter, null, null, null).getEntities(); } protected List getEntitiesFromIdWithWindow(String entityType, Long windowEnd, String fromId) throws IOException { return store.getEntities(entityType, null, null, windowEnd, fromId, null, - null, null, null).getEntities(); + null, null, null, null).getEntities(); } protected List getEntitiesFromIdWithPrimaryFilterAndWindow( String entityType, Long windowEnd, String fromId, NameValuePair primaryFilter) throws IOException { return store.getEntities(entityType, null, null, windowEnd, fromId, null, - primaryFilter, null, null).getEntities(); + primaryFilter, null, null, null).getEntities(); } protected List getEntitiesWithFilters(String entityType, NameValuePair primaryFilter, Collection secondaryFilters) throws IOException { return store.getEntities(entityType, null, null, null, null, null, - primaryFilter, secondaryFilters, null).getEntities(); + primaryFilter, secondaryFilters, null, null).getEntities(); } protected List getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, NameValuePair primaryFilter, EnumSet fields) throws IOException { return store.getEntities(entityType, limit, windowStart, windowEnd, null, - null, primaryFilter, null, fields).getEntities(); + null, primaryFilter, null, fields, null).getEntities(); } public void testGetEntities() throws IOException { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java index 53d8c8132e9f9..c93e8f2bc9f0a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java @@ -34,6 +34,7 @@ import org.apache.hadoop.minikdc.MiniKdc; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authentication.KerberosTestUtils; +import org.apache.hadoop.security.authentication.client.AuthenticationException; import org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler; import org.apache.hadoop.security.authorize.AuthorizationException; import org.apache.hadoop.security.ssl.KeyStoreTestUtil; @@ -47,9 +48,9 @@ import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer; import org.apache.hadoop.yarn.server.timeline.MemoryTimelineStore; import org.apache.hadoop.yarn.server.timeline.TimelineStore; -import org.junit.After; +import org.junit.AfterClass; import org.junit.Assert; -import org.junit.Before; +import org.junit.BeforeClass; import org.junit.Test; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; @@ -77,20 +78,19 @@ public static Collection withSsl() { return Arrays.asList(new Object[][] { { false }, { true } }); } - private MiniKdc testMiniKDC; - private String keystoresDir; - private String sslConfDir; - private ApplicationHistoryServer testTimelineServer; - private Configuration conf; - private TimelineClient client; - private boolean withSsl; + private static MiniKdc testMiniKDC; + private static String keystoresDir; + private static String sslConfDir; + private static ApplicationHistoryServer testTimelineServer; + private static Configuration conf; + private static boolean withSsl; public TestTimelineAuthenticationFilter(boolean withSsl) { - this.withSsl = withSsl; + TestTimelineAuthenticationFilter.withSsl = withSsl; } - @Before - public void setup() { + @BeforeClass + public static void setup() { try { testMiniKDC = new MiniKdc(MiniKdc.createConf(), testRootDir); testMiniKDC.start(); @@ -127,6 +127,7 @@ public void setup() { "localhost:8190"); conf.set("hadoop.proxyuser.HTTP.hosts", "*"); conf.set("hadoop.proxyuser.HTTP.users", FOO_USER); + conf.setInt(YarnConfiguration.TIMELINE_SERVICE_CLIENT_MAX_RETRIES, 1); if (withSsl) { conf.set(YarnConfiguration.YARN_HTTP_POLICY_KEY, @@ -146,14 +147,17 @@ public void setup() { } catch (Exception e) { assertTrue("Couldn't setup TimelineServer", false); } + } - client = TimelineClient.createTimelineClient(); + private TimelineClient createTimelineClientForUGI() { + TimelineClient client = TimelineClient.createTimelineClient(); client.init(conf); client.start(); + return client; } - @After - public void tearDown() throws Exception { + @AfterClass + public static void tearDown() throws Exception { if (testMiniKDC != null) { testMiniKDC.stop(); } @@ -162,10 +166,6 @@ public void tearDown() throws Exception { testTimelineServer.stop(); } - if (client != null) { - client.stop(); - } - if (withSsl) { KeyStoreTestUtil.cleanupSSLConfig(keystoresDir, sslConfDir); File base = new File(BASEDIR); @@ -178,6 +178,7 @@ public void testPutTimelineEntities() throws Exception { KerberosTestUtils.doAs(HTTP_USER + "/localhost", new Callable() { @Override public Void call() throws Exception { + TimelineClient client = createTimelineClientForUGI(); TimelineEntity entityToStore = new TimelineEntity(); entityToStore.setEntityType( TestTimelineAuthenticationFilter.class.getName()); @@ -199,6 +200,7 @@ public void testPutDomains() throws Exception { KerberosTestUtils.doAs(HTTP_USER + "/localhost", new Callable() { @Override public Void call() throws Exception { + TimelineClient client = createTimelineClientForUGI(); TimelineDomain domainToStore = new TimelineDomain(); domainToStore.setId(TestTimelineAuthenticationFilter.class.getName()); domainToStore.setReaders("*"); @@ -215,119 +217,96 @@ public Void call() throws Exception { @Test public void testDelegationTokenOperations() throws Exception { - KerberosTestUtils.doAs(HTTP_USER + "/localhost", new Callable() { - @Override - public Void call() throws Exception { - // Let HTTP user to get the delegation for itself - Token token = - client.getDelegationToken( - UserGroupInformation.getCurrentUser().getShortUserName()); - Assert.assertNotNull(token); - TimelineDelegationTokenIdentifier tDT = token.decodeIdentifier(); - Assert.assertNotNull(tDT); - Assert.assertEquals(new Text(HTTP_USER), tDT.getOwner()); + TimelineClient httpUserClient = + KerberosTestUtils.doAs(HTTP_USER + "/localhost", new Callable() { + @Override + public TimelineClient call() throws Exception { + return createTimelineClientForUGI(); + } + }); + UserGroupInformation httpUser = + KerberosTestUtils.doAs(HTTP_USER + "/localhost", new Callable() { + @Override + public UserGroupInformation call() throws Exception { + return UserGroupInformation.getCurrentUser(); + } + }); + // Let HTTP user to get the delegation for itself + Token token = + httpUserClient.getDelegationToken(httpUser.getShortUserName()); + Assert.assertNotNull(token); + TimelineDelegationTokenIdentifier tDT = token.decodeIdentifier(); + Assert.assertNotNull(tDT); + Assert.assertEquals(new Text(HTTP_USER), tDT.getOwner()); - // Renew token - long renewTime1 = client.renewDelegationToken(token); - Thread.sleep(100); - long renewTime2 = client.renewDelegationToken(token); - Assert.assertTrue(renewTime1 < renewTime2); + // Renew token + long renewTime1 = httpUserClient.renewDelegationToken(token); + Thread.sleep(100); + long renewTime2 = httpUserClient.renewDelegationToken(token); + Assert.assertTrue(renewTime1 < renewTime2); - // Cancel token - client.cancelDelegationToken(token); - // Renew should not be successful because the token is canceled - try { - client.renewDelegationToken(token); - Assert.fail(); - } catch (Exception e) { - Assert.assertTrue(e.getMessage().contains( - "Renewal request for unknown token")); - } + // Cancel token + httpUserClient.cancelDelegationToken(token); + // Renew should not be successful because the token is canceled + try { + httpUserClient.renewDelegationToken(token); + Assert.fail(); + } catch (Exception e) { + Assert.assertTrue(e.getMessage().contains( + "Renewal request for unknown token")); + } - // Let HTTP user to get the delegation token for FOO user - UserGroupInformation fooUgi = UserGroupInformation.createProxyUser( - FOO_USER, UserGroupInformation.getCurrentUser()); - token = fooUgi.doAs( - new PrivilegedExceptionAction>() { + // Let HTTP user to get the delegation token for FOO user + UserGroupInformation fooUgi = UserGroupInformation.createProxyUser( + FOO_USER, httpUser); + TimelineClient fooUserClient = fooUgi.doAs( + new PrivilegedExceptionAction() { @Override - public Token run() - throws Exception { - return client.getDelegationToken( - UserGroupInformation.getCurrentUser().getShortUserName()); + public TimelineClient run() throws Exception { + return createTimelineClientForUGI(); } }); - Assert.assertNotNull(token); - tDT = token.decodeIdentifier(); - Assert.assertNotNull(tDT); - Assert.assertEquals(new Text(FOO_USER), tDT.getOwner()); - Assert.assertEquals(new Text(HTTP_USER), tDT.getRealUser()); + token = fooUserClient.getDelegationToken(httpUser.getShortUserName()); + Assert.assertNotNull(token); + tDT = token.decodeIdentifier(); + Assert.assertNotNull(tDT); + Assert.assertEquals(new Text(FOO_USER), tDT.getOwner()); + Assert.assertEquals(new Text(HTTP_USER), tDT.getRealUser()); - // Renew token - final Token tokenToRenew = token; - renewTime1 = fooUgi.doAs( - new PrivilegedExceptionAction() { - @Override - public Long run() throws Exception { - return client.renewDelegationToken(tokenToRenew); - } - }); - renewTime2 = fooUgi.doAs( - new PrivilegedExceptionAction() { - @Override - public Long run() throws Exception { - return client.renewDelegationToken(tokenToRenew); - } - }); - Assert.assertTrue(renewTime1 < renewTime2); + // Renew token as the renewer + final Token tokenToRenew = token; + renewTime1 = httpUserClient.renewDelegationToken(tokenToRenew); + renewTime2 = httpUserClient.renewDelegationToken(tokenToRenew); + Assert.assertTrue(renewTime1 < renewTime2); - // Cancel token - fooUgi.doAs( - new PrivilegedExceptionAction() { - @Override - public Void run() throws Exception { - client.cancelDelegationToken(tokenToRenew); - return null; - } - }); - // Renew should not be successful because the token is canceled - try { - fooUgi.doAs( - new PrivilegedExceptionAction() { - @Override - public Void run() throws Exception { - client.renewDelegationToken(tokenToRenew); - return null; - } - }); - Assert.fail(); - } catch (Exception e) { - Assert.assertTrue(e.getMessage().contains( - "Renewal request for unknown token")); - } + // Cancel token + fooUserClient.cancelDelegationToken(tokenToRenew); + + // Renew should not be successful because the token is canceled + try { + httpUserClient.renewDelegationToken(tokenToRenew); + Assert.fail(); + } catch (Exception e) { + Assert.assertTrue( + e.getMessage().contains("Renewal request for unknown token")); + } - // Let HTTP user to get the delegation token for BAR user - UserGroupInformation barUgi = UserGroupInformation.createProxyUser( - BAR_USER, UserGroupInformation.getCurrentUser()); - token = barUgi.doAs( - new PrivilegedExceptionAction>() { + // Let HTTP user to get the delegation token for BAR user + UserGroupInformation barUgi = UserGroupInformation.createProxyUser( + BAR_USER, httpUser); + TimelineClient barUserClient = barUgi.doAs( + new PrivilegedExceptionAction() { @Override - public Token run() - throws Exception { - try { - Token token = - client.getDelegationToken( - UserGroupInformation.getCurrentUser().getShortUserName()); - Assert.fail(); - return token; - } catch (Exception e) { - Assert.assertTrue(e instanceof AuthorizationException); - return null; - } + public TimelineClient run() { + return createTimelineClientForUGI(); } }); - return null; - } - }); - } + try { + barUserClient.getDelegationToken(httpUser.getShortUserName()); + Assert.fail(); + } catch (Exception e) { + Assert.assertTrue(e.getCause() instanceof AuthorizationException || e.getCause() instanceof AuthenticationException); + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java index e67530874f34d..ba130c61ba0bf 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/lib/ZKClient.java @@ -40,7 +40,7 @@ public class ZKClient { * the zookeeper client library to * talk to zookeeper * @param string the host - * @throws throws IOException + * @throws IOException */ public ZKClient(String string) throws IOException { zkClient = new ZooKeeper(string, 30000, new ZKWatcher()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ApplicationContext.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ApplicationContext.java deleted file mode 100644 index 0e2ffdfd909b4..0000000000000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ApplicationContext.java +++ /dev/null @@ -1,122 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.yarn.server.api; - -import java.io.IOException; -import java.util.Map; - -import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; -import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; -import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.ApplicationReport; -import org.apache.hadoop.yarn.api.records.ContainerId; -import org.apache.hadoop.yarn.api.records.ContainerReport; -import org.apache.hadoop.yarn.exceptions.YarnException; - -public interface ApplicationContext { - /** - * This method returns Application {@link ApplicationReport} for the specified - * {@link ApplicationId}. - * - * @param appId - * - * @return {@link ApplicationReport} for the ApplicationId. - * @throws YarnException - * @throws IOException - */ - ApplicationReport getApplication(ApplicationId appId) - throws YarnException, IOException; - - /** - * This method returns all Application {@link ApplicationReport}s - * - * @return map of {@link ApplicationId} to {@link ApplicationReport}s. - * @throws YarnException - * @throws IOException - */ - Map getAllApplications() - throws YarnException, IOException; - - /** - * Application can have multiple application attempts - * {@link ApplicationAttemptReport}. This method returns the all - * {@link ApplicationAttemptReport}s for the Application. - * - * @param appId - * - * @return all {@link ApplicationAttemptReport}s for the Application. - * @throws YarnException - * @throws IOException - */ - Map getApplicationAttempts( - ApplicationId appId) throws YarnException, IOException; - - /** - * This method returns {@link ApplicationAttemptReport} for specified - * {@link ApplicationId}. - * - * @param appAttemptId - * {@link ApplicationAttemptId} - * @return {@link ApplicationAttemptReport} for ApplicationAttemptId - * @throws YarnException - * @throws IOException - */ - ApplicationAttemptReport getApplicationAttempt( - ApplicationAttemptId appAttemptId) throws YarnException, IOException; - - /** - * This method returns {@link ContainerReport} for specified - * {@link ContainerId}. - * - * @param containerId - * {@link ContainerId} - * @return {@link ContainerReport} for ContainerId - * @throws YarnException - * @throws IOException - */ - ContainerReport getContainer(ContainerId containerId) - throws YarnException, IOException; - - /** - * This method returns {@link ContainerReport} for specified - * {@link ApplicationAttemptId}. - * - * @param appAttemptId - * {@link ApplicationAttemptId} - * @return {@link ContainerReport} for ApplicationAttemptId - * @throws YarnException - * @throws IOException - */ - ContainerReport getAMContainer(ApplicationAttemptId appAttemptId) - throws YarnException, IOException; - - /** - * This method returns Map of {@link ContainerId} to {@link ContainerReport} - * for specified {@link ApplicationAttemptId}. - * - * @param appAttemptId - * {@link ApplicationAttemptId} - * @return Map of {@link ContainerId} to {@link ContainerReport} for - * ApplicationAttemptId - * @throws YarnException - * @throws IOException - */ - Map getContainers( - ApplicationAttemptId appAttemptId) throws YarnException, IOException; -} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java index addd3fe6815b3..b80d9cedd02ed 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatRequest.java @@ -18,6 +18,8 @@ package org.apache.hadoop.yarn.server.api.protocolrecords; +import java.util.Set; + import org.apache.hadoop.yarn.server.api.records.MasterKey; import org.apache.hadoop.yarn.server.api.records.NodeStatus; import org.apache.hadoop.yarn.util.Records; @@ -26,7 +28,7 @@ public abstract class NodeHeartbeatRequest { public static NodeHeartbeatRequest newInstance(NodeStatus nodeStatus, MasterKey lastKnownContainerTokenMasterKey, - MasterKey lastKnownNMTokenMasterKey) { + MasterKey lastKnownNMTokenMasterKey, Set nodeLabels) { NodeHeartbeatRequest nodeHeartbeatRequest = Records.newRecord(NodeHeartbeatRequest.class); nodeHeartbeatRequest.setNodeStatus(nodeStatus); @@ -34,6 +36,7 @@ public static NodeHeartbeatRequest newInstance(NodeStatus nodeStatus, .setLastKnownContainerTokenMasterKey(lastKnownContainerTokenMasterKey); nodeHeartbeatRequest .setLastKnownNMTokenMasterKey(lastKnownNMTokenMasterKey); + nodeHeartbeatRequest.setNodeLabels(nodeLabels); return nodeHeartbeatRequest; } @@ -45,4 +48,7 @@ public static NodeHeartbeatRequest newInstance(NodeStatus nodeStatus, public abstract MasterKey getLastKnownNMTokenMasterKey(); public abstract void setLastKnownNMTokenMasterKey(MasterKey secretKey); + + public abstract Set getNodeLabels(); + public abstract void setNodeLabels(Set nodeLabels); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java index 9fb44caf63ec1..1498a0c16d1d8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java @@ -67,4 +67,7 @@ public interface NodeHeartbeatResponse { void setSystemCredentialsForApps( Map systemCredentials); + + boolean getAreNodeLabelsAcceptedByRM(); + void setAreNodeLabelsAcceptedByRM(boolean areNodeLabelsAcceptedByRM); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java index 0e3d7e4c9f32f..bf09b33f8afd6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java @@ -19,6 +19,7 @@ package org.apache.hadoop.yarn.server.api.protocolrecords; import java.util.List; +import java.util.Set; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.NodeId; @@ -31,6 +32,14 @@ public static RegisterNodeManagerRequest newInstance(NodeId nodeId, int httpPort, Resource resource, String nodeManagerVersionId, List containerStatuses, List runningApplications) { + return newInstance(nodeId, httpPort, resource, nodeManagerVersionId, + containerStatuses, runningApplications, null); + } + + public static RegisterNodeManagerRequest newInstance(NodeId nodeId, + int httpPort, Resource resource, String nodeManagerVersionId, + List containerStatuses, + List runningApplications, Set nodeLabels) { RegisterNodeManagerRequest request = Records.newRecord(RegisterNodeManagerRequest.class); request.setHttpPort(httpPort); @@ -39,6 +48,7 @@ public static RegisterNodeManagerRequest newInstance(NodeId nodeId, request.setNMVersion(nodeManagerVersionId); request.setContainerStatuses(containerStatuses); request.setRunningApplications(runningApplications); + request.setNodeLabels(nodeLabels); return request; } @@ -47,12 +57,15 @@ public static RegisterNodeManagerRequest newInstance(NodeId nodeId, public abstract Resource getResource(); public abstract String getNMVersion(); public abstract List getNMContainerStatuses(); + public abstract Set getNodeLabels(); + public abstract void setNodeLabels(Set nodeLabels); /** * We introduce this here because currently YARN RM doesn't persist nodes info * for application running. When RM restart happened, we cannot determinate if * a node should do application cleanup (like log-aggregation, status update, - * etc.) or not.

    + * etc.) or not. + *

    * When we have this running application list in node manager register * request, we can recover nodes info for running applications. And then we * can take actions accordingly diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java index b20803fb9cca8..c8678f6cd33e6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerResponse.java @@ -45,4 +45,7 @@ public interface RegisterNodeManagerResponse { void setRMVersion(String version); String getRMVersion(); + + boolean getAreNodeLabelsAcceptedByRM(); + void setAreNodeLabelsAcceptedByRM(boolean areNodeLabelsAcceptedByRM); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java index 26d1f190de31d..16d47f9df1ca1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatRequestPBImpl.java @@ -18,6 +18,11 @@ package org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb; +import java.util.HashSet; +import java.util.Set; + +import org.apache.hadoop.yarn.proto.YarnProtos.NodeIdToLabelsProto; +import org.apache.hadoop.yarn.proto.YarnProtos.StringArrayProto; import org.apache.hadoop.yarn.proto.YarnServerCommonProtos.MasterKeyProto; import org.apache.hadoop.yarn.proto.YarnServerCommonProtos.NodeStatusProto; import org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos.NodeHeartbeatRequestProto; @@ -36,6 +41,7 @@ public class NodeHeartbeatRequestPBImpl extends NodeHeartbeatRequest { private NodeStatus nodeStatus = null; private MasterKey lastKnownContainerTokenMasterKey = null; private MasterKey lastKnownNMTokenMasterKey = null; + private Set labels = null; public NodeHeartbeatRequestPBImpl() { builder = NodeHeartbeatRequestProto.newBuilder(); @@ -80,6 +86,11 @@ private void mergeLocalToBuilder() { builder.setLastKnownNmTokenMasterKey( convertToProtoFormat(this.lastKnownNMTokenMasterKey)); } + if (this.labels != null) { + builder.clearNodeLabels(); + builder.setNodeLabels(StringArrayProto.newBuilder() + .addAllElements(this.labels).build()); + } } private void mergeLocalToProto() { @@ -178,4 +189,30 @@ private MasterKeyPBImpl convertFromProtoFormat(MasterKeyProto p) { private MasterKeyProto convertToProtoFormat(MasterKey t) { return ((MasterKeyPBImpl)t).getProto(); } + + @Override + public Set getNodeLabels() { + initNodeLabels(); + return this.labels; + } + + @Override + public void setNodeLabels(Set nodeLabels) { + maybeInitBuilder(); + builder.clearNodeLabels(); + this.labels = nodeLabels; + } + + private void initNodeLabels() { + if (this.labels != null) { + return; + } + NodeHeartbeatRequestProtoOrBuilder p = viaProto ? proto : builder; + if (!p.hasNodeLabels()) { + labels = null; + return; + } + StringArrayProto nodeLabels = p.getNodeLabels(); + labels = new HashSet(nodeLabels.getElementsList()); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java index 630a5bf58891d..e27d8ca007b70 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java @@ -483,5 +483,18 @@ private MasterKeyPBImpl convertFromProtoFormat(MasterKeyProto p) { private MasterKeyProto convertToProtoFormat(MasterKey t) { return ((MasterKeyPBImpl) t).getProto(); } + + @Override + public boolean getAreNodeLabelsAcceptedByRM() { + NodeHeartbeatResponseProtoOrBuilder p = + this.viaProto ? this.proto : this.builder; + return p.getAreNodeLabelsAcceptedByRM(); + } + + @Override + public void setAreNodeLabelsAcceptedByRM(boolean areNodeLabelsAcceptedByRM) { + maybeInitBuilder(); + this.builder.setAreNodeLabelsAcceptedByRM(areNodeLabelsAcceptedByRM); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java index ce4faec575034..1d2bb829a3734 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java @@ -20,32 +20,27 @@ import java.util.ArrayList; +import java.util.HashSet; import java.util.Iterator; import java.util.List; +import java.util.Set; -import org.apache.hadoop.yarn.api.records.NodeId; -import org.apache.hadoop.yarn.api.records.Resource; -import org.apache.hadoop.yarn.api.records.impl.pb.NodeIdPBImpl; -import org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl; import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.ContainerStatus; import org.apache.hadoop.yarn.api.records.NodeId; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl; -import org.apache.hadoop.yarn.api.records.impl.pb.ContainerStatusPBImpl; import org.apache.hadoop.yarn.api.records.impl.pb.NodeIdPBImpl; import org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl; import org.apache.hadoop.yarn.proto.YarnProtos.ApplicationIdProto; -import org.apache.hadoop.yarn.proto.YarnProtos.ContainerStatusProto; import org.apache.hadoop.yarn.proto.YarnProtos.NodeIdProto; +import org.apache.hadoop.yarn.proto.YarnProtos.NodeIdToLabelsProto; import org.apache.hadoop.yarn.proto.YarnProtos.ResourceProto; +import org.apache.hadoop.yarn.proto.YarnProtos.StringArrayProto; import org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos.NMContainerStatusProto; import org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos.RegisterNodeManagerRequestProto; import org.apache.hadoop.yarn.proto.YarnServerCommonServiceProtos.RegisterNodeManagerRequestProtoOrBuilder; import org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus; import org.apache.hadoop.yarn.server.api.protocolrecords.RegisterNodeManagerRequest; - - public class RegisterNodeManagerRequestPBImpl extends RegisterNodeManagerRequest { RegisterNodeManagerRequestProto proto = RegisterNodeManagerRequestProto.getDefaultInstance(); @@ -56,7 +51,8 @@ public class RegisterNodeManagerRequestPBImpl extends RegisterNodeManagerRequest private NodeId nodeId = null; private List containerStatuses = null; private List runningApplications = null; - + private Set labels = null; + public RegisterNodeManagerRequestPBImpl() { builder = RegisterNodeManagerRequestProto.newBuilder(); } @@ -86,7 +82,11 @@ private void mergeLocalToBuilder() { if (this.nodeId != null) { builder.setNodeId(convertToProtoFormat(this.nodeId)); } - + if (this.labels != null) { + builder.clearNodeLabels(); + builder.setNodeLabels(StringArrayProto.newBuilder() + .addAllElements(this.labels).build()); + } } private synchronized void addNMContainerStatusesToProto() { @@ -292,6 +292,32 @@ public void setNMVersion(String version) { builder.setNmVersion(version); } + @Override + public Set getNodeLabels() { + initNodeLabels(); + return this.labels; + } + + @Override + public void setNodeLabels(Set nodeLabels) { + maybeInitBuilder(); + builder.clearNodeLabels(); + this.labels = nodeLabels; + } + + private void initNodeLabels() { + if (this.labels != null) { + return; + } + RegisterNodeManagerRequestProtoOrBuilder p = viaProto ? proto : builder; + if (!p.hasNodeLabels()) { + labels=null; + return; + } + StringArrayProto nodeLabels = p.getNodeLabels(); + labels = new HashSet(nodeLabels.getElementsList()); + } + private ApplicationIdPBImpl convertFromProtoFormat(ApplicationIdProto p) { return new ApplicationIdPBImpl(p); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java index ac329edd6dad1..391d00dc1092b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerResponsePBImpl.java @@ -216,4 +216,17 @@ private MasterKeyPBImpl convertFromProtoFormat(MasterKeyProto p) { private MasterKeyProto convertToProtoFormat(MasterKey t) { return ((MasterKeyPBImpl)t).getProto(); } + + @Override + public boolean getAreNodeLabelsAcceptedByRM() { + RegisterNodeManagerResponseProtoOrBuilder p = + this.viaProto ? this.proto : this.builder; + return p.getAreNodeLabelsAcceptedByRM(); + } + + @Override + public void setAreNodeLabelsAcceptedByRM(boolean areNodeLabelsAcceptedByRM) { + maybeInitBuilder(); + this.builder.setAreNodeLabelsAcceptedByRM(areNodeLabelsAcceptedByRM); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeHealthStatus.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeHealthStatus.java index bc5825a9b9500..b21b88071f5e3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeHealthStatus.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeHealthStatus.java @@ -26,19 +26,17 @@ import org.apache.hadoop.yarn.util.Records; /** - *

    NodeHealthStatus is a summary of the health status of the - * node.

    - * - *

    It includes information such as: - *

      - *
    • - * An indicator of whether the node is healthy, as determined by the - * health-check script. - *
      • - *
    • The previous time at which the health status was reported.
      • - *
    • A diagnostic report on the health status.
      • - *
    - *

    + * {@code NodeHealthStatus} is a summary of the health status of the node. + *

    + * It includes information such as: + *

      + *
    • + * An indicator of whether the node is healthy, as determined by the + * health-check script. + *
      • + *
    • The previous time at which the health status was reported.
      • + *
    • A diagnostic report on the health status.
      • + *
    * * @see NodeReport * @see ApplicationClientProtocol#getClusterNodes(org.apache.hadoop.yarn.api.protocolrecords.GetClusterNodesRequest) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java index 8791da4f219da..0d5540df6bc8f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java @@ -57,4 +57,6 @@ public class ContainerMetricsConstants { public static final String STATE_EVENT_INFO = "YARN_CONTAINER_STATE"; + public static final String ALLOCATED_HOST_HTTP_ADDRESS_ENTITY_INFO = + "YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS"; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java index a62cda39a3018..9fc13348c132a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java @@ -43,14 +43,11 @@ public class RMAuthenticationFilterInitializer extends FilterInitializer { String configPrefix; - String signatureSecretFileProperty; String kerberosPrincipalProperty; String cookiePath; public RMAuthenticationFilterInitializer() { this.configPrefix = "hadoop.http.authentication."; - this.signatureSecretFileProperty = - AuthenticationFilter.SIGNATURE_SECRET + ".file"; this.kerberosPrincipalProperty = KerberosAuthenticationHandler.PRINCIPAL; this.cookiePath = "/"; } @@ -77,34 +74,6 @@ protected Map createFilterConfig(Configuration conf) { } } - String signatureSecretFile = filterConfig.get(signatureSecretFileProperty); - if (signatureSecretFile != null) { - Reader reader = null; - try { - StringBuilder secret = new StringBuilder(); - reader = - new InputStreamReader(new FileInputStream(signatureSecretFile), - "UTF-8"); - int c = reader.read(); - while (c > -1) { - secret.append((char) c); - c = reader.read(); - } - filterConfig.put(AuthenticationFilter.SIGNATURE_SECRET, - secret.toString()); - } catch (IOException ex) { - // if running in non-secure mode, this filter only gets added - // because the user has not setup his own filter so just generate - // a random secret. in secure mode, the user needs to setup security - if (UserGroupInformation.isSecurityEnabled()) { - throw new RuntimeException( - "Could not read HTTP signature secret file: " + signatureSecretFile); - } - } finally { - IOUtils.closeQuietly(reader); - } - } - // Resolve _HOST into bind address String bindAddress = conf.get(HttpServer2.BIND_ADDRESS); String principal = filterConfig.get(kerberosPrincipalProperty); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppAttemptBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppAttemptBlock.java index 4a02892f7e20b..dca39d6d3c02c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppAttemptBlock.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppAttemptBlock.java @@ -19,7 +19,6 @@ import static org.apache.hadoop.yarn.util.StringHelper.join; import static org.apache.hadoop.yarn.webapp.YarnWebParams.APPLICATION_ATTEMPT_ID; - import java.security.PrivilegedExceptionAction; import java.util.Collection; @@ -27,10 +26,14 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; +import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerReport; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState; import org.apache.hadoop.yarn.server.webapp.dao.AppAttemptInfo; import org.apache.hadoop.yarn.server.webapp.dao.ContainerInfo; import org.apache.hadoop.yarn.util.ConverterUtils; @@ -39,17 +42,18 @@ import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TBODY; import org.apache.hadoop.yarn.webapp.view.HtmlBlock; import org.apache.hadoop.yarn.webapp.view.InfoBlock; - import com.google.inject.Inject; public class AppAttemptBlock extends HtmlBlock { private static final Log LOG = LogFactory.getLog(AppAttemptBlock.class); - private final ApplicationContext appContext; + protected ApplicationBaseProtocol appBaseProt; + protected ApplicationAttemptId appAttemptId = null; @Inject - public AppAttemptBlock(ApplicationContext appContext) { - this.appContext = appContext; + public AppAttemptBlock(ApplicationBaseProtocol appBaseProt, ViewContext ctx) { + super(ctx); + this.appBaseProt = appBaseProt; } @Override @@ -60,7 +64,6 @@ protected void render(Block html) { return; } - ApplicationAttemptId appAttemptId = null; try { appAttemptId = ConverterUtils.toApplicationAttemptId(attemptid); } catch (IllegalArgumentException e) { @@ -68,18 +71,22 @@ protected void render(Block html) { return; } - final ApplicationAttemptId appAttemptIdFinal = appAttemptId; UserGroupInformation callerUGI = getCallerUGI(); - ApplicationAttemptReport appAttemptReport; + ApplicationAttemptReport appAttemptReport = null; try { + final GetApplicationAttemptReportRequest request = + GetApplicationAttemptReportRequest.newInstance(appAttemptId); if (callerUGI == null) { - appAttemptReport = appContext.getApplicationAttempt(appAttemptId); + appAttemptReport = + appBaseProt.getApplicationAttemptReport(request) + .getApplicationAttemptReport(); } else { appAttemptReport = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ApplicationAttemptReport run() throws Exception { - return appContext.getApplicationAttempt(appAttemptIdFinal); + return appBaseProt.getApplicationAttemptReport(request) + .getApplicationAttemptReport(); } }); } @@ -90,10 +97,35 @@ public ApplicationAttemptReport run() throws Exception { html.p()._(message)._(); return; } + if (appAttemptReport == null) { puts("Application Attempt not found: " + attemptid); return; } + + boolean exceptionWhenGetContainerReports = false; + Collection containers = null; + try { + final GetContainersRequest request = + GetContainersRequest.newInstance(appAttemptId); + if (callerUGI == null) { + containers = appBaseProt.getContainers(request).getContainerList(); + } else { + containers = callerUGI.doAs( + new PrivilegedExceptionAction> () { + @Override + public Collection run() throws Exception { + return appBaseProt.getContainers(request).getContainerList(); + } + }); + } + } catch (RuntimeException e) { + // have this block to suppress the findbugs warning + exceptionWhenGetContainerReports = true; + } catch (Exception e) { + exceptionWhenGetContainerReports = true; + } + AppAttemptInfo appAttempt = new AppAttemptInfo(appAttemptReport); setTitle(join("Application Attempt ", attemptid)); @@ -104,43 +136,35 @@ public ApplicationAttemptReport run() throws Exception { node = appAttempt.getHost() + ":" + appAttempt.getRpcPort(); } info("Application Attempt Overview") - ._("State", appAttempt.getAppAttemptState()) ._( - "Master Container", - appAttempt.getAmContainerId() == null ? "#" : root_url("container", - appAttempt.getAmContainerId()), + "Application Attempt State:", + appAttempt.getAppAttemptState() == null ? UNAVAILABLE : appAttempt + .getAppAttemptState()) + ._( + "AM Container:", + appAttempt.getAmContainerId() == null || containers == null + || !hasAMContainer(appAttemptReport.getAMContainerId(), containers) + ? null : root_url("container", appAttempt.getAmContainerId()), String.valueOf(appAttempt.getAmContainerId())) ._("Node:", node) ._( "Tracking URL:", - appAttempt.getTrackingUrl() == null ? "#" : root_url(appAttempt - .getTrackingUrl()), "History") - ._("Diagnostics Info:", appAttempt.getDiagnosticsInfo()); - - html._(InfoBlock.class); - - Collection containers; - try { - if (callerUGI == null) { - containers = appContext.getContainers(appAttemptId).values(); - } else { - containers = callerUGI.doAs( - new PrivilegedExceptionAction> () { - @Override - public Collection run() throws Exception { - return appContext.getContainers(appAttemptIdFinal).values(); - } - }); - } - } catch (RuntimeException e) { - // have this block to suppress the findbugs warning - html - .p() - ._( - "Sorry, Failed to get containers for application attempt" + attemptid - + ".")._(); - return; - } catch (Exception e) { + appAttempt.getTrackingUrl() == null + || appAttempt.getTrackingUrl() == UNAVAILABLE ? null + : root_url(appAttempt.getTrackingUrl()), + appAttempt.getTrackingUrl() == null + || appAttempt.getTrackingUrl() == UNAVAILABLE + ? "Unassigned" + : appAttempt.getAppAttemptState() == YarnApplicationAttemptState.FINISHED + || appAttempt.getAppAttemptState() == YarnApplicationAttemptState.FAILED + || appAttempt.getAppAttemptState() == YarnApplicationAttemptState.KILLED + ? "History" : "ApplicationMaster") + ._("Diagnostics Info:", appAttempt.getDiagnosticsInfo() == null ? + "" : appAttempt.getDiagnosticsInfo()); + + + + if (exceptionWhenGetContainerReports) { html .p() ._( @@ -149,6 +173,9 @@ public Collection run() throws Exception { return; } + createAttemptHeadRoomTable(html); + html._(InfoBlock.class); + // Container Table TBODY> tbody = html.table("#containers").thead().tr().th(".id", "Container ID") @@ -165,12 +192,15 @@ public Collection run() throws Exception { .append(url("container", container.getContainerId())) .append("'>") .append(container.getContainerId()) - .append("\",\"") + .append("\",\"\",\"") + container.getNodeHttpAddress() == null ? "#" : "href='" + + container.getNodeHttpAddress()) + .append("'>") + .append(container.getNodeHttpAddress() == null ? "N/A" : + StringEscapeUtils.escapeJavaScript(StringEscapeUtils + .escapeHtml(container.getNodeHttpAddress()))) + .append("\",\"") .append(container.getContainerExitStatus()).append("\",\"") @@ -187,4 +217,18 @@ public Collection run() throws Exception { tbody._()._(); } -} \ No newline at end of file + + private boolean hasAMContainer(ContainerId containerId, + Collection containers) { + for (ContainerReport container : containers) { + if (containerId.equals(container.getContainerId())) { + return true; + } + } + return false; + } + + protected void createAttemptHeadRoomTable(Block html) { + + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java index 8fa40860b82a9..abb6b9cebf381 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java @@ -20,23 +20,34 @@ import static org.apache.hadoop.yarn.util.StringHelper.join; import static org.apache.hadoop.yarn.webapp.YarnWebParams.APPLICATION_ID; - +import static org.apache.hadoop.yarn.webapp.YarnWebParams.WEB_UI_TYPE; import java.security.PrivilegedExceptionAction; import java.util.Collection; - import org.apache.commons.lang.StringEscapeUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest; import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ApplicationReport; +import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerReport; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import org.apache.hadoop.yarn.api.records.FinalApplicationStatus; +import org.apache.hadoop.yarn.api.records.YarnApplicationState; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException; import org.apache.hadoop.yarn.server.webapp.dao.AppAttemptInfo; import org.apache.hadoop.yarn.server.webapp.dao.AppInfo; import org.apache.hadoop.yarn.server.webapp.dao.ContainerInfo; import org.apache.hadoop.yarn.util.Apps; import org.apache.hadoop.yarn.util.Times; +import org.apache.hadoop.yarn.webapp.YarnWebParams; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TBODY; @@ -47,23 +58,28 @@ public class AppBlock extends HtmlBlock { - protected ApplicationContext appContext; + private static final Log LOG = LogFactory.getLog(AppBlock.class); + protected ApplicationBaseProtocol appBaseProt; + protected Configuration conf; + protected ApplicationId appID = null; @Inject - AppBlock(ApplicationContext appContext, ViewContext ctx) { + protected AppBlock(ApplicationBaseProtocol appBaseProt, ViewContext ctx, + Configuration conf) { super(ctx); - this.appContext = appContext; + this.appBaseProt = appBaseProt; + this.conf = conf; } @Override protected void render(Block html) { + String webUiType = $(WEB_UI_TYPE); String aid = $(APPLICATION_ID); if (aid.isEmpty()) { puts("Bad request: requires Application ID"); return; } - ApplicationId appID = null; try { appID = Apps.toAppID(aid); } catch (Exception e) { @@ -71,18 +87,21 @@ protected void render(Block html) { return; } - final ApplicationId appIDFinal = appID; UserGroupInformation callerUGI = getCallerUGI(); - ApplicationReport appReport; + ApplicationReport appReport = null; try { + final GetApplicationReportRequest request = + GetApplicationReportRequest.newInstance(appID); if (callerUGI == null) { - appReport = appContext.getApplication(appID); + appReport = + appBaseProt.getApplicationReport(request).getApplicationReport(); } else { appReport = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ApplicationReport run() throws Exception { - return appContext.getApplication(appIDFinal); + return appBaseProt.getApplicationReport(request) + .getApplicationReport(); } }); } @@ -92,41 +111,90 @@ public ApplicationReport run() throws Exception { html.p()._(message)._(); return; } + if (appReport == null) { puts("Application not found: " + aid); return; } + AppInfo app = new AppInfo(appReport); setTitle(join("Application ", aid)); + if (webUiType != null + && webUiType.equals(YarnWebParams.RM_WEB_UI) + && conf.getBoolean(YarnConfiguration.RM_WEBAPP_UI_ACTIONS_ENABLED, + YarnConfiguration.DEFAULT_RM_WEBAPP_UI_ACTIONS_ENABLED)) { + // Application Kill + html.div() + .button() + .$onclick("confirmAction()").b("Kill Application")._() + ._(); + + StringBuilder script = new StringBuilder(); + script.append("function confirmAction() {") + .append(" b = confirm(\"Are you sure?\");") + .append(" if (b == true) {") + .append(" $.ajax({") + .append(" type: 'PUT',") + .append(" url: '/ws/v1/cluster/apps/").append(aid).append("/state',") + .append(" contentType: 'application/json',") + .append(" data: '{\"state\":\"KILLED\"}',") + .append(" dataType: 'json'") + .append(" }).done(function(data){") + .append(" setTimeout(function(){") + .append(" location.href = '/cluster/app/").append(aid).append("';") + .append(" }, 1000);") + .append(" }).fail(function(data){") + .append(" console.log(data);") + .append(" });") + .append(" }") + .append("}"); + + html.script().$type("text/javascript")._(script.toString())._(); + } + info("Application Overview") ._("User:", app.getUser()) ._("Name:", app.getName()) ._("Application Type:", app.getType()) - ._("State:", app.getAppState()) - ._("FinalStatus:", app.getFinalAppStatus()) + ._("Application Tags:", + app.getApplicationTags() == null ? "" : app.getApplicationTags()) + ._("YarnApplicationState:", + app.getAppState() == null ? UNAVAILABLE : clarifyAppState(app + .getAppState())) + ._("FinalStatus Reported by AM:", + clairfyAppFinalStatus(app.getFinalAppStatus())) ._("Started:", Times.format(app.getStartedTime())) ._( "Elapsed:", StringUtils.formatTime(Times.elapsed(app.getStartedTime(), app.getFinishedTime()))) ._("Tracking URL:", - app.getTrackingUrl() == null ? "#" : root_url(app.getTrackingUrl()), - "History")._("Diagnostics:", app.getDiagnosticsInfo()); - - html._(InfoBlock.class); + app.getTrackingUrl() == null || app.getTrackingUrl() == UNAVAILABLE + ? null : root_url(app.getTrackingUrl()), + app.getTrackingUrl() == null || app.getTrackingUrl() == UNAVAILABLE + ? "Unassigned" : app.getAppState() == YarnApplicationState.FINISHED + || app.getAppState() == YarnApplicationState.FAILED + || app.getAppState() == YarnApplicationState.KILLED ? "History" + : "ApplicationMaster") + ._("Diagnostics:", + app.getDiagnosticsInfo() == null ? "" : app.getDiagnosticsInfo()); Collection attempts; try { + final GetApplicationAttemptsRequest request = + GetApplicationAttemptsRequest.newInstance(appID); if (callerUGI == null) { - attempts = appContext.getApplicationAttempts(appID).values(); + attempts = appBaseProt.getApplicationAttempts(request) + .getApplicationAttemptList(); } else { attempts = callerUGI.doAs( new PrivilegedExceptionAction> () { @Override public Collection run() throws Exception { - return appContext.getApplicationAttempts(appIDFinal).values(); + return appBaseProt.getApplicationAttempts(request) + .getApplicationAttemptList(); } }); } @@ -138,6 +206,10 @@ public Collection run() throws Exception { return; } + createApplicationMetricsTable(html); + + html._(InfoBlock.class); + // Application Attempt Table TBODY> tbody = html.table("#attempts").thead().tr().th(".id", "Attempt ID") @@ -147,18 +219,28 @@ public Collection run() throws Exception { StringBuilder attemptsTableData = new StringBuilder("[\n"); for (final ApplicationAttemptReport appAttemptReport : attempts) { AppAttemptInfo appAttempt = new AppAttemptInfo(appAttemptReport); - ContainerReport containerReport; + ContainerReport containerReport = null; try { + // AM container is always the first container of the attempt + final GetContainerReportRequest request = + GetContainerReportRequest.newInstance(ContainerId.newContainerId( + appAttemptReport.getApplicationAttemptId(), 1)); if (callerUGI == null) { - containerReport = appContext.getAMContainer(appAttemptReport - .getApplicationAttemptId()); + containerReport = + appBaseProt.getContainerReport(request).getContainerReport(); } else { containerReport = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ContainerReport run() throws Exception { - return appContext.getAMContainer(appAttemptReport - .getApplicationAttemptId()); + ContainerReport report = null; + try { + report = appBaseProt.getContainerReport(request) + .getContainerReport(); + } catch (ContainerNotFoundException ex) { + LOG.warn(ex.getMessage()); + } + return report; } }); } @@ -170,17 +252,14 @@ public ContainerReport run() throws Exception { html.p()._(message)._(); return; } - long startTime = Long.MAX_VALUE; + long startTime = 0L; String logsLink = null; + String nodeLink = null; if (containerReport != null) { ContainerInfo container = new ContainerInfo(containerReport); startTime = container.getStartedTime(); logsLink = containerReport.getLogUrl(); - } - String nodeLink = null; - if (appAttempt.getHost() != null && appAttempt.getRpcPort() >= 0 - && appAttempt.getRpcPort() < 65536) { - nodeLink = appAttempt.getHost() + ":" + appAttempt.getRpcPort(); + nodeLink = containerReport.getNodeHttpAddress(); } // AppAttemptID numerical value parsed by parseHadoopID in // yarn.dt.plugins.js @@ -191,15 +270,13 @@ public ContainerReport run() throws Exception { .append(appAttempt.getAppAttemptId()) .append("\",\"") .append(startTime) - .append("\",\"") - .append( - nodeLink == null ? "N/A" : StringEscapeUtils + .append(nodeLink == null ? "N/A" : StringEscapeUtils .escapeJavaScript(StringEscapeUtils.escapeHtml(nodeLink))) - .append("\",\"") + .append("\",\"") .append(logsLink == null ? "N/A" : "Logs").append("\"],\n"); } if (attemptsTableData.charAt(attemptsTableData.length() - 2) == ',') { @@ -212,4 +289,35 @@ public ContainerReport run() throws Exception { tbody._()._(); } + + private String clarifyAppState(YarnApplicationState state) { + String ret = state.toString(); + switch (state) { + case NEW: + return ret + ": waiting for application to be initialized"; + case NEW_SAVING: + return ret + ": waiting for application to be persisted in state-store."; + case SUBMITTED: + return ret + ": waiting for application to be accepted by scheduler."; + case ACCEPTED: + return ret + ": waiting for AM container to be allocated, launched and" + + " register with RM."; + case RUNNING: + return ret + ": AM has registered with RM and started running."; + default: + return ret; + } + } + + private String clairfyAppFinalStatus(FinalApplicationStatus status) { + if (status == FinalApplicationStatus.UNDEFINED) { + return "Application has not completed yet."; + } + return status.toString(); + } + + // The preemption metrics only need to be shown in RM WebUI + protected void createApplicationMetricsTable(Block html) { + + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java index f341cf6e55db2..161486df1b2e3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java @@ -25,13 +25,16 @@ import java.security.PrivilegedExceptionAction; import java.util.Collection; -import java.util.HashSet; +import java.util.EnumSet; import org.apache.commons.lang.StringEscapeUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest; import org.apache.hadoop.yarn.api.records.ApplicationReport; import org.apache.hadoop.yarn.api.records.YarnApplicationState; -import org.apache.hadoop.yarn.server.api.ApplicationContext; import org.apache.hadoop.yarn.server.webapp.dao.AppInfo; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; @@ -42,12 +45,13 @@ public class AppsBlock extends HtmlBlock { - protected ApplicationContext appContext; + private static final Log LOG = LogFactory.getLog(AppsBlock.class); + protected ApplicationBaseProtocol appBaseProt; @Inject - AppsBlock(ApplicationContext appContext, ViewContext ctx) { + AppsBlock(ApplicationBaseProtocol appBaseProt, ViewContext ctx) { super(ctx); - this.appContext = appContext; + this.appBaseProt = appBaseProt; } @Override @@ -61,27 +65,29 @@ public void render(Block html) { .th(".finishtime", "FinishTime").th(".state", "State") .th(".finalstatus", "FinalStatus").th(".progress", "Progress") .th(".ui", "Tracking UI")._()._().tbody(); - Collection reqAppStates = null; + EnumSet reqAppStates = + EnumSet.noneOf(YarnApplicationState.class); String reqStateString = $(APP_STATE); if (reqStateString != null && !reqStateString.isEmpty()) { String[] appStateStrings = reqStateString.split(","); - reqAppStates = new HashSet(appStateStrings.length); for (String stateString : appStateStrings) { - reqAppStates.add(YarnApplicationState.valueOf(stateString)); + reqAppStates.add(YarnApplicationState.valueOf(stateString.trim())); } } UserGroupInformation callerUGI = getCallerUGI(); - Collection appReports; + Collection appReports = null; try { + final GetApplicationsRequest request = + GetApplicationsRequest.newInstance(reqAppStates); if (callerUGI == null) { - appReports = appContext.getAllApplications().values(); + appReports = appBaseProt.getApplications(request).getApplicationList(); } else { appReports = callerUGI.doAs( new PrivilegedExceptionAction> () { @Override public Collection run() throws Exception { - return appContext.getAllApplications().values(); + return appBaseProt.getApplications(request).getApplicationList(); } }); } @@ -93,12 +99,15 @@ public Collection run() throws Exception { } StringBuilder appsTableData = new StringBuilder("[\n"); for (ApplicationReport appReport : appReports) { - if (reqAppStates != null + // TODO: remove the following condition. It is still here because + // the history side implementation of ApplicationBaseProtocol + // hasn't filtering capability (YARN-1819). + if (!reqAppStates.isEmpty() && !reqAppStates.contains(appReport.getYarnApplicationState())) { continue; } AppInfo app = new AppInfo(appReport); - String percent = String.format("%.1f", app.getProgress() * 100.0F); + String percent = String.format("%.1f", app.getProgress()); // AppID numerical value parsed by parseHadoopID in yarn.dt.plugins.js appsTableData .append("[\" ").append("
    ").append("\",\"     ").append("\",\"").append("History") - .append("\"],\n"); + String trackingUI = + app.getTrackingUrl() == null || app.getTrackingUrl() == UNAVAILABLE + ? "Unassigned" + : app.getAppState() == YarnApplicationState.FINISHED + || app.getAppState() == YarnApplicationState.FAILED + || app.getAppState() == YarnApplicationState.KILLED + ? "History" : "ApplicationMaster"; + appsTableData.append(trackingURL == null ? "#" : "href='" + trackingURL) + .append("'>").append(trackingUI).append("\"],\n"); } if (appsTableData.charAt(appsTableData.length() - 2) == ',') { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ContainerBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ContainerBlock.java index 2bb48a8b1cc9d..cae8d2e6fb509 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ContainerBlock.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ContainerBlock.java @@ -26,9 +26,10 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest; import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerReport; -import org.apache.hadoop.yarn.server.api.ApplicationContext; import org.apache.hadoop.yarn.server.webapp.dao.ContainerInfo; import org.apache.hadoop.yarn.util.ConverterUtils; import org.apache.hadoop.yarn.util.Times; @@ -40,12 +41,12 @@ public class ContainerBlock extends HtmlBlock { private static final Log LOG = LogFactory.getLog(ContainerBlock.class); - private final ApplicationContext appContext; + protected ApplicationBaseProtocol appBaseProt; @Inject - public ContainerBlock(ApplicationContext appContext, ViewContext ctx) { + public ContainerBlock(ApplicationBaseProtocol appBaseProt, ViewContext ctx) { super(ctx); - this.appContext = appContext; + this.appBaseProt = appBaseProt; } @Override @@ -64,18 +65,21 @@ protected void render(Block html) { return; } - final ContainerId containerIdFinal = containerId; UserGroupInformation callerUGI = getCallerUGI(); - ContainerReport containerReport; + ContainerReport containerReport = null; try { + final GetContainerReportRequest request = + GetContainerReportRequest.newInstance(containerId); if (callerUGI == null) { - containerReport = appContext.getContainer(containerId); + containerReport = appBaseProt.getContainerReport(request) + .getContainerReport(); } else { containerReport = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ContainerReport run() throws Exception { - return appContext.getContainer(containerIdFinal); + return appBaseProt.getContainerReport(request) + .getContainerReport(); } }); } @@ -85,6 +89,7 @@ public ContainerReport run() throws Exception { html.p()._(message)._(); return; } + if (containerReport == null) { puts("Container not found: " + containerid); return; @@ -94,9 +99,17 @@ public ContainerReport run() throws Exception { setTitle(join("Container ", containerid)); info("Container Overview") - ._("State:", container.getContainerState()) + ._( + "Container State:", + container.getContainerState() == null ? UNAVAILABLE : container + .getContainerState()) ._("Exit Status:", container.getContainerExitStatus()) - ._("Node:", container.getAssignedNodeId()) + ._( + "Node:", + container.getNodeHttpAddress() == null ? "#" : container + .getNodeHttpAddress(), + container.getNodeHttpAddress() == null ? "N/A" : container + .getNodeHttpAddress()) ._("Priority:", container.getPriority()) ._("Started:", Times.format(container.getStartedTime())) ._( @@ -109,7 +122,8 @@ public ContainerReport run() throws Exception { + container.getAllocatedVCores() + " VCores") ._("Logs:", container.getLogUrl() == null ? "#" : container.getLogUrl(), container.getLogUrl() == null ? "N/A" : "Logs") - ._("Diagnostics:", container.getDiagnosticsInfo()); + ._("Diagnostics:", container.getDiagnosticsInfo() == null ? + "" : container.getDiagnosticsInfo()); html._(InfoBlock.class); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java new file mode 100644 index 0000000000000..5acabf5f3e4f3 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebPageUtils.java @@ -0,0 +1,86 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.webapp; + +import static org.apache.hadoop.yarn.webapp.view.JQueryUI.tableInit; + + +public class WebPageUtils { + + public static String appsTableInit() { + return appsTableInit(false); + } + + public static String appsTableInit(boolean isFairSchedulerPage) { + // id, user, name, queue, starttime, finishtime, state, status, progress, ui + // FairSchedulerPage's table is a bit different + return tableInit() + .append(", 'aaData': appsTableData") + .append(", bDeferRender: true") + .append(", bProcessing: true") + .append("\n, aoColumnDefs: ") + .append(getAppsTableColumnDefs(isFairSchedulerPage)) + // Sort by id upon page load + .append(", aaSorting: [[0, 'desc']]}").toString(); + } + + private static String getAppsTableColumnDefs(boolean isFairSchedulerPage) { + StringBuilder sb = new StringBuilder(); + return sb + .append("[\n") + .append("{'sType':'string', 'aTargets': [0]") + .append(", 'mRender': parseHadoopID }") + .append("\n, {'sType':'numeric', 'aTargets': " + + (isFairSchedulerPage ? "[6, 7]": "[5, 6]")) + .append(", 'mRender': renderHadoopDate }") + .append("\n, {'sType':'numeric', bSearchable:false, 'aTargets': [9]") + .append(", 'mRender': parseHadoopProgress }]").toString(); + } + + public static String attemptsTableInit() { + return tableInit().append(", 'aaData': attemptsTableData") + .append(", bDeferRender: true").append(", bProcessing: true") + .append("\n, aoColumnDefs: ").append(getAttemptsTableColumnDefs()) + // Sort by id upon page load + .append(", aaSorting: [[0, 'desc']]}").toString(); + } + + private static String getAttemptsTableColumnDefs() { + StringBuilder sb = new StringBuilder(); + return sb.append("[\n").append("{'sType':'string', 'aTargets': [0]") + .append(", 'mRender': parseHadoopID }") + .append("\n, {'sType':'numeric', 'aTargets': [1]") + .append(", 'mRender': renderHadoopDate }]").toString(); + } + + public static String containersTableInit() { + return tableInit().append(", 'aaData': containersTableData") + .append(", bDeferRender: true").append(", bProcessing: true") + .append("\n, aoColumnDefs: ").append(getContainersTableColumnDefs()) + // Sort by id upon page load + .append(", aaSorting: [[0, 'desc']]}").toString(); + } + + private static String getContainersTableColumnDefs() { + StringBuilder sb = new StringBuilder(); + return sb.append("[\n").append("{'sType':'string', 'aTargets': [0]") + .append(", 'mRender': parseHadoopID }]").toString(); + } + +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java index 6d94737f90ed6..909bf1d022cb5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServices.java @@ -40,7 +40,13 @@ import org.apache.hadoop.yarn.api.records.ContainerReport; import org.apache.hadoop.yarn.api.records.FinalApplicationStatus; import org.apache.hadoop.yarn.api.records.YarnApplicationState; -import org.apache.hadoop.yarn.server.api.ApplicationContext; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationAttemptsRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetContainersRequest; import org.apache.hadoop.yarn.server.webapp.dao.AppAttemptInfo; import org.apache.hadoop.yarn.server.webapp.dao.AppAttemptsInfo; import org.apache.hadoop.yarn.server.webapp.dao.AppInfo; @@ -54,10 +60,10 @@ public class WebServices { - protected ApplicationContext appContext; + protected ApplicationBaseProtocol appBaseProt; - public WebServices(ApplicationContext appContext) { - this.appContext = appContext; + public WebServices(ApplicationBaseProtocol appBaseProt) { + this.appBaseProt = appBaseProt; } public AppsInfo getApps(HttpServletRequest req, HttpServletResponse res, @@ -144,13 +150,17 @@ public AppsInfo getApps(HttpServletRequest req, HttpServletResponse res, Collection appReports = null; try { if (callerUGI == null) { - appReports = appContext.getAllApplications().values(); + // TODO: the request should take the params like what RMWebServices does + // in YARN-1819. + GetApplicationsRequest request = GetApplicationsRequest.newInstance(); + appReports = appBaseProt.getApplications(request).getApplicationList(); } else { appReports = callerUGI.doAs( new PrivilegedExceptionAction> () { @Override public Collection run() throws Exception { - return appContext.getAllApplications().values(); + return appBaseProt.getApplications( + GetApplicationsRequest.newInstance()).getApplicationList(); } }); } @@ -214,13 +224,17 @@ public AppInfo getApp(HttpServletRequest req, HttpServletResponse res, ApplicationReport app = null; try { if (callerUGI == null) { - app = appContext.getApplication(id); + GetApplicationReportRequest request = + GetApplicationReportRequest.newInstance(id); + app = appBaseProt.getApplicationReport(request).getApplicationReport(); } else { app = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ApplicationReport run() throws Exception { - return appContext.getApplication(id); + GetApplicationReportRequest request = + GetApplicationReportRequest.newInstance(id); + return appBaseProt.getApplicationReport(request).getApplicationReport(); } }); } @@ -240,13 +254,20 @@ public AppAttemptsInfo getAppAttempts(HttpServletRequest req, Collection appAttemptReports = null; try { if (callerUGI == null) { - appAttemptReports = appContext.getApplicationAttempts(id).values(); + GetApplicationAttemptsRequest request = + GetApplicationAttemptsRequest.newInstance(id); + appAttemptReports = + appBaseProt.getApplicationAttempts(request) + .getApplicationAttemptList(); } else { appAttemptReports = callerUGI.doAs( new PrivilegedExceptionAction> () { @Override public Collection run() throws Exception { - return appContext.getApplicationAttempts(id).values(); + GetApplicationAttemptsRequest request = + GetApplicationAttemptsRequest.newInstance(id); + return appBaseProt.getApplicationAttempts(request) + .getApplicationAttemptList(); } }); } @@ -271,13 +292,20 @@ public AppAttemptInfo getAppAttempt(HttpServletRequest req, ApplicationAttemptReport appAttempt = null; try { if (callerUGI == null) { - appAttempt = appContext.getApplicationAttempt(aaid); + GetApplicationAttemptReportRequest request = + GetApplicationAttemptReportRequest.newInstance(aaid); + appAttempt = + appBaseProt.getApplicationAttemptReport(request) + .getApplicationAttemptReport(); } else { appAttempt = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ApplicationAttemptReport run() throws Exception { - return appContext.getApplicationAttempt(aaid); + GetApplicationAttemptReportRequest request = + GetApplicationAttemptReportRequest.newInstance(aaid); + return appBaseProt.getApplicationAttemptReport(request) + .getApplicationAttemptReport(); } }); } @@ -300,13 +328,16 @@ public ContainersInfo getContainers(HttpServletRequest req, Collection containerReports = null; try { if (callerUGI == null) { - containerReports = appContext.getContainers(aaid).values(); + GetContainersRequest request = GetContainersRequest.newInstance(aaid); + containerReports = + appBaseProt.getContainers(request).getContainerList(); } else { containerReports = callerUGI.doAs( new PrivilegedExceptionAction> () { @Override public Collection run() throws Exception { - return appContext.getContainers(aaid).values(); + GetContainersRequest request = GetContainersRequest.newInstance(aaid); + return appBaseProt.getContainers(request).getContainerList(); } }); } @@ -332,13 +363,18 @@ public ContainerInfo getContainer(HttpServletRequest req, ContainerReport container = null; try { if (callerUGI == null) { - container = appContext.getContainer(cid); + GetContainerReportRequest request = + GetContainerReportRequest.newInstance(cid); + container = + appBaseProt.getContainerReport(request).getContainerReport(); } else { container = callerUGI.doAs( new PrivilegedExceptionAction () { @Override public ContainerReport run() throws Exception { - return appContext.getContainer(cid); + GetContainerReportRequest request = + GetContainerReportRequest.newInstance(cid); + return appBaseProt.getContainerReport(request).getContainerReport(); } }); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/AppInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/AppInfo.java index d78f9287ffb61..e8b1acce49809 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/AppInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/AppInfo.java @@ -18,6 +18,8 @@ package org.apache.hadoop.yarn.server.webapp.dao; +import static org.apache.hadoop.yarn.util.StringHelper.CSV_JOINER; + import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; import javax.xml.bind.annotation.XmlRootElement; @@ -49,6 +51,7 @@ public class AppInfo { protected long startedTime; protected long finishedTime; protected long elapsedTime; + protected String applicationTags; public AppInfo() { // JAXB needs this @@ -74,7 +77,10 @@ public AppInfo(ApplicationReport app) { finishedTime = app.getFinishTime(); elapsedTime = Times.elapsed(startedTime, finishedTime); finalAppStatus = app.getFinalApplicationStatus(); - progress = app.getProgress(); + progress = app.getProgress() * 100; // in percent + if (app.getApplicationTags() != null && !app.getApplicationTags().isEmpty()) { + this.applicationTags = CSV_JOINER.join(app.getApplicationTags()); + } } public String getAppId() { @@ -149,4 +155,7 @@ public long getElapsedTime() { return elapsedTime; } + public String getApplicationTags() { + return applicationTags; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/ContainerInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/ContainerInfo.java index bdcc7b2e30ec8..0d18e7af47526 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/ContainerInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/ContainerInfo.java @@ -42,6 +42,7 @@ public class ContainerInfo { protected String logUrl; protected int containerExitStatus; protected ContainerState containerState; + protected String nodeHttpAddress; public ContainerInfo() { // JAXB needs this @@ -64,6 +65,7 @@ public ContainerInfo(ContainerReport container) { logUrl = container.getLogUrl(); containerExitStatus = container.getContainerExitStatus(); containerState = container.getContainerState(); + nodeHttpAddress = container.getNodeHttpAddress(); } public String getContainerId() { @@ -114,4 +116,7 @@ public ContainerState getContainerState() { return containerState; } + public String getNodeHttpAddress() { + return nodeHttpAddress; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto index 91473c5a2dece..d8c92c4aaffbc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto @@ -32,6 +32,7 @@ message RegisterNodeManagerRequestProto { optional string nm_version = 5; repeated NMContainerStatusProto container_statuses = 6; repeated ApplicationIdProto runningApplications = 7; + optional StringArrayProto nodeLabels = 8; } message RegisterNodeManagerResponseProto { @@ -41,12 +42,14 @@ message RegisterNodeManagerResponseProto { optional int64 rm_identifier = 4; optional string diagnostics_message = 5; optional string rm_version = 6; + optional bool areNodeLabelsAcceptedByRM = 7 [default = false]; } message NodeHeartbeatRequestProto { optional NodeStatusProto node_status = 1; optional MasterKeyProto last_known_container_token_master_key = 2; optional MasterKeyProto last_known_nm_token_master_key = 3; + optional StringArrayProto nodeLabels = 4; } message NodeHeartbeatResponseProto { @@ -60,6 +63,7 @@ message NodeHeartbeatResponseProto { optional string diagnostics_message = 8; repeated ContainerIdProto containers_to_be_removed_from_nm = 9; repeated SystemCredentialsForAppsProto system_credentials_for_apps = 10; + optional bool areNodeLabelsAcceptedByRM = 11 [default = false]; } message SystemCredentialsForAppsProto { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java index 20983b6109ffb..d42b2c7d0c3a9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java @@ -19,11 +19,13 @@ package org.apache.hadoop.yarn; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; import java.nio.ByteBuffer; import java.util.ArrayList; import java.util.Arrays; +import java.util.HashSet; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; @@ -36,6 +38,7 @@ import org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl; import org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; +import org.apache.hadoop.yarn.server.api.protocolrecords.RegisterNodeManagerRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatResponsePBImpl; import org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RegisterNodeManagerRequestPBImpl; @@ -46,6 +49,7 @@ import org.apache.hadoop.yarn.server.api.records.NodeStatus; import org.apache.hadoop.yarn.server.api.records.impl.pb.MasterKeyPBImpl; import org.apache.hadoop.yarn.server.api.records.impl.pb.NodeStatusPBImpl; +import org.junit.Assert; import org.junit.Test; /** @@ -77,7 +81,17 @@ public void testRegisterNodeManagerResponsePBImpl() { assertEquals(1, copy.getNMTokenMasterKey().getKeyId()); assertEquals(NodeAction.NORMAL, copy.getNodeAction()); assertEquals("testDiagnosticMessage", copy.getDiagnosticsMessage()); + assertFalse(copy.getAreNodeLabelsAcceptedByRM()); + } + @Test + public void testRegisterNodeManagerResponsePBImplWithRMAcceptLbls() { + RegisterNodeManagerResponsePBImpl original = + new RegisterNodeManagerResponsePBImpl(); + original.setAreNodeLabelsAcceptedByRM(true); + RegisterNodeManagerResponsePBImpl copy = + new RegisterNodeManagerResponsePBImpl(original.getProto()); + assertTrue(copy.getAreNodeLabelsAcceptedByRM()); } /** @@ -89,11 +103,32 @@ public void testNodeHeartbeatRequestPBImpl() { original.setLastKnownContainerTokenMasterKey(getMasterKey()); original.setLastKnownNMTokenMasterKey(getMasterKey()); original.setNodeStatus(getNodeStatus()); + original.setNodeLabels(getValidNodeLabels()); NodeHeartbeatRequestPBImpl copy = new NodeHeartbeatRequestPBImpl( original.getProto()); assertEquals(1, copy.getLastKnownContainerTokenMasterKey().getKeyId()); assertEquals(1, copy.getLastKnownNMTokenMasterKey().getKeyId()); assertEquals("localhost", copy.getNodeStatus().getNodeId().getHost()); + // check labels are coming with valid values + Assert.assertTrue(original.getNodeLabels() + .containsAll(copy.getNodeLabels())); + // check for empty labels + original.setNodeLabels(new HashSet ()); + copy = new NodeHeartbeatRequestPBImpl( + original.getProto()); + Assert.assertNotNull(copy.getNodeLabels()); + Assert.assertEquals(0, copy.getNodeLabels().size()); + } + + /** + * Test NodeHeartbeatRequestPBImpl. + */ + @Test + public void testNodeHeartbeatRequestPBImplWithNullLabels() { + NodeHeartbeatRequestPBImpl original = new NodeHeartbeatRequestPBImpl(); + NodeHeartbeatRequestPBImpl copy = + new NodeHeartbeatRequestPBImpl(original.getProto()); + Assert.assertNull(copy.getNodeLabels()); } /** @@ -119,6 +154,16 @@ public void testNodeHeartbeatResponsePBImpl() { assertEquals(1, copy.getContainerTokenMasterKey().getKeyId()); assertEquals(1, copy.getNMTokenMasterKey().getKeyId()); assertEquals("testDiagnosticMessage", copy.getDiagnosticsMessage()); + assertEquals(false, copy.getAreNodeLabelsAcceptedByRM()); + } + + @Test + public void testNodeHeartbeatResponsePBImplWithRMAcceptLbls() { + NodeHeartbeatResponsePBImpl original = new NodeHeartbeatResponsePBImpl(); + original.setAreNodeLabelsAcceptedByRM(true); + NodeHeartbeatResponsePBImpl copy = + new NodeHeartbeatResponsePBImpl(original.getProto()); + assertTrue(copy.getAreNodeLabelsAcceptedByRM()); } /** @@ -208,6 +253,55 @@ public void testNodeStatusPBImpl() { } + @Test + public void testRegisterNodeManagerRequestWithNullLabels() { + RegisterNodeManagerRequest request = + RegisterNodeManagerRequest.newInstance( + NodeId.newInstance("host", 1234), 1234, Resource.newInstance(0, 0), + "version", null, null); + + // serialze to proto, and get request from proto + RegisterNodeManagerRequest request1 = + new RegisterNodeManagerRequestPBImpl( + ((RegisterNodeManagerRequestPBImpl) request).getProto()); + + // check labels are coming with no values + Assert.assertNull(request1.getNodeLabels()); + } + + @Test + public void testRegisterNodeManagerRequestWithValidLabels() { + HashSet nodeLabels = getValidNodeLabels(); + RegisterNodeManagerRequest request = + RegisterNodeManagerRequest.newInstance( + NodeId.newInstance("host", 1234), 1234, Resource.newInstance(0, 0), + "version", null, null, nodeLabels); + + // serialze to proto, and get request from proto + RegisterNodeManagerRequest copy = + new RegisterNodeManagerRequestPBImpl( + ((RegisterNodeManagerRequestPBImpl) request).getProto()); + + // check labels are coming with valid values + Assert.assertEquals(true, nodeLabels.containsAll(copy.getNodeLabels())); + + // check for empty labels + request.setNodeLabels(new HashSet ()); + copy = new RegisterNodeManagerRequestPBImpl( + ((RegisterNodeManagerRequestPBImpl) request).getProto()); + Assert.assertNotNull(copy.getNodeLabels()); + Assert.assertEquals(0, copy.getNodeLabels().size()); + } + + private HashSet getValidNodeLabels() { + HashSet nodeLabels = new HashSet(); + nodeLabels.add("java"); + nodeLabels.add("windows"); + nodeLabels.add("gpu"); + nodeLabels.add("x86"); + return nodeLabels; + } + private ContainerStatus getContainerStatus(int applicationId, int containerID, int appAttemptId) { ContainerStatus status = recordFactory diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java index 77193df328096..1c670a16ae9de 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java @@ -41,6 +41,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerDiagnosticsUpdateEvent; @@ -101,13 +102,15 @@ public Path localizeClasspathJar(Path classPathJar, Path pwd, String owner) /** * Prepare the environment for containers in this application to execute. + * 
        * For $x in local.dirs
    *   create $x/$user/$appId
-   * Copy $nmLocal/appTokens -> $N/$user/$appId
+   * Copy $nmLocal/appTokens {@literal ->} $N/$user/$appId
    * For $rsrc in private resources
-   *   Copy $rsrc -> $N/$user/filecache/[idef]
+   *   Copy $rsrc {@literal ->} $N/$user/filecache/[idef]
    * For $rsrc in job resources
-   *   Copy $rsrc -> $N/$user/$appId/filecache/idef
+   *   Copy $rsrc {@literal ->} $N/$user/$appId/filecache/idef
+   *
    * @param user user name of application owner * @param appId id of the application * @param nmPrivateContainerTokens path to localized credentials, rsrc by NM @@ -207,8 +210,22 @@ public int reacquireContainer(String user, ContainerId containerId) } } - public void writeLaunchEnv(OutputStream out, Map environment, Map> resources, List command) throws IOException{ - ContainerLaunch.ShellScriptBuilder sb = ContainerLaunch.ShellScriptBuilder.create(); + /** + * This method writes out the launch environment of a container. This can be + * overridden by extending ContainerExecutors to provide different behaviors + * @param out the output stream to which the environment is written (usually + * a script file which will be executed by the Launcher) + * @param environment The environment variables and their values + * @param resources The resources which have been localized for this container + * Symlinks will be created to these localized resources + * @param command The command that will be run. + * @throws IOException if any errors happened writing to the OutputStream, + * while creating symlinks + */ + public void writeLaunchEnv(OutputStream out, Map environment, + Map> resources, List command) throws IOException{ + ContainerLaunch.ShellScriptBuilder sb = + ContainerLaunch.ShellScriptBuilder.create(); if (environment != null) { for (Map.Entry env : environment.entrySet()) { sb.env(env.getKey().toString(), env.getValue().toString()); @@ -298,6 +315,11 @@ protected Path getPidFilePath(ContainerId containerId) { readLock.unlock(); } } + + protected String[] getRunCommand(String command, String groupId, + String userName, Path pidFile, Configuration conf) { + return getRunCommand(command, groupId, userName, pidFile, conf, null); + } /** * Return a command to execute the given command in OS shell. @@ -306,7 +328,7 @@ protected Path getPidFilePath(ContainerId containerId) { * non-Windows, groupId is ignored. */ protected String[] getRunCommand(String command, String groupId, - String userName, Path pidFile, Configuration conf) { + String userName, Path pidFile, Configuration conf, Resource resource) { boolean containerSchedPriorityIsSet = false; int containerSchedPriorityAdjustment = YarnConfiguration.DEFAULT_NM_CONTAINER_EXECUTOR_SCHED_PRIORITY; @@ -320,7 +342,46 @@ protected String[] getRunCommand(String command, String groupId, } if (Shell.WINDOWS) { - return new String[] { Shell.WINUTILS, "task", "create", groupId, + int cpuRate = -1; + int memory = -1; + if (resource != null) { + if (conf + .getBoolean( + YarnConfiguration.NM_WINDOWS_CONTAINER_MEMORY_LIMIT_ENABLED, + YarnConfiguration.DEFAULT_NM_WINDOWS_CONTAINER_MEMORY_LIMIT_ENABLED)) { + memory = resource.getMemory(); + } + + if (conf.getBoolean( + YarnConfiguration.NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED, + YarnConfiguration.DEFAULT_NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED)) { + int containerVCores = resource.getVirtualCores(); + int nodeVCores = conf.getInt(YarnConfiguration.NM_VCORES, + YarnConfiguration.DEFAULT_NM_VCORES); + // cap overall usage to the number of cores allocated to YARN + int nodeCpuPercentage = Math + .min( + conf.getInt( + YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT, + YarnConfiguration.DEFAULT_NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT), + 100); + nodeCpuPercentage = Math.max(0, nodeCpuPercentage); + if (nodeCpuPercentage == 0) { + String message = "Illegal value for " + + YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT + + ". Value cannot be less than or equal to 0."; + throw new IllegalArgumentException(message); + } + float yarnVCores = (nodeCpuPercentage * nodeVCores) / 100.0f; + // CPU should be set to a percentage * 100, e.g. 20% cpu rate limit + // should be set as 20 * 100. The following setting is equal to: + // 100 * (100 * (vcores / Total # of cores allocated to YARN)) + cpuRate = Math.min(10000, + (int) ((containerVCores * 10000) / yarnVCores)); + } + } + return new String[] { Shell.WINUTILS, "task", "create", "-m", + String.valueOf(memory), "-c", String.valueOf(cpuRate), groupId, "cmd /c " + command }; } else { List retCommand = new ArrayList(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java index f3d21210b8552..e0ecea322eb83 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java @@ -48,6 +48,7 @@ import org.apache.hadoop.util.Shell.ShellCommandExecutor; import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; @@ -202,7 +203,7 @@ public int launchContainer(Container container, setScriptExecutable(sb.getWrapperScriptPath(), user); shExec = buildCommandExecutor(sb.getWrapperScriptPath().toString(), - containerIdStr, user, pidFile, + containerIdStr, user, pidFile, container.getResource(), new File(containerWorkDir.toUri().getPath()), container.getLaunchContext().getEnvironment()); @@ -256,12 +257,12 @@ public int launchContainer(Container container, } protected CommandExecutor buildCommandExecutor(String wrapperScriptPath, - String containerIdStr, String user, Path pidFile, File wordDir, - Map environment) + String containerIdStr, String user, Path pidFile, Resource resource, + File wordDir, Map environment) throws IOException { String[] command = getRunCommand(wrapperScriptPath, - containerIdStr, user, pidFile, this.getConf()); + containerIdStr, user, pidFile, this.getConf(), resource); LOG.info("launchContainer: " + Arrays.toString(command)); return new ShellCommandExecutor( diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java index c854173fad36e..71eaa04b3c6c1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java @@ -18,10 +18,24 @@ package org.apache.hadoop.yarn.server.nodemanager; -import com.google.common.annotations.VisibleForTesting; -import com.google.common.base.Joiner; -import com.google.common.base.Preconditions; -import com.google.common.base.Strings; +import static org.apache.hadoop.fs.CreateFlag.CREATE; +import static org.apache.hadoop.fs.CreateFlag.OVERWRITE; + +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; +import java.io.File; +import java.io.IOException; +import java.io.OutputStream; +import java.io.PrintStream; +import java.net.InetSocketAddress; +import java.util.ArrayList; +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; import org.apache.commons.lang.math.RandomUtils; import org.apache.commons.logging.Log; @@ -45,38 +59,35 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer; import org.apache.hadoop.yarn.util.ConverterUtils; -import java.io.ByteArrayOutputStream; -import java.io.DataOutputStream; -import java.io.File; -import java.io.IOException; -import java.io.OutputStream; -import java.io.PrintStream; -import java.util.ArrayList; -import java.util.Collections; -import java.util.EnumSet; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Random; -import java.util.Set; -import java.util.regex.Pattern; -import java.net.InetSocketAddress; -import static org.apache.hadoop.fs.CreateFlag.CREATE; -import static org.apache.hadoop.fs.CreateFlag.OVERWRITE; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Joiner; +import com.google.common.base.Preconditions; +import com.google.common.base.Strings; /** - * This executor will launch a docker container and run the task inside the container. + * This executor will launch and run tasks inside Docker containers. It + * currently only supports simple authentication mode. It shares a lot of code + * with the DefaultContainerExecutor (and it may make sense to pull out those + * common pieces later). */ public class DockerContainerExecutor extends ContainerExecutor { - private static final Log LOG = LogFactory - .getLog(DockerContainerExecutor.class); - public static final String DOCKER_CONTAINER_EXECUTOR_SCRIPT = "docker_container_executor"; - public static final String DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT = "docker_container_executor_session"; - - // This validates that the image is a proper docker image and would not crash docker. - public static final String DOCKER_IMAGE_PATTERN = "^(([\\w\\.-]+)(:\\d+)*\\/)?[\\w\\.:-]+$"; - + .getLog(DockerContainerExecutor.class); + //The name of the script file that will launch the Docker containers + public static final String DOCKER_CONTAINER_EXECUTOR_SCRIPT = + "docker_container_executor"; + //The name of the session script that the DOCKER_CONTAINER_EXECUTOR_SCRIPT + //launches in turn + public static final String DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT = + "docker_container_executor_session"; + + //This validates that the image is a proper docker image and would not crash + //docker. The image name is not allowed to contain spaces. e.g. + //registry.somecompany.com:9999/containername:0.1 or + //containername:0.1 or + //containername + public static final String DOCKER_IMAGE_PATTERN = + "^(([\\w\\.-]+)(:\\d+)*\\/)?[\\w\\.:-]+$"; private final FileContext lfs; private final Pattern dockerImagePattern; @@ -96,23 +107,26 @@ protected void copyFile(Path src, Path dst, String owner) throws IOException { @Override public void init() throws IOException { - String auth = getConf().get(CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION); + String auth = + getConf().get(CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION); if (auth != null && !auth.equals("simple")) { - throw new IllegalStateException("DockerContainerExecutor only works with simple authentication mode"); + throw new IllegalStateException( + "DockerContainerExecutor only works with simple authentication mode"); } - String dockerExecutor = getConf().get(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, + String dockerExecutor = getConf().get( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, YarnConfiguration.NM_DEFAULT_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME); if (!new File(dockerExecutor).exists()) { - throw new IllegalStateException("Invalid docker exec path: " + dockerExecutor); + throw new IllegalStateException( + "Invalid docker exec path: " + dockerExecutor); } } @Override public synchronized void startLocalizer(Path nmPrivateContainerTokensPath, - InetSocketAddress nmAddr, String user, String appId, String locId, - LocalDirsHandlerService dirsHandler) + InetSocketAddress nmAddr, String user, String appId, String locId, + LocalDirsHandlerService dirsHandler) throws IOException, InterruptedException { - List localDirs = dirsHandler.getLocalDirs(); List logDirs = dirsHandler.getLogDirs(); @@ -128,7 +142,8 @@ public synchronized void startLocalizer(Path nmPrivateContainerTokensPath, // randomly choose the local directory Path appStorageDir = getWorkingDir(localDirs, user, appId); - String tokenFn = String.format(ContainerLocalizer.TOKEN_FILE_NAME_FMT, locId); + String tokenFn = + String.format(ContainerLocalizer.TOKEN_FILE_NAME_FMT, locId); Path tokenDst = new Path(appStorageDir, tokenFn); copyFile(nmPrivateContainerTokensPath, tokenDst, user); LOG.info("Copying from " + nmPrivateContainerTokensPath + " to " + tokenDst); @@ -140,31 +155,34 @@ public synchronized void startLocalizer(Path nmPrivateContainerTokensPath, @Override - public int launchContainer(Container container, - Path nmPrivateContainerScriptPath, Path nmPrivateTokensPath, - String userName, String appId, Path containerWorkDir, - List localDirs, List logDirs) throws IOException { + public int launchContainer(Container container, Path + nmPrivateContainerScriptPath, Path nmPrivateTokensPath, String userName, + String appId, Path containerWorkDir, List localDirs, List + logDirs) throws IOException { + //Variables for the launch environment can be injected from the command-line + //while submitting the application String containerImageName = container.getLaunchContext().getEnvironment() - .get(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME); + .get(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME); if (LOG.isDebugEnabled()) { LOG.debug("containerImageName from launchContext: " + containerImageName); } - Preconditions.checkArgument(!Strings.isNullOrEmpty(containerImageName), "Container image must not be null"); + Preconditions.checkArgument(!Strings.isNullOrEmpty(containerImageName), + "Container image must not be null"); containerImageName = containerImageName.replaceAll("['\"]", ""); - Preconditions.checkArgument(saneDockerImage(containerImageName), "Image: " + containerImageName + " is not a proper docker image"); - String dockerExecutor = getConf().get(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, - YarnConfiguration.NM_DEFAULT_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME); + Preconditions.checkArgument(saneDockerImage(containerImageName), "Image: " + + containerImageName + " is not a proper docker image"); + String dockerExecutor = getConf().get( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, + YarnConfiguration.NM_DEFAULT_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME); FsPermission dirPerm = new FsPermission(APPDIR_PERM); ContainerId containerId = container.getContainerId(); // create container dirs on all disks String containerIdStr = ConverterUtils.toString(containerId); - String appIdStr = - ConverterUtils.toString( - containerId.getApplicationAttemptId(). - getApplicationId()); + String appIdStr = ConverterUtils.toString( + containerId.getApplicationAttemptId().getApplicationId()); for (String sLocalDir : localDirs) { Path usersdir = new Path(sLocalDir, ContainerLocalizer.USERCACHE); Path userdir = new Path(usersdir, userName); @@ -178,46 +196,57 @@ public int launchContainer(Container container, createContainerLogDirs(appIdStr, containerIdStr, logDirs, userName); Path tmpDir = new Path(containerWorkDir, - YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR); + YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR); createDir(tmpDir, dirPerm, false, userName); // copy launch script to work dir Path launchDst = - new Path(containerWorkDir, ContainerLaunch.CONTAINER_SCRIPT); + new Path(containerWorkDir, ContainerLaunch.CONTAINER_SCRIPT); lfs.util().copy(nmPrivateContainerScriptPath, launchDst); // copy container tokens to work dir Path tokenDst = - new Path(containerWorkDir, ContainerLaunch.FINAL_CONTAINER_TOKENS_FILE); + new Path(containerWorkDir, ContainerLaunch.FINAL_CONTAINER_TOKENS_FILE); lfs.util().copy(nmPrivateTokensPath, tokenDst); - - String localDirMount = toMount(localDirs); String logDirMount = toMount(logDirs); - String containerWorkDirMount = toMount(Collections.singletonList(containerWorkDir.toUri().getPath())); + String containerWorkDirMount = toMount(Collections.singletonList( + containerWorkDir.toUri().getPath())); StringBuilder commands = new StringBuilder(); + //Use docker run to launch the docker container. See man pages for + //docker-run + //--rm removes the container automatically once the container finishes + //--net=host allows the container to take on the host's network stack + //--name sets the Docker Container name to the YARN containerId string + //-v is used to bind mount volumes for local, log and work dirs. String commandStr = commands.append(dockerExecutor) - .append(" ") - .append("run") - .append(" ") - .append("--rm --net=host") - .append(" ") - .append(" --name " + containerIdStr) - .append(localDirMount) - .append(logDirMount) - .append(containerWorkDirMount) - .append(" ") - .append(containerImageName) - .toString(); - String dockerPidScript = "`" + dockerExecutor + " inspect --format {{.State.Pid}} " + containerIdStr + "`"; + .append(" ") + .append("run") + .append(" ") + .append("--rm --net=host") + .append(" ") + .append(" --name " + containerIdStr) + .append(localDirMount) + .append(logDirMount) + .append(containerWorkDirMount) + .append(" ") + .append(containerImageName) + .toString(); + //Get the pid of the process which has been launched as a docker container + //using docker inspect + String dockerPidScript = "`" + dockerExecutor + + " inspect --format {{.State.Pid}} " + containerIdStr + "`"; + // Create new local launch wrapper script - LocalWrapperScriptBuilder sb = - new UnixLocalWrapperScriptBuilder(containerWorkDir, commandStr, dockerPidScript); + LocalWrapperScriptBuilder sb = new UnixLocalWrapperScriptBuilder( + containerWorkDir, commandStr, dockerPidScript); Path pidFile = getPidFilePath(containerId); if (pidFile != null) { sb.writeLocalWrapperScript(launchDst, pidFile); } else { + //Although the container was activated by ContainerLaunch before exec() + //was called, since then deactivateContainer() has been called. LOG.info("Container " + containerIdStr + " was marked as inactive. Returning terminated error"); return ExitCode.TERMINATED.getExitCode(); @@ -234,12 +263,13 @@ public int launchContainer(Container container, String[] command = getRunCommand(sb.getWrapperScriptPath().toString(), containerIdStr, userName, pidFile, this.getConf()); if (LOG.isDebugEnabled()) { - LOG.debug("launchContainer: " + commandStr + " " + Joiner.on(" ").join(command)); + LOG.debug("launchContainer: " + commandStr + " " + + Joiner.on(" ").join(command)); } shExec = new ShellCommandExecutor( - command, - new File(containerWorkDir.toUri().getPath()), - container.getLaunchContext().getEnvironment()); // sanitized env + command, + new File(containerWorkDir.toUri().getPath()), + container.getLaunchContext().getEnvironment()); // sanitized env if (isContainerActive(containerId)) { shExec.execute(); } else { @@ -279,9 +309,17 @@ public int launchContainer(Container container, } @Override - public void writeLaunchEnv(OutputStream out, Map environment, Map> resources, List command) throws IOException { - ContainerLaunch.ShellScriptBuilder sb = ContainerLaunch.ShellScriptBuilder.create(); + /** + * Filter the environment variables that may conflict with the ones set in + * the docker image and write them out to an OutputStream. + */ + public void writeLaunchEnv(OutputStream out, Map environment, + Map> resources, List command) + throws IOException { + ContainerLaunch.ShellScriptBuilder sb = + ContainerLaunch.ShellScriptBuilder.create(); + //Remove environments that may conflict with the ones in Docker image. Set exclusionSet = new HashSet(); exclusionSet.add(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME); exclusionSet.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name()); @@ -427,6 +465,9 @@ private String toMount(List dirs) { return builder.toString(); } + //This class facilitates (only) the creation of platform-specific scripts that + //will be used to launch the containers + //TODO: This should be re-used from the DefaultContainerExecutor. private abstract class LocalWrapperScriptBuilder { private final Path wrapperScriptPath; @@ -435,7 +476,8 @@ public Path getWrapperScriptPath() { return wrapperScriptPath; } - public void writeLocalWrapperScript(Path launchDst, Path pidFile) throws IOException { + public void writeLocalWrapperScript(Path launchDst, Path pidFile) + throws IOException { DataOutputStream out = null; PrintStream pout = null; @@ -448,8 +490,8 @@ public void writeLocalWrapperScript(Path launchDst, Path pidFile) throws IOExcep } } - protected abstract void writeLocalWrapperScript(Path launchDst, Path pidFile, - PrintStream pout); + protected abstract void writeLocalWrapperScript(Path launchDst, + Path pidFile, PrintStream pout); protected LocalWrapperScriptBuilder(Path containerWorkDir) { this.wrapperScriptPath = new Path(containerWorkDir, @@ -457,13 +499,15 @@ protected LocalWrapperScriptBuilder(Path containerWorkDir) { } } + //TODO: This class too should be used from DefaultContainerExecutor. private final class UnixLocalWrapperScriptBuilder - extends LocalWrapperScriptBuilder { + extends LocalWrapperScriptBuilder { private final Path sessionScriptPath; private final String dockerCommand; private final String dockerPidScript; - public UnixLocalWrapperScriptBuilder(Path containerWorkDir, String dockerCommand, String dockerPidScript) { + public UnixLocalWrapperScriptBuilder(Path containerWorkDir, + String dockerCommand, String dockerPidScript) { super(containerWorkDir); this.dockerCommand = dockerCommand; this.dockerPidScript = dockerPidScript; @@ -480,8 +524,7 @@ public void writeLocalWrapperScript(Path launchDst, Path pidFile) @Override public void writeLocalWrapperScript(Path launchDst, Path pidFile, - PrintStream pout) { - + PrintStream pout) { String exitCodeFile = ContainerLaunch.getExitCodeFile( pidFile.toString()); String tmpFile = exitCodeFile + ".tmp"; @@ -505,7 +548,8 @@ private void writeSessionScript(Path launchDst, Path pidFile) // hence write pid to tmp file first followed by a mv pout.println("#!/usr/bin/env bash"); pout.println(); - pout.println("echo "+ dockerPidScript +" > " + pidFile.toString() + ".tmp"); + pout.println("echo "+ dockerPidScript +" > " + pidFile.toString() + + ".tmp"); pout.println("/bin/mv -f " + pidFile.toString() + ".tmp " + pidFile); pout.println(dockerCommand + " bash \"" + launchDst.toUri().getPath().toString() + "\""); @@ -518,7 +562,7 @@ private void writeSessionScript(Path launchDst, Path pidFile) } protected void createDir(Path dirPath, FsPermission perms, - boolean createParent, String user) throws IOException { + boolean createParent, String user) throws IOException { lfs.mkdir(dirPath, perms, createParent); if (!perms.equals(perms.applyUMask(lfs.getUMask()))) { lfs.setPermission(dirPath, perms); @@ -532,13 +576,14 @@ protected void createDir(Path dirPath, FsPermission perms, * */ void createUserLocalDirs(List localDirs, String user) - throws IOException { + throws IOException { boolean userDirStatus = false; FsPermission userperms = new FsPermission(USER_PERM); for (String localDir : localDirs) { // create $local.dir/usercache/$user and its immediate parent try { - createDir(getUserCacheDir(new Path(localDir), user), userperms, true, user); + createDir(getUserCacheDir(new Path(localDir), user), userperms, true, + user); } catch (IOException e) { LOG.warn("Unable to create the user directory : " + localDir, e); continue; @@ -633,7 +678,7 @@ void createAppDirs(List localDirs, String user, String appId) * Create application log directories on all disks. */ void createContainerLogDirs(String appId, String containerId, - List logDirs, String user) throws IOException { + List logDirs, String user) throws IOException { boolean containerLogDirStatus = false; FsPermission containerLogDirPerms = new FsPermission(LOGDIR_PERM); @@ -707,7 +752,7 @@ private Path getFileCacheDir(Path base, String user) { } protected Path getWorkingDir(List localDirs, String user, - String appId) throws IOException { + String appId) throws IOException { Path appStorageDir = null; long totalAvailable = 0L; long[] availableOnDisk = new long[localDirs.size()]; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java index a4be120c2c963..f95a7adfc1630 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java @@ -58,6 +58,7 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; import org.apache.hadoop.yarn.server.nodemanager.metrics.NodeManagerMetrics; +import org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider; import org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService; import org.apache.hadoop.yarn.server.nodemanager.recovery.NMNullStateStoreService; import org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService; @@ -80,6 +81,7 @@ public class NodeManager extends CompositeService protected final NodeManagerMetrics metrics = NodeManagerMetrics.create(); private ApplicationACLsManager aclsManager; private NodeHealthCheckerService nodeHealthChecker; + private NodeLabelsProvider nodeLabelsProvider; private LocalDirsHandlerService dirsHandler; private Context context; private AsyncDispatcher dispatcher; @@ -98,7 +100,22 @@ public NodeManager() { protected NodeStatusUpdater createNodeStatusUpdater(Context context, Dispatcher dispatcher, NodeHealthCheckerService healthChecker) { return new NodeStatusUpdaterImpl(context, dispatcher, healthChecker, - metrics); + metrics, nodeLabelsProvider); + } + + protected NodeStatusUpdater createNodeStatusUpdater(Context context, + Dispatcher dispatcher, NodeHealthCheckerService healthChecker, + NodeLabelsProvider nodeLabelsProvider) { + return new NodeStatusUpdaterImpl(context, dispatcher, healthChecker, + metrics, nodeLabelsProvider); + } + + @VisibleForTesting + protected NodeLabelsProvider createNodeLabelsProvider( + Configuration conf) throws IOException { + // TODO as part of YARN-2729 + // Need to get the implementation of provider service and return + return null; } protected NodeResourceMonitor createNodeResourceMonitor() { @@ -245,9 +262,18 @@ protected void serviceInit(Configuration conf) throws Exception { this.context = createNMContext(containerTokenSecretManager, nmTokenSecretManager, nmStore); - - nodeStatusUpdater = - createNodeStatusUpdater(context, dispatcher, nodeHealthChecker); + + nodeLabelsProvider = createNodeLabelsProvider(conf); + + if (null == nodeLabelsProvider) { + nodeStatusUpdater = + createNodeStatusUpdater(context, dispatcher, nodeHealthChecker); + } else { + addService(nodeLabelsProvider); + nodeStatusUpdater = + createNodeStatusUpdater(context, dispatcher, nodeHealthChecker, + nodeLabelsProvider); + } NodeResourceMonitor nodeResourceMonitor = createNodeResourceMonitor(); addService(nodeResourceMonitor); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java index 6ddd7e4af5d37..2549e0ff4d8b7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java @@ -42,6 +42,7 @@ import org.apache.hadoop.security.Credentials; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.util.VersionUtil; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.ContainerId; @@ -53,6 +54,7 @@ import org.apache.hadoop.yarn.event.Dispatcher; import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; +import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; import org.apache.hadoop.yarn.server.api.ResourceManagerConstants; import org.apache.hadoop.yarn.server.api.ResourceTracker; import org.apache.hadoop.yarn.server.api.ServerRMProxy; @@ -70,6 +72,7 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationState; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; import org.apache.hadoop.yarn.server.nodemanager.metrics.NodeManagerMetrics; +import org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider; import org.apache.hadoop.yarn.util.YarnVersionInfo; import com.google.common.annotations.VisibleForTesting; @@ -120,15 +123,25 @@ public class NodeStatusUpdaterImpl extends AbstractService implements private long rmIdentifier = ResourceManagerConstants.RM_INVALID_IDENTIFIER; Set pendingContainersToRemove = new HashSet(); + private final NodeLabelsProvider nodeLabelsProvider; + private final boolean hasNodeLabelsProvider; + public NodeStatusUpdaterImpl(Context context, Dispatcher dispatcher, NodeHealthCheckerService healthChecker, NodeManagerMetrics metrics) { + this(context, dispatcher, healthChecker, metrics, null); + } + + public NodeStatusUpdaterImpl(Context context, Dispatcher dispatcher, + NodeHealthCheckerService healthChecker, NodeManagerMetrics metrics, + NodeLabelsProvider nodeLabelsProvider) { super(NodeStatusUpdaterImpl.class.getName()); this.healthChecker = healthChecker; + this.nodeLabelsProvider = nodeLabelsProvider; + this.hasNodeLabelsProvider = (nodeLabelsProvider != null); this.context = context; this.dispatcher = dispatcher; this.metrics = metrics; - this.recentlyStoppedContainers = - new LinkedHashMap(); + this.recentlyStoppedContainers = new LinkedHashMap(); this.pendingCompletedContainers = new HashMap(); } @@ -253,22 +266,30 @@ protected ResourceTracker getRMClient() throws IOException { protected void registerWithRM() throws YarnException, IOException { List containerReports = getNMContainerStatuses(); + Set nodeLabels = null; + if (hasNodeLabelsProvider) { + nodeLabels = nodeLabelsProvider.getNodeLabels(); + nodeLabels = + (null == nodeLabels) ? CommonNodeLabelsManager.EMPTY_STRING_SET + : nodeLabels; + } RegisterNodeManagerRequest request = RegisterNodeManagerRequest.newInstance(nodeId, httpPort, totalResource, - nodeManagerVersionId, containerReports, getRunningApplications()); + nodeManagerVersionId, containerReports, getRunningApplications(), + nodeLabels); if (containerReports != null) { LOG.info("Registering with RM using containers :" + containerReports); } RegisterNodeManagerResponse regNMResponse = resourceTracker.registerNodeManager(request); this.rmIdentifier = regNMResponse.getRMIdentifier(); - // if the Resourcemanager instructs NM to shutdown. + // if the Resource Manager instructs NM to shutdown. if (NodeAction.SHUTDOWN.equals(regNMResponse.getNodeAction())) { String message = "Message from ResourceManager: " + regNMResponse.getDiagnosticsMessage(); throw new YarnRuntimeException( - "Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, " + "Recieved SHUTDOWN signal from Resourcemanager, Registration of NodeManager failed, " + message); } @@ -306,8 +327,21 @@ protected void registerWithRM() this.context.getNMTokenSecretManager().setMasterKey(masterKey); } - LOG.info("Registered with ResourceManager as " + this.nodeId - + " with total resource of " + this.totalResource); + StringBuilder successfullRegistrationMsg = new StringBuilder(); + successfullRegistrationMsg.append("Registered with ResourceManager as ") + .append(this.nodeId).append(" with total resource of ") + .append(this.totalResource); + + if (regNMResponse.getAreNodeLabelsAcceptedByRM()) { + successfullRegistrationMsg + .append(" and with following Node label(s) : {") + .append(StringUtils.join(",", nodeLabels)).append("}"); + } else if (hasNodeLabelsProvider) { + //case where provider is set but RM did not accept the Node Labels + LOG.error(regNMResponse.getDiagnosticsMessage()); + } + + LOG.info(successfullRegistrationMsg); LOG.info("Notifying ContainerManager to unblock new container-requests"); ((ContainerManagerImpl) this.context.getContainerManager()) .setBlockNewContainerRequests(false); @@ -580,19 +614,41 @@ protected void startStatusUpdater() { @Override @SuppressWarnings("unchecked") public void run() { - int lastHeartBeatID = 0; + int lastHeartbeatID = 0; + Set lastUpdatedNodeLabelsToRM = null; + if (hasNodeLabelsProvider) { + lastUpdatedNodeLabelsToRM = nodeLabelsProvider.getNodeLabels(); + lastUpdatedNodeLabelsToRM = + (null == lastUpdatedNodeLabelsToRM) ? CommonNodeLabelsManager.EMPTY_STRING_SET + : lastUpdatedNodeLabelsToRM; + } while (!isStopped) { // Send heartbeat try { NodeHeartbeatResponse response = null; - NodeStatus nodeStatus = getNodeStatus(lastHeartBeatID); - + Set nodeLabelsForHeartbeat = null; + NodeStatus nodeStatus = getNodeStatus(lastHeartbeatID); + + if (hasNodeLabelsProvider) { + nodeLabelsForHeartbeat = nodeLabelsProvider.getNodeLabels(); + //if the provider returns null then consider empty labels are set + nodeLabelsForHeartbeat = + (nodeLabelsForHeartbeat == null) ? CommonNodeLabelsManager.EMPTY_STRING_SET + : nodeLabelsForHeartbeat; + if (!areNodeLabelsUpdated(nodeLabelsForHeartbeat, + lastUpdatedNodeLabelsToRM)) { + //if nodelabels have not changed then no need to send + nodeLabelsForHeartbeat = null; + } + } + NodeHeartbeatRequest request = NodeHeartbeatRequest.newInstance(nodeStatus, - NodeStatusUpdaterImpl.this.context - .getContainerTokenSecretManager().getCurrentKey(), - NodeStatusUpdaterImpl.this.context.getNMTokenSecretManager() - .getCurrentKey()); + NodeStatusUpdaterImpl.this.context + .getContainerTokenSecretManager().getCurrentKey(), + NodeStatusUpdaterImpl.this.context + .getNMTokenSecretManager().getCurrentKey(), + nodeLabelsForHeartbeat); response = resourceTracker.nodeHeartbeat(request); //get next heartbeat interval from response nextHeartBeatInterval = response.getNextHeartBeatInterval(); @@ -623,6 +679,17 @@ public void run() { break; } + if (response.getAreNodeLabelsAcceptedByRM()) { + lastUpdatedNodeLabelsToRM = nodeLabelsForHeartbeat; + LOG.info("Node Labels {" + + StringUtils.join(",", nodeLabelsForHeartbeat) + + "} were Accepted by RM "); + } else if (nodeLabelsForHeartbeat != null) { + // case where NodeLabelsProvider is set and updated labels were + // sent to RM and RM rejected the labels + LOG.error(response.getDiagnosticsMessage()); + } + // Explicitly put this method after checking the resync response. We // don't want to remove the completed containers before resync // because these completed containers will be reported back to RM @@ -631,7 +698,7 @@ public void run() { removeOrTrackCompletedContainersFromContext(response .getContainersToBeRemovedFromNM()); - lastHeartBeatID = response.getResponseId(); + lastHeartbeatID = response.getResponseId(); List containersToCleanup = response .getContainersToCleanup(); if (!containersToCleanup.isEmpty()) { @@ -680,6 +747,23 @@ public void run() { } } + /** + * Caller should take care of sending non null nodelabels for both + * arguments + * + * @param nodeLabelsNew + * @param nodeLabelsOld + * @return if the New node labels are diff from the older one. + */ + private boolean areNodeLabelsUpdated(Set nodeLabelsNew, + Set nodeLabelsOld) { + if (nodeLabelsNew.size() != nodeLabelsOld.size() + || !nodeLabelsOld.containsAll(nodeLabelsNew)) { + return true; + } + return false; + } + private void updateMasterKeys(NodeHeartbeatResponse response) { // See if the master-key has rolled over MasterKey updatedMasterKey = response.getContainerTokenMasterKey(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java index cd3e71a8d623c..b7bec5f0cff25 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/WindowsSecureContainerExecutor.java @@ -52,6 +52,7 @@ import org.apache.hadoop.util.NativeCodeLoader; import org.apache.hadoop.util.Shell; import org.apache.hadoop.util.Shell.CommandExecutor; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer; import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService; @@ -727,11 +728,9 @@ public void startLocalizer(Path nmPrivateContainerTokens, } @Override - protected CommandExecutor buildCommandExecutor(String wrapperScriptPath, - String containerIdStr, - String userName, Path pidFile,File wordDir, Map environment) - throws IOException { - + protected CommandExecutor buildCommandExecutor(String wrapperScriptPath, + String containerIdStr, String userName, Path pidFile, Resource resource, + File wordDir, Map environment) throws IOException { return new WintuilsProcessStubExecutor( wordDir.toString(), containerIdStr, userName, pidFile.toString(), diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java index a87238d7932a6..5a9229b1d6218 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java @@ -267,11 +267,11 @@ public Integer call() { // Sanitize the container's environment sanitizeEnv(environment, containerWorkDir, appDirs, containerLogDirs, localResources, nmPrivateClasspathJarDir); - + // Write out the environment - exec.writeLaunchEnv(containerScriptOutStream, environment, localResources, - launchContext.getCommands()); - + exec.writeLaunchEnv(containerScriptOutStream, environment, + localResources, launchContext.getCommands()); + // /////////// End of writing out container-script // /////////// Write out the container-tokens in the nmPrivate space. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java index dd50eadf45850..423639219109a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java @@ -448,6 +448,10 @@ private void handleInitContainerResources( .getApplicationId()); for (LocalResourceRequest req : e.getValue()) { tracker.handle(new ResourceRequestEvent(req, e.getKey(), ctxt)); + if (LOG.isDebugEnabled()) { + LOG.debug("Localizing " + req.getPath() + + " for container " + c.getContainerId()); + } } } } @@ -456,10 +460,14 @@ private void handleCacheCleanup(LocalizationEvent event) { ResourceRetentionSet retain = new ResourceRetentionSet(delService, cacheTargetSize); retain.addResources(publicRsrc); - LOG.debug("Resource cleanup (public) " + retain); + if (LOG.isDebugEnabled()) { + LOG.debug("Resource cleanup (public) " + retain); + } for (LocalResourcesTracker t : privateRsrc.values()) { retain.addResources(t); - LOG.debug("Resource cleanup " + t.getUser() + ":" + retain); + if (LOG.isDebugEnabled()) { + LOG.debug("Resource cleanup " + t.getUser() + ":" + retain); + } } //TODO Check if appRsrcs should also be added to the retention set. } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java index 787422b8c5d0d..393576bfa4a9c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java @@ -44,7 +44,6 @@ import org.apache.hadoop.fs.UnsupportedFileSystemException; import org.apache.hadoop.security.Credentials; import org.apache.hadoop.security.UserGroupInformation; -import org.apache.hadoop.security.token.Token; import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.api.records.ApplicationAccessType; import org.apache.hadoop.yarn.api.records.ApplicationId; @@ -116,6 +115,7 @@ public class AppLogAggregatorImpl implements AppLogAggregator { private final Context context; private final int retentionSize; private final long rollingMonitorInterval; + private final boolean logAggregationInRolling; private final NodeId nodeId; // This variable is only for testing private final AtomicBoolean waiting = new AtomicBoolean(false); @@ -193,9 +193,14 @@ public AppLogAggregatorImpl(Dispatcher dispatcher, } this.rollingMonitorInterval = configuredRollingMonitorInterval; } + this.logAggregationInRolling = + this.rollingMonitorInterval <= 0 || this.logAggregationContext == null + || this.logAggregationContext.getRolledLogsIncludePattern() == null + || this.logAggregationContext.getRolledLogsIncludePattern() + .isEmpty() ? false : true; } - private void uploadLogsForContainers() { + private void uploadLogsForContainers(boolean appFinished) { if (this.logAggregationDisabled) { return; } @@ -262,7 +267,7 @@ private void uploadLogsForContainers() { containerLogAggregators.put(container, aggregator); } Set uploadedFilePathsInThisCycle = - aggregator.doContainerLogAggregation(writer); + aggregator.doContainerLogAggregation(writer, appFinished); if (uploadedFilePathsInThisCycle.size() > 0) { uploadedLogsInThisCycle = true; } @@ -298,7 +303,7 @@ private void uploadLogsForContainers() { userUgi.doAs(new PrivilegedExceptionAction() { @Override public Object run() throws Exception { - FileSystem remoteFS = FileSystem.get(conf); + FileSystem remoteFS = remoteNodeLogFileForApp.getFileSystem(conf); if (remoteFS.exists(remoteNodeTmpLogFileForApp)) { if (rename) { remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath); @@ -394,12 +399,12 @@ private void doAppLogAggregation() { synchronized(this) { try { waiting.set(true); - if (this.rollingMonitorInterval > 0) { + if (logAggregationInRolling) { wait(this.rollingMonitorInterval * 1000); if (this.appFinishing.get() || this.aborted.get()) { break; } - uploadLogsForContainers(); + uploadLogsForContainers(false); } else { wait(THREAD_SLEEP_TIME); } @@ -415,7 +420,7 @@ private void doAppLogAggregation() { } // App is finished, upload the container logs. - uploadLogsForContainers(); + uploadLogsForContainers(true); // Remove the local app-log-dirs List localAppLogDirs = new ArrayList(); @@ -536,7 +541,8 @@ public ContainerLogAggregator(ContainerId containerId) { this.containerId = containerId; } - public Set doContainerLogAggregation(LogWriter writer) { + public Set doContainerLogAggregation(LogWriter writer, + boolean appFinished) { LOG.info("Uploading logs for container " + containerId + ". Current good log dirs are " + StringUtils.join(",", dirsHandler.getLogDirs())); @@ -544,7 +550,7 @@ public Set doContainerLogAggregation(LogWriter writer) { final LogValue logValue = new LogValue(dirsHandler.getLogDirs(), containerId, userUgi.getShortUserName(), logAggregationContext, - this.uploadedFileMeta); + this.uploadedFileMeta, appFinished); try { writer.append(logKey, logValue); } catch (Exception e) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java index bd3e847579d1a..0018d56421470 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java @@ -176,7 +176,7 @@ private void stopAggregators() { } protected FileSystem getFileSystem(Configuration conf) throws IOException { - return FileSystem.get(conf); + return this.remoteRootLogDir.getFileSystem(conf); } void verifyAndCreateRemoteLogDir(Configuration conf) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java index 7850688a6d2aa..ffa72a415d550 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java @@ -42,14 +42,29 @@ @Metrics(context="container") public class ContainerMetrics implements MetricsSource { - public static final String PMEM_LIMIT_METRIC_NAME = "pMemLimit"; - public static final String VMEM_LIMIT_METRIC_NAME = "vMemLimit"; + public static final String PMEM_LIMIT_METRIC_NAME = "pMemLimitMBs"; + public static final String VMEM_LIMIT_METRIC_NAME = "vMemLimitMBs"; public static final String VCORE_LIMIT_METRIC_NAME = "vCoreLimit"; - public static final String PMEM_USAGE_METRIC_NAME = "pMemUsage"; + public static final String PMEM_USAGE_METRIC_NAME = "pMemUsageMBs"; + private static final String PHY_CPU_USAGE_METRIC_NAME = "pCpuUsagePercent"; + + // Use a multiplier of 1000 to avoid losing too much precision when + // converting to integers + private static final String VCORE_USAGE_METRIC_NAME = "milliVcoreUsage"; @Metric public MutableStat pMemMBsStat; + // This tracks overall CPU percentage of the machine in terms of percentage + // of 1 core similar to top + // Thus if you use 2 cores completely out of 4 available cores this value + // will be 200 + @Metric + public MutableStat cpuCoreUsagePercent; + + @Metric + public MutableStat milliVcoresUsed; + @Metric public MutableGaugeInt pMemLimitMbs; @@ -57,7 +72,7 @@ public class ContainerMetrics implements MetricsSource { public MutableGaugeInt vMemLimitMbs; @Metric - public MutableGaugeInt cpuVcores; + public MutableGaugeInt cpuVcoreLimit; static final MetricsInfo RECORD_INFO = info("ContainerResource", "Resource limit and usage by container"); @@ -95,11 +110,17 @@ public class ContainerMetrics implements MetricsSource { this.pMemMBsStat = registry.newStat( PMEM_USAGE_METRIC_NAME, "Physical memory stats", "Usage", "MBs", true); + this.cpuCoreUsagePercent = registry.newStat( + PHY_CPU_USAGE_METRIC_NAME, "Physical Cpu core percent usage stats", + "Usage", "Percents", true); + this.milliVcoresUsed = registry.newStat( + VCORE_USAGE_METRIC_NAME, "1000 times Vcore usage", "Usage", + "MilliVcores", true); this.pMemLimitMbs = registry.newGauge( PMEM_LIMIT_METRIC_NAME, "Physical memory limit in MBs", 0); this.vMemLimitMbs = registry.newGauge( VMEM_LIMIT_METRIC_NAME, "Virtual memory limit in MBs", 0); - this.cpuVcores = registry.newGauge( + this.cpuVcoreLimit = registry.newGauge( VCORE_LIMIT_METRIC_NAME, "CPU limit in number of vcores", 0); } @@ -167,7 +188,19 @@ public synchronized void finished() { } public void recordMemoryUsage(int memoryMBs) { - this.pMemMBsStat.add(memoryMBs); + if (memoryMBs >= 0) { + this.pMemMBsStat.add(memoryMBs); + } + } + + public void recordCpuUsage( + int totalPhysicalCpuPercent, int milliVcoresUsed) { + if (totalPhysicalCpuPercent >=0) { + this.cpuCoreUsagePercent.add(totalPhysicalCpuPercent); + } + if (milliVcoresUsed >= 0) { + this.milliVcoresUsed.add(milliVcoresUsed); + } } public void recordProcessId(String processId) { @@ -177,7 +210,7 @@ public void recordProcessId(String processId) { public void recordResourceLimit(int vmemLimit, int pmemLimit, int cpuVcores) { this.vMemLimitMbs.set(vmemLimit); this.pMemLimitMbs.set(pmemLimit); - this.cpuVcores.set(cpuVcores); + this.cpuVcoreLimit.set(cpuVcores); } private synchronized void scheduleTimerTaskIfRequired() { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java index 2cecda6cf2d4b..51530517563d4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java @@ -38,6 +38,7 @@ import org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor; import org.apache.hadoop.yarn.server.nodemanager.Context; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerKillEvent; +import org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils; import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree; import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin; @@ -75,6 +76,7 @@ public class ContainersMonitorImpl extends AbstractService implements private long maxVCoresAllottedForContainers; private static final long UNKNOWN_MEMORY_LIMIT = -1L; + private int nodeCpuPercentageForYARN; public ContainersMonitorImpl(ContainerExecutor exec, AsyncDispatcher dispatcher, Context context) { @@ -145,6 +147,9 @@ protected void serviceInit(Configuration conf) throws Exception { LOG.info("Physical memory check enabled: " + pmemCheckEnabled); LOG.info("Virtual memory check enabled: " + vmemCheckEnabled); + nodeCpuPercentageForYARN = + NodeManagerHardwareUtils.getNodeCpuPercentage(conf); + if (pmemCheckEnabled) { // Logging if actual pmem cannot be determined. long totalPhysicalMemoryOnNM = UNKNOWN_MEMORY_LIMIT; @@ -328,10 +333,10 @@ boolean isProcessTreeOverLimit(String containerId, // method provided just for easy testing purposes boolean isProcessTreeOverLimit(ResourceCalculatorProcessTree pTree, String containerId, long limit) { - long currentMemUsage = pTree.getCumulativeVmem(); + long currentMemUsage = pTree.getVirtualMemorySize(); // as processes begin with an age 1, we want to see if there are processes // more than 1 iteration old. - long curMemUsageOfAgedProcesses = pTree.getCumulativeVmem(1); + long curMemUsageOfAgedProcesses = pTree.getVirtualMemorySize(1); return isProcessTreeOverLimit(containerId, currentMemUsage, curMemUsageOfAgedProcesses, limit); } @@ -432,12 +437,22 @@ public void run() { + " ContainerId = " + containerId); ResourceCalculatorProcessTree pTree = ptInfo.getProcessTree(); pTree.updateProcessTree(); // update process-tree - long currentVmemUsage = pTree.getCumulativeVmem(); - long currentPmemUsage = pTree.getCumulativeRssmem(); + long currentVmemUsage = pTree.getVirtualMemorySize(); + long currentPmemUsage = pTree.getRssMemorySize(); + // if machine has 6 cores and 3 are used, + // cpuUsagePercentPerCore should be 300% and + // cpuUsageTotalCoresPercentage should be 50% + float cpuUsagePercentPerCore = pTree.getCpuUsagePercent(); + float cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore / + resourceCalculatorPlugin.getNumProcessors(); + + // Multiply by 1000 to avoid losing data when converting to int + int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000 + * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN); // as processes begin with an age 1, we want to see if there // are processes more than 1 iteration old. - long curMemUsageOfAgedProcesses = pTree.getCumulativeVmem(1); - long curRssMemUsageOfAgedProcesses = pTree.getCumulativeRssmem(1); + long curMemUsageOfAgedProcesses = pTree.getVirtualMemorySize(1); + long curRssMemUsageOfAgedProcesses = pTree.getRssMemorySize(1); long vmemLimit = ptInfo.getVmemLimit(); long pmemLimit = ptInfo.getPmemLimit(); LOG.info(String.format( @@ -451,6 +466,9 @@ public void run() { ContainerMetrics.forContainer( containerId, containerMetricsPeriodMs).recordMemoryUsage( (int) (currentPmemUsage >> 20)); + ContainerMetrics.forContainer( + containerId, containerMetricsPeriodMs).recordCpuUsage + ((int)cpuUsagePercentPerCore, milliVcoresUsed); } boolean isMemoryOverLimit = false; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java new file mode 100644 index 0000000000000..4b34d76387391 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/nodelabels/NodeLabelsProvider.java @@ -0,0 +1,43 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager.nodelabels; + +import java.util.Set; + +import org.apache.hadoop.service.AbstractService; + +/** + * Interface which will be responsible for fetching the labels + * + */ +public abstract class NodeLabelsProvider extends AbstractService { + + public NodeLabelsProvider(String name) { + super(name); + } + + /** + * Provides the labels. LabelProvider is expected to give same Labels + * continuously until there is a change in labels. + * If null is returned then Empty label set is assumed by the caller. + * + * @return Set of node label strings applicable for a node + */ + public abstract Set getNodeLabels(); +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java index 07cf698429c04..77db1e32621f3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java @@ -59,6 +59,19 @@ public static float getContainersCores(Configuration conf) { public static float getContainersCores(ResourceCalculatorPlugin plugin, Configuration conf) { int numProcessors = plugin.getNumProcessors(); + int nodeCpuPercentage = getNodeCpuPercentage(conf); + + return (nodeCpuPercentage * numProcessors) / 100.0f; + } + + /** + * Gets the percentage of physical CPU that is configured for YARN containers. + * This is percent {@literal >} 0 and {@literal <=} 100 based on + * {@link YarnConfiguration#NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT} + * @param conf Configuration object + * @return percent {@literal >} 0 and {@literal <=} 100 + */ + public static int getNodeCpuPercentage(Configuration conf) { int nodeCpuPercentage = Math.min(conf.getInt( YarnConfiguration.NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT, @@ -73,7 +86,6 @@ public static float getContainersCores(ResourceCalculatorPlugin plugin, + ". Value cannot be less than or equal to 0."; throw new IllegalArgumentException(message); } - - return (nodeCpuPercentage * numProcessors) / 100.0f; + return nodeCpuPercentage; } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java index 1c3b63b4c8697..b8a10f1ce7e1b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java @@ -38,7 +38,7 @@ public NavBlock(Configuration conf) { protected void render(Block html) { String RMWebAppURL = - WebAppUtils.getResolvedRMWebAppURLWithScheme(this.conf); + WebAppUtils.getResolvedRemoteRMWebAppURLWithScheme(this.conf); html .div("#nav") .h3()._("ResourceManager")._() diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java index fd3634bd1c9a9..dc3e94180349e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java @@ -18,13 +18,21 @@ package org.apache.hadoop.yarn.server.nodemanager; +import java.util.Arrays; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.util.Shell; +import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils; +import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin; +import org.junit.Assert; import org.junit.Test; + import static org.junit.Assert.*; +import static org.junit.Assume.assumeTrue; public class TestContainerExecutor { @@ -69,4 +77,49 @@ public void testRunCommandwithPriority() throws Exception { } } + @Test (timeout = 5000) + public void testRunCommandWithNoResources() { + // Windows only test + assumeTrue(Shell.WINDOWS); + Configuration conf = new Configuration(); + String[] command = containerExecutor.getRunCommand("echo", "group1", null, null, + conf, Resource.newInstance(1024, 1)); + // Assert the cpu and memory limits are set correctly in the command + String[] expected = { Shell.WINUTILS, "task", "create", "-m", "-1", "-c", + "-1", "group1", "cmd /c " + "echo" }; + Assert.assertTrue(Arrays.equals(expected, command)); + } + + @Test (timeout = 5000) + public void testRunCommandWithMemoryOnlyResources() { + // Windows only test + assumeTrue(Shell.WINDOWS); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.NM_WINDOWS_CONTAINER_MEMORY_LIMIT_ENABLED, "true"); + String[] command = containerExecutor.getRunCommand("echo", "group1", null, null, + conf, Resource.newInstance(1024, 1)); + // Assert the cpu and memory limits are set correctly in the command + String[] expected = { Shell.WINUTILS, "task", "create", "-m", "1024", "-c", + "-1", "group1", "cmd /c " + "echo" }; + Assert.assertTrue(Arrays.equals(expected, command)); + } + + @Test (timeout = 5000) + public void testRunCommandWithCpuAndMemoryResources() { + // Windows only test + assumeTrue(Shell.WINDOWS); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED, "true"); + conf.set(YarnConfiguration.NM_WINDOWS_CONTAINER_MEMORY_LIMIT_ENABLED, "true"); + String[] command = containerExecutor.getRunCommand("echo", "group1", null, null, + conf, Resource.newInstance(1024, 1)); + float yarnProcessors = NodeManagerHardwareUtils.getContainersCores( + ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf), + conf); + int cpuRate = Math.min(10000, (int) ((1 * 10000) / yarnProcessors)); + // Assert the cpu and memory limits are set correctly in the command + String[] expected = { Shell.WINUTILS, "task", "create", "-m", "1024", "-c", + String.valueOf(cpuRate), "group1", "cmd /c " + "echo" }; + Assert.assertTrue(Arrays.equals(expected, command)); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java index e43ac2eb672c8..65e381c52651e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java @@ -18,7 +18,18 @@ package org.apache.hadoop.yarn.server.nodemanager; -import com.google.common.base.Strings; +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.RETURNS_DEEP_STUBS; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.PrintWriter; +import java.util.HashMap; +import java.util.Map; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; @@ -26,48 +37,29 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.util.Shell; -import org.apache.hadoop.yarn.api.ApplicationConstants; import org.apache.hadoop.yarn.api.records.ContainerId; import org.apache.hadoop.yarn.api.records.ContainerLaunchContext; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container; -import org.apache.hadoop.yarn.util.ConverterUtils; import org.junit.After; import org.junit.Before; import org.junit.Test; -import java.io.File; -import java.io.FileOutputStream; -import java.io.FileReader; -import java.io.IOException; -import java.io.LineNumberReader; -import java.io.PrintWriter; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; -import static org.junit.Assume.assumeTrue; -import static org.mockito.Mockito.RETURNS_DEEP_STUBS; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; +import com.google.common.base.Strings; /** - * This is intended to test the DockerContainerExecutor code, but it requires docker - * to be installed. + * This is intended to test the DockerContainerExecutor code, but it requires + * docker to be installed. *
      - *
    1. Install docker, and Compile the code with docker-service-url set to the host and port - * where docker service is running. + *
    2. Install docker, and Compile the code with docker-service-url set to the + * host and port where docker service is running. * 
      
- * > mvn clean install -Ddocker-service-url=tcp://0.0.0.0:4243
- *                          -DskipTests
+ * > mvn clean install -Ddocker-service-url=tcp://0.0.0.0:4243 -DskipTests
  *
      */ public class TestDockerContainerExecutor { private static final Log LOG = LogFactory - .getLog(TestDockerContainerExecutor.class); + .getLog(TestDockerContainerExecutor.class); private static File workSpace = null; private DockerContainerExecutor exec = null; private LocalDirsHandlerService dirsHandler; @@ -75,14 +67,10 @@ public class TestDockerContainerExecutor { private FileContext lfs; private String yarnImage; - private int id = 0; private String appSubmitter; private String dockerUrl; - private String testImage = "centos"; + private String testImage = "centos:latest"; private String dockerExec; - private String containerIdStr; - - private ContainerId getNextContainerId() { ContainerId cId = mock(ContainerId.class, RETURNS_DEEP_STUBS); String id = "CONTAINER_" + System.currentTimeMillis(); @@ -91,6 +79,8 @@ private ContainerId getNextContainerId() { } @Before + //Initialize a new DockerContainerExecutor that will be used to launch mocked + //containers. public void setup() { try { lfs = FileContext.getLocalFSFileContext(); @@ -113,8 +103,10 @@ public void setup() { } dockerUrl = " -H " + dockerUrl; dockerExec = "docker " + dockerUrl; - conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, yarnImage); - conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, dockerExec); + conf.set( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, yarnImage); + conf.set( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, dockerExec); exec = new DockerContainerExecutor(); dirsHandler = new LocalDirsHandlerService(); dirsHandler.init(conf); @@ -129,11 +121,10 @@ public void setup() { private Shell.ShellCommandExecutor shellExec(String command) { try { - Shell.ShellCommandExecutor shExec = new Shell.ShellCommandExecutor( - command.split("\\s+"), - new File(workDir.toUri().getPath()), - System.getenv()); + command.split("\\s+"), + new File(workDir.toUri().getPath()), + System.getenv()); shExec.execute(); return shExec; } catch (IOException e) { @@ -145,14 +136,24 @@ private boolean shouldRun() { return exec != null; } - private int runAndBlock(ContainerId cId, Map launchCtxEnv, String... cmd) throws IOException { + /** + * Test that a docker container can be launched to run a command + * @param cId a fake ContainerID + * @param launchCtxEnv + * @param cmd the command to launch inside the docker container + * @return the exit code of the process used to launch the docker container + * @throws IOException + */ + private int runAndBlock(ContainerId cId, Map launchCtxEnv, + String... cmd) throws IOException { String appId = "APP_" + System.currentTimeMillis(); Container container = mock(Container.class); ContainerLaunchContext context = mock(ContainerLaunchContext.class); when(container.getContainerId()).thenReturn(cId); when(container.getLaunchContext()).thenReturn(context); - when(cId.getApplicationAttemptId().getApplicationId().toString()).thenReturn(appId); + when(cId.getApplicationAttemptId().getApplicationId().toString()) + .thenReturn(appId); when(context.getEnvironment()).thenReturn(launchCtxEnv); String script = writeScriptFile(launchCtxEnv, cmd); @@ -164,11 +165,13 @@ private int runAndBlock(ContainerId cId, Map launchCtxEnv, Strin exec.activateContainer(cId, pidFile); return exec.launchContainer(container, scriptPath, tokensPath, - appSubmitter, appId, workDir, dirsHandler.getLocalDirs(), - dirsHandler.getLogDirs()); + appSubmitter, appId, workDir, dirsHandler.getLocalDirs(), + dirsHandler.getLogDirs()); } - private String writeScriptFile(Map launchCtxEnv, String... cmd) throws IOException { + // Write the script used to launch the docker container in a temp file + private String writeScriptFile(Map launchCtxEnv, + String... cmd) throws IOException { File f = File.createTempFile("TestDockerContainerExecutor", ".sh"); f.deleteOnExit(); PrintWriter p = new PrintWriter(new FileOutputStream(f)); @@ -193,6 +196,10 @@ public void tearDown() { } } + /** + * Test that a touch command can be launched successfully in a docker + * container + */ @Test public void testLaunchContainer() throws IOException { if (!shouldRun()) { @@ -201,12 +208,13 @@ public void testLaunchContainer() throws IOException { } Map env = new HashMap(); - env.put(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); + env.put(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, + testImage); String touchFileName = "touch-file-" + System.currentTimeMillis(); File touchFile = new File(dirsHandler.getLocalDirs().get(0), touchFileName); ContainerId cId = getNextContainerId(); - int ret = runAndBlock( - cId, env, "touch", touchFile.getAbsolutePath(), "&&", "cp", touchFile.getAbsolutePath(), "/"); + int ret = runAndBlock(cId, env, "touch", touchFile.getAbsolutePath(), "&&", + "cp", touchFile.getAbsolutePath(), "/"); assertEquals(0, ret); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java index 3584fedde43af..8acd9caaa97b7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java @@ -18,7 +18,23 @@ package org.apache.hadoop.yarn.server.nodemanager; -import com.google.common.base.Strings; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assume.assumeTrue; +import static org.mockito.Mockito.RETURNS_DEEP_STUBS; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.io.LineNumberReader; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; + import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; @@ -36,30 +52,13 @@ import org.junit.Before; import org.junit.Test; -import java.io.File; -import java.io.FileReader; -import java.io.IOException; -import java.io.LineNumberReader; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; -import static org.junit.Assume.assumeTrue; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; -import static org.mockito.Mockito.RETURNS_DEEP_STUBS; - /** * Mock tests for docker container executor */ public class TestDockerContainerExecutorWithMocks { private static final Log LOG = LogFactory - .getLog(TestDockerContainerExecutorWithMocks.class); + .getLog(TestDockerContainerExecutorWithMocks.class); public static final String DOCKER_LAUNCH_COMMAND = "/bin/true"; private DockerContainerExecutor dockerContainerExecutor = null; private LocalDirsHandlerService dirsHandler; @@ -81,8 +80,10 @@ public void setup() { conf.set(YarnConfiguration.NM_LINUX_CONTAINER_EXECUTOR_PATH, executorPath); conf.set(YarnConfiguration.NM_LOCAL_DIRS, "/tmp/nm-local-dir" + time); conf.set(YarnConfiguration.NM_LOG_DIRS, "/tmp/userlogs" + time); - conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, yarnImage); - conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME , DOCKER_LAUNCH_COMMAND); + conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, + yarnImage); + conf.set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_EXEC_NAME, + DOCKER_LAUNCH_COMMAND); dockerContainerExecutor = new DockerContainerExecutor(); dirsHandler = new LocalDirsHandlerService(); dirsHandler.init(conf); @@ -95,7 +96,6 @@ public void setup() { } catch (IOException e) { throw new RuntimeException(e); } - } @After @@ -110,13 +110,17 @@ public void tearDown() { } @Test(expected = IllegalStateException.class) + //Test that DockerContainerExecutor doesn't successfully init on a secure + //cluster public void testContainerInitSecure() throws IOException { dockerContainerExecutor.getConf().set( - CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION, "kerberos"); + CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION, "kerberos"); dockerContainerExecutor.init(); } @Test(expected = IllegalArgumentException.class) + //Test that when the image name is null, the container launch throws an + //IllegalArgumentException public void testContainerLaunchNullImage() throws IOException { String appSubmitter = "nobody"; String appId = "APP_ID"; @@ -126,17 +130,19 @@ public void testContainerLaunchNullImage() throws IOException { Container container = mock(Container.class, RETURNS_DEEP_STUBS); ContainerId cId = mock(ContainerId.class, RETURNS_DEEP_STUBS); ContainerLaunchContext context = mock(ContainerLaunchContext.class); - HashMap env = new HashMap(); + HashMap env = new HashMap(); when(container.getContainerId()).thenReturn(cId); when(container.getLaunchContext()).thenReturn(context); - when(cId.getApplicationAttemptId().getApplicationId().toString()).thenReturn(appId); + when(cId.getApplicationAttemptId().getApplicationId().toString()) + .thenReturn(appId); when(cId.toString()).thenReturn(containerId); when(context.getEnvironment()).thenReturn(env); - env.put(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); - dockerContainerExecutor.getConf() - .set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); + env.put( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); + dockerContainerExecutor.getConf().set( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); Path scriptPath = new Path("file:///bin/echo"); Path tokensPath = new Path("file:///dev/null"); @@ -149,6 +155,8 @@ public void testContainerLaunchNullImage() throws IOException { } @Test(expected = IllegalArgumentException.class) + //Test that when the image name is invalid, the container launch throws an + //IllegalArgumentException public void testContainerLaunchInvalidImage() throws IOException { String appSubmitter = "nobody"; String appId = "APP_ID"; @@ -162,13 +170,15 @@ public void testContainerLaunchInvalidImage() throws IOException { when(container.getContainerId()).thenReturn(cId); when(container.getLaunchContext()).thenReturn(context); - when(cId.getApplicationAttemptId().getApplicationId().toString()).thenReturn(appId); + when(cId.getApplicationAttemptId().getApplicationId().toString()) + .thenReturn(appId); when(cId.toString()).thenReturn(containerId); when(context.getEnvironment()).thenReturn(env); - env.put(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); - dockerContainerExecutor.getConf() - .set(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); + env.put( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); + dockerContainerExecutor.getConf().set( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); Path scriptPath = new Path("file:///bin/echo"); Path tokensPath = new Path("file:///dev/null"); @@ -181,6 +191,8 @@ public void testContainerLaunchInvalidImage() throws IOException { } @Test + //Test that a container launch correctly wrote the session script with the + //commands we expected public void testContainerLaunch() throws IOException { String appSubmitter = "nobody"; String appId = "APP_ID"; @@ -194,40 +206,48 @@ public void testContainerLaunch() throws IOException { when(container.getContainerId()).thenReturn(cId); when(container.getLaunchContext()).thenReturn(context); - when(cId.getApplicationAttemptId().getApplicationId().toString()).thenReturn(appId); + when(cId.getApplicationAttemptId().getApplicationId().toString()) + .thenReturn(appId); when(cId.toString()).thenReturn(containerId); when(context.getEnvironment()).thenReturn(env); - env.put(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); + env.put( + YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME, testImage); Path scriptPath = new Path("file:///bin/echo"); Path tokensPath = new Path("file:///dev/null"); Path pidFile = new Path(workDir, "pid"); dockerContainerExecutor.activateContainer(cId, pidFile); - int ret = dockerContainerExecutor.launchContainer(container, scriptPath, tokensPath, - appSubmitter, appId, workDir, dirsHandler.getLocalDirs(), - dirsHandler.getLogDirs()); + int ret = dockerContainerExecutor.launchContainer(container, scriptPath, + tokensPath, appSubmitter, appId, workDir, dirsHandler.getLocalDirs(), + dirsHandler.getLogDirs()); assertEquals(0, ret); //get the script Path sessionScriptPath = new Path(workDir, - Shell.appendScriptExtension( - DockerContainerExecutor.DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT)); - LineNumberReader lnr = new LineNumberReader(new FileReader(sessionScriptPath.toString())); + Shell.appendScriptExtension( + DockerContainerExecutor.DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT)); + LineNumberReader lnr = new LineNumberReader(new FileReader( + sessionScriptPath.toString())); boolean cmdFound = false; List localDirs = dirsToMount(dirsHandler.getLocalDirs()); List logDirs = dirsToMount(dirsHandler.getLogDirs()); - List workDirMount = dirsToMount(Collections.singletonList(workDir.toUri().getPath())); - List expectedCommands = new ArrayList( - Arrays.asList(DOCKER_LAUNCH_COMMAND, "run", "--rm", "--net=host", "--name", containerId)); + List workDirMount = dirsToMount(Collections.singletonList( + workDir.toUri().getPath())); + List expectedCommands = new ArrayList(Arrays.asList( + DOCKER_LAUNCH_COMMAND, "run", "--rm", "--net=host", "--name", + containerId)); expectedCommands.addAll(localDirs); expectedCommands.addAll(logDirs); expectedCommands.addAll(workDirMount); String shellScript = workDir + "/launch_container.sh"; - expectedCommands.addAll(Arrays.asList(testImage.replaceAll("['\"]", ""), "bash","\"" + shellScript + "\"")); + expectedCommands.addAll(Arrays.asList(testImage.replaceAll("['\"]", ""), + "bash","\"" + shellScript + "\"")); - String expectedPidString = "echo `/bin/true inspect --format {{.State.Pid}} " + containerId+"` > "+ pidFile.toString() + ".tmp"; + String expectedPidString = + "echo `/bin/true inspect --format {{.State.Pid}} " + containerId+"` > "+ + pidFile.toString() + ".tmp"; boolean pidSetterFound = false; while(lnr.ready()){ String line = lnr.readLine(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java index 71a420eba4597..fc404de741452 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java @@ -1182,7 +1182,7 @@ protected NodeStatusUpdater createUpdater(Context context, } }; verifyNodeStartFailure( - "Recieved SHUTDOWN signal from Resourcemanager ," + "Recieved SHUTDOWN signal from Resourcemanager, " + "Registration of NodeManager failed, " + "Message from ResourceManager: RM Shutting Down Node"); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java new file mode 100644 index 0000000000000..437e4c89d754b --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdaterForLabels.java @@ -0,0 +1,281 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.nodemanager; + +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Collections; +import java.util.Set; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.service.ServiceOperations; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.event.Dispatcher; +import org.apache.hadoop.yarn.exceptions.YarnException; +import org.apache.hadoop.yarn.factories.RecordFactory; +import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; +import org.apache.hadoop.yarn.nodelabels.NodeLabelTestBase; +import org.apache.hadoop.yarn.server.api.ResourceTracker; +import org.apache.hadoop.yarn.server.api.protocolrecords.NodeHeartbeatRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.NodeHeartbeatResponse; +import org.apache.hadoop.yarn.server.api.protocolrecords.RegisterNodeManagerRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.RegisterNodeManagerResponse; +import org.apache.hadoop.yarn.server.api.records.MasterKey; +import org.apache.hadoop.yarn.server.api.records.NodeAction; +import org.apache.hadoop.yarn.server.api.records.NodeStatus; +import org.apache.hadoop.yarn.server.api.records.impl.pb.MasterKeyPBImpl; +import org.apache.hadoop.yarn.server.nodemanager.nodelabels.NodeLabelsProvider; +import org.apache.hadoop.yarn.server.utils.YarnServerBuilderUtils; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +public class TestNodeStatusUpdaterForLabels extends NodeLabelTestBase { + private static final RecordFactory recordFactory = RecordFactoryProvider + .getRecordFactory(null); + + private NodeManager nm; + protected DummyNodeLabelsProvider dummyLabelsProviderRef; + + @Before + public void setup() { + dummyLabelsProviderRef = new DummyNodeLabelsProvider(); + } + + @After + public void tearDown() { + if (null != nm) { + ServiceOperations.stop(nm); + } + } + + private class ResourceTrackerForLabels implements ResourceTracker { + int heartbeatID = 0; + Set labels; + + private boolean receivedNMHeartbeat = false; + private boolean receivedNMRegister = false; + + private MasterKey createMasterKey() { + MasterKey masterKey = new MasterKeyPBImpl(); + masterKey.setKeyId(123); + masterKey.setBytes(ByteBuffer.wrap(new byte[] { new Integer(123) + .byteValue() })); + return masterKey; + } + + @Override + public RegisterNodeManagerResponse registerNodeManager( + RegisterNodeManagerRequest request) throws YarnException, IOException { + labels = request.getNodeLabels(); + RegisterNodeManagerResponse response = + recordFactory.newRecordInstance(RegisterNodeManagerResponse.class); + response.setNodeAction(NodeAction.NORMAL); + response.setContainerTokenMasterKey(createMasterKey()); + response.setNMTokenMasterKey(createMasterKey()); + response.setAreNodeLabelsAcceptedByRM(labels != null); + synchronized (ResourceTrackerForLabels.class) { + receivedNMRegister = true; + ResourceTrackerForLabels.class.notifyAll(); + } + return response; + } + + public void waitTillHeartbeat() { + if (receivedNMHeartbeat) { + return; + } + int i = 500; + while (!receivedNMHeartbeat && i > 0) { + synchronized (ResourceTrackerForLabels.class) { + if (!receivedNMHeartbeat) { + try { + System.out + .println("In ResourceTrackerForLabels waiting for heartbeat : " + + System.currentTimeMillis()); + ResourceTrackerForLabels.class.wait(500l); + // to avoid race condition, i.e. sendOutofBandHeartBeat can be + // sent before NSU thread has gone to sleep, hence we wait and try + // to resend heartbeat again + nm.getNodeStatusUpdater().sendOutofBandHeartBeat(); + ResourceTrackerForLabels.class.wait(500l); + i--; + } catch (InterruptedException e) { + Assert.fail("Exception caught while waiting for Heartbeat"); + e.printStackTrace(); + } + } + } + } + if (!receivedNMHeartbeat) { + Assert.fail("Heartbeat dint receive even after waiting"); + } + } + + public void waitTillRegister() { + if (receivedNMRegister) { + return; + } + while (!receivedNMRegister) { + synchronized (ResourceTrackerForLabels.class) { + try { + ResourceTrackerForLabels.class.wait(); + } catch (InterruptedException e) { + Assert.fail("Exception caught while waiting for register"); + e.printStackTrace(); + } + } + } + } + + /** + * Flag to indicate received any + */ + public void resetNMHeartbeatReceiveFlag() { + synchronized (ResourceTrackerForLabels.class) { + receivedNMHeartbeat = false; + } + } + + @Override + public NodeHeartbeatResponse nodeHeartbeat(NodeHeartbeatRequest request) + throws YarnException, IOException { + System.out.println("RTS receive heartbeat : " + + System.currentTimeMillis()); + labels = request.getNodeLabels(); + NodeStatus nodeStatus = request.getNodeStatus(); + nodeStatus.setResponseId(heartbeatID++); + + NodeHeartbeatResponse nhResponse = + YarnServerBuilderUtils.newNodeHeartbeatResponse(heartbeatID, + NodeAction.NORMAL, null, null, null, null, 1000L); + + // to ensure that heartbeats are sent only when required. + nhResponse.setNextHeartBeatInterval(Long.MAX_VALUE); + nhResponse.setAreNodeLabelsAcceptedByRM(labels != null); + + synchronized (ResourceTrackerForLabels.class) { + receivedNMHeartbeat = true; + ResourceTrackerForLabels.class.notifyAll(); + } + return nhResponse; + } + } + + public static class DummyNodeLabelsProvider extends NodeLabelsProvider { + + @SuppressWarnings("unchecked") + private Set nodeLabels = Collections.EMPTY_SET; + + public DummyNodeLabelsProvider() { + super(DummyNodeLabelsProvider.class.getName()); + } + + @Override + public synchronized Set getNodeLabels() { + return nodeLabels; + } + + synchronized void setNodeLabels(Set nodeLabels) { + this.nodeLabels = nodeLabels; + } + } + + private YarnConfiguration createNMConfigForDistributeNodeLabels() { + YarnConfiguration conf = new YarnConfiguration(); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE); + return conf; + } + + @Test + public void testNodeStatusUpdaterForNodeLabels() throws InterruptedException, + IOException { + final ResourceTrackerForLabels resourceTracker = + new ResourceTrackerForLabels(); + nm = new NodeManager() { + @Override + protected NodeLabelsProvider createNodeLabelsProvider( + Configuration conf) throws IOException { + return dummyLabelsProviderRef; + } + + @Override + protected NodeStatusUpdater createNodeStatusUpdater(Context context, + Dispatcher dispatcher, NodeHealthCheckerService healthChecker, + NodeLabelsProvider labelsProvider) { + + return new NodeStatusUpdaterImpl(context, dispatcher, healthChecker, + metrics, labelsProvider) { + @Override + protected ResourceTracker getRMClient() { + return resourceTracker; + } + + @Override + protected void stopRMProxy() { + return; + } + }; + } + }; + + YarnConfiguration conf = createNMConfigForDistributeNodeLabels(); + nm.init(conf); + resourceTracker.resetNMHeartbeatReceiveFlag(); + nm.start(); + resourceTracker.waitTillRegister(); + assertCollectionEquals(resourceTracker.labels, + dummyLabelsProviderRef.getNodeLabels()); + + resourceTracker.waitTillHeartbeat();// wait till the first heartbeat + resourceTracker.resetNMHeartbeatReceiveFlag(); + + // heartbeat with updated labels + dummyLabelsProviderRef.setNodeLabels(toSet("P")); + + nm.getNodeStatusUpdater().sendOutofBandHeartBeat(); + resourceTracker.waitTillHeartbeat(); + assertCollectionEquals(resourceTracker.labels, + dummyLabelsProviderRef.getNodeLabels()); + resourceTracker.resetNMHeartbeatReceiveFlag(); + + // heartbeat without updating labels + nm.getNodeStatusUpdater().sendOutofBandHeartBeat(); + resourceTracker.waitTillHeartbeat(); + resourceTracker.resetNMHeartbeatReceiveFlag(); + assertNull( + "If no change in labels then null should be sent as part of request", + resourceTracker.labels); + + // provider return with null labels + dummyLabelsProviderRef.setNodeLabels(null); + nm.getNodeStatusUpdater().sendOutofBandHeartBeat(); + resourceTracker.waitTillHeartbeat(); + assertTrue("If provider sends null then empty labels should be sent", + resourceTracker.labels.isEmpty()); + resourceTracker.resetNMHeartbeatReceiveFlag(); + + nm.stop(); + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java index a73d58341bbdf..c45ffbb93ddef 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java @@ -130,8 +130,10 @@ public void testApplicationRecovery() throws Exception { containerTokens, acls); // create the logAggregationContext LogAggregationContext logAggregationContext = - LogAggregationContext.newInstance("includePattern", "excludePattern"); - StartContainersResponse startResponse = startContainer(context, cm, cid, + LogAggregationContext.newInstance("includePattern", "excludePattern", + "includePatternInRollingAggregation", + "excludePatternInRollingAggregation"); + StartContainersResponse startResponse = startContainer(context, cm, cid, clc, logAggregationContext); assertTrue(startResponse.getFailedRequests().isEmpty()); assertEquals(1, context.getApplications().size()); @@ -171,6 +173,10 @@ public void testApplicationRecovery() throws Exception { recovered.getIncludePattern()); assertEquals(logAggregationContext.getExcludePattern(), recovered.getExcludePattern()); + assertEquals(logAggregationContext.getRolledLogsIncludePattern(), + recovered.getRolledLogsIncludePattern()); + assertEquals(logAggregationContext.getRolledLogsExcludePattern(), + recovered.getRolledLogsExcludePattern()); waitForAppState(app, ApplicationState.INITING); assertTrue(context.getApplicationACLsManager().checkAccess( diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java index 901e45a57bbb6..b1de9cb259b86 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java @@ -698,7 +698,7 @@ private void writeContainerLogs(File appLogDir, ContainerId containerId, } } - private String verifyContainerLogs(LogAggregationService logAggregationService, + private LogFileStatusInLastCycle verifyContainerLogs(LogAggregationService logAggregationService, ApplicationId appId, ContainerId[] expectedContainerIds, String[] logFiles, int numOfContainerLogs, boolean multiLogs) throws IOException { @@ -743,7 +743,9 @@ private String verifyContainerLogs(LogAggregationService logAggregationService, new AggregatedLogFormat.LogReader(this.conf, targetNodeFile.getPath()); Assert.assertEquals(this.user, reader.getApplicationOwner()); verifyAcls(reader.getApplicationAcls()); - + + List fileTypes = new ArrayList(); + try { Map> logMap = new HashMap>(); @@ -769,6 +771,7 @@ private String verifyContainerLogs(LogAggregationService logAggregationService, Assert.assertEquals("LogType:", writtenLines[0].substring(0, 8)); String fileType = writtenLines[0].substring(8); + fileTypes.add(fileType); Assert.assertEquals("LogLength:", writtenLines[1].substring(0, 10)); String fileLengthStr = writtenLines[1].substring(10); @@ -801,7 +804,9 @@ private String verifyContainerLogs(LogAggregationService logAggregationService, Map thisContainerMap = logMap.remove(containerStr); Assert.assertEquals(numOfContainerLogs, thisContainerMap.size()); for (String fileType : logFiles) { - String expectedValue = containerStr + " Hello " + fileType + "!"; + String expectedValue = + containerStr + " Hello " + fileType + "!End of LogType:" + + fileType; LOG.info("Expected log-content : " + new String(expectedValue)); String foundValue = thisContainerMap.remove(fileType); Assert.assertNotNull(cId + " " + fileType @@ -811,7 +816,7 @@ private String verifyContainerLogs(LogAggregationService logAggregationService, Assert.assertEquals(0, thisContainerMap.size()); } Assert.assertEquals(0, logMap.size()); - return targetNodeFile.getPath().getName(); + return new LogFileStatusInLastCycle(targetNodeFile.getPath().getName(), fileTypes); } finally { reader.close(); } @@ -1289,9 +1294,17 @@ private void testLogAggregationService(boolean retentionSizeLimitation) throws Exception { LogAggregationContext logAggregationContextWithInterval = Records.newRecord(LogAggregationContext.class); + // set IncludePattern/excludePattern in rolling fashion + // we expect all the logs except std_final will be uploaded + // when app is running. The std_final will be uploaded when + // the app finishes. + logAggregationContextWithInterval.setRolledLogsIncludePattern(".*"); + logAggregationContextWithInterval.setRolledLogsExcludePattern("std_final"); this.conf.set(YarnConfiguration.NM_LOG_DIRS, localLogDir.getAbsolutePath()); + //configure YarnConfiguration.NM_REMOTE_APP_LOG_DIR to + //have fully qualified path this.conf.set(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, - this.remoteRootLogDir.getAbsolutePath()); + "file://" + this.remoteRootLogDir.getAbsolutePath()); this.conf.setLong( YarnConfiguration.NM_LOG_AGGREGATION_ROLL_MONITORING_INTERVAL_SECONDS, 3600); @@ -1338,9 +1351,14 @@ private void testLogAggregationService(boolean retentionSizeLimitation) this.user, null, ContainerLogsRetentionPolicy.ALL_CONTAINERS, this.acls, logAggregationContextWithInterval)); + LogFileStatusInLastCycle logFileStatusInLastCycle = null; // Simulate log-file creation - String[] logFiles1 = new String[] { "stdout", "stderr", "syslog" }; - writeContainerLogs(appLogDir, container, logFiles1); + // create std_final in log directory which will not be aggregated + // until the app finishes. + String[] logFiles1WithFinalLog = + new String[] { "stdout", "stderr", "syslog", "std_final" }; + String[] logFiles1 = new String[] { "stdout", "stderr", "syslog"}; + writeContainerLogs(appLogDir, container, logFiles1WithFinalLog); // Do log aggregation AppLogAggregatorImpl aggregator = @@ -1355,10 +1373,16 @@ private void testLogAggregationService(boolean retentionSizeLimitation) Assert.assertTrue(waitAndCheckLogNum(logAggregationService, application, 50, 1, false, null)); } - String logFileInLastCycle = null; // Container logs should be uploaded - logFileInLastCycle = verifyContainerLogs(logAggregationService, application, + logFileStatusInLastCycle = verifyContainerLogs(logAggregationService, application, new ContainerId[] { container }, logFiles1, 3, true); + for(String logFile : logFiles1) { + Assert.assertTrue(logFileStatusInLastCycle.getLogFileTypesInLastCycle() + .contains(logFile)); + } + // Make sure the std_final is not uploaded. + Assert.assertFalse(logFileStatusInLastCycle.getLogFileTypesInLastCycle() + .contains("std_final")); Thread.sleep(2000); @@ -1380,15 +1404,23 @@ private void testLogAggregationService(boolean retentionSizeLimitation) if (retentionSizeLimitation) { Assert.assertTrue(waitAndCheckLogNum(logAggregationService, application, - 50, 1, true, logFileInLastCycle)); + 50, 1, true, logFileStatusInLastCycle.getLogFilePathInLastCycle())); } else { Assert.assertTrue(waitAndCheckLogNum(logAggregationService, application, 50, 2, false, null)); } // Container logs should be uploaded - logFileInLastCycle = verifyContainerLogs(logAggregationService, application, + logFileStatusInLastCycle = verifyContainerLogs(logAggregationService, application, new ContainerId[] { container }, logFiles2, 3, true); + for(String logFile : logFiles2) { + Assert.assertTrue(logFileStatusInLastCycle.getLogFileTypesInLastCycle() + .contains(logFile)); + } + // Make sure the std_final is not uploaded. + Assert.assertFalse(logFileStatusInLastCycle.getLogFileTypesInLastCycle() + .contains("std_final")); + Thread.sleep(2000); // create another logs @@ -1402,13 +1434,17 @@ private void testLogAggregationService(boolean retentionSizeLimitation) logAggregationService.handle(new LogHandlerAppFinishedEvent(application)); if (retentionSizeLimitation) { Assert.assertTrue(waitAndCheckLogNum(logAggregationService, application, - 50, 1, true, logFileInLastCycle)); + 50, 1, true, logFileStatusInLastCycle.getLogFilePathInLastCycle())); } else { Assert.assertTrue(waitAndCheckLogNum(logAggregationService, application, 50, 3, false, null)); } + + // the app is finished. The log "std_final" should be aggregated this time. + String[] logFiles3WithFinalLog = + new String[] { "stdout_2", "stderr_2", "syslog_2", "std_final" }; verifyContainerLogs(logAggregationService, application, - new ContainerId[] { container }, logFiles3, 3, true); + new ContainerId[] { container }, logFiles3WithFinalLog, 4, true); logAggregationService.stop(); assertEquals(0, logAggregationService.getNumAggregators()); } @@ -1512,4 +1548,23 @@ private boolean waitAndCheckLogNum( return numOfLogsAvailable(logAggregationService, application, sizeLimited, lastLogFile) == expectNum; } + + private static class LogFileStatusInLastCycle { + private String logFilePathInLastCycle; + private List logFileTypesInLastCycle; + + public LogFileStatusInLastCycle(String logFilePathInLastCycle, + List logFileTypesInLastCycle) { + this.logFilePathInLastCycle = logFilePathInLastCycle; + this.logFileTypesInLastCycle = logFileTypesInLastCycle; + } + + public String getLogFilePathInLastCycle() { + return this.logFilePathInLastCycle; + } + + public List getLogFileTypesInLastCycle() { + return this.logFileTypesInLastCycle; + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java index 618099546da6e..12714de2e89be 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java @@ -77,6 +77,8 @@ import org.apache.hadoop.yarn.server.api.protocolrecords.RemoveFromClusterNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.ReplaceLabelsOnNodeResponse; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsRequest; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeLabelsResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceResponse; import org.apache.hadoop.yarn.server.resourcemanager.reservation.ReservationSystem; @@ -663,6 +665,28 @@ public ReplaceLabelsOnNodeResponse replaceLabelsOnNode( throw logAndWrapException(ioe, user.getShortUserName(), argName, msg); } } + + @Override + public UpdateNodeLabelsResponse updateNodeLabels( + UpdateNodeLabelsRequest request) throws YarnException, IOException { + String argName = "updateNodeLabels"; + final String msg = "update node labels"; + UserGroupInformation user = checkAcls(argName); + + checkRMStatus(user.getShortUserName(), argName, msg); + + UpdateNodeLabelsResponse response = UpdateNodeLabelsResponse.newInstance(); + + try { + rmContext.getNodeLabelManager().updateNodeLabels( + request.getNodeLabels()); + RMAuditLogger + .logSuccess(user.getShortUserName(), argName, "AdminService"); + return response; + } catch (YarnException ioe) { + throw logAndWrapException(ioe, user.getShortUserName(), argName, msg); + } + } private void checkRMStatus(String user, String argName, String msg) throws StandbyException { @@ -673,11 +697,11 @@ private void checkRMStatus(String user, String argName, String msg) } } - private YarnException logAndWrapException(IOException ioe, String user, + private YarnException logAndWrapException(Exception exception, String user, String argName, String msg) throws YarnException { - LOG.info("Exception " + msg, ioe); + LOG.warn("Exception " + msg, exception); RMAuditLogger.logFailure(user, argName, "", "AdminService", "Exception " + msg); - return RPCUtil.getRemoteException(ioe); + return RPCUtil.getRemoteException(exception); } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java index 8dcfe6799608c..91976302ce3b3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java @@ -390,7 +390,11 @@ private ResourceRequest validateAndCreateResourceRequest( + " for application " + submissionContext.getApplicationId(), e); throw e; } - + SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(), + scheduler.getClusterResource(), + scheduler.getMinimumResourceCapability(), + scheduler.getMaximumResourceCapability(), + scheduler.getMinimumResourceCapability()); return amReq; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java index 0de556bc1b7f4..22efe250d0ece 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java @@ -21,6 +21,9 @@ import java.io.InputStream; import java.net.InetSocketAddress; import java.nio.ByteBuffer; +import java.util.HashMap; +import java.util.Map; +import java.util.Set; import java.util.concurrent.ConcurrentMap; import org.apache.commons.logging.Log; @@ -31,6 +34,7 @@ import org.apache.hadoop.net.Node; import org.apache.hadoop.security.authorize.PolicyProvider; import org.apache.hadoop.service.AbstractService; +import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.util.VersionUtil; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; @@ -100,6 +104,8 @@ public class ResourceTrackerService extends AbstractService implements private int minAllocMb; private int minAllocVcores; + private boolean isDistributesNodeLabelsConf; + static { resync.setNodeAction(NodeAction.RESYNC); @@ -149,6 +155,14 @@ protected void serviceInit(Configuration conf) throws Exception { YarnConfiguration.RM_NODEMANAGER_MINIMUM_VERSION, YarnConfiguration.DEFAULT_RM_NODEMANAGER_MINIMUM_VERSION); + String nodeLabelConfigurationType = + conf.get(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DEFAULT_NODELABEL_CONFIGURATION_TYPE); + + isDistributesNodeLabelsConf = + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE + .equals(nodeLabelConfigurationType); + super.serviceInit(conf); } @@ -336,11 +350,31 @@ public RegisterNodeManagerResponse registerNodeManager( } } - String message = - "NodeManager from node " + host + "(cmPort: " + cmPort + " httpPort: " - + httpPort + ") " + "registered with capability: " + capability - + ", assigned nodeId " + nodeId; - LOG.info(message); + // Update node's labels to RM's NodeLabelManager. + Set nodeLabels = request.getNodeLabels(); + if (isDistributesNodeLabelsConf && nodeLabels != null) { + try { + updateNodeLabelsFromNMReport(nodeLabels, nodeId); + response.setAreNodeLabelsAcceptedByRM(true); + } catch (IOException ex) { + // Ensure the exception is captured in the response + response.setDiagnosticsMessage(ex.getMessage()); + response.setAreNodeLabelsAcceptedByRM(false); + } + } + + StringBuilder message = new StringBuilder(); + message.append("NodeManager from node ").append(host).append("(cmPort: ") + .append(cmPort).append(" httpPort: "); + message.append(httpPort).append(") ") + .append("registered with capability: ").append(capability); + message.append(", assigned nodeId ").append(nodeId); + if (response.getAreNodeLabelsAcceptedByRM()) { + message.append(", node labels { ").append( + StringUtils.join(",", nodeLabels) + " } "); + } + + LOG.info(message.toString()); response.setNodeAction(NodeAction.NORMAL); response.setRMIdentifier(ResourceManager.getClusterTimeStamp()); response.setRMVersion(YarnVersionInfo.getVersion()); @@ -359,6 +393,7 @@ public NodeHeartbeatResponse nodeHeartbeat(NodeHeartbeatRequest request) * 2. Check if it's a registered node * 3. Check if it's a 'fresh' heartbeat i.e. not duplicate heartbeat * 4. Send healthStatus to RMNode + * 5. Update node's labels if distributed Node Labels configuration is enabled */ NodeId nodeId = remoteNodeStatus.getNodeId(); @@ -428,9 +463,44 @@ public NodeHeartbeatResponse nodeHeartbeat(NodeHeartbeatRequest request) remoteNodeStatus.getContainersStatuses(), remoteNodeStatus.getKeepAliveApplications(), nodeHeartBeatResponse)); + // 5. Update node's labels to RM's NodeLabelManager. + if (isDistributesNodeLabelsConf && request.getNodeLabels() != null) { + try { + updateNodeLabelsFromNMReport(request.getNodeLabels(), nodeId); + nodeHeartBeatResponse.setAreNodeLabelsAcceptedByRM(true); + } catch (IOException ex) { + //ensure the error message is captured and sent across in response + nodeHeartBeatResponse.setDiagnosticsMessage(ex.getMessage()); + nodeHeartBeatResponse.setAreNodeLabelsAcceptedByRM(false); + } + } + return nodeHeartBeatResponse; } + private void updateNodeLabelsFromNMReport(Set nodeLabels, + NodeId nodeId) throws IOException { + try { + Map> labelsUpdate = + new HashMap>(); + labelsUpdate.put(nodeId, nodeLabels); + this.rmContext.getNodeLabelManager().replaceLabelsOnNode(labelsUpdate); + if (LOG.isDebugEnabled()) { + LOG.debug("Node Labels {" + StringUtils.join(",", nodeLabels) + + "} from Node " + nodeId + " were Accepted from RM"); + } + } catch (IOException ex) { + StringBuilder errorMessage = new StringBuilder(); + errorMessage.append("Node Labels {") + .append(StringUtils.join(",", nodeLabels)) + .append("} reported from NM with ID ").append(nodeId) + .append(" was rejected from RM with exception message as : ") + .append(ex.getMessage()); + LOG.error(errorMessage, ex); + throw new IOException(errorMessage.toString(), ex); + } + } + private void populateKeys(NodeHeartbeatRequest request, NodeHeartbeatResponse nodeHeartBeatResponse) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ContainerCreatedEvent.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ContainerCreatedEvent.java index eeda18199b55e..05b6781113826 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ContainerCreatedEvent.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/ContainerCreatedEvent.java @@ -29,18 +29,21 @@ public class ContainerCreatedEvent extends SystemMetricsEvent { private Resource allocatedResource; private NodeId allocatedNode; private Priority allocatedPriority; + private String nodeHttpAddress; public ContainerCreatedEvent( ContainerId containerId, Resource allocatedResource, NodeId allocatedNode, Priority allocatedPriority, - long createdTime) { + long createdTime, + String nodeHttpAddress) { super(SystemMetricsEventType.CONTAINER_CREATED, createdTime); this.containerId = containerId; this.allocatedResource = allocatedResource; this.allocatedNode = allocatedNode; this.allocatedPriority = allocatedPriority; + this.nodeHttpAddress = nodeHttpAddress; } @Override @@ -64,4 +67,7 @@ public Priority getAllocatedPriority() { return allocatedPriority; } + public String getNodeHttpAddress() { + return nodeHttpAddress; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java index 3adf519a6c3cd..b849b0035dba0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java @@ -181,7 +181,7 @@ public void containerCreated(RMContainer container, long createdTime) { container.getAllocatedResource(), container.getAllocatedNode(), container.getAllocatedPriority(), - createdTime)); + createdTime, container.getNodeHttpAddress())); } } @@ -388,6 +388,9 @@ private void publishContainerCreatedEvent(ContainerCreatedEvent event) { event.getAllocatedNode().getPort()); entityInfo.put(ContainerMetricsConstants.ALLOCATED_PRIORITY_ENTITY_INFO, event.getAllocatedPriority().getPriority()); + entityInfo.put( + ContainerMetricsConstants.ALLOCATED_HOST_HTTP_ADDRESS_ENTITY_INFO, + event.getNodeHttpAddress()); entity.setOtherInfo(entityInfo); TimelineEvent tEvent = new TimelineEvent(); tEvent.setEventType(ContainerMetricsConstants.CREATED_EVENT_TYPE); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java index 738f5272e2e45..87a2a00dd7391 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java @@ -527,6 +527,17 @@ private Map> getContainersToPreempt( List skippedAMContainerlist = new ArrayList(); for (TempQueue qT : queues) { + if (qT.preemptionDisabled && qT.leafQueue != null) { + if (LOG.isDebugEnabled()) { + if (Resources.greaterThan(rc, clusterResource, + qT.toBePreempted, Resource.newInstance(0, 0))) { + LOG.debug("Tried to preempt the following " + + "resources from non-preemptable queue: " + + qT.queueName + " - Resources: " + qT.toBePreempted); + } + } + continue; + } // we act only if we are violating balance by more than // maxIgnoredOverCapacity if (Resources.greaterThan(rc, clusterResource, qT.current, @@ -734,6 +745,7 @@ private TempQueue cloneQueues(CSQueue root, Resource clusterResources) { float absUsed = root.getAbsoluteUsedCapacity(); float absCap = root.getAbsoluteCapacity(); float absMaxCap = root.getAbsoluteMaximumCapacity(); + boolean preemptionDisabled = root.getPreemptionDisabled(); Resource current = Resources.multiply(clusterResources, absUsed); Resource guaranteed = Resources.multiply(clusterResources, absCap); @@ -747,8 +759,8 @@ private TempQueue cloneQueues(CSQueue root, Resource clusterResources) { LeafQueue l = (LeafQueue) root; Resource pending = l.getTotalResourcePending(); ret = new TempQueue(queueName, current, pending, guaranteed, - maxCapacity); - if (root.getPreemptionDisabled()) { + maxCapacity, preemptionDisabled); + if (preemptionDisabled) { ret.untouchableExtra = extra; } else { ret.preemptableExtra = extra; @@ -757,7 +769,7 @@ private TempQueue cloneQueues(CSQueue root, Resource clusterResources) { } else { Resource pending = Resource.newInstance(0, 0); ret = new TempQueue(root.getQueueName(), current, pending, guaranteed, - maxCapacity); + maxCapacity, false); Resource childrensPreemptable = Resource.newInstance(0, 0); for (CSQueue c : root.getChildQueues()) { TempQueue subq = cloneQueues(c, clusterResources); @@ -816,9 +828,10 @@ static class TempQueue { final ArrayList children; LeafQueue leafQueue; + boolean preemptionDisabled; TempQueue(String queueName, Resource current, Resource pending, - Resource guaranteed, Resource maxCapacity) { + Resource guaranteed, Resource maxCapacity, boolean preemptionDisabled) { this.queueName = queueName; this.current = current; this.pending = pending; @@ -831,6 +844,7 @@ static class TempQueue { this.children = new ArrayList(); this.untouchableExtra = Resource.newInstance(0, 0); this.preemptableExtra = Resource.newInstance(0, 0); + this.preemptionDisabled = preemptionDisabled; } public void setLeafQueue(LeafQueue l){ @@ -862,10 +876,13 @@ public ArrayList getChildren(){ // the unused ones Resource offer(Resource avail, ResourceCalculator rc, Resource clusterResource) { + Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax( + Resources.subtract(maxCapacity, idealAssigned), + Resource.newInstance(0, 0)); // remain = avail - min(avail, (max - assigned), (current + pending - assigned)) Resource accepted = Resources.min(rc, clusterResource, - Resources.subtract(maxCapacity, idealAssigned), + absMaxCapIdealAssignedDelta, Resources.min(rc, clusterResource, avail, Resources.subtract( Resources.add(current, pending), idealAssigned))); Resource remain = Resources.subtract(avail, accepted); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java index e5abdc99f9705..574e24c3512fd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/RMNodeLabelsManager.java @@ -36,7 +36,7 @@ import org.apache.hadoop.yarn.api.records.NodeId; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; -import org.apache.hadoop.yarn.nodelabels.NodeLabel; +import org.apache.hadoop.yarn.nodelabels.RMNodeLabel; import org.apache.hadoop.yarn.security.YarnAuthorizationProvider; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeLabelsUpdateSchedulerEvent; @@ -45,7 +45,6 @@ import com.google.common.collect.ImmutableSet; public class RMNodeLabelsManager extends CommonNodeLabelsManager { - protected static class Queue { protected Set acccessibleNodeLabels; protected Resource resource; @@ -156,7 +155,7 @@ public void replaceLabelsOnNode(Map> replaceLabelsToNode) throws IOException { try { writeLock.lock(); - + // get nodesCollection before edition Map before = cloneNodeMap(replaceLabelsToNode.keySet()); @@ -171,7 +170,6 @@ public void replaceLabelsOnNode(Map> replaceLabelsToNode) writeLock.unlock(); } } - /* * Following methods are used for setting if a node is up and running, and it @@ -201,7 +199,7 @@ public void activateNode(NodeId nodeId, Resource resource) { Set labelsForNode = getLabelsByNode(nodeId); if (labelsForNode != null) { for (String label : labelsForNode) { - NodeLabel labelInfo = labelCollections.get(label); + RMNodeLabel labelInfo = labelCollections.get(label); if(labelInfo != null) { labelInfo.addNodeId(nodeId); } @@ -383,7 +381,7 @@ private void updateResourceMappings(Map before, // no label in the past if (oldLabels.isEmpty()) { // update labels - NodeLabel label = labelCollections.get(NO_LABEL); + RMNodeLabel label = labelCollections.get(NO_LABEL); label.removeNode(oldNM.resource); // update queues, all queue can access this node @@ -393,7 +391,7 @@ private void updateResourceMappings(Map before, } else { // update labels for (String labelName : oldLabels) { - NodeLabel label = labelCollections.get(labelName); + RMNodeLabel label = labelCollections.get(labelName); if (null == label) { continue; } @@ -418,7 +416,7 @@ private void updateResourceMappings(Map before, // no label in the past if (newLabels.isEmpty()) { // update labels - NodeLabel label = labelCollections.get(NO_LABEL); + RMNodeLabel label = labelCollections.get(NO_LABEL); label.addNode(newNM.resource); // update queues, all queue can access this node @@ -428,7 +426,7 @@ private void updateResourceMappings(Map before, } else { // update labels for (String labelName : newLabels) { - NodeLabel label = labelCollections.get(labelName); + RMNodeLabel label = labelCollections.get(labelName); label.addNode(newNM.resource); } @@ -499,13 +497,13 @@ public void setRMContext(RMContext rmContext) { this.rmContext = rmContext; } - public List pullRMNodeLabelsInfo() { + public List pullRMNodeLabelsInfo() { try { readLock.lock(); - List infos = new ArrayList(); + List infos = new ArrayList(); - for (Entry entry : labelCollections.entrySet()) { - NodeLabel label = entry.getValue(); + for (Entry entry : labelCollections.entrySet()) { + RMNodeLabel label = entry.getValue(); infos.add(label.getCopy()); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java index 8147597dcad78..7652a070189fe 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java @@ -84,7 +84,10 @@ public class FileSystemRMStateStore extends RMStateStore { protected static final String AMRMTOKEN_SECRET_MANAGER_NODE = "AMRMTokenSecretManagerNode"; + @VisibleForTesting protected FileSystem fs; + @VisibleForTesting + protected Configuration fsConf; private Path rootDirPath; @Private @@ -121,14 +124,23 @@ protected synchronized void startInternal() throws Exception { // create filesystem only now, as part of service-start. By this time, RM is // authenticated with kerberos so we are good to create a file-system // handle. - Configuration conf = new Configuration(getConfig()); - conf.setBoolean("dfs.client.retry.policy.enabled", true); + fsConf = new Configuration(getConfig()); + fsConf.setBoolean("dfs.client.retry.policy.enabled", true); String retryPolicy = - conf.get(YarnConfiguration.FS_RM_STATE_STORE_RETRY_POLICY_SPEC, + fsConf.get(YarnConfiguration.FS_RM_STATE_STORE_RETRY_POLICY_SPEC, YarnConfiguration.DEFAULT_FS_RM_STATE_STORE_RETRY_POLICY_SPEC); - conf.set("dfs.client.retry.policy.spec", retryPolicy); + fsConf.set("dfs.client.retry.policy.spec", retryPolicy); + + String scheme = fsWorkingPath.toUri().getScheme(); + if (scheme == null) { + scheme = FileSystem.getDefaultUri(fsConf).getScheme(); + } + if (scheme != null) { + String disableCacheName = String.format("fs.%s.impl.disable.cache", scheme); + fsConf.setBoolean(disableCacheName, true); + } - fs = fsWorkingPath.getFileSystem(conf); + fs = fsWorkingPath.getFileSystem(fsConf); mkdirsWithRetries(rmDTSecretManagerRoot); mkdirsWithRetries(rmAppRoot); mkdirsWithRetries(amrmTokenSecretManagerRoot); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java index 591a5511785d1..614ef15e43ef9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java @@ -153,7 +153,13 @@ public class ZKRMStateStore extends RMStateStore { @VisibleForTesting protected ZooKeeper zkClient; - private ZooKeeper oldZkClient; + + /* activeZkClient is not used to do actual operations, + * it is only used to verify client session for watched events and + * it gets activated into zkClient on connection event. + */ + @VisibleForTesting + ZooKeeper activeZkClient; /** Fencing related variables */ private static final String FENCING_LOCK = "RM_ZK_FENCING_LOCK"; @@ -355,21 +361,14 @@ public Void run() throws KeeperException, InterruptedException { } private synchronized void closeZkClients() throws IOException { - if (zkClient != null) { + zkClient = null; + if (activeZkClient != null) { try { - zkClient.close(); + activeZkClient.close(); } catch (InterruptedException e) { throw new IOException("Interrupted while closing ZK", e); } - zkClient = null; - } - if (oldZkClient != null) { - try { - oldZkClient.close(); - } catch (InterruptedException e) { - throw new IOException("Interrupted while closing old ZK", e); - } - oldZkClient = null; + activeZkClient = null; } } @@ -830,11 +829,16 @@ public synchronized void deleteStore() throws Exception { * hides the ZK methods of the store from its public interface */ private final class ForwardingWatcher implements Watcher { + private ZooKeeper watchedZkClient; + + public ForwardingWatcher(ZooKeeper client) { + this.watchedZkClient = client; + } @Override public void process(WatchedEvent event) { try { - ZKRMStateStore.this.processWatchEvent(event); + ZKRMStateStore.this.processWatchEvent(watchedZkClient, event); } catch (Throwable t) { LOG.error("Failed to process watcher event " + event + ": " + StringUtils.stringifyException(t)); @@ -845,8 +849,16 @@ public void process(WatchedEvent event) { @VisibleForTesting @Private @Unstable - public synchronized void processWatchEvent(WatchedEvent event) - throws Exception { + public synchronized void processWatchEvent(ZooKeeper zk, + WatchedEvent event) throws Exception { + // only process watcher event from current ZooKeeper Client session. + if (zk != activeZkClient) { + LOG.info("Ignore watcher event type: " + event.getType() + + " with state:" + event.getState() + " for path:" + + event.getPath() + " from old session"); + return; + } + Event.EventType eventType = event.getType(); LOG.info("Watcher event type: " + eventType + " with state:" + event.getState() + " for path:" + event.getPath() + " for " + this); @@ -857,17 +869,15 @@ public synchronized void processWatchEvent(WatchedEvent event) switch (event.getState()) { case SyncConnected: LOG.info("ZKRMStateStore Session connected"); - if (oldZkClient != null) { + if (zkClient == null) { // the SyncConnected must be from the client that sent Disconnected - zkClient = oldZkClient; - oldZkClient = null; + zkClient = activeZkClient; ZKRMStateStore.this.notifyAll(); LOG.info("ZKRMStateStore Session restored"); } break; case Disconnected: LOG.info("ZKRMStateStore Session disconnected"); - oldZkClient = zkClient; zkClient = null; break; case Expired: @@ -1100,7 +1110,8 @@ private synchronized void createConnection() for (int retries = 0; retries < numRetries && zkClient == null; retries++) { try { - zkClient = getNewZooKeeper(); + activeZkClient = getNewZooKeeper(); + zkClient = activeZkClient; for (ZKUtil.ZKAuthInfo zkAuth : zkAuths) { zkClient.addAuthInfo(zkAuth.getScheme(), zkAuth.getAuth()); } @@ -1130,7 +1141,7 @@ private synchronized void createConnection() protected synchronized ZooKeeper getNewZooKeeper() throws IOException, InterruptedException { ZooKeeper zk = new ZooKeeper(zkHostPort, zkSessionTimeout, null); - zk.register(new ForwardingWatcher()); + zk.register(new ForwardingWatcher(zk)); return zk; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java index cf8c2bbc86576..b85174efcf6d2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java @@ -216,11 +216,14 @@ public interface RMAppAttempt extends EventHandler { /** * Return the flag which indicates whether the attempt failure should be * counted to attempt retry count. - *
        + *

        * There failure types should not be counted to attempt retry count: - *

      • preempted by the scheduler.
        • - *
      • hardware failures, such as NM failing, lost NM and NM disk errors.
        • - *
      • killed by RM because of RM restart or failover.
        • + *
          + *
        • preempted by the scheduler.
          • + *
        • + * hardware failures, such as NM failing, lost NM and NM disk errors. + *
          • + *
        • killed by RM because of RM restart or failover.
          • *
        */ boolean shouldCountTowardsMaxAttemptRetry(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java index 0e60fd5abbb9b..0a3638bf691cc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java @@ -32,6 +32,7 @@ import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType; import org.apache.hadoop.yarn.util.resource.Resources; public class RMAppAttemptMetrics { @@ -40,6 +41,8 @@ public class RMAppAttemptMetrics { private ApplicationAttemptId attemptId = null; // preemption info private Resource resourcePreempted = Resource.newInstance(0, 0); + // application headroom + private volatile Resource applicationHeadroom = Resource.newInstance(0, 0); private AtomicInteger numNonAMContainersPreempted = new AtomicInteger(0); private AtomicBoolean isPreempted = new AtomicBoolean(false); @@ -49,6 +52,10 @@ public class RMAppAttemptMetrics { private AtomicLong finishedVcoreSeconds = new AtomicLong(0); private RMContext rmContext; + private int[][] localityStatistics = + new int[NodeType.values().length][NodeType.values().length]; + private volatile int totalAllocatedContainers; + public RMAppAttemptMetrics(ApplicationAttemptId attemptId, RMContext rmContext) { this.attemptId = attemptId; @@ -57,7 +64,7 @@ public RMAppAttemptMetrics(ApplicationAttemptId attemptId, this.writeLock = lock.writeLock(); this.rmContext = rmContext; } - + public void updatePreemptionInfo(Resource resource, RMContainer container) { try { writeLock.lock(); @@ -126,4 +133,26 @@ public void updateAggregateAppResourceUsage(long finishedMemorySeconds, this.finishedMemorySeconds.addAndGet(finishedMemorySeconds); this.finishedVcoreSeconds.addAndGet(finishedVcoreSeconds); } + + public void incNumAllocatedContainers(NodeType containerType, + NodeType requestType) { + localityStatistics[containerType.index][requestType.index]++; + totalAllocatedContainers++; + } + + public int[][] getLocalityStatistics() { + return this.localityStatistics; + } + + public int getTotalAllocatedContainers() { + return this.totalAllocatedContainers; + } + + public Resource getApplicationAttemptHeadroom() { + return applicationHeadroom; + } + + public void setApplicationAttemptHeadRoom(Resource headRoom) { + this.applicationHeadroom = headRoom; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java index 9e9dcb9aa6f1b..20087f5fc0e25 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java @@ -79,4 +79,5 @@ public interface RMContainer extends EventHandler { List getResourceRequests(); + String getNodeHttpAddress(); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java index e37d8fd3c8f49..38a03aef14511 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java @@ -41,7 +41,6 @@ import org.apache.hadoop.yarn.event.EventHandler; import org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRunningOnNodeEvent; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.event.RMAppAttemptContainerAllocatedEvent; @@ -573,10 +572,29 @@ public ContainerReport createContainerReport() { this.getAllocatedResource(), this.getAllocatedNode(), this.getAllocatedPriority(), this.getCreationTime(), this.getFinishTime(), this.getDiagnosticsInfo(), this.getLogURL(), - this.getContainerExitStatus(), this.getContainerState()); + this.getContainerExitStatus(), this.getContainerState(), + this.getNodeHttpAddress()); } finally { this.readLock.unlock(); } return containerReport; } + + @Override + public String getNodeHttpAddress() { + try { + readLock.lock(); + if (container.getNodeHttpAddress() != null) { + StringBuilder httpAddress = new StringBuilder(); + httpAddress.append(WebAppUtils.getHttpSchemePrefix(rmContext + .getYarnConfiguration())); + httpAddress.append(container.getNodeHttpAddress()); + return httpAddress.toString(); + } else { + return null; + } + } finally { + readLock.unlock(); + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java index 9701775994253..c556b80469a0e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java @@ -571,12 +571,12 @@ public void transition(RMNodeImpl rmNode, RMNodeEvent event) { rmNode.nodeUpdateQueue.clear(); rmNode.context.getDispatcher().getEventHandler().handle( new NodeRemovedSchedulerEvent(rmNode)); - + if (rmNode.getHttpPort() == newNode.getHttpPort()) { // Reset heartbeat ID since node just restarted. rmNode.getLastNodeHeartBeatResponse().setResponseId(0); - if (rmNode.getState() != NodeState.UNHEALTHY) { - // Only add new node if old state is not UNHEALTHY + if (rmNode.getState().equals(NodeState.RUNNING)) { + // Only add new node if old state is RUNNING rmNode.context.getDispatcher().getEventHandler().handle( new NodeAddedSchedulerEvent(newNode)); } @@ -599,30 +599,32 @@ public void transition(RMNodeImpl rmNode, RMNodeEvent event) { } else { rmNode.httpPort = newNode.getHttpPort(); rmNode.httpAddress = newNode.getHttpAddress(); - rmNode.totalCapability = newNode.getTotalCapability(); + boolean isCapabilityChanged = false; + if (rmNode.getTotalCapability() != newNode.getTotalCapability()) { + rmNode.totalCapability = newNode.getTotalCapability(); + isCapabilityChanged = true; + } handleNMContainerStatus(reconnectEvent.getNMContainerStatuses(), rmNode); // Reset heartbeat ID since node just restarted. rmNode.getLastNodeHeartBeatResponse().setResponseId(0); - } - if (null != reconnectEvent.getRunningApplications()) { for (ApplicationId appId : reconnectEvent.getRunningApplications()) { handleRunningAppOnNode(rmNode, rmNode.context, appId, rmNode.nodeId); } - } - rmNode.context.getDispatcher().getEventHandler().handle( - new NodesListManagerEvent( - NodesListManagerEventType.NODE_USABLE, rmNode)); - if (rmNode.getState().equals(NodeState.RUNNING)) { - // Update scheduler node's capacity for reconnect node. - rmNode.context.getDispatcher().getEventHandler().handle( - new NodeResourceUpdateSchedulerEvent(rmNode, - ResourceOption.newInstance(newNode.getTotalCapability(), -1))); + if (isCapabilityChanged + && rmNode.getState().equals(NodeState.RUNNING)) { + // Update scheduler node's capacity for reconnect node. + rmNode.context + .getDispatcher() + .getEventHandler() + .handle( + new NodeResourceUpdateSchedulerEvent(rmNode, ResourceOption + .newInstance(newNode.getTotalCapability(), -1))); + } } - } private void handleNMContainerStatus( diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java index 968a767f517dd..e1f94cf40c195 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java @@ -358,14 +358,15 @@ public synchronized void recoverContainersOnNode( container)); // recover scheduler node - nodes.get(nm.getNodeID()).recoverContainer(rmContainer); + SchedulerNode schedulerNode = nodes.get(nm.getNodeID()); + schedulerNode.recoverContainer(rmContainer); // recover queue: update headroom etc. Queue queue = schedulerAttempt.getQueue(); queue.recoverContainer(clusterResource, schedulerAttempt, rmContainer); // recover scheduler attempt - schedulerAttempt.recoverContainer(rmContainer); + schedulerAttempt.recoverContainer(schedulerNode, rmContainer); // set master container for the current running AMContainer for this // attempt. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java index 97dc231815d11..84ebe9cf70b90 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java @@ -20,8 +20,6 @@ import java.util.ArrayList; import java.util.Collection; -import java.util.Collections; -import java.util.HashMap; import java.util.HashSet; import java.util.List; import java.util.Map; @@ -191,6 +189,16 @@ synchronized public void updateResourceRequests( request.getCapability()); metrics.decrPendingResources(user, lastRequestContainers, lastRequestCapability); + + // update queue: + queue.incPendingResource( + request.getNodeLabelExpression(), + Resources.multiply(request.getCapability(), + request.getNumContainers())); + if (lastRequest != null) { + queue.decPendingResource(lastRequest.getNodeLabelExpression(), + Resources.multiply(lastRequestCapability, lastRequestContainers)); + } } } } @@ -239,7 +247,7 @@ synchronized public ResourceRequest getResourceRequest(Priority priority, public synchronized Resource getResource(Priority priority) { ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); - return request.getCapability(); + return (request == null) ? null : request.getCapability(); } public synchronized boolean isBlacklisted(String resourceName) { @@ -376,15 +384,20 @@ synchronized private void decrementOutstanding( if (numOffSwitchContainers == 0) { checkForDeactivation(); } + + queue.decPendingResource(offSwitchRequest.getNodeLabelExpression(), + offSwitchRequest.getCapability()); } synchronized private void checkForDeactivation() { boolean deactivate = true; for (Priority priority : getPriorities()) { ResourceRequest request = getResourceRequest(priority, ResourceRequest.ANY); - if (request.getNumContainers() > 0) { - deactivate = false; - break; + if (request != null) { + if (request.getNumContainers() > 0) { + deactivate = false; + break; + } } } if (deactivate) { @@ -402,6 +415,12 @@ synchronized public void move(Queue newQueue) { request.getCapability()); newMetrics.incrPendingResources(user, request.getNumContainers(), request.getCapability()); + + Resource delta = Resources.multiply(request.getCapability(), + request.getNumContainers()); + // Update Queue + queue.decPendingResource(request.getNodeLabelExpression(), delta); + newQueue.incPendingResource(request.getNodeLabelExpression(), delta); } } oldMetrics.moveAppFrom(this); @@ -421,6 +440,12 @@ synchronized public void stop(RMAppAttemptState rmAppAttemptFinalState) { if (request != null) { metrics.decrPendingResources(user, request.getNumContainers(), request.getCapability()); + + // Update Queue + queue.decPendingResource( + request.getNodeLabelExpression(), + Resources.multiply(request.getCapability(), + request.getNumContainers())); } } metrics.finishAppAttempt(applicationId, pending, user); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java index 821ec2411a441..2b193bbb86f83 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java @@ -22,7 +22,10 @@ * Resource classification. */ public enum NodeType { - NODE_LOCAL, - RACK_LOCAL, - OFF_SWITCH + NODE_LOCAL(0), RACK_LOCAL(1), OFF_SWITCH(2); + public int index; + + private NodeType(int index) { + this.index = index; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Queue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Queue.java index 4663a914c5946..02003c145aec0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Queue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Queue.java @@ -90,4 +90,24 @@ public void recoverContainer(Resource clusterResource, * @return default label expression */ public String getDefaultNodeLabelExpression(); + + /** + * When new outstanding resource is asked, calling this will increase pending + * resource in a queue. + * + * @param nodeLabel asked by application + * @param resourceToInc new resource asked + */ + public void incPendingResource(String nodeLabel, Resource resourceToInc); + + /** + * When an outstanding resource is fulfilled or canceled, calling this will + * decrease pending resource in a queue. + * + * @param nodeLabel + * asked by application + * @param resourceToDec + * new resource asked + */ + public void decPendingResource(String nodeLabel, Resource resourceToDec); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java index 507b798a56280..58b1ed1151374 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java @@ -38,6 +38,7 @@ import org.apache.hadoop.metrics2.lib.MutableCounterInt; import org.apache.hadoop.metrics2.lib.MutableCounterLong; import org.apache.hadoop.metrics2.lib.MutableGaugeInt; +import org.apache.hadoop.metrics2.lib.MutableRate; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; @@ -74,6 +75,7 @@ public class QueueMetrics implements MetricsSource { @Metric("# of reserved containers") MutableGaugeInt reservedContainers; @Metric("# of active users") MutableGaugeInt activeUsers; @Metric("# of active applications") MutableGaugeInt activeApplications; + @Metric("App Attempt First Container Allocation Delay") MutableRate appAttemptFirstContainerAllocationDelay; private final MutableGaugeInt[] runningTime; private TimeBucketMetrics runBuckets; @@ -462,7 +464,11 @@ public void deactivateApp(String user) { parent.deactivateApp(user); } } - + + public void addAppAttemptFirstContainerAllocationDelay(long latency) { + appAttemptFirstContainerAllocationDelay.add(latency); + } + public int getAppsSubmitted() { return appsSubmitted.value(); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java index de44bbe497626..36ee4daa1edbc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java @@ -20,6 +20,7 @@ import java.util.HashMap; import java.util.Map; +import java.util.Map.Entry; import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock; import java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock; @@ -75,14 +76,17 @@ public UsageByLabel(String label) { }; } + public Resource getUsed() { + return resArr[ResourceType.USED.idx]; + } + @Override public String toString() { StringBuilder sb = new StringBuilder(); sb.append("{used=" + resArr[0] + "%, "); sb.append("pending=" + resArr[1] + "%, "); sb.append("am_used=" + resArr[2] + "%, "); - sb.append("reserved=" + resArr[3] + "%, "); - sb.append("headroom=" + resArr[4] + "%}"); + sb.append("reserved=" + resArr[3] + "%}"); return sb.toString(); } } @@ -117,6 +121,17 @@ public void decUsed(String label, Resource res) { public void setUsed(Resource res) { setUsed(NL, res); } + + public void copyAllUsed(ResourceUsage other) { + try { + writeLock.lock(); + for (Entry entry : other.usages.entrySet()) { + setUsed(entry.getKey(), Resources.clone(entry.getValue().getUsed())); + } + } finally { + writeLock.unlock(); + } + } public void setUsed(String label, Resource res) { _set(label, ResourceType.USED, res); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java index 532df05d592ba..bf5641d1c3aad 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java @@ -25,6 +25,7 @@ import java.util.List; import java.util.Map; import java.util.Set; +import java.util.concurrent.atomic.AtomicLong; import org.apache.commons.lang.time.DateUtils; import org.apache.commons.logging.Log; @@ -46,6 +47,7 @@ import org.apache.hadoop.yarn.api.records.ResourceRequest; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.AggregateAppResourceUsage; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptState; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerEvent; @@ -78,7 +80,7 @@ public class SchedulerApplicationAttempt { private long lastVcoreSeconds = 0; protected final AppSchedulingInfo appSchedulingInfo; - + protected ApplicationAttemptId attemptId; protected Map liveContainers = new HashMap(); protected final Map> reservedContainers = @@ -86,13 +88,14 @@ public class SchedulerApplicationAttempt { private final Multiset reReservations = HashMultiset.create(); - protected final Resource currentReservation = Resource.newInstance(0, 0); private Resource resourceLimit = Resource.newInstance(0, 0); - protected Resource currentConsumption = Resource.newInstance(0, 0); - private Resource amResource = Resources.none(); private boolean unmanagedAM = true; private boolean amRunning = false; private LogAggregationContext logAggregationContext; + + protected ResourceUsage attemptResourceUsage = new ResourceUsage(); + private AtomicLong firstAllocationRequestSentTime = new AtomicLong(0); + private AtomicLong firstContainerAllocatedTime = new AtomicLong(0); protected List newlyAllocatedContainers = new ArrayList(); @@ -132,6 +135,7 @@ public SchedulerApplicationAttempt(ApplicationAttemptId applicationAttemptId, activeUsersManager, rmContext.getEpoch()); this.queue = queue; this.pendingRelease = new HashSet(); + this.attemptId = applicationAttemptId; if (rmContext.getRMApps() != null && rmContext.getRMApps() .containsKey(applicationAttemptId.getApplicationId())) { @@ -215,11 +219,11 @@ public String getQueueName() { } public Resource getAMResource() { - return amResource; + return attemptResourceUsage.getAMUsed(); } public void setAMResource(Resource amResource) { - this.amResource = amResource; + attemptResourceUsage.setAMUsed(amResource); } public boolean isAmRunning() { @@ -258,7 +262,7 @@ public synchronized int getReReservations(Priority priority) { @Stable @Private public synchronized Resource getCurrentReservation() { - return currentReservation; + return attemptResourceUsage.getReserved(); } public Queue getQueue() { @@ -309,8 +313,8 @@ public synchronized RMContainer reserve(SchedulerNode node, Priority priority, rmContainer = new RMContainerImpl(container, getApplicationAttemptId(), node.getNodeID(), appSchedulingInfo.getUser(), rmContext); - - Resources.addTo(currentReservation, container.getResource()); + attemptResourceUsage.incReserved(node.getPartition(), + container.getResource()); // Reset the re-reservation count resetReReservations(priority); @@ -334,7 +338,7 @@ public synchronized RMContainer reserve(SchedulerNode node, Priority priority, + " reserved container " + rmContainer + " on node " + node + ". This attempt currently has " + reservedContainers.size() + " reserved containers at priority " + priority - + "; currentReservation " + currentReservation.getMemory()); + + "; currentReservation " + container.getResource()); } return rmContainer; @@ -400,9 +404,9 @@ public synchronized void showRequests() { for (Priority priority : getPriorities()) { Map requests = getResourceRequests(priority); if (requests != null) { - LOG.debug("showRequests:" + " application=" + getApplicationId() + - " headRoom=" + getHeadroom() + - " currentConsumption=" + currentConsumption.getMemory()); + LOG.debug("showRequests:" + " application=" + getApplicationId() + + " headRoom=" + getHeadroom() + " currentConsumption=" + + attemptResourceUsage.getUsed().getMemory()); for (ResourceRequest request : requests.values()) { LOG.debug("showRequests:" + " application=" + getApplicationId() + " request=" + request); @@ -413,7 +417,7 @@ public synchronized void showRequests() { } public Resource getCurrentConsumption() { - return currentConsumption; + return attemptResourceUsage.getUsed(); } public static class ContainersAndNMTokensAllocation { @@ -546,12 +550,17 @@ synchronized AggregateAppResourceUsage getRunningAggregateAppResourceUsage() { } public synchronized ApplicationResourceUsageReport getResourceUsageReport() { - AggregateAppResourceUsage resUsage = getRunningAggregateAppResourceUsage(); + AggregateAppResourceUsage runningResourceUsage = + getRunningAggregateAppResourceUsage(); + Resource usedResourceClone = + Resources.clone(attemptResourceUsage.getUsed()); + Resource reservedResourceClone = + Resources.clone(attemptResourceUsage.getReserved()); return ApplicationResourceUsageReport.newInstance(liveContainers.size(), - reservedContainers.size(), Resources.clone(currentConsumption), - Resources.clone(currentReservation), - Resources.add(currentConsumption, currentReservation), - resUsage.getMemorySeconds(), resUsage.getVcoreSeconds()); + reservedContainers.size(), usedResourceClone, reservedResourceClone, + Resources.add(usedResourceClone, reservedResourceClone), + runningResourceUsage.getMemorySeconds(), + runningResourceUsage.getVcoreSeconds()); } public synchronized Map getLiveContainersMap() { @@ -570,7 +579,7 @@ public synchronized void transferStateFromPreviousAttempt( SchedulerApplicationAttempt appAttempt) { this.liveContainers = appAttempt.getLiveContainersMap(); // this.reReservations = appAttempt.reReservations; - this.currentConsumption = appAttempt.getCurrentConsumption(); + this.attemptResourceUsage.copyAllUsed(appAttempt.attemptResourceUsage); this.resourceLimit = appAttempt.getResourceLimit(); // this.currentReservation = appAttempt.currentReservation; // this.newlyAllocatedContainers = appAttempt.newlyAllocatedContainers; @@ -601,7 +610,8 @@ public synchronized void move(Queue newQueue) { this.queue = newQueue; } - public synchronized void recoverContainer(RMContainer rmContainer) { + public synchronized void recoverContainer(SchedulerNode node, + RMContainer rmContainer) { // recover app scheduling info appSchedulingInfo.recoverContainer(rmContainer); @@ -611,12 +621,48 @@ public synchronized void recoverContainer(RMContainer rmContainer) { LOG.info("SchedulerAttempt " + getApplicationAttemptId() + " is recovering container " + rmContainer.getContainerId()); liveContainers.put(rmContainer.getContainerId(), rmContainer); - Resources.addTo(currentConsumption, rmContainer.getContainer() - .getResource()); + attemptResourceUsage.incUsed(node.getPartition(), rmContainer + .getContainer().getResource()); + // resourceLimit: updated when LeafQueue#recoverContainer#allocateResource // is called. // newlyAllocatedContainers.add(rmContainer); // schedulingOpportunities // lastScheduledContainer } + + public void incNumAllocatedContainers(NodeType containerType, + NodeType requestType) { + RMAppAttempt attempt = + rmContext.getRMApps().get(attemptId.getApplicationId()) + .getCurrentAppAttempt(); + if (attempt != null) { + attempt.getRMAppAttemptMetrics().incNumAllocatedContainers(containerType, + requestType); + } + } + + public void setApplicationHeadroomForMetrics(Resource headroom) { + RMAppAttempt attempt = + rmContext.getRMApps().get(attemptId.getApplicationId()) + .getCurrentAppAttempt(); + if (attempt != null) { + attempt.getRMAppAttemptMetrics().setApplicationAttemptHeadRoom( + Resources.clone(headroom)); + } + } + + public void recordContainerRequestTime(long value) { + firstAllocationRequestSentTime.compareAndSet(0, value); + } + + public void recordContainerAllocationTime(long value) { + if (firstContainerAllocatedTime.compareAndSet(0, value)) { + long timediff = firstContainerAllocatedTime.longValue() - + firstAllocationRequestSentTime.longValue(); + if (timediff > 0) { + queue.getMetrics().addAppAttemptFirstContainerAllocationDelay(timediff); + } + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java index 957e8f650a402..f03663a832b75 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java @@ -35,6 +35,7 @@ import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; +import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerState; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; @@ -114,7 +115,7 @@ public String getHttpAddress() { /** * Get the name of the node for scheduling matching decisions. - *

        + *

        * Typically this is the 'hostname' reported by the node, but it could be * configured to be 'hostname:port' reported by the node via the * {@link YarnConfiguration#RM_SCHEDULER_INCLUDE_PORT_IN_NODE_NAME} constant. @@ -294,4 +295,17 @@ public Set getLabels() { public void updateLabels(Set labels) { this.labels = labels; } + + /** + * Get partition of which the node belongs to, if node-labels of this node is + * empty or null, it belongs to NO_LABEL partition. And since we only support + * one partition for each node (YARN-2694), first label will be its partition. + */ + public String getPartition() { + if (this.labels == null || this.labels.isEmpty()) { + return RMNodeLabelsManager.NO_LABEL; + } else { + return this.labels.iterator().next(); + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java index 65d68598aead0..248cc08b74853 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java @@ -194,8 +194,7 @@ public static void normalizeRequest( * Utility method to validate a resource request, by insuring that the * requested memory/vcore is non-negative and not greater than max * - * @throws InvalidResourceRequestException when there is invalid - * request + * @throws InvalidResourceRequestException when there is invalid request */ public static void validateResourceRequest(ResourceRequest resReq, Resource maximumResource, String queueName, YarnScheduler scheduler) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java index d8007097a2084..3cd85ae42f8d7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java @@ -20,10 +20,13 @@ import java.io.IOException; import java.util.HashMap; +import java.util.HashSet; import java.util.Map; import java.util.Set; import org.apache.commons.lang.StringUtils; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.security.UserGroupInformation; import org.apache.hadoop.security.authorize.AccessControlList; @@ -34,6 +37,7 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.factories.RecordFactory; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; +import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; import org.apache.hadoop.yarn.security.AccessType; import org.apache.hadoop.yarn.security.PrivilegedEntity; import org.apache.hadoop.yarn.security.PrivilegedEntity.EntityType; @@ -49,6 +53,7 @@ import com.google.common.collect.Sets; public abstract class AbstractCSQueue implements CSQueue { + private static final Log LOG = LogFactory.getLog(AbstractCSQueue.class); CSQueue parent; final String queueName; @@ -406,21 +411,126 @@ private boolean isQueueHierarchyPreemptionDisabled(CSQueue q) { parentQ.getPreemptionDisabled()); } - protected Resource getCurrentResourceLimit(Resource clusterResource, - ResourceLimits currentResourceLimits) { + private Resource getCurrentLimitResource(String nodeLabel, + Resource clusterResource, ResourceLimits currentResourceLimits) { /* - * Queue's max available resource = min(my.max, my.limit) - * my.limit is set by my parent, considered used resource of my siblings + * Current limit resource: For labeled resource: limit = queue-max-resource + * (TODO, this part need update when we support labeled-limit) For + * non-labeled resource: limit = min(queue-max-resource, + * limit-set-by-parent) */ Resource queueMaxResource = - Resources.multiplyAndNormalizeDown(resourceCalculator, clusterResource, - queueCapacities.getAbsoluteMaximumCapacity(), minimumAllocation); - Resource queueCurrentResourceLimit = - Resources.min(resourceCalculator, clusterResource, queueMaxResource, - currentResourceLimits.getLimit()); - queueCurrentResourceLimit = - Resources.roundDown(resourceCalculator, queueCurrentResourceLimit, - minimumAllocation); - return queueCurrentResourceLimit; + Resources.multiplyAndNormalizeDown(resourceCalculator, + labelManager.getResourceByLabel(nodeLabel, clusterResource), + queueCapacities.getAbsoluteMaximumCapacity(nodeLabel), minimumAllocation); + if (nodeLabel.equals(RMNodeLabelsManager.NO_LABEL)) { + return Resources.min(resourceCalculator, clusterResource, + queueMaxResource, currentResourceLimits.getLimit()); + } + return queueMaxResource; + } + + synchronized boolean canAssignToThisQueue(Resource clusterResource, + Set nodeLabels, ResourceLimits currentResourceLimits, + Resource nowRequired, Resource resourceCouldBeUnreserved) { + // Get label of this queue can access, it's (nodeLabel AND queueLabel) + Set labelCanAccess; + if (null == nodeLabels || nodeLabels.isEmpty()) { + labelCanAccess = new HashSet(); + // Any queue can always access any node without label + labelCanAccess.add(RMNodeLabelsManager.NO_LABEL); + } else { + labelCanAccess = new HashSet( + accessibleLabels.contains(CommonNodeLabelsManager.ANY) ? nodeLabels + : Sets.intersection(accessibleLabels, nodeLabels)); + } + + for (String label : labelCanAccess) { + // New total resource = used + required + Resource newTotalResource = + Resources.add(queueUsage.getUsed(label), nowRequired); + + Resource currentLimitResource = + getCurrentLimitResource(label, clusterResource, currentResourceLimits); + + // if reservation continous looking enabled, check to see if could we + // potentially use this node instead of a reserved node if the application + // has reserved containers. + // TODO, now only consider reservation cases when the node has no label + if (this.reservationsContinueLooking + && label.equals(RMNodeLabelsManager.NO_LABEL) + && Resources.greaterThan(resourceCalculator, clusterResource, + resourceCouldBeUnreserved, Resources.none())) { + // resource-without-reserved = used - reserved + Resource newTotalWithoutReservedResource = + Resources.subtract(newTotalResource, resourceCouldBeUnreserved); + + // when total-used-without-reserved-resource < currentLimit, we still + // have chance to allocate on this node by unreserving some containers + if (Resources.lessThan(resourceCalculator, clusterResource, + newTotalWithoutReservedResource, currentLimitResource)) { + if (LOG.isDebugEnabled()) { + LOG.debug("try to use reserved: " + getQueueName() + + " usedResources: " + queueUsage.getUsed() + + ", clusterResources: " + clusterResource + + ", reservedResources: " + resourceCouldBeUnreserved + + ", capacity-without-reserved: " + + newTotalWithoutReservedResource + ", maxLimitCapacity: " + + currentLimitResource); + } + return true; + } + } + + // Otherwise, if any of the label of this node beyond queue limit, we + // cannot allocate on this node. Consider a small epsilon here. + if (Resources.greaterThan(resourceCalculator, clusterResource, + newTotalResource, currentLimitResource)) { + return false; + } + + if (LOG.isDebugEnabled()) { + LOG.debug(getQueueName() + + "Check assign to queue, label=" + label + + " usedResources: " + queueUsage.getUsed(label) + + " clusterResources: " + clusterResource + + " currentUsedCapacity " + + Resources.divide(resourceCalculator, clusterResource, + queueUsage.getUsed(label), + labelManager.getResourceByLabel(label, clusterResource)) + + " max-capacity: " + + queueCapacities.getAbsoluteMaximumCapacity(label) + + ")"); + } + return true; + } + + // Actually, this will not happen, since labelCanAccess will be always + // non-empty + return false; + } + + @Override + public void incPendingResource(String nodeLabel, Resource resourceToInc) { + if (nodeLabel == null) { + nodeLabel = RMNodeLabelsManager.NO_LABEL; + } + // ResourceUsage has its own lock, no addition lock needs here. + queueUsage.incPending(nodeLabel, resourceToInc); + if (null != parent) { + parent.incPendingResource(nodeLabel, resourceToInc); + } + } + + @Override + public void decPendingResource(String nodeLabel, Resource resourceToDec) { + if (nodeLabel == null) { + nodeLabel = RMNodeLabelsManager.NO_LABEL; + } + // ResourceUsage has its own lock, no addition lock needs here. + queueUsage.decPending(nodeLabel, resourceToDec); + if (null != parent) { + parent.decPendingResource(nodeLabel, resourceToDec); + } } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java index 0a60acc9fdc13..1a9448acaa148 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java @@ -189,13 +189,11 @@ public void finishApplicationAttempt(FiCaSchedulerApp application, * Assign containers to applications in the queue or it's children (if any). * @param clusterResource the resource of the cluster. * @param node node on which resources are available - * @param needToUnreserve assign container only if it can unreserve one first * @param resourceLimits how much overall resource of this queue can use. * @return the assignment */ public CSAssignment assignContainers(Resource clusterResource, - FiCaSchedulerNode node, boolean needToUnreserve, - ResourceLimits resourceLimits); + FiCaSchedulerNode node, ResourceLimits resourceLimits); /** * A container assigned to the queue has completed. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java index 28ce26448e221..c86c0ff0cd311 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java @@ -1061,9 +1061,14 @@ private synchronized void allocateContainersToNode(FiCaSchedulerNode node) { node.getNodeID()); LeafQueue queue = ((LeafQueue)reservedApplication.getQueue()); - CSAssignment assignment = queue.assignContainers(clusterResource, node, - false, new ResourceLimits( - clusterResource)); + CSAssignment assignment = + queue.assignContainers( + clusterResource, + node, + // TODO, now we only consider limits for parent for non-labeled + // resources, should consider labeled resources as well. + new ResourceLimits(labelManager.getResourceByLabel( + RMNodeLabelsManager.NO_LABEL, clusterResource))); RMContainer excessReservation = assignment.getExcessReservation(); if (excessReservation != null) { @@ -1087,8 +1092,13 @@ false, new ResourceLimits( LOG.debug("Trying to schedule on node: " + node.getNodeName() + ", available: " + node.getAvailableResource()); } - root.assignContainers(clusterResource, node, false, new ResourceLimits( - clusterResource)); + root.assignContainers( + clusterResource, + node, + // TODO, now we only consider limits for parent for non-labeled + // resources, should consider labeled resources as well. + new ResourceLimits(labelManager.getResourceByLabel( + RMNodeLabelsManager.NO_LABEL, clusterResource))); } } else { LOG.info("Skipping scheduling since node " + node.getNodeID() + @@ -1209,6 +1219,13 @@ private synchronized void addNode(RMNode nodeManager) { usePortForNodeName, nodeManager.getNodeLabels()); this.nodes.put(nodeManager.getNodeID(), schedulerNode); Resources.addTo(clusterResource, nodeManager.getTotalCapability()); + + // update this node to node label manager + if (labelManager != null) { + labelManager.activateNode(nodeManager.getNodeID(), + nodeManager.getTotalCapability()); + } + root.updateClusterResource(clusterResource, new ResourceLimits( clusterResource)); int numNodes = numNodeManagers.incrementAndGet(); @@ -1220,12 +1237,6 @@ private synchronized void addNode(RMNode nodeManager) { if (scheduleAsynchronously && numNodes == 1) { asyncSchedulerThread.beginSchedule(); } - - // update this node to node label manager - if (labelManager != null) { - labelManager.activateNode(nodeManager.getNodeID(), - nodeManager.getTotalCapability()); - } } private synchronized void removeNode(RMNode nodeInfo) { @@ -1279,7 +1290,8 @@ private synchronized void removeNode(RMNode nodeInfo) { protected synchronized void completedContainer(RMContainer rmContainer, ContainerStatus containerStatus, RMContainerEventType event) { if (rmContainer == null) { - LOG.info("Null container completed..."); + LOG.info("Container " + containerStatus.getContainerId() + + " completed with event " + event); return; } @@ -1291,7 +1303,7 @@ protected synchronized void completedContainer(RMContainer rmContainer, ApplicationId appId = container.getId().getApplicationAttemptId().getApplicationId(); if (application == null) { - LOG.info("Container " + container + " of" + " unknown application " + LOG.info("Container " + container + " of" + " finished application " + appId + " completed with event " + event); return; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java index 3910ac87530dd..3e5405dd5ed02 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java @@ -32,6 +32,7 @@ import java.util.TreeSet; import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang.mutable.MutableObject; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience.Private; @@ -75,7 +76,6 @@ import org.apache.hadoop.yarn.util.resource.Resources; import com.google.common.annotations.VisibleForTesting; -import com.google.common.collect.Sets; @Private @Unstable @@ -156,7 +156,7 @@ protected synchronized void setupQueueConfigs(Resource clusterResource) // and all queues may not be realized yet, we'll use (optimistic) // absoluteMaxCapacity (it will be replaced with the more accurate // absoluteMaxAvailCapacity during headroom/userlimit/allocation events) - computeQueueCurrentLimitAndSetHeadroomInfo(clusterResource); + setQueueResourceLimitsInfo(clusterResource); CapacitySchedulerConfiguration conf = csContext.getConfiguration(); userLimit = conf.getUserLimit(getQueuePath()); @@ -419,10 +419,13 @@ public synchronized User getUser(String userName) { */ public synchronized ArrayList getUsers() { ArrayList usersToReturn = new ArrayList(); - for (Map.Entry entry: users.entrySet()) { - usersToReturn.add(new UserInfo(entry.getKey(), Resources.clone( - entry.getValue().getUsed()), entry.getValue().getActiveApplications(), - entry.getValue().getPendingApplications())); + for (Map.Entry entry : users.entrySet()) { + User user = entry.getValue(); + usersToReturn.add(new UserInfo(entry.getKey(), Resources.clone(user + .getUsed()), user.getActiveApplications(), user + .getPendingApplications(), Resources.clone(user + .getConsumedAMResources()), Resources.clone(user + .getUserResourceLimit()))); } return usersToReturn; } @@ -736,11 +739,19 @@ private static Set getRequestLabelSetByExpression( return labels; } + private boolean checkResourceRequestMatchingNodeLabel(ResourceRequest offswitchResourceRequest, + FiCaSchedulerNode node) { + String askedNodeLabel = offswitchResourceRequest.getNodeLabelExpression(); + if (null == askedNodeLabel) { + askedNodeLabel = RMNodeLabelsManager.NO_LABEL; + } + return askedNodeLabel.equals(node.getPartition()); + } + @Override public synchronized CSAssignment assignContainers(Resource clusterResource, - FiCaSchedulerNode node, boolean needToUnreserve, - ResourceLimits currentResourceLimits) { - this.currentResourceLimits = currentResourceLimits; + FiCaSchedulerNode node, ResourceLimits currentResourceLimits) { + updateCurrentResourceLimits(currentResourceLimits, clusterResource); if(LOG.isDebugEnabled()) { LOG.debug("assignContainers: node=" + node.getNodeName() @@ -794,8 +805,16 @@ public synchronized CSAssignment assignContainers(Resource clusterResource, if (application.getTotalRequiredResources(priority) <= 0) { continue; } + + // Is the node-label-expression of this offswitch resource request + // matches the node's label? + // If not match, jump to next priority. + if (!checkResourceRequestMatchingNodeLabel(anyRequest, node)) { + continue; + } + if (!this.reservationsContinueLooking) { - if (!needContainers(application, priority, required)) { + if (!shouldAllocOrReserveNewContainer(application, priority, required)) { if (LOG.isDebugEnabled()) { LOG.debug("doesn't need containers based on reservation algo!"); } @@ -817,13 +836,13 @@ public synchronized CSAssignment assignContainers(Resource clusterResource, required, requestedNodeLabels); // Check queue max-capacity limit - if (!canAssignToThisQueue(clusterResource, required, - node.getLabels(), application, true)) { + if (!super.canAssignToThisQueue(clusterResource, node.getLabels(), + this.currentResourceLimits, required, application.getCurrentReservation())) { return NULL_ASSIGNMENT; } // Check user limit - if (!assignToUser(clusterResource, application.getUser(), userLimit, + if (!canAssignToUser(clusterResource, application.getUser(), userLimit, application, true, requestedNodeLabels)) { break; } @@ -834,7 +853,7 @@ public synchronized CSAssignment assignContainers(Resource clusterResource, // Try to schedule CSAssignment assignment = assignContainersOnNode(clusterResource, node, application, priority, - null, needToUnreserve); + null); // Did the application skip this node? if (assignment.getSkipped()) { @@ -895,7 +914,7 @@ private synchronized CSAssignment assignReservedContainer( // Try to assign if we have sufficient resources assignContainersOnNode(clusterResource, node, application, priority, - rmContainer, false); + rmContainer); // Doesn't matter... since it's already charged for at time of reservation // "re-reservation" is *free* @@ -937,102 +956,14 @@ private Resource getHeadroom(User user, Resource currentResourceLimit, Resources.roundDown(resourceCalculator, headroom, minimumAllocation); return headroom; } - - synchronized boolean canAssignToThisQueue(Resource clusterResource, - Resource required, Set nodeLabels, FiCaSchedulerApp application, - boolean checkReservations) { - // Get label of this queue can access, it's (nodeLabel AND queueLabel) - Set labelCanAccess; - if (null == nodeLabels || nodeLabels.isEmpty()) { - labelCanAccess = new HashSet(); - // Any queue can always access any node without label - labelCanAccess.add(RMNodeLabelsManager.NO_LABEL); - } else { - labelCanAccess = new HashSet(Sets.intersection(accessibleLabels, nodeLabels)); - } - - boolean canAssign = true; - for (String label : labelCanAccess) { - Resource potentialTotalCapacity = - Resources.add(queueUsage.getUsed(label), required); - - float potentialNewCapacity = - Resources.divide(resourceCalculator, clusterResource, - potentialTotalCapacity, - labelManager.getResourceByLabel(label, clusterResource)); - // if enabled, check to see if could we potentially use this node instead - // of a reserved node if the application has reserved containers - // TODO, now only consider reservation cases when the node has no label - if (this.reservationsContinueLooking && checkReservations - && label.equals(RMNodeLabelsManager.NO_LABEL)) { - float potentialNewWithoutReservedCapacity = Resources.divide( - resourceCalculator, - clusterResource, - Resources.subtract(potentialTotalCapacity, - application.getCurrentReservation()), - labelManager.getResourceByLabel(label, clusterResource)); - - if (potentialNewWithoutReservedCapacity <= queueCapacities - .getAbsoluteMaximumCapacity()) { - if (LOG.isDebugEnabled()) { - LOG.debug("try to use reserved: " - + getQueueName() - + " usedResources: " - + queueUsage.getUsed() - + " clusterResources: " - + clusterResource - + " reservedResources: " - + application.getCurrentReservation() - + " currentCapacity " - + Resources.divide(resourceCalculator, clusterResource, - queueUsage.getUsed(), clusterResource) + " required " + required - + " potentialNewWithoutReservedCapacity: " - + potentialNewWithoutReservedCapacity + " ( " - + " max-capacity: " - + queueCapacities.getAbsoluteMaximumCapacity() + ")"); - } - // we could potentially use this node instead of reserved node - return true; - } - } - - // Otherwise, if any of the label of this node beyond queue limit, we - // cannot allocate on this node. Consider a small epsilon here. - if (potentialNewCapacity > queueCapacities - .getAbsoluteMaximumCapacity(label) + 1e-4) { - canAssign = false; - break; - } - - if (LOG.isDebugEnabled()) { - LOG.debug(getQueueName() - + "Check assign to queue, label=" + label - + " usedResources: " + queueUsage.getUsed(label) - + " clusterResources: " + clusterResource - + " currentCapacity " - + Resources.divide(resourceCalculator, clusterResource, - queueUsage.getUsed(label), - labelManager.getResourceByLabel(label, clusterResource)) - + " potentialNewCapacity: " + potentialNewCapacity + " ( " - + " max-capacity: " + queueCapacities.getAbsoluteMaximumCapacity() - + ")"); - } - } - - return canAssign; - } - private Resource computeQueueCurrentLimitAndSetHeadroomInfo( + private void setQueueResourceLimitsInfo( Resource clusterResource) { - Resource queueCurrentResourceLimit = - getCurrentResourceLimit(clusterResource, currentResourceLimits); - synchronized (queueResourceLimitsInfo) { - queueResourceLimitsInfo.setQueueCurrentLimit(queueCurrentResourceLimit); + queueResourceLimitsInfo.setQueueCurrentLimit(currentResourceLimits + .getLimit()); queueResourceLimitsInfo.setClusterResource(clusterResource); } - - return queueCurrentResourceLimit; } @Lock({LeafQueue.class, FiCaSchedulerApp.class}) @@ -1047,16 +978,16 @@ Resource computeUserLimitAndSetHeadroom(FiCaSchedulerApp application, computeUserLimit(application, clusterResource, required, queueUser, requestedLabels); - Resource currentResourceLimit = - computeQueueCurrentLimitAndSetHeadroomInfo(clusterResource); + setQueueResourceLimitsInfo(clusterResource); Resource headroom = - getHeadroom(queueUser, currentResourceLimit, clusterResource, userLimit); + getHeadroom(queueUser, currentResourceLimits.getLimit(), + clusterResource, userLimit); if (LOG.isDebugEnabled()) { LOG.debug("Headroom calculation for user " + user + ": " + " userLimit=" + userLimit + - " queueMaxAvailRes=" + currentResourceLimit + + " queueMaxAvailRes=" + currentResourceLimits.getLimit() + " consumed=" + queueUser.getUsed() + " headroom=" + headroom); } @@ -1157,12 +1088,12 @@ private Resource computeUserLimit(FiCaSchedulerApp application, " clusterCapacity: " + clusterResource ); } - + user.setUserResourceLimit(limit); return limit; } @Private - protected synchronized boolean assignToUser(Resource clusterResource, + protected synchronized boolean canAssignToUser(Resource clusterResource, String userName, Resource limit, FiCaSchedulerApp application, boolean checkReservations, Set requestLabels) { User user = getUser(userName); @@ -1180,7 +1111,8 @@ protected synchronized boolean assignToUser(Resource clusterResource, limit)) { // if enabled, check to see if could we potentially use this node instead // of a reserved node if the application has reserved containers - if (this.reservationsContinueLooking && checkReservations) { + if (this.reservationsContinueLooking && checkReservations + && label.equals(CommonNodeLabelsManager.NO_LABEL)) { if (Resources.lessThanOrEqual( resourceCalculator, clusterResource, @@ -1206,8 +1138,8 @@ protected synchronized boolean assignToUser(Resource clusterResource, return true; } - boolean needContainers(FiCaSchedulerApp application, Priority priority, - Resource required) { + boolean shouldAllocOrReserveNewContainer(FiCaSchedulerApp application, + Priority priority, Resource required) { int requiredContainers = application.getTotalRequiredResources(priority); int reservedContainers = application.getNumReservedContainers(priority); int starvation = 0; @@ -1239,18 +1171,28 @@ resourceCalculator, required, getMaximumAllocation() private CSAssignment assignContainersOnNode(Resource clusterResource, FiCaSchedulerNode node, FiCaSchedulerApp application, Priority priority, - RMContainer reservedContainer, boolean needToUnreserve) { + RMContainer reservedContainer) { Resource assigned = Resources.none(); + NodeType requestType = null; + MutableObject allocatedContainer = new MutableObject(); // Data-local ResourceRequest nodeLocalResourceRequest = application.getResourceRequest(priority, node.getNodeName()); if (nodeLocalResourceRequest != null) { - assigned = - assignNodeLocalContainers(clusterResource, nodeLocalResourceRequest, - node, application, priority, reservedContainer, needToUnreserve); - if (Resources.greaterThan(resourceCalculator, clusterResource, + requestType = NodeType.NODE_LOCAL; + assigned = + assignNodeLocalContainers(clusterResource, nodeLocalResourceRequest, + node, application, priority, reservedContainer, + allocatedContainer); + if (Resources.greaterThan(resourceCalculator, clusterResource, assigned, Resources.none())) { + + //update locality statistics + if (allocatedContainer.getValue() != null) { + application.incNumAllocatedContainers(NodeType.NODE_LOCAL, + requestType); + } return new CSAssignment(assigned, NodeType.NODE_LOCAL); } } @@ -1262,12 +1204,23 @@ private CSAssignment assignContainersOnNode(Resource clusterResource, if (!rackLocalResourceRequest.getRelaxLocality()) { return SKIP_ASSIGNMENT; } - - assigned = - assignRackLocalContainers(clusterResource, rackLocalResourceRequest, - node, application, priority, reservedContainer, needToUnreserve); - if (Resources.greaterThan(resourceCalculator, clusterResource, + + if (requestType != NodeType.NODE_LOCAL) { + requestType = NodeType.RACK_LOCAL; + } + + assigned = + assignRackLocalContainers(clusterResource, rackLocalResourceRequest, + node, application, priority, reservedContainer, + allocatedContainer); + if (Resources.greaterThan(resourceCalculator, clusterResource, assigned, Resources.none())) { + + //update locality statistics + if (allocatedContainer.getValue() != null) { + application.incNumAllocatedContainers(NodeType.RACK_LOCAL, + requestType); + } return new CSAssignment(assigned, NodeType.RACK_LOCAL); } } @@ -1279,22 +1232,43 @@ private CSAssignment assignContainersOnNode(Resource clusterResource, if (!offSwitchResourceRequest.getRelaxLocality()) { return SKIP_ASSIGNMENT; } + if (requestType != NodeType.NODE_LOCAL + && requestType != NodeType.RACK_LOCAL) { + requestType = NodeType.OFF_SWITCH; + } - return new CSAssignment(assignOffSwitchContainers(clusterResource, - offSwitchResourceRequest, node, application, priority, - reservedContainer, needToUnreserve), - NodeType.OFF_SWITCH); + assigned = + assignOffSwitchContainers(clusterResource, offSwitchResourceRequest, + node, application, priority, reservedContainer, + allocatedContainer); + + // update locality statistics + if (allocatedContainer.getValue() != null) { + application.incNumAllocatedContainers(NodeType.OFF_SWITCH, requestType); + } + return new CSAssignment(assigned, NodeType.OFF_SWITCH); } return SKIP_ASSIGNMENT; } + + private Resource getMinimumResourceNeedUnreserved(Resource askedResource) { + // First we need to get minimum resource we need unreserve + // minimum-resource-need-unreserve = used + asked - limit + Resource minimumUnreservedResource = + Resources.subtract(Resources.add(queueUsage.getUsed(), askedResource), + currentResourceLimits.getLimit()); + return minimumUnreservedResource; + } @Private protected boolean findNodeToUnreserve(Resource clusterResource, FiCaSchedulerNode node, FiCaSchedulerApp application, Priority priority, - Resource capability) { + Resource askedResource, Resource minimumUnreservedResource) { // need to unreserve some other container first - NodeId idToUnreserve = application.getNodeIdToUnreserve(priority, capability); + NodeId idToUnreserve = + application.getNodeIdToUnreserve(priority, minimumUnreservedResource, + resourceCalculator, clusterResource); if (idToUnreserve == null) { if (LOG.isDebugEnabled()) { LOG.debug("checked to see if could unreserve for app but nothing " @@ -1311,7 +1285,7 @@ protected boolean findNodeToUnreserve(Resource clusterResource, LOG.debug("unreserving for app: " + application.getApplicationId() + " on nodeId: " + idToUnreserve + " in order to replace reserved application and place it on node: " - + node.getNodeID() + " needing: " + capability); + + node.getNodeID() + " needing: " + askedResource); } // headroom @@ -1332,15 +1306,7 @@ protected boolean findNodeToUnreserve(Resource clusterResource, @Private protected boolean checkLimitsToReserve(Resource clusterResource, - FiCaSchedulerApp application, Resource capability, - boolean needToUnreserve) { - if (needToUnreserve) { - if (LOG.isDebugEnabled()) { - LOG.debug("we needed to unreserve to be able to allocate"); - } - return false; - } - + FiCaSchedulerApp application, Resource capability) { // we can't reserve if we got here based on the limit // checks assuming we could unreserve!!! Resource userLimit = computeUserLimitAndSetHeadroom(application, @@ -1348,7 +1314,8 @@ protected boolean checkLimitsToReserve(Resource clusterResource, // Check queue max-capacity limit, // TODO: Consider reservation on labels - if (!canAssignToThisQueue(clusterResource, capability, null, application, false)) { + if (!canAssignToThisQueue(clusterResource, null, + this.currentResourceLimits, capability, Resources.none())) { if (LOG.isDebugEnabled()) { LOG.debug("was going to reserve but hit queue limit"); } @@ -1356,7 +1323,7 @@ protected boolean checkLimitsToReserve(Resource clusterResource, } // Check user limit - if (!assignToUser(clusterResource, application.getUser(), userLimit, + if (!canAssignToUser(clusterResource, application.getUser(), userLimit, application, false, null)) { if (LOG.isDebugEnabled()) { LOG.debug("was going to reserve but hit user limit"); @@ -1370,12 +1337,12 @@ protected boolean checkLimitsToReserve(Resource clusterResource, private Resource assignNodeLocalContainers(Resource clusterResource, ResourceRequest nodeLocalResourceRequest, FiCaSchedulerNode node, FiCaSchedulerApp application, Priority priority, - RMContainer reservedContainer, boolean needToUnreserve) { - if (canAssign(application, priority, node, NodeType.NODE_LOCAL, + RMContainer reservedContainer, MutableObject allocatedContainer) { + if (canAssign(application, priority, node, NodeType.NODE_LOCAL, reservedContainer)) { return assignContainer(clusterResource, node, application, priority, nodeLocalResourceRequest, NodeType.NODE_LOCAL, reservedContainer, - needToUnreserve); + allocatedContainer); } return Resources.none(); @@ -1384,12 +1351,12 @@ private Resource assignNodeLocalContainers(Resource clusterResource, private Resource assignRackLocalContainers(Resource clusterResource, ResourceRequest rackLocalResourceRequest, FiCaSchedulerNode node, FiCaSchedulerApp application, Priority priority, - RMContainer reservedContainer, boolean needToUnreserve) { - if (canAssign(application, priority, node, NodeType.RACK_LOCAL, + RMContainer reservedContainer, MutableObject allocatedContainer) { + if (canAssign(application, priority, node, NodeType.RACK_LOCAL, reservedContainer)) { return assignContainer(clusterResource, node, application, priority, rackLocalResourceRequest, NodeType.RACK_LOCAL, reservedContainer, - needToUnreserve); + allocatedContainer); } return Resources.none(); @@ -1398,12 +1365,12 @@ private Resource assignRackLocalContainers(Resource clusterResource, private Resource assignOffSwitchContainers(Resource clusterResource, ResourceRequest offSwitchResourceRequest, FiCaSchedulerNode node, FiCaSchedulerApp application, Priority priority, - RMContainer reservedContainer, boolean needToUnreserve) { - if (canAssign(application, priority, node, NodeType.OFF_SWITCH, + RMContainer reservedContainer, MutableObject allocatedContainer) { + if (canAssign(application, priority, node, NodeType.OFF_SWITCH, reservedContainer)) { return assignContainer(clusterResource, node, application, priority, offSwitchResourceRequest, NodeType.OFF_SWITCH, reservedContainer, - needToUnreserve); + allocatedContainer); } return Resources.none(); @@ -1487,13 +1454,12 @@ Container createContainer(FiCaSchedulerApp application, FiCaSchedulerNode node, private Resource assignContainer(Resource clusterResource, FiCaSchedulerNode node, FiCaSchedulerApp application, Priority priority, ResourceRequest request, NodeType type, RMContainer rmContainer, - boolean needToUnreserve) { + MutableObject createdContainer) { if (LOG.isDebugEnabled()) { LOG.debug("assignContainers: node=" + node.getNodeName() + " application=" + application.getApplicationId() + " priority=" + priority.getPriority() - + " request=" + request + " type=" + type - + " needToUnreserve= " + needToUnreserve); + + " request=" + request + " type=" + type); } // check if the resource request can access the label @@ -1513,12 +1479,14 @@ private Resource assignContainer(Resource clusterResource, FiCaSchedulerNode nod Resource available = node.getAvailableResource(); Resource totalResource = node.getTotalResource(); - if (!Resources.fitsIn(capability, totalResource)) { + if (!Resources.lessThanOrEqual(resourceCalculator, clusterResource, + capability, totalResource)) { LOG.warn("Node : " + node.getNodeID() + " does not have sufficient resource for request : " + request + " node total capability : " + node.getTotalResource()); return Resources.none(); } + assert Resources.greaterThan( resourceCalculator, clusterResource, available, Resources.none()); @@ -1531,18 +1499,9 @@ private Resource assignContainer(Resource clusterResource, FiCaSchedulerNode nod LOG.warn("Couldn't get container for allocation!"); return Resources.none(); } - - // default to true since if reservation continue look feature isn't on - // needContainers is checked earlier and we wouldn't have gotten this far - boolean canAllocContainer = true; - if (this.reservationsContinueLooking) { - // based on reservations can we allocate/reserve more or do we need - // to unreserve one first - canAllocContainer = needContainers(application, priority, capability); - if (LOG.isDebugEnabled()) { - LOG.debug("can alloc container is: " + canAllocContainer); - } - } + + boolean shouldAllocOrReserveNewContainer = shouldAllocOrReserveNewContainer( + application, priority, capability); // Can we allocate a container on this node? int availableContainers = @@ -1553,25 +1512,25 @@ private Resource assignContainer(Resource clusterResource, FiCaSchedulerNode nod // Did we previously reserve containers at this 'priority'? if (rmContainer != null) { unreserve(application, priority, node, rmContainer); - } else if (this.reservationsContinueLooking - && (!canAllocContainer || needToUnreserve)) { - // need to unreserve some other container first - boolean res = findNodeToUnreserve(clusterResource, node, application, - priority, capability); - if (!res) { - return Resources.none(); - } - } else { - // we got here by possibly ignoring queue capacity limits. If the - // parameter needToUnreserve is true it means we ignored one of those - // limits in the chance we could unreserve. If we are here we aren't - // trying to unreserve so we can't allocate anymore due to that parent - // limit. - if (needToUnreserve) { - if (LOG.isDebugEnabled()) { - LOG.debug("we needed to unreserve to be able to allocate, skipping"); + } else if (this.reservationsContinueLooking && node.getLabels().isEmpty()) { + // when reservationsContinueLooking is set, we may need to unreserve + // some containers to meet this queue and its parents' resource limits + // TODO, need change here when we want to support continuous reservation + // looking for labeled partitions. + Resource minimumUnreservedResource = + getMinimumResourceNeedUnreserved(capability); + if (!shouldAllocOrReserveNewContainer + || Resources.greaterThan(resourceCalculator, clusterResource, + minimumUnreservedResource, Resources.none())) { + boolean containerUnreserved = + findNodeToUnreserve(clusterResource, node, application, priority, + capability, minimumUnreservedResource); + // When (minimum-unreserved-resource > 0 OR we cannot allocate new/reserved + // container (That means we *have to* unreserve some resource to + // continue)). If we failed to unreserve some resource, + if (!containerUnreserved) { + return Resources.none(); } - return Resources.none(); } } @@ -1592,22 +1551,21 @@ private Resource assignContainer(Resource clusterResource, FiCaSchedulerNode nod " container=" + container + " queue=" + this + " clusterResource=" + clusterResource); - + createdContainer.setValue(allocatedContainer); return container.getResource(); } else { // if we are allowed to allocate but this node doesn't have space, reserve it or // if this was an already a reserved container, reserve it again - if ((canAllocContainer) || (rmContainer != null)) { - - if (reservationsContinueLooking) { - // we got here by possibly ignoring parent queue capacity limits. If - // the parameter needToUnreserve is true it means we ignored one of - // those limits in the chance we could unreserve. If we are here - // we aren't trying to unreserve so we can't allocate - // anymore due to that parent limit - boolean res = checkLimitsToReserve(clusterResource, application, capability, - needToUnreserve); - if (!res) { + if (shouldAllocOrReserveNewContainer || rmContainer != null) { + + if (reservationsContinueLooking && rmContainer == null) { + // we could possibly ignoring parent queue capacity limits when + // reservationsContinueLooking is set. + // If we're trying to reserve a container here, not container will be + // unreserved for reserving the new one. Check limits again before + // reserve the new container + if (!checkLimitsToReserve(clusterResource, + application, capability)) { return Resources.none(); } } @@ -1682,7 +1640,8 @@ public void completedContainer(Resource clusterResource, node, rmContainer); } else { removed = - application.containerCompleted(rmContainer, containerStatus, event); + application.containerCompleted(rmContainer, containerStatus, + event, node.getPartition()); node.releaseContainer(container); } @@ -1749,18 +1708,36 @@ private void updateAbsoluteCapacityResource(Resource clusterResource) { Resources.multiplyAndNormalizeUp(resourceCalculator, clusterResource, queueCapacities.getAbsoluteCapacity(), minimumAllocation); } + + private void updateCurrentResourceLimits( + ResourceLimits currentResourceLimits, Resource clusterResource) { + // TODO: need consider non-empty node labels when resource limits supports + // node labels + // Even if ParentQueue will set limits respect child's max queue capacity, + // but when allocating reserved container, CapacityScheduler doesn't do + // this. So need cap limits by queue's max capacity here. + this.currentResourceLimits = currentResourceLimits; + Resource queueMaxResource = + Resources.multiplyAndNormalizeDown(resourceCalculator, labelManager + .getResourceByLabel(RMNodeLabelsManager.NO_LABEL, clusterResource), + queueCapacities + .getAbsoluteMaximumCapacity(RMNodeLabelsManager.NO_LABEL), + minimumAllocation); + this.currentResourceLimits.setLimit(Resources.min(resourceCalculator, + clusterResource, queueMaxResource, currentResourceLimits.getLimit())); + } @Override public synchronized void updateClusterResource(Resource clusterResource, ResourceLimits currentResourceLimits) { - this.currentResourceLimits = currentResourceLimits; + updateCurrentResourceLimits(currentResourceLimits, clusterResource); lastClusterResource = clusterResource; updateAbsoluteCapacityResource(clusterResource); // Update headroom info based on new cluster resource value // absoluteMaxCapacity now, will be replaced with absoluteMaxAvailCapacity // during allocation - computeQueueCurrentLimitAndSetHeadroomInfo(clusterResource); + setQueueResourceLimitsInfo(clusterResource); // Update metrics CSQueueUtils.updateQueueStatistics( @@ -1783,6 +1760,7 @@ resourceCalculator, this, getParent(), clusterResource, @VisibleForTesting public static class User { ResourceUsage userResourceUsage = new ResourceUsage(); + volatile Resource userResourceLimit = Resource.newInstance(0, 0); int pendingApplications = 0; int activeApplications = 0; @@ -1852,6 +1830,14 @@ public void releaseContainer(Resource resource, Set nodeLabels) { } } } + + public Resource getUserResourceLimit() { + return userResourceLimit; + } + + public void setUserResourceLimit(Resource userResourceLimit) { + this.userResourceLimit = userResourceLimit; + } } @Override diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java index 7feaa152fbbb1..5ed6bb8c93263 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java @@ -23,7 +23,6 @@ import java.util.Collection; import java.util.Comparator; import java.util.HashMap; -import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Map; @@ -48,7 +47,6 @@ import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.factories.RecordFactory; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; -import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; import org.apache.hadoop.yarn.security.AccessType; import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; @@ -63,8 +61,6 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode; import org.apache.hadoop.yarn.util.resource.Resources; -import com.google.common.collect.Sets; - @Private @Evolving public class ParentQueue extends AbstractCSQueue { @@ -380,8 +376,7 @@ private synchronized void removeApplication(ApplicationId applicationId, @Override public synchronized CSAssignment assignContainers(Resource clusterResource, - FiCaSchedulerNode node, boolean needToUnreserve, - ResourceLimits resourceLimits) { + FiCaSchedulerNode node, ResourceLimits resourceLimits) { CSAssignment assignment = new CSAssignment(Resources.createResource(0, 0), NodeType.NODE_LOCAL); Set nodeLabels = node.getLabels(); @@ -397,21 +392,18 @@ public synchronized CSAssignment assignContainers(Resource clusterResource, + getQueueName()); } - boolean localNeedToUnreserve = false; - // Are we over maximum-capacity for this queue? - if (!canAssignToThisQueue(clusterResource, nodeLabels)) { - // check to see if we could if we unreserve first - localNeedToUnreserve = assignToQueueIfUnreserve(clusterResource); - if (!localNeedToUnreserve) { - break; - } + // This will also consider parent's limits and also continuous reservation + // looking + if (!super.canAssignToThisQueue(clusterResource, nodeLabels, resourceLimits, + minimumAllocation, Resources.createResource(getMetrics() + .getReservedMB(), getMetrics().getReservedVirtualCores()))) { + break; } // Schedule CSAssignment assignedToChild = - assignContainersToChildQueues(clusterResource, node, - localNeedToUnreserve | needToUnreserve, resourceLimits); + assignContainersToChildQueues(clusterResource, node, resourceLimits); assignment.setType(assignedToChild.getType()); // Done if no child-queue assigned anything @@ -459,74 +451,6 @@ public synchronized CSAssignment assignContainers(Resource clusterResource, return assignment; } - private synchronized boolean canAssignToThisQueue(Resource clusterResource, - Set nodeLabels) { - Set labelCanAccess = - new HashSet( - accessibleLabels.contains(CommonNodeLabelsManager.ANY) ? nodeLabels - : Sets.intersection(accessibleLabels, nodeLabels)); - if (nodeLabels.isEmpty()) { - // Any queue can always access any node without label - labelCanAccess.add(RMNodeLabelsManager.NO_LABEL); - } - - boolean canAssign = true; - for (String label : labelCanAccess) { - float currentAbsoluteLabelUsedCapacity = - Resources.divide(resourceCalculator, clusterResource, - queueUsage.getUsed(label), - labelManager.getResourceByLabel(label, clusterResource)); - // if any of the label doesn't beyond limit, we can allocate on this node - if (currentAbsoluteLabelUsedCapacity >= - queueCapacities.getAbsoluteMaximumCapacity(label)) { - if (LOG.isDebugEnabled()) { - LOG.debug(getQueueName() + " used=" + queueUsage.getUsed() - + " current-capacity (" + queueUsage.getUsed(label) + ") " - + " >= max-capacity (" - + labelManager.getResourceByLabel(label, clusterResource) + ")"); - } - canAssign = false; - break; - } - } - - return canAssign; - } - - - private synchronized boolean assignToQueueIfUnreserve(Resource clusterResource) { - if (this.reservationsContinueLooking) { - // check to see if we could potentially use this node instead of a reserved - // node - - Resource reservedResources = Resources.createResource(getMetrics() - .getReservedMB(), getMetrics().getReservedVirtualCores()); - float capacityWithoutReservedCapacity = Resources.divide( - resourceCalculator, clusterResource, - Resources.subtract(queueUsage.getUsed(), reservedResources), - clusterResource); - - if (capacityWithoutReservedCapacity <= queueCapacities - .getAbsoluteMaximumCapacity()) { - if (LOG.isDebugEnabled()) { - LOG.debug("parent: try to use reserved: " + getQueueName() - + " usedResources: " + queueUsage.getUsed().getMemory() - + " clusterResources: " + clusterResource.getMemory() - + " reservedResources: " + reservedResources.getMemory() - + " currentCapacity " + ((float) queueUsage.getUsed().getMemory()) - / clusterResource.getMemory() - + " potentialNewWithoutReservedCapacity: " - + capacityWithoutReservedCapacity + " ( " + " max-capacity: " - + queueCapacities.getAbsoluteMaximumCapacity() + ")"); - } - // we could potentially use this node instead of reserved node - return true; - } - } - return false; - } - - private boolean canAssign(Resource clusterResource, FiCaSchedulerNode node) { return (node.getReservedContainer() == null) && Resources.greaterThanOrEqual(resourceCalculator, clusterResource, @@ -534,28 +458,38 @@ private boolean canAssign(Resource clusterResource, FiCaSchedulerNode node) { } private ResourceLimits getResourceLimitsOfChild(CSQueue child, - Resource clusterResource, ResourceLimits myLimits) { - /* - * Set head-room of a given child, limit = - * min(minimum-of-limit-of-this-queue-and-ancestors, this.max) - this.used - * + child.used. To avoid any of this queue's and its ancestors' limit - * being violated - */ - Resource myCurrentLimit = - getCurrentResourceLimit(clusterResource, myLimits); - // My available resource = my-current-limit - my-used-resource - Resource myMaxAvailableResource = Resources.subtract(myCurrentLimit, - getUsedResources()); - // Child's limit = my-available-resource + resource-already-used-by-child + Resource clusterResource, ResourceLimits parentLimits) { + // Set resource-limit of a given child, child.limit = + // min(my.limit - my.used + child.used, child.max) + + // Parent available resource = parent-limit - parent-used-resource + Resource parentMaxAvailableResource = + Resources.subtract(parentLimits.getLimit(), getUsedResources()); + + // Child's limit = parent-available-resource + child-used Resource childLimit = - Resources.add(myMaxAvailableResource, child.getUsedResources()); - + Resources.add(parentMaxAvailableResource, child.getUsedResources()); + + // Get child's max resource + Resource childConfiguredMaxResource = + Resources.multiplyAndNormalizeDown(resourceCalculator, labelManager + .getResourceByLabel(RMNodeLabelsManager.NO_LABEL, clusterResource), + child.getAbsoluteMaximumCapacity(), minimumAllocation); + + // Child's limit should be capped by child configured max resource + childLimit = + Resources.min(resourceCalculator, clusterResource, childLimit, + childConfiguredMaxResource); + + // Normalize before return + childLimit = + Resources.roundDown(resourceCalculator, childLimit, minimumAllocation); + return new ResourceLimits(childLimit); } private synchronized CSAssignment assignContainersToChildQueues( - Resource cluster, FiCaSchedulerNode node, boolean needToUnreserve, - ResourceLimits limits) { + Resource cluster, FiCaSchedulerNode node, ResourceLimits limits) { CSAssignment assignment = new CSAssignment(Resources.createResource(0, 0), NodeType.NODE_LOCAL); @@ -573,9 +507,7 @@ private synchronized CSAssignment assignContainersToChildQueues( ResourceLimits childLimits = getResourceLimitsOfChild(childQueue, cluster, limits); - assignment = - childQueue.assignContainers(cluster, node, needToUnreserve, - childLimits); + assignment = childQueue.assignContainers(cluster, node, childLimits); if(LOG.isDebugEnabled()) { LOG.debug("Assigned to queue: " + childQueue.getQueuePath() + " stats: " + childQueue + " --> " + diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java index 65c911bbb9ea7..3725e21b342b6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java @@ -32,14 +32,19 @@ public class UserInfo { protected ResourceInfo resourcesUsed; protected int numPendingApplications; protected int numActiveApplications; + protected ResourceInfo AMResourceUsed; + protected ResourceInfo userResourceLimit; UserInfo() {} - UserInfo(String username, Resource resUsed, int activeApps, int pendingApps) { + UserInfo(String username, Resource resUsed, int activeApps, int pendingApps, + Resource amResUsed, Resource resourceLimit) { this.username = username; this.resourcesUsed = new ResourceInfo(resUsed); this.numActiveApplications = activeApps; this.numPendingApplications = pendingApps; + this.AMResourceUsed = new ResourceInfo(amResUsed); + this.userResourceLimit = new ResourceInfo(resourceLimit); } public String getUsername() { @@ -57,4 +62,12 @@ public int getNumPendingApplications() { public int getNumActiveApplications() { return numActiveApplications; } + + public ResourceInfo getAMResourcesUsed() { + return AMResourceUsed; + } + + public ResourceInfo getUserResourceLimit() { + return userResourceLimit; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java index 9f97b137f2ddb..e0413895d46c4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java @@ -90,7 +90,8 @@ public FiCaSchedulerApp(ApplicationAttemptId applicationAttemptId, } synchronized public boolean containerCompleted(RMContainer rmContainer, - ContainerStatus containerStatus, RMContainerEventType event) { + ContainerStatus containerStatus, RMContainerEventType event, + String partition) { // Remove from the list of containers if (null == liveContainers.remove(rmContainer.getContainerId())) { @@ -122,7 +123,7 @@ synchronized public boolean containerCompleted(RMContainer rmContainer, // Update usage metrics Resource containerResource = rmContainer.getContainer().getResource(); queue.getMetrics().releaseResources(getUser(), 1, containerResource); - Resources.subtractFrom(currentConsumption, containerResource); + attemptResourceUsage.decUsed(partition, containerResource); // Clear resource utilization metrics cache. lastMemoryAggregateAllocationUpdateTime = -1; @@ -156,7 +157,8 @@ synchronized public RMContainer allocate(NodeType type, FiCaSchedulerNode node, // Update consumption and track allocations List resourceRequestList = appSchedulingInfo.allocate( type, node, priority, request, container); - Resources.addTo(currentConsumption, container.getResource()); + attemptResourceUsage.incUsed(node.getPartition(), + container.getResource()); // Update resource requests related to "request" and store in RMContainer ((RMContainerImpl)rmContainer).setResourceRequests(resourceRequestList); @@ -198,12 +200,13 @@ public synchronized boolean unreserve(FiCaSchedulerNode node, Priority priority) resetReReservations(priority); Resource resource = reservedContainer.getContainer().getResource(); - Resources.subtractFrom(currentReservation, resource); + this.attemptResourceUsage.decReserved(node.getPartition(), resource); LOG.info("Application " + getApplicationId() + " unreserved " - + " on node " + node + ", currently has " + reservedContainers.size() - + " at priority " + priority + "; currentReservation " - + currentReservation); + + " on node " + node + ", currently has " + + reservedContainers.size() + " at priority " + priority + + "; currentReservation " + this.attemptResourceUsage.getReserved() + + " on node-label=" + node.getPartition()); return true; } } @@ -268,13 +271,16 @@ public synchronized Allocation getAllocation(ResourceCalculator rc, minimumAllocation, numCont); ContainersAndNMTokensAllocation allocation = pullNewlyAllocatedContainersAndNMTokens(); - return new Allocation(allocation.getContainerList(), getHeadroom(), null, + Resource headroom = getHeadroom(); + setApplicationHeadroomForMetrics(headroom); + return new Allocation(allocation.getContainerList(), headroom, null, currentContPreemption, Collections.singletonList(rr), allocation.getNMTokenList()); } synchronized public NodeId getNodeIdToUnreserve(Priority priority, - Resource capability) { + Resource resourceNeedUnreserve, ResourceCalculator rc, + Resource clusterResource) { // first go around make this algorithm simple and just grab first // reservation that has enough resources @@ -283,16 +289,19 @@ synchronized public NodeId getNodeIdToUnreserve(Priority priority, if ((reservedContainers != null) && (!reservedContainers.isEmpty())) { for (Map.Entry entry : reservedContainers.entrySet()) { + NodeId nodeId = entry.getKey(); + Resource containerResource = entry.getValue().getContainer().getResource(); + // make sure we unreserve one with at least the same amount of // resources, otherwise could affect capacity limits - if (Resources.fitsIn(capability, entry.getValue().getContainer() - .getResource())) { + if (Resources.lessThanOrEqual(rc, clusterResource, + resourceNeedUnreserve, containerResource)) { if (LOG.isDebugEnabled()) { LOG.debug("unreserving node with reservation size: " - + entry.getValue().getContainer().getResource() - + " in order to allocate container with size: " + capability); + + containerResource + + " in order to allocate container with size: " + resourceNeedUnreserve); } - return entry.getKey(); + return nodeId; } } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java index 9cb767d38a5d5..0ea731403029e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java @@ -33,9 +33,6 @@ import com.google.common.annotations.VisibleForTesting; -import javax.annotation.concurrent.ThreadSafe; - -@ThreadSafe public class AllocationConfiguration extends ReservationSchedulerConfiguration { private static final AccessControlList EVERYBODY_ACL = new AccessControlList("*"); private static final AccessControlList NOBODY_ACL = new AccessControlList(" "); @@ -207,16 +204,12 @@ public float getFairSharePreemptionThreshold(String queueName) { } public ResourceWeights getQueueWeight(String queue) { - synchronized (queueWeights) { - ResourceWeights weight = queueWeights.get(queue); - return (weight == null) ? ResourceWeights.NEUTRAL : weight; - } + ResourceWeights weight = queueWeights.get(queue); + return (weight == null) ? ResourceWeights.NEUTRAL : weight; } public void setQueueWeight(String queue, ResourceWeights weight) { - synchronized (queueWeights) { - queueWeights.put(queue, weight); - } + queueWeights.put(queue, weight); } public int getUserMaxApps(String user) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java index c19aa513e1c1d..dab6d9f0bd13f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java @@ -201,7 +201,7 @@ public synchronized void setReloadListener(Listener reloadListener) { * @throws ParserConfigurationException if XML parser is misconfigured. * @throws SAXException if config file is malformed. */ - public void reloadAllocations() throws IOException, + public synchronized void reloadAllocations() throws IOException, ParserConfigurationException, SAXException, AllocationConfigurationException { if (allocFile == null) { return; @@ -426,13 +426,19 @@ private void loadQueue(String parentName, Element element, Map> configuredQueues, Set reservableQueues) throws AllocationConfigurationException { - String queueName = element.getAttribute("name"); + String queueName = element.getAttribute("name").trim(); if (queueName.contains(".")) { throw new AllocationConfigurationException("Bad fair scheduler config " + "file: queue name (" + queueName + ") shouldn't contain period."); } + if (queueName.isEmpty()) { + throw new AllocationConfigurationException("Bad fair scheduler config " + + "file: queue name shouldn't be empty or " + + "consist only of whitespace."); + } + if (parentName != null) { queueName = parentName + "." + queueName; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java index 67103d187b3fe..46617ff5cda01 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java @@ -142,7 +142,7 @@ synchronized public void containerCompleted(RMContainer rmContainer, // Update usage metrics Resource containerResource = rmContainer.getContainer().getResource(); queue.getMetrics().releaseResources(getUser(), 1, containerResource); - Resources.subtractFrom(currentConsumption, containerResource); + this.attemptResourceUsage.decUsed(containerResource); // remove from preemption map if it is completed preemptionMap.remove(rmContainer); @@ -164,11 +164,12 @@ private synchronized void unreserveInternal( resetReReservations(priority); Resource resource = reservedContainer.getContainer().getResource(); - Resources.subtractFrom(currentReservation, resource); + this.attemptResourceUsage.decReserved(resource); LOG.info("Application " + getApplicationId() + " unreserved " + " on node " - + node + ", currently has " + reservedContainers.size() + " at priority " - + priority + "; currentReservation " + currentReservation); + + node + ", currently has " + reservedContainers.size() + + " at priority " + priority + "; currentReservation " + + this.attemptResourceUsage.getReserved()); } @Override @@ -339,7 +340,7 @@ else if (allowed.equals(NodeType.RACK_LOCAL) && // Update consumption and track allocations List resourceRequestList = appSchedulingInfo.allocate( type, node, priority, request, container); - Resources.addTo(currentConsumption, container.getResource()); + this.attemptResourceUsage.incUsed(container.getResource()); // Update resource requests related to "request" and store in RMContainer ((RMContainerImpl) rmContainer).setResourceRequests(resourceRequestList); @@ -569,6 +570,10 @@ private Resource assignContainer(FSSchedulerNode node, boolean reserved) { // Check the AM resource usage for the leaf queue if (getLiveContainers().size() == 0 && !getUnmanagedAM()) { if (!getQueue().canRunAppAM(getAMResource())) { + if (LOG.isDebugEnabled()) { + LOG.debug("Skipping allocation because maxAMShare limit would " + + "be exceeded"); + } return Resources.none(); } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java index 3c975356dfe4c..c49a3231d9d04 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java @@ -284,6 +284,8 @@ public void updateDemand() { if (LOG.isDebugEnabled()) { LOG.debug("The updated demand for " + getName() + " is " + demand + "; the max is " + maxRes); + LOG.debug("The updated fairshare for " + getName() + " is " + + getFairShare()); } } @@ -304,7 +306,7 @@ public Resource assignContainer(FSSchedulerNode node) { Resource assigned = Resources.none(); if (LOG.isDebugEnabled()) { LOG.debug("Node " + node.getNodeName() + " offered to queue: " + - getName()); + getName() + " fairShare: " + getFairShare()); } if (!assignContainerPreCheck(node)) { @@ -330,6 +332,10 @@ public Resource assignContainer(FSSchedulerNode node) { assigned = sched.assignContainer(node); if (!assigned.equals(Resources.none())) { + if (LOG.isDebugEnabled()) { + LOG.debug("Assigned container in queue:" + getName() + " " + + "container:" + assigned); + } break; } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java index c50f281cb6645..c2282fdb736ca 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSOpDurations.java @@ -31,8 +31,6 @@ import static org.apache.hadoop.metrics2.lib.Interns.info; import org.apache.hadoop.metrics2.lib.MutableRate; -import javax.annotation.concurrent.ThreadSafe; - /** * Class to capture the performance metrics of FairScheduler. * This should be a singleton. @@ -40,7 +38,6 @@ @InterfaceAudience.Private @InterfaceStability.Unstable @Metrics(context="fairscheduler-op-durations") -@ThreadSafe public class FSOpDurations implements MetricsSource { @Metric("Duration for a continuous scheduling run") diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java index 349464e1ef2f3..ee47d9a6006b2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java @@ -23,6 +23,8 @@ import java.util.List; import java.util.Set; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; import org.apache.hadoop.classification.InterfaceAudience.Private; import org.apache.hadoop.classification.InterfaceStability.Unstable; import org.apache.hadoop.security.UserGroupInformation; @@ -41,6 +43,9 @@ @Private @Unstable public abstract class FSQueue implements Queue, Schedulable { + private static final Log LOG = LogFactory.getLog( + FSQueue.class.getName()); + private Resource fairShare = Resources.createResource(0, 0); private Resource steadyFairShare = Resources.createResource(0, 0); private final String name; @@ -164,6 +169,9 @@ public Resource getFairShare() { public void setFairShare(Resource fairShare) { this.fairShare = fairShare; metrics.setFairShare(fairShare); + if (LOG.isDebugEnabled()) { + LOG.debug("The updated fairShare for " + getName() + " is " + fairShare); + } } /** Get the steady fair share assigned to this Schedulable. */ @@ -291,4 +299,12 @@ public String getDefaultNodeLabelExpression() { // TODO, add implementation for FS return null; } + + @Override + public void incPendingResource(String nodeLabel, Resource resourceToInc) { + } + + @Override + public void decPendingResource(String nodeLabel, Resource resourceToDec) { + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java index 2b597164baf70..04c7f70e6f3bd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java @@ -701,6 +701,8 @@ FSLeafQueue assignToQueue(RMApp rmApp, String queueName, String user) { appRejectMsg = queueName + " is not a leaf queue"; } } + } catch (InvalidQueueNameException qne) { + appRejectMsg = qne.getMessage(); } catch (IOException ioe) { appRejectMsg = "Error assigning app to queue " + queueName; } @@ -896,6 +898,9 @@ public Allocation allocate(ApplicationAttemptId appAttemptId, clusterResource, minimumAllocation, getMaximumResourceCapability(), incrAllocation); + // Record container allocation start time + application.recordContainerRequestTime(getClock().getTime()); + // Set amResource for this app if (!application.getUnmanagedAM() && ask.size() == 1 && application.getLiveContainers().isEmpty()) { @@ -929,7 +934,7 @@ clusterResource, minimumAllocation, getMaximumResourceCapability(), LOG.debug("Preempting " + application.getPreemptionContainers().size() + " container(s)"); } - + Set preemptionContainerIds = new HashSet(); for (RMContainer container : application.getPreemptionContainers()) { preemptionContainerIds.add(container.getContainerId()); @@ -938,9 +943,16 @@ clusterResource, minimumAllocation, getMaximumResourceCapability(), application.updateBlacklist(blacklistAdditions, blacklistRemovals); ContainersAndNMTokensAllocation allocation = application.pullNewlyAllocatedContainersAndNMTokens(); - return new Allocation(allocation.getContainerList(), - application.getHeadroom(), preemptionContainerIds, null, null, - allocation.getNMTokenList()); + + // Record container allocation time + if (!(allocation.getContainerList().isEmpty())) { + application.recordContainerAllocationTime(getClock().getTime()); + } + + Resource headroom = application.getHeadroom(); + application.setApplicationHeadroomForMetrics(headroom); + return new Allocation(allocation.getContainerList(), headroom, + preemptionContainerIds, null, null, allocation.getNMTokenList()); } } @@ -1477,6 +1489,7 @@ public void onReload(AllocationConfiguration queueInfo) { allocConf = queueInfo; allocConf.getDefaultSchedulingPolicy().initialize(clusterResource); queueMgr.updateAllocationConfiguration(allocConf); + maxRunningEnforcer.updateRunnabilityOnReload(); } } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/InvalidQueueNameException.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/InvalidQueueNameException.java new file mode 100644 index 0000000000000..fc5ba161e4a89 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/InvalidQueueNameException.java @@ -0,0 +1,39 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Unstable; + +/** + * Thrown when Queue Name is malformed. + */ +@Private +@Unstable +public class InvalidQueueNameException extends IllegalArgumentException { + private static final long serialVersionUID = -7306320927804540011L; + + public InvalidQueueNameException(String message) { + super(message); + } + + public InvalidQueueNameException(String message, Throwable t) { + super(message, t); + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java index 2c90edd400a47..f75043814d0c2 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/MaxRunningAppsEnforcer.java @@ -104,6 +104,26 @@ public void trackNonRunnableApp(FSAppAttempt app) { usersNonRunnableApps.put(user, app); } + /** + * This is called after reloading the allocation configuration when the + * scheduler is reinitilized + * + * Checks to see whether any non-runnable applications become runnable + * now that the max running apps of given queue has been changed + * + * Runs in O(n) where n is the number of apps that are non-runnable and in + * the queues that went from having no slack to having slack. + */ + public void updateRunnabilityOnReload() { + FSParentQueue rootQueue = scheduler.getQueueManager().getRootQueue(); + List> appsNowMaybeRunnable = + new ArrayList>(); + + gatherPossiblyRunnableAppLists(rootQueue, appsNowMaybeRunnable); + + updateAppsRunnability(appsNowMaybeRunnable, Integer.MAX_VALUE); + } + /** * Checks to see whether any other applications runnable now that the given * application has been removed from the given queue. And makes them so. @@ -156,6 +176,19 @@ public void updateRunnabilityOnAppRemoval(FSAppAttempt app, FSLeafQueue queue) { } } + updateAppsRunnability(appsNowMaybeRunnable, + appsNowMaybeRunnable.size()); + } + + /** + * Checks to see whether applications are runnable now by iterating + * through each one of them and check if the queue and user have slack + * + * if we know how many apps can be runnable, there is no need to iterate + * through all apps, maxRunnableApps is used to break out of the iteration + */ + private void updateAppsRunnability(List> + appsNowMaybeRunnable, int maxRunnableApps) { // Scan through and check whether this means that any apps are now runnable Iterator iter = new MultiListStartTimeIterator( appsNowMaybeRunnable); @@ -173,9 +206,7 @@ public void updateRunnabilityOnAppRemoval(FSAppAttempt app, FSLeafQueue queue) { next.getQueue().addApp(appSched, true); noLongerPendingApps.add(appSched); - // No more than one app per list will be able to be made runnable, so - // we can stop looking after we've found that many - if (noLongerPendingApps.size() >= appsNowMaybeRunnable.size()) { + if (noLongerPendingApps.size() >= maxRunnableApps) { break; } } @@ -194,11 +225,10 @@ public void updateRunnabilityOnAppRemoval(FSAppAttempt app, FSLeafQueue queue) { if (!usersNonRunnableApps.remove(appSched.getUser(), appSched)) { LOG.error("Waiting app " + appSched + " expected to be in " - + "usersNonRunnableApps, but was not. This should never happen."); + + "usersNonRunnableApps, but was not. This should never happen."); } } } - /** * Updates the relevant tracking variables after a runnable app with the given * queue and user has been removed. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java index 27e571e4f6691..64442abf2cc82 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java @@ -36,6 +36,7 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.xml.sax.SAXException; +import com.google.common.annotations.VisibleForTesting; /** * Maintains a list of queues as well as scheduling parameters for each queue, * such as guaranteed share allocations, from the fair scheduler config file. @@ -155,7 +156,13 @@ private FSQueue createQueue(String name, FSQueueType queueType) { // Move up the queue tree until we reach one that exists. while (sepIndex != -1) { + int prevSepIndex = sepIndex; sepIndex = name.lastIndexOf('.', sepIndex-1); + String node = name.substring(sepIndex+1, prevSepIndex); + if (!isQueueNameValid(node)) { + throw new InvalidQueueNameException("Illegal node name at offset " + + (sepIndex+1) + " for queue name " + name); + } FSQueue queue; String curName = null; curName = name.substring(0, sepIndex); @@ -401,4 +408,13 @@ public void updateAllocationConfiguration(AllocationConfiguration queueConf) { // recursively rootQueue.updatePreemptionVariables(); } + + /** + * Check whether queue name is valid, + * return true if it is valid, otherwise return false. + */ + @VisibleForTesting + boolean isQueueNameValid(String node) { + return !node.isEmpty() && node.equals(node.trim()); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java index 3bea985edbe7a..f4fad32db5b94 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/ComputeFairShares.java @@ -71,7 +71,7 @@ public static void computeSteadyShares( * fair shares. The min and max shares and of the Schedulables are assumed to * be set beforehand. We compute the fairest possible allocation of shares to * the Schedulables that respects their min and max shares. - * + *

        * To understand what this method does, we must first define what weighted * fair sharing means in the presence of min and max shares. If there * were no minimum or maximum shares, then weighted fair sharing would be @@ -79,30 +79,31 @@ public static void computeSteadyShares( * Schedulable and all slots were assigned. Minimum and maximum shares add a * further twist - Some Schedulables may have a min share higher than their * assigned share or a max share lower than their assigned share. - * + *

        * To deal with these possibilities, we define an assignment of slots as being * fair if there exists a ratio R such that: Schedulables S where S.minShare - * > R * S.weight are given share S.minShare - Schedulables S where S.maxShare - * < R * S.weight are given S.maxShare - All other Schedulables S are - * assigned share R * S.weight - The sum of all the shares is totalSlots. - * + * {@literal >} R * S.weight are given share S.minShare - Schedulables S + * where S.maxShare {@literal <} R * S.weight are given S.maxShare - + * All other Schedulables S are assigned share R * S.weight - + * The sum of all the shares is totalSlots. + *

        * We call R the weight-to-slots ratio because it converts a Schedulable's * weight to the number of slots it is assigned. - * + *

        * We compute a fair allocation by finding a suitable weight-to-slot ratio R. * To do this, we use binary search. Given a ratio R, we compute the number of * slots that would be used in total with this ratio (the sum of the shares * computed using the conditions above). If this number of slots is less than * totalSlots, then R is too small and more slots could be assigned. If the * number of slots is more than totalSlots, then R is too large. - * + *

        * We begin the binary search with a lower bound on R of 0 (which means that * all Schedulables are only given their minShare) and an upper bound computed * to be large enough that too many slots are given (by doubling R until we * use more than totalResources resources). The helper method * resourceUsedWithWeightToResourceRatio computes the total resources used with a * given value of R. - * + *

        * The running time of this algorithm is linear in the number of Schedulables, * because resourceUsedWithWeightToResourceRatio is linear-time and the number of * iterations of binary search is a constant (dependent on desired precision). diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java index e00671536500b..b8c419cdec0d9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java @@ -54,6 +54,7 @@ import org.apache.hadoop.yarn.factories.RecordFactory; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; +import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.RMState; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEvent; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType; @@ -200,6 +201,14 @@ public String getDefaultNodeLabelExpression() { // TODO add implementation for FIFO scheduler return null; } + + @Override + public void incPendingResource(String nodeLabel, Resource resourceToInc) { + } + + @Override + public void decPendingResource(String nodeLabel, Resource resourceToDec) { + } }; public FifoScheduler() { @@ -343,9 +352,10 @@ public Allocation allocate( application.updateBlacklist(blacklistAdditions, blacklistRemovals); ContainersAndNMTokensAllocation allocation = application.pullNewlyAllocatedContainersAndNMTokens(); - return new Allocation(allocation.getContainerList(), - application.getHeadroom(), null, null, null, - allocation.getNMTokenList()); + Resource headroom = application.getHeadroom(); + application.setApplicationHeadroomForMetrics(headroom); + return new Allocation(allocation.getContainerList(), headroom, null, + null, null, allocation.getNMTokenList()); } } @@ -869,7 +879,8 @@ protected synchronized void completedContainer(RMContainer rmContainer, } // Inform the application - application.containerCompleted(rmContainer, containerStatus, event); + application.containerCompleted(rmContainer, containerStatus, event, + RMNodeLabelsManager.NO_LABEL); // Inform the node node.releaseContainer(container); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java index dfcceb8612268..261997180c477 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java @@ -364,7 +364,6 @@ public Set> getDelegationTokens() { * @param shouldCancelAtEnd true if tokens should be canceled when the app is * done else false. * @param user user - * @throws IOException */ public void addApplicationAsync(ApplicationId applicationId, Credentials ts, boolean shouldCancelAtEnd, String user) { @@ -606,6 +605,7 @@ private void requestNewHdfsDelegationToken(ApplicationId applicationId, rmContext.getSystemCredentialsForApps().put(applicationId, byteBuffer); } + @VisibleForTesting protected Token[] obtainSystemTokensForUser(String user, final Credentials credentials) throws IOException, InterruptedException { // Get new hdfs tokens on behalf of this user @@ -616,8 +616,16 @@ protected Token[] obtainSystemTokensForUser(String user, proxyUser.doAs(new PrivilegedExceptionAction[]>() { @Override public Token[] run() throws Exception { - return FileSystem.get(getConfig()).addDelegationTokens( - UserGroupInformation.getLoginUser().getUserName(), credentials); + FileSystem fs = FileSystem.get(getConfig()); + try { + return fs.addDelegationTokens( + UserGroupInformation.getLoginUser().getUserName(), + credentials); + } finally { + // Close the FileSystem created by the new proxy user, + // So that we don't leave an entry in the FileSystem cache + fs.close(); + } } }); return newTokens; @@ -634,7 +642,6 @@ private void cancelToken(DelegationTokenToRenew t) { /** * removing failed DT - * @param applicationId */ private void removeFailedDelegationToken(DelegationTokenToRenew t) { ApplicationId applicationId = t.applicationId; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppAttemptPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppAttemptPage.java new file mode 100644 index 0000000000000..df5fb9e8a5984 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppAttemptPage.java @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.resourcemanager.webapp; + +import static org.apache.hadoop.yarn.util.StringHelper.join; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES_ID; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI.initID; + +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; +import org.apache.hadoop.yarn.webapp.SubView; +import org.apache.hadoop.yarn.webapp.YarnWebParams; + + +public class AppAttemptPage extends RmView { + + @Override + protected void preHead(Page.HTML<_> html) { + commonPreHead(html); + + String appAttemptId = $(YarnWebParams.APPLICATION_ATTEMPT_ID); + set( + TITLE, + appAttemptId.isEmpty() ? "Bad request: missing application attempt ID" + : join("Application Attempt ", + $(YarnWebParams.APPLICATION_ATTEMPT_ID))); + + set(DATATABLES_ID, "containers"); + set(initID(DATATABLES, "containers"), WebPageUtils.containersTableInit()); + setTableStyles(html, "containers", ".queue {width:6em}", ".ui {width:8em}"); + + set(YarnWebParams.WEB_UI_TYPE, YarnWebParams.RM_WEB_UI); + } + + @Override + protected Class content() { + return RMAppAttemptBlock.class; + } + +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java deleted file mode 100644 index 62ad8dfce44ab..0000000000000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java +++ /dev/null @@ -1,272 +0,0 @@ -/** -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, software -* distributed under the License is distributed on an "AS IS" BASIS, -* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -* See the License for the specific language governing permissions and -* limitations under the License. -*/ - -package org.apache.hadoop.yarn.server.resourcemanager.webapp; - -import static org.apache.hadoop.yarn.util.StringHelper.join; -import static org.apache.hadoop.yarn.webapp.YarnWebParams.APPLICATION_ID; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI._EVEN; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI._ODD; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI._TH; - -import java.util.Collection; - -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.security.UserGroupInformation; -import org.apache.hadoop.util.StringUtils; -import org.apache.hadoop.yarn.api.records.ApplicationAccessType; -import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.FinalApplicationStatus; -import org.apache.hadoop.yarn.api.records.QueueACL; -import org.apache.hadoop.yarn.api.records.Resource; -import org.apache.hadoop.yarn.api.records.ResourceRequest; -import org.apache.hadoop.yarn.api.records.YarnApplicationState; -import org.apache.hadoop.yarn.server.resourcemanager.RMContext; -import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppMetrics; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics; -import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppAttemptInfo; -import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo; -import org.apache.hadoop.yarn.util.Apps; -import org.apache.hadoop.yarn.util.Times; -import org.apache.hadoop.yarn.util.resource.Resources; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TBODY; -import org.apache.hadoop.yarn.webapp.util.WebAppUtils; -import org.apache.hadoop.yarn.webapp.view.HtmlBlock; -import org.apache.hadoop.yarn.webapp.view.InfoBlock; - -import com.google.inject.Inject; - -public class AppBlock extends HtmlBlock { - - private final Configuration conf; - private final ResourceManager rm; - - @Inject - AppBlock(ResourceManager rm, ViewContext ctx, Configuration conf) { - super(ctx); - this.conf = conf; - this.rm = rm; - } - - @Override - protected void render(Block html) { - String aid = $(APPLICATION_ID); - if (aid.isEmpty()) { - puts("Bad request: requires application ID"); - return; - } - - ApplicationId appID = null; - try { - appID = Apps.toAppID(aid); - } catch (Exception e) { - puts("Invalid Application ID: " + aid); - return; - } - - RMContext context = this.rm.getRMContext(); - RMApp rmApp = context.getRMApps().get(appID); - if (rmApp == null) { - puts("Application not found: "+ aid); - return; - } - AppInfo app = - new AppInfo(rm, rmApp, true, WebAppUtils.getHttpSchemePrefix(conf)); - - // Check for the authorization. - String remoteUser = request().getRemoteUser(); - UserGroupInformation callerUGI = null; - if (remoteUser != null) { - callerUGI = UserGroupInformation.createRemoteUser(remoteUser); - } - if (callerUGI != null - && !(this.rm.getApplicationACLsManager().checkAccess(callerUGI, - ApplicationAccessType.VIEW_APP, app.getUser(), appID) || this.rm - .getQueueACLsManager().checkAccess(callerUGI, - QueueACL.ADMINISTER_QUEUE, app.getQueue()))) { - puts("You (User " + remoteUser - + ") are not authorized to view application " + appID); - return; - } - - setTitle(join("Application ", aid)); - - RMAppMetrics appMerics = rmApp.getRMAppMetrics(); - - // Get attempt metrics and fields, it is possible currentAttempt of RMApp is - // null. In that case, we will assume resource preempted and number of Non - // AM container preempted on that attempt is 0 - RMAppAttemptMetrics attemptMetrics; - if (null == rmApp.getCurrentAppAttempt()) { - attemptMetrics = null; - } else { - attemptMetrics = rmApp.getCurrentAppAttempt().getRMAppAttemptMetrics(); - } - Resource attemptResourcePreempted = - attemptMetrics == null ? Resources.none() : attemptMetrics - .getResourcePreempted(); - int attemptNumNonAMContainerPreempted = - attemptMetrics == null ? 0 : attemptMetrics - .getNumNonAMContainersPreempted(); - - info("Application Overview") - ._("User:", app.getUser()) - ._("Name:", app.getName()) - ._("Application Type:", app.getApplicationType()) - ._("Application Tags:", app.getApplicationTags()) - ._("YarnApplicationState:", clarifyAppState(app.getState())) - ._("FinalStatus Reported by AM:", - clairfyAppFinalStatus(app.getFinalStatus())) - ._("Started:", Times.format(app.getStartTime())) - ._("Elapsed:", - StringUtils.formatTime(Times.elapsed(app.getStartTime(), - app.getFinishTime()))) - ._("Tracking URL:", - !app.isTrackingUrlReady() ? "#" : app.getTrackingUrlPretty(), - app.getTrackingUI()) - ._("Diagnostics:", app.getNote()); - - DIV pdiv = html. - _(InfoBlock.class). - div(_INFO_WRAP); - info("Application Overview").clear(); - info("Application Metrics") - ._("Total Resource Preempted:", - appMerics.getResourcePreempted()) - ._("Total Number of Non-AM Containers Preempted:", - String.valueOf(appMerics.getNumNonAMContainersPreempted())) - ._("Total Number of AM Containers Preempted:", - String.valueOf(appMerics.getNumAMContainersPreempted())) - ._("Resource Preempted from Current Attempt:", - attemptResourcePreempted) - ._("Number of Non-AM Containers Preempted from Current Attempt:", - attemptNumNonAMContainerPreempted) - ._("Aggregate Resource Allocation:", - String.format("%d MB-seconds, %d vcore-seconds", - appMerics.getMemorySeconds(), appMerics.getVcoreSeconds())); - pdiv._(); - - Collection attempts = rmApp.getAppAttempts().values(); - String amString = - attempts.size() == 1 ? "ApplicationMaster" : "ApplicationMasters"; - - DIV div = html. - _(InfoBlock.class). - div(_INFO_WRAP); - // MRAppMasters Table - TABLE> table = div.table("#app"); - table. - tr(). - th(amString). - _(). - tr(). - th(_TH, "Attempt Number"). - th(_TH, "Start Time"). - th(_TH, "Node"). - th(_TH, "Logs"). - _(); - - boolean odd = false; - for (RMAppAttempt attempt : attempts) { - AppAttemptInfo attemptInfo = new AppAttemptInfo(attempt, app.getUser()); - table.tr((odd = !odd) ? _ODD : _EVEN). - td(String.valueOf(attemptInfo.getAttemptId())). - td(Times.format(attemptInfo.getStartTime())). - td().a(".nodelink", url("//", - attemptInfo.getNodeHttpAddress()), - attemptInfo.getNodeHttpAddress())._(). - td().a(".logslink", url(attemptInfo.getLogsLink()), "logs")._(). - _(); - } - - table._(); - div._(); - - createResourceRequestsTable(html, app); - } - - private void createResourceRequestsTable(Block html, AppInfo app) { - TBODY> tbody = - html.table("#ResourceRequests").thead().tr() - .th(".priority", "Priority") - .th(".resourceName", "ResourceName") - .th(".totalResource", "Capability") - .th(".numContainers", "NumContainers") - .th(".relaxLocality", "RelaxLocality") - .th(".nodeLabelExpression", "NodeLabelExpression")._()._().tbody(); - - Resource totalResource = Resource.newInstance(0, 0); - if (app.getResourceRequests() != null) { - for (ResourceRequest request : app.getResourceRequests()) { - if (request.getNumContainers() == 0) { - continue; - } - - tbody.tr() - .td(String.valueOf(request.getPriority())) - .td(request.getResourceName()) - .td(String.valueOf(request.getCapability())) - .td(String.valueOf(request.getNumContainers())) - .td(String.valueOf(request.getRelaxLocality())) - .td(request.getNodeLabelExpression() == null ? "N/A" : request - .getNodeLabelExpression())._(); - if (request.getResourceName().equals(ResourceRequest.ANY)) { - Resources.addTo(totalResource, - Resources.multiply(request.getCapability(), - request.getNumContainers())); - } - } - } - html.div().$class("totalResourceRequests") - .h3("Total Outstanding Resource Requests: " + totalResource)._(); - tbody._()._(); - } - - private String clarifyAppState(YarnApplicationState state) { - String ret = state.toString(); - switch (state) { - case NEW: - return ret + ": waiting for application to be initialized"; - case NEW_SAVING: - return ret + ": waiting for application to be persisted in state-store."; - case SUBMITTED: - return ret + ": waiting for application to be accepted by scheduler."; - case ACCEPTED: - return ret + ": waiting for AM container to be allocated, launched and" - + " register with RM."; - case RUNNING: - return ret + ": AM has registered with RM and started running."; - default: - return ret; - } - } - - private String clairfyAppFinalStatus(FinalApplicationStatus status) { - if (status == FinalApplicationStatus.UNDEFINED) { - return "Application has not completed yet."; - } - return status.toString(); - } -} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java index 899332477f491..0c5516a304ade 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java @@ -18,19 +18,37 @@ package org.apache.hadoop.yarn.server.resourcemanager.webapp; +import static org.apache.hadoop.yarn.util.StringHelper.join; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES; import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES_ID; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI.initID; +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; import org.apache.hadoop.yarn.webapp.SubView; +import org.apache.hadoop.yarn.webapp.YarnWebParams; public class AppPage extends RmView { - @Override protected void preHead(Page.HTML<_> html) { + @Override + protected void preHead(Page.HTML<_> html) { commonPreHead(html); - set(DATATABLES_ID, "ResourceRequests"); + String appId = $(YarnWebParams.APPLICATION_ID); + set( + TITLE, + appId.isEmpty() ? "Bad request: missing application ID" : join( + "Application ", $(YarnWebParams.APPLICATION_ID))); + + set(DATATABLES_ID, "attempts ResourceRequests"); + set(initID(DATATABLES, "attempts"), WebPageUtils.attemptsTableInit()); + setTableStyles(html, "attempts", ".queue {width:6em}", ".ui {width:8em}"); + setTableStyles(html, "ResourceRequests"); + + set(YarnWebParams.WEB_UI_TYPE, YarnWebParams.RM_WEB_UI); } - @Override protected Class content() { - return AppBlock.class; + @Override + protected Class content() { + return RMAppBlock.class; } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java deleted file mode 100644 index 935be612c72c0..0000000000000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java +++ /dev/null @@ -1,132 +0,0 @@ -/** -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, software -* distributed under the License is distributed on an "AS IS" BASIS, -* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -* See the License for the specific language governing permissions and -* limitations under the License. -*/ - -package org.apache.hadoop.yarn.server.resourcemanager.webapp; - -import static org.apache.hadoop.yarn.util.StringHelper.join; -import static org.apache.hadoop.yarn.webapp.YarnWebParams.APP_STATE; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI.C_PROGRESSBAR; -import static org.apache.hadoop.yarn.webapp.view.JQueryUI.C_PROGRESSBAR_VALUE; - -import java.util.Collection; -import java.util.HashSet; -import java.util.concurrent.ConcurrentMap; - -import org.apache.commons.lang.StringEscapeUtils; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.FinalApplicationStatus; -import org.apache.hadoop.yarn.api.records.YarnApplicationState; -import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; -import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; -import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TBODY; -import org.apache.hadoop.yarn.webapp.util.WebAppUtils; -import org.apache.hadoop.yarn.webapp.view.HtmlBlock; - -import com.google.inject.Inject; - -class AppsBlock extends HtmlBlock { - final ConcurrentMap apps; - private final Configuration conf; - final ResourceManager rm; - @Inject - AppsBlock(ResourceManager rm, ViewContext ctx, Configuration conf) { - super(ctx); - apps = rm.getRMContext().getRMApps(); - this.conf = conf; - this.rm = rm; - } - - @Override public void render(Block html) { - TBODY> tbody = html. - table("#apps"). - thead(). - tr(). - th(".id", "ID"). - th(".user", "User"). - th(".name", "Name"). - th(".type", "Application Type"). - th(".queue", "Queue"). - th(".starttime", "StartTime"). - th(".finishtime", "FinishTime"). - th(".state", "YarnApplicationState"). - th(".finalstatus", "FinalStatus"). - th(".progress", "Progress"). - th(".ui", "Tracking UI")._()._(). - tbody(); - Collection reqAppStates = null; - String reqStateString = $(APP_STATE); - if (reqStateString != null && !reqStateString.isEmpty()) { - String[] appStateStrings = reqStateString.split(","); - reqAppStates = new HashSet(appStateStrings.length); - for(String stateString : appStateStrings) { - reqAppStates.add(YarnApplicationState.valueOf(stateString)); - } - } - StringBuilder appsTableData = new StringBuilder("[\n"); - for (RMApp app : apps.values()) { - if (reqAppStates != null && !reqAppStates.contains(app.createApplicationState())) { - continue; - } - AppInfo appInfo = new AppInfo(rm, app, true, WebAppUtils.getHttpSchemePrefix(conf)); - String percent = String.format("%.1f", appInfo.getProgress()); - //AppID numerical value parsed by parseHadoopID in yarn.dt.plugins.js - appsTableData.append("[\"") - .append(appInfo.getAppId()).append("\",\"") - .append(StringEscapeUtils.escapeJavaScript(StringEscapeUtils.escapeHtml( - appInfo.getUser()))).append("\",\"") - .append(StringEscapeUtils.escapeJavaScript(StringEscapeUtils.escapeHtml( - appInfo.getName()))).append("\",\"") - .append(StringEscapeUtils.escapeJavaScript(StringEscapeUtils.escapeHtml( - appInfo.getApplicationType()))).append("\",\"") - .append(StringEscapeUtils.escapeJavaScript(StringEscapeUtils.escapeHtml( - appInfo.getQueue()))).append("\",\"") - .append(appInfo.getStartTime()).append("\",\"") - .append(appInfo.getFinishTime()).append("\",\"") - .append(appInfo.getState()).append("\",\"") - .append(appInfo.getFinalStatus() == FinalApplicationStatus.UNDEFINED ? - "N/A" : appInfo.getFinalStatus()).append("\",\"") - // Progress bar - .append("

        ").append("
        ") - .append("\",\"") - .append(appInfo.getTrackingUI()).append("\"],\n"); - - } - if(appsTableData.charAt(appsTableData.length() - 2) == ',') { - appsTableData.delete(appsTableData.length()-2, appsTableData.length()-1); - } - appsTableData.append("]"); - html.script().$type("text/javascript"). - _("var appsTableData=" + appsTableData)._(); - - tbody._()._(); - } -} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java index 6d461f659c743..cb0836a7e5f95 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlockWithMetrics.java @@ -18,6 +18,7 @@ package org.apache.hadoop.yarn.server.resourcemanager.webapp; +import org.apache.hadoop.yarn.server.webapp.AppsBlock; import org.apache.hadoop.yarn.webapp.view.HtmlBlock; /** diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java index 83df72b0376ac..c6c410c3ec6d3 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java @@ -31,11 +31,14 @@ import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.CapacitySchedulerLeafQueueInfo; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.CapacitySchedulerQueueInfo; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ResourceInfo; +import org.apache.hadoop.yarn.server.webapp.AppsBlock; import org.apache.hadoop.yarn.webapp.ResponseInfo; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.LI; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TBODY; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.UL; import org.apache.hadoop.yarn.webapp.view.HtmlBlock; import org.apache.hadoop.yarn.webapp.view.InfoBlock; @@ -66,41 +69,8 @@ static class LeafQueueInfoBlock extends HtmlBlock { lqinfo = (CapacitySchedulerLeafQueueInfo) info.qinfo; } - //Return a string describing one resource as a percentage of another - private String getPercentage(ResourceInfo numerator, ResourceInfo denominator) { - StringBuilder percentString = new StringBuilder("Memory: "); - if (numerator != null) { - percentString.append(numerator.getMemory()); - } - if (denominator.getMemory() != 0) { - percentString.append(" (") - .append(StringUtils.format("%.2f", numerator.getMemory() * 100.0 / - denominator.getMemory()) + "%)"); - } - percentString.append(", vCores: "); - if (numerator != null) { - percentString.append(numerator.getvCores()); - } - if (denominator.getvCores() != 0) { - percentString.append(" (") - .append(StringUtils.format("%.2f", numerator.getvCores() * 100.0 / - denominator.getvCores()) + "%)"); - } - return percentString.toString(); - } - @Override protected void render(Block html) { - StringBuilder activeUserList = new StringBuilder(""); - ResourceInfo usedResources = lqinfo.getResourcesUsed(); - ArrayList users = lqinfo.getUsers().getUsersList(); - for (UserInfo entry: users) { - activeUserList.append(entry.getUsername()).append(" <") - .append(getPercentage(entry.getResourcesUsed(), usedResources)) - .append(", Schedulable Apps: " + entry.getNumActiveApplications()) - .append(", Non-Schedulable Apps: " + entry.getNumPendingApplications()) - .append(">
        "); //Force line break - } ResponseInfo ri = info("\'" + lqinfo.getQueuePath().substring(5) + "\' Queue Status"). _("Queue State:", lqinfo.getQueueState()). @@ -115,12 +85,12 @@ protected void render(Block html) { _("Max Applications:", Integer.toString(lqinfo.getMaxApplications())). _("Max Applications Per User:", Integer.toString(lqinfo.getMaxApplicationsPerUser())). _("Max Application Master Resources:", lqinfo.getAMResourceLimit().toString()). + _("Used Application Master Resources:", lqinfo.getUsedAMResource().toString()). _("Max Application Master Resources Per User:", lqinfo.getUserAMResourceLimit().toString()). _("Configured Capacity:", percent(lqinfo.getCapacity() / 100)). _("Configured Max Capacity:", percent(lqinfo.getMaxCapacity() / 100)). _("Configured Minimum User Limit Percent:", Integer.toString(lqinfo.getUserLimit()) + "%"). _("Configured User Limit Factor:", String.format("%.1f", lqinfo.getUserLimitFactor())). - _r("Active Users: ", activeUserList.toString()). _("Accessible Node Labels:", StringUtils.join(",", lqinfo.getNodeLabels())). _("Preemption:", lqinfo.getPreemptionDisabled() ? "disabled" : "enabled"); @@ -131,6 +101,44 @@ protected void render(Block html) { } } + static class QueueUsersInfoBlock extends HtmlBlock { + final CapacitySchedulerLeafQueueInfo lqinfo; + + @Inject + QueueUsersInfoBlock(ViewContext ctx, CSQInfo info) { + super(ctx); + lqinfo = (CapacitySchedulerLeafQueueInfo) info.qinfo; + } + + @Override + protected void render(Block html) { + TBODY> tbody = + html.table("#userinfo").thead().$class("ui-widget-header").tr().th() + .$class("ui-state-default")._("User Name")._().th() + .$class("ui-state-default")._("Max Resource")._().th() + .$class("ui-state-default")._("Used Resource")._().th() + .$class("ui-state-default")._("Max AM Resource")._().th() + .$class("ui-state-default")._("Used AM Resource")._().th() + .$class("ui-state-default")._("Schedulable Apps")._().th() + .$class("ui-state-default")._("Non-Schedulable Apps")._()._()._() + .tbody(); + + ArrayList users = lqinfo.getUsers().getUsersList(); + for (UserInfo userInfo : users) { + tbody.tr().td(userInfo.getUsername()) + .td(userInfo.getUserResourceLimit().toString()) + .td(userInfo.getResourcesUsed().toString()) + .td(lqinfo.getUserAMResourceLimit().toString()) + .td(userInfo.getAMResourcesUsed().toString()) + .td(Integer.toString(userInfo.getNumActiveApplications())) + .td(Integer.toString(userInfo.getNumPendingApplications()))._(); + } + + html.div().$class("usersinfo").h5("Active Users Info")._(); + tbody._()._(); + } + } + public static class QueueBlock extends HtmlBlock { final CSQInfo csqinfo; @@ -165,6 +173,7 @@ public void render(Block html) { csqinfo.qinfo = info; if (info.getQueues() == null) { li.ul("#lq").li()._(LeafQueueInfoBlock.class)._()._(); + li.ul("#lq").li()._(QueueUsersInfoBlock.class)._()._(); } else { li._(QueueBlock.class); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/ContainerPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/ContainerPage.java new file mode 100644 index 0000000000000..b8cd1adbd5827 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/ContainerPage.java @@ -0,0 +1,44 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.resourcemanager.webapp; + +import static org.apache.hadoop.yarn.util.StringHelper.join; + +import org.apache.hadoop.yarn.server.webapp.ContainerBlock; +import org.apache.hadoop.yarn.webapp.SubView; +import org.apache.hadoop.yarn.webapp.YarnWebParams; + + +public class ContainerPage extends RmView { + + @Override + protected void preHead(Page.HTML<_> html) { + commonPreHead(html); + + String containerId = $(YarnWebParams.CONTAINER_ID); + set(TITLE, containerId.isEmpty() ? "Bad request: missing container ID" + : join("Container ", $(YarnWebParams.CONTAINER_ID))); + } + + @Override + protected Class content() { + return ContainerBlock.class; + } + +} \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java index e05987bb5949c..1c8828cf7e0da 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java @@ -23,6 +23,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.FifoSchedulerInfo; +import org.apache.hadoop.yarn.server.webapp.AppsBlock; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java index 8c54f4e62ee99..97ab872d15a59 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/FairSchedulerPage.java @@ -27,6 +27,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.FairSchedulerInfo; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.FairSchedulerLeafQueueInfo; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.FairSchedulerQueueInfo; +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; import org.apache.hadoop.yarn.webapp.ResponseInfo; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; @@ -234,6 +235,11 @@ public void render(Block html) { return QueuesBlock.class; } + @Override + protected String initAppsTable() { + return WebPageUtils.appsTableInit(true); + } + static String percent(float f) { return String.format("%.1f%%", f * 100); } @@ -245,19 +251,4 @@ static String width(float f) { static String left(float f) { return String.format("left:%.1f%%", f * 100); } - - @Override - protected String getAppsTableColumnDefs() { - StringBuilder sb = new StringBuilder(); - return sb - .append("[\n") - .append("{'sType':'numeric', 'aTargets': [0]") - .append(", 'mRender': parseHadoopID }") - - .append("\n, {'sType':'numeric', 'aTargets': [6, 7]") - .append(", 'mRender': renderHadoopDate }") - - .append("\n, {'sType':'numeric', bSearchable:false, 'aTargets': [9]") - .append(", 'mRender': parseHadoopProgress }]").toString(); - } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java index 88efe47e71f63..7ee2ca4e10baf 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java @@ -21,6 +21,7 @@ import org.apache.hadoop.util.StringUtils; import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo; +import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.SchedulerInfo; import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.UserMetricsInfo; import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; @@ -153,6 +154,27 @@ protected void render(Block html) { } } + + SchedulerInfo schedulerInfo=new SchedulerInfo(this.rm); + + div.h3("Scheduler Metrics"). + table("#schedulermetricsoverview"). + thead().$class("ui-widget-header"). + tr(). + th().$class("ui-state-default")._("Scheduler Type")._(). + th().$class("ui-state-default")._("Scheduling Resource Type")._(). + th().$class("ui-state-default")._("Minimum Allocation")._(). + th().$class("ui-state-default")._("Maximum Allocation")._(). + _(). + _(). + tbody().$class("ui-widget-content"). + tr(). + td(String.valueOf(schedulerInfo.getSchedulerType())). + td(String.valueOf(schedulerInfo.getSchedulerResourceTypes())). + td(schedulerInfo.getMinAllocation().toString()). + td(schedulerInfo.getMaxAllocation().toString()). + _(). + _()._(); div._(); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodeLabelsPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodeLabelsPage.java index 5e8c1ed5bf2ec..7458558755311 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodeLabelsPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodeLabelsPage.java @@ -20,7 +20,7 @@ import static org.apache.hadoop.yarn.webapp.view.JQueryUI.DATATABLES_ID; -import org.apache.hadoop.yarn.nodelabels.NodeLabel; +import org.apache.hadoop.yarn.nodelabels.RMNodeLabel; import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.webapp.SubView; @@ -55,7 +55,7 @@ protected void render(Block html) { tbody(); RMNodeLabelsManager nlm = rm.getRMContext().getNodeLabelManager(); - for (NodeLabel info : nlm.pullRMNodeLabelsInfo()) { + for (RMNodeLabel info : nlm.pullRMNodeLabelsInfo()) { TR>> row = tbody.tr().td( info.getLabelName().isEmpty() ? "" : info diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java new file mode 100644 index 0000000000000..419c0ce67539e --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppAttemptBlock.java @@ -0,0 +1,177 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.resourcemanager.webapp; + +import static org.apache.hadoop.yarn.webapp.view.JQueryUI._EVEN; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI._ODD; +import static org.apache.hadoop.yarn.webapp.view.JQueryUI._TH; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.Resource; +import org.apache.hadoop.yarn.api.records.ResourceRequest; +import org.apache.hadoop.yarn.api.records.YarnApplicationAttemptState; +import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics; +import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo; +import org.apache.hadoop.yarn.server.webapp.AppAttemptBlock; +import org.apache.hadoop.yarn.util.resource.Resources; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TABLE; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.TBODY; +import org.apache.hadoop.yarn.webapp.util.WebAppUtils; +import org.apache.hadoop.yarn.webapp.view.InfoBlock; +import com.google.inject.Inject; + +public class RMAppAttemptBlock extends AppAttemptBlock{ + + private final ResourceManager rm; + protected Configuration conf; + + @Inject + RMAppAttemptBlock(ViewContext ctx, ResourceManager rm, Configuration conf) { + super(rm.getClientRMService(), ctx); + this.rm = rm; + this.conf = conf; + } + + @Override + protected void render(Block html) { + super.render(html); + createContainerLocalityTable(html); + createResourceRequestsTable(html); + } + + private void createResourceRequestsTable(Block html) { + AppInfo app = + new AppInfo(rm, rm.getRMContext().getRMApps() + .get(this.appAttemptId.getApplicationId()), true, + WebAppUtils.getHttpSchemePrefix(conf)); + TBODY> tbody = + html.table("#ResourceRequests").thead().tr() + .th(".priority", "Priority") + .th(".resourceName", "ResourceName") + .th(".totalResource", "Capability") + .th(".numContainers", "NumContainers") + .th(".relaxLocality", "RelaxLocality") + .th(".nodeLabelExpression", "NodeLabelExpression")._()._().tbody(); + + Resource totalResource = Resource.newInstance(0, 0); + if (app.getResourceRequests() != null) { + for (ResourceRequest request : app.getResourceRequests()) { + if (request.getNumContainers() == 0) { + continue; + } + + tbody.tr() + .td(String.valueOf(request.getPriority())) + .td(request.getResourceName()) + .td(String.valueOf(request.getCapability())) + .td(String.valueOf(request.getNumContainers())) + .td(String.valueOf(request.getRelaxLocality())) + .td(request.getNodeLabelExpression() == null ? "N/A" : request + .getNodeLabelExpression())._(); + if (request.getResourceName().equals(ResourceRequest.ANY)) { + Resources.addTo(totalResource, + Resources.multiply(request.getCapability(), + request.getNumContainers())); + } + } + } + html.div().$class("totalResourceRequests") + .h3("Total Outstanding Resource Requests: " + totalResource)._(); + tbody._()._(); + } + + private void createContainerLocalityTable(Block html) { + RMAppAttemptMetrics attemptMetrics = null; + RMAppAttempt attempt = getRMAppAttempt(); + if (attempt != null) { + attemptMetrics = attempt.getRMAppAttemptMetrics(); + } + + if (attemptMetrics == null) { + return; + } + + DIV div = html.div(_INFO_WRAP); + TABLE> table = + div.h3( + "Total Allocated Containers: " + + attemptMetrics.getTotalAllocatedContainers()).h3("Each table cell" + + " represents the number of NodeLocal/RackLocal/OffSwitch containers" + + " satisfied by NodeLocal/RackLocal/OffSwitch resource requests.").table( + "#containerLocality"); + table. + tr(). + th(_TH, ""). + th(_TH, "Node Local Request"). + th(_TH, "Rack Local Request"). + th(_TH, "Off Switch Request"). + _(); + + String[] containersType = + { "Num Node Local Containers (satisfied by)", "Num Rack Local Containers (satisfied by)", + "Num Off Switch Containers (satisfied by)" }; + boolean odd = false; + for (int i = 0; i < attemptMetrics.getLocalityStatistics().length; i++) { + table.tr((odd = !odd) ? _ODD : _EVEN).td(containersType[i]) + .td(String.valueOf(attemptMetrics.getLocalityStatistics()[i][0])) + .td(i == 0 ? "" : String.valueOf(attemptMetrics.getLocalityStatistics()[i][1])) + .td(i <= 1 ? "" : String.valueOf(attemptMetrics.getLocalityStatistics()[i][2]))._(); + } + table._(); + div._(); + } + + private boolean isApplicationInFinalState(YarnApplicationAttemptState state) { + return state == YarnApplicationAttemptState.FINISHED + || state == YarnApplicationAttemptState.FAILED + || state == YarnApplicationAttemptState.KILLED; + } + + @Override + protected void createAttemptHeadRoomTable(Block html) { + RMAppAttempt attempt = getRMAppAttempt(); + if (attempt != null) { + if (!isApplicationInFinalState(YarnApplicationAttemptState + .valueOf(attempt.getAppAttemptState().toString()))) { + DIV pdiv = html._(InfoBlock.class).div(_INFO_WRAP); + info("Application Attempt Overview").clear(); + info("Application Attempt Metrics")._( + "Application Attempt Headroom : ", 0); + pdiv._(); + } + } + } + + private RMAppAttempt getRMAppAttempt() { + ApplicationId appId = this.appAttemptId.getApplicationId(); + RMAppAttempt attempt = null; + RMApp rmApp = rm.getRMContext().getRMApps().get(appId); + if (rmApp != null) { + attempt = rmApp.getAppAttempts().get(appAttemptId); + } + return attempt; + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java new file mode 100644 index 0000000000000..64c57476f7951 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.resourcemanager.webapp; + +import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.yarn.api.records.Resource; +import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppMetrics; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics; +import org.apache.hadoop.yarn.server.webapp.AppBlock; +import org.apache.hadoop.yarn.util.resource.Resources; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet; +import org.apache.hadoop.yarn.webapp.hamlet.Hamlet.DIV; +import org.apache.hadoop.yarn.webapp.view.InfoBlock; + +import com.google.inject.Inject; + +public class RMAppBlock extends AppBlock{ + + private final ResourceManager rm; + + @Inject + RMAppBlock(ViewContext ctx, Configuration conf, ResourceManager rm) { + super(rm.getClientRMService(), ctx, conf); + this.rm = rm; + } + + @Override + protected void render(Block html) { + super.render(html); + } + + @Override + protected void createApplicationMetricsTable(Block html){ + RMApp rmApp = this.rm.getRMContext().getRMApps().get(appID); + RMAppMetrics appMetrics = rmApp == null ? null : rmApp.getRMAppMetrics(); + // Get attempt metrics and fields, it is possible currentAttempt of RMApp is + // null. In that case, we will assume resource preempted and number of Non + // AM container preempted on that attempt is 0 + RMAppAttemptMetrics attemptMetrics; + if (rmApp == null || null == rmApp.getCurrentAppAttempt()) { + attemptMetrics = null; + } else { + attemptMetrics = rmApp.getCurrentAppAttempt().getRMAppAttemptMetrics(); + } + Resource attemptResourcePreempted = + attemptMetrics == null ? Resources.none() : attemptMetrics + .getResourcePreempted(); + int attemptNumNonAMContainerPreempted = + attemptMetrics == null ? 0 : attemptMetrics + .getNumNonAMContainersPreempted(); + DIV pdiv = html. + _(InfoBlock.class). + div(_INFO_WRAP); + info("Application Overview").clear(); + info("Application Metrics") + ._("Total Resource Preempted:", + appMetrics == null ? "N/A" : appMetrics.getResourcePreempted()) + ._("Total Number of Non-AM Containers Preempted:", + appMetrics == null ? "N/A" + : appMetrics.getNumNonAMContainersPreempted()) + ._("Total Number of AM Containers Preempted:", + appMetrics == null ? "N/A" + : appMetrics.getNumAMContainersPreempted()) + ._("Resource Preempted from Current Attempt:", + attemptResourcePreempted) + ._("Number of Non-AM Containers Preempted from Current Attempt:", + attemptNumNonAMContainerPreempted) + ._("Aggregate Resource Allocation:", + String.format("%d MB-seconds, %d vcore-seconds", + appMetrics == null ? "N/A" : appMetrics.getMemorySeconds(), + appMetrics == null ? "N/A" : appMetrics.getVcoreSeconds())); + pdiv._(); + } +} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java index c0e68349e4ab5..4189053d16c27 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java @@ -23,6 +23,7 @@ import java.net.InetSocketAddress; import org.apache.hadoop.ha.HAServiceProtocol.HAServiceState; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.util.RMHAUtils; import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; @@ -53,6 +54,7 @@ public void setup() { if (rm != null) { bind(ResourceManager.class).toInstance(rm); + bind(ApplicationBaseProtocol.class).toInstance(rm.getClientRMService()); } route("/", RmController.class); route(pajoin("/nodes", NODE_STATE), RmController.class, "nodes"); @@ -62,6 +64,9 @@ public void setup() { route("/scheduler", RmController.class, "scheduler"); route(pajoin("/queue", QUEUE_NAME), RmController.class, "queue"); route("/nodelabels", RmController.class, "nodelabels"); + route(pajoin("/appattempt", APPLICATION_ATTEMPT_ID), RmController.class, + "appattempt"); + route(pajoin("/container", CONTAINER_ID), RmController.class, "container"); } @Override diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java index 972432b9882d1..cc5232ec81483 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java @@ -54,6 +54,14 @@ public void app() { render(AppPage.class); } + public void appattempt() { + render(AppAttemptPage.class); + } + + public void container() { + render(ContainerPage.class); + } + public void nodes() { render(NodesPage.class); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java index 769c4da692198..1a437f834b723 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java @@ -18,6 +18,7 @@ package org.apache.hadoop.yarn.server.resourcemanager.webapp; +import org.apache.hadoop.yarn.server.webapp.WebPageUtils; import org.apache.hadoop.yarn.webapp.SubView; import org.apache.hadoop.yarn.webapp.view.TwoColumnLayout; @@ -35,7 +36,7 @@ public class RmView extends TwoColumnLayout { protected void preHead(Page.HTML<_> html) { commonPreHead(html); set(DATATABLES_ID, "apps"); - set(initID(DATATABLES, "apps"), appsTableInit()); + set(initID(DATATABLES, "apps"), initAppsTable()); setTableStyles(html, "apps", ".queue {width:6em}", ".ui {width:8em}"); // Set the correct title. @@ -59,31 +60,7 @@ protected Class content() { return AppsBlockWithMetrics.class; } - private String appsTableInit() { - // id, user, name, queue, starttime, finishtime, state, status, progress, ui - return tableInit() - .append(", 'aaData': appsTableData") - .append(", bDeferRender: true") - .append(", bProcessing: true") - - .append("\n, aoColumnDefs: ") - .append(getAppsTableColumnDefs()) - - // Sort by id upon page load - .append(", aaSorting: [[0, 'desc']]}").toString(); - } - - protected String getAppsTableColumnDefs() { - StringBuilder sb = new StringBuilder(); - return sb - .append("[\n") - .append("{'sType':'string', 'aTargets': [0]") - .append(", 'mRender': parseHadoopID }") - - .append("\n, {'sType':'numeric', 'aTargets': [5, 6]") - .append(", 'mRender': renderHadoopDate }") - - .append("\n, {'sType':'numeric', bSearchable:false, 'aTargets': [9]") - .append(", 'mRender': parseHadoopProgress }]").toString(); + protected String initAppsTable() { + return WebPageUtils.appsTableInit(); } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java index a8b0d32343e7c..5258b3d1a2cc9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java @@ -35,7 +35,8 @@ public class CapacitySchedulerLeafQueueInfo extends CapacitySchedulerQueueInfo { protected int userLimit; protected UsersInfo users; // To add another level in the XML protected float userLimitFactor; - protected ResourceInfo aMResourceLimit; + protected ResourceInfo AMResourceLimit; + protected ResourceInfo usedAMResource; protected ResourceInfo userAMResourceLimit; protected boolean preemptionDisabled; @@ -52,7 +53,8 @@ public class CapacitySchedulerLeafQueueInfo extends CapacitySchedulerQueueInfo { userLimit = q.getUserLimit(); users = new UsersInfo(q.getUsers()); userLimitFactor = q.getUserLimitFactor(); - aMResourceLimit = new ResourceInfo(q.getAMResourceLimit()); + AMResourceLimit = new ResourceInfo(q.getAMResourceLimit()); + usedAMResource = new ResourceInfo(q.getQueueResourceUsage().getAMUsed()); userAMResourceLimit = new ResourceInfo(q.getUserAMResourceLimit()); preemptionDisabled = q.getPreemptionDisabled(); } @@ -91,9 +93,13 @@ public float getUserLimitFactor() { } public ResourceInfo getAMResourceLimit() { - return aMResourceLimit; + return AMResourceLimit; } + public ResourceInfo getUsedAMResource() { + return usedAMResource; + } + public ResourceInfo getUserAMResourceLimit() { return userAMResourceLimit; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodesInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodesInfo.java index 7be9a6f743509..7dacd1021a724 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodesInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/NodesInfo.java @@ -36,4 +36,7 @@ public void add(NodeInfo nodeinfo) { node.add(nodeinfo); } + public ArrayList getNodes() { + return node; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java index 898aef76823bb..185b449dbcf56 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerInfo.java @@ -18,15 +18,59 @@ package org.apache.hadoop.yarn.server.resourcemanager.webapp.dao; +import java.util.EnumSet; + import javax.xml.bind.annotation.XmlRootElement; import javax.xml.bind.annotation.XmlSeeAlso; +import org.apache.hadoop.yarn.proto.YarnServiceProtos.SchedulerResourceTypes; +import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler; + @XmlRootElement @XmlSeeAlso({ CapacitySchedulerInfo.class, FairSchedulerInfo.class, FifoSchedulerInfo.class }) public class SchedulerInfo { + protected String schedulerName; + protected ResourceInfo minAllocResource; + protected ResourceInfo maxAllocResource; + protected EnumSet schedulingResourceTypes; public SchedulerInfo() { } // JAXB needs this + public SchedulerInfo(final ResourceManager rm) { + ResourceScheduler rs = rm.getResourceScheduler(); + + if (rs instanceof CapacityScheduler) { + this.schedulerName = "Capacity Scheduler"; + } else if (rs instanceof FairScheduler) { + this.schedulerName = "Fair Scheduler"; + } else if (rs instanceof FifoScheduler) { + this.schedulerName = "Fifo Scheduler"; + } + this.minAllocResource = new ResourceInfo(rs.getMinimumResourceCapability()); + this.maxAllocResource = new ResourceInfo(rs.getMaximumResourceCapability()); + this.schedulingResourceTypes = rs.getSchedulingResourceTypes(); + } + + public String getSchedulerType() { + return this.schedulerName; + } + + public ResourceInfo getMinAllocation() { + return this.minAllocResource; + } + + public ResourceInfo getMaxAllocation() { + return this.maxAllocResource; + } + + public String getSchedulerResourceTypes() { + return this.schedulingResourceTypes.toString(); + } + } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerTypeInfo.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerTypeInfo.java index 34078f120d64a..22018d0561b9c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerTypeInfo.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/SchedulerTypeInfo.java @@ -34,4 +34,7 @@ public SchedulerTypeInfo(final SchedulerInfo scheduler) { this.schedulerInfo = scheduler; } + public SchedulerInfo getSchedulerInfo() { + return schedulerInfo; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java index 494f5a412f642..f62fdb3dcee22 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java @@ -164,17 +164,19 @@ public List createReq(String[] hosts, int memory, int priority, public List createReq(String[] hosts, int memory, int priority, int containers, String labelExpression) throws Exception { List reqs = new ArrayList(); - for (String host : hosts) { - // only add host/rack request when asked host isn't ANY - if (!host.equals(ResourceRequest.ANY)) { - ResourceRequest hostReq = - createResourceReq(host, memory, priority, containers, - labelExpression); - reqs.add(hostReq); - ResourceRequest rackReq = - createResourceReq("/default-rack", memory, priority, containers, - labelExpression); - reqs.add(rackReq); + if (hosts != null) { + for (String host : hosts) { + // only add host/rack request when asked host isn't ANY + if (!host.equals(ResourceRequest.ANY)) { + ResourceRequest hostReq = + createResourceReq(host, memory, priority, containers, + labelExpression); + reqs.add(hostReq); + ResourceRequest rackReq = + createResourceReq("/default-rack", memory, priority, containers, + labelExpression); + reqs.add(rackReq); + } } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java index 5f53805c21a4e..c917f7976b006 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java @@ -51,7 +51,7 @@ public class MockNM { private final int memory; private final int vCores; private ResourceTrackerService resourceTracker; - private final int httpPort = 2; + private int httpPort = 2; private MasterKey currentContainerTokenMasterKey; private MasterKey currentNMTokenMasterKey; private String version; @@ -87,6 +87,10 @@ public int getHttpPort() { return httpPort; } + public void setHttpPort(int port) { + httpPort = port; + } + public void setResourceTrackerService(ResourceTrackerService resourceTracker) { this.resourceTracker = resourceTracker; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java index d2ac4efb9572b..5ebc68ca18fe8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java @@ -67,6 +67,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler; import org.apache.hadoop.yarn.server.resourcemanager.security.ClientToAMTokenSecretManagerInRM; import org.apache.hadoop.yarn.server.security.ApplicationACLsManager; +import org.apache.hadoop.yarn.util.resource.ResourceCalculator; import org.apache.hadoop.yarn.util.resource.Resources; import org.junit.After; import org.junit.Before; @@ -604,6 +605,10 @@ private static ResourceScheduler mockResourceScheduler() { when(scheduler.getMaximumResourceCapability()).thenReturn( Resources.createResource( YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB)); + + ResourceCalculator rs = mock(ResourceCalculator.class); + when(scheduler.getResourceCalculator()).thenReturn(rs); + return scheduler; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java index d1f0ede4a3329..ca5c7a41e55d9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java @@ -55,7 +55,8 @@ import static java.lang.Thread.sleep; public class TestApplicationMasterService { - private static final Log LOG = LogFactory.getLog(TestFifoScheduler.class); + private static final Log LOG = LogFactory + .getLog(TestApplicationMasterService.class); private final int GB = 1024; private static YarnConfiguration conf; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java index dd2b3f837955d..430763cc3efa5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java @@ -146,6 +146,7 @@ import org.apache.hadoop.yarn.util.Clock; import org.apache.hadoop.yarn.util.Records; import org.apache.hadoop.yarn.util.UTCClock; +import org.apache.hadoop.yarn.util.resource.ResourceCalculator; import org.apache.hadoop.yarn.util.resource.Resources; import org.junit.AfterClass; import org.junit.Assert; @@ -1289,6 +1290,10 @@ private static YarnScheduler mockYarnScheduler() { Arrays.asList(getApplicationAttemptId(103))); ApplicationAttemptId attemptId = getApplicationAttemptId(1); when(yarnScheduler.getAppResourceUsageReport(attemptId)).thenReturn(null); + + ResourceCalculator rs = mock(ResourceCalculator.class); + when(yarnScheduler.getResourceCalculator()).thenReturn(rs); + return yarnScheduler; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java deleted file mode 100644 index 95aa856fcefa5..0000000000000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java +++ /dev/null @@ -1,607 +0,0 @@ -/** -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, software -* distributed under the License is distributed on an "AS IS" BASIS, -* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -* See the License for the specific language governing permissions and -* limitations under the License. -*/ - -package org.apache.hadoop.yarn.server.resourcemanager; - -import static org.junit.Assert.assertTrue; -import static org.junit.Assert.fail; - -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse; -import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; -import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.api.records.Container; -import org.apache.hadoop.yarn.api.records.ContainerId; -import org.apache.hadoop.yarn.api.records.ContainerState; -import org.apache.hadoop.yarn.api.records.ContainerStatus; -import org.apache.hadoop.yarn.api.records.NodeId; -import org.apache.hadoop.yarn.api.records.Resource; -import org.apache.hadoop.yarn.api.records.ResourceOption; -import org.apache.hadoop.yarn.api.records.ResourceRequest; -import org.apache.hadoop.yarn.conf.YarnConfiguration; -import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; -import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceRequest; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; -import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; -import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.Allocation; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNodeReport; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAddedSchedulerEvent; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAttemptAddedSchedulerEvent; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeAddedSchedulerEvent; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeRemovedSchedulerEvent; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeUpdateSchedulerEvent; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEvent; -import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler; -import org.apache.hadoop.yarn.server.utils.BuilderUtils; -import org.apache.hadoop.yarn.util.resource.Resources; -import org.apache.log4j.Level; -import org.apache.log4j.LogManager; -import org.apache.log4j.Logger; -import org.junit.Assert; -import org.junit.BeforeClass; -import org.junit.Test; -import static org.mockito.Mockito.mock; - -public class TestFifoScheduler { - private static final Log LOG = LogFactory.getLog(TestFifoScheduler.class); - - private final int GB = 1024; - private static YarnConfiguration conf; - - @BeforeClass - public static void setup() { - conf = new YarnConfiguration(); - conf.setClass(YarnConfiguration.RM_SCHEDULER, - FifoScheduler.class, ResourceScheduler.class); - } - - @Test (timeout = 30000) - public void testConfValidation() throws Exception { - FifoScheduler scheduler = new FifoScheduler(); - Configuration conf = new YarnConfiguration(); - conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 2048); - conf.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 1024); - try { - scheduler.serviceInit(conf); - fail("Exception is expected because the min memory allocation is" + - " larger than the max memory allocation."); - } catch (YarnRuntimeException e) { - // Exception is expected. - assertTrue("The thrown exception is not the expected one.", - e.getMessage().startsWith( - "Invalid resource scheduler memory")); - } - } - - @Test - public void testAllocateContainerOnNodeWithoutOffSwitchSpecified() - throws Exception { - Logger rootLogger = LogManager.getRootLogger(); - rootLogger.setLevel(Level.DEBUG); - - MockRM rm = new MockRM(conf); - rm.start(); - MockNM nm1 = rm.registerNode("127.0.0.1:1234", 6 * GB); - - RMApp app1 = rm.submitApp(2048); - // kick the scheduling, 2 GB given to AM1, remaining 4GB on nm1 - nm1.nodeHeartbeat(true); - RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); - MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); - am1.registerAppAttempt(); - - // add request for containers - List requests = new ArrayList(); - requests.add(am1.createResourceReq("127.0.0.1", 1 * GB, 1, 1)); - requests.add(am1.createResourceReq("/default-rack", 1 * GB, 1, 1)); - am1.allocate(requests, null); // send the request - - try { - // kick the schedule - nm1.nodeHeartbeat(true); - } catch (NullPointerException e) { - Assert.fail("NPE when allocating container on node but " - + "forget to set off-switch request should be handled"); - } - rm.stop(); - } - - @Test - public void test() throws Exception { - Logger rootLogger = LogManager.getRootLogger(); - rootLogger.setLevel(Level.DEBUG); - MockRM rm = new MockRM(conf); - rm.start(); - MockNM nm1 = rm.registerNode("127.0.0.1:1234", 6 * GB); - MockNM nm2 = rm.registerNode("127.0.0.2:5678", 4 * GB); - - RMApp app1 = rm.submitApp(2048); - // kick the scheduling, 2 GB given to AM1, remaining 4GB on nm1 - nm1.nodeHeartbeat(true); - RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); - MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); - am1.registerAppAttempt(); - SchedulerNodeReport report_nm1 = rm.getResourceScheduler().getNodeReport( - nm1.getNodeId()); - Assert.assertEquals(2 * GB, report_nm1.getUsedResource().getMemory()); - - RMApp app2 = rm.submitApp(2048); - // kick the scheduling, 2GB given to AM, remaining 2 GB on nm2 - nm2.nodeHeartbeat(true); - RMAppAttempt attempt2 = app2.getCurrentAppAttempt(); - MockAM am2 = rm.sendAMLaunched(attempt2.getAppAttemptId()); - am2.registerAppAttempt(); - SchedulerNodeReport report_nm2 = rm.getResourceScheduler().getNodeReport( - nm2.getNodeId()); - Assert.assertEquals(2 * GB, report_nm2.getUsedResource().getMemory()); - - // add request for containers - am1.addRequests(new String[] { "127.0.0.1", "127.0.0.2" }, GB, 1, 1); - AllocateResponse alloc1Response = am1.schedule(); // send the request - // add request for containers - am2.addRequests(new String[] { "127.0.0.1", "127.0.0.2" }, 3 * GB, 0, 1); - AllocateResponse alloc2Response = am2.schedule(); // send the request - - // kick the scheduler, 1 GB and 3 GB given to AM1 and AM2, remaining 0 - nm1.nodeHeartbeat(true); - while (alloc1Response.getAllocatedContainers().size() < 1) { - LOG.info("Waiting for containers to be created for app 1..."); - Thread.sleep(1000); - alloc1Response = am1.schedule(); - } - while (alloc2Response.getAllocatedContainers().size() < 1) { - LOG.info("Waiting for containers to be created for app 2..."); - Thread.sleep(1000); - alloc2Response = am2.schedule(); - } - // kick the scheduler, nothing given remaining 2 GB. - nm2.nodeHeartbeat(true); - - List allocated1 = alloc1Response.getAllocatedContainers(); - Assert.assertEquals(1, allocated1.size()); - Assert.assertEquals(1 * GB, allocated1.get(0).getResource().getMemory()); - Assert.assertEquals(nm1.getNodeId(), allocated1.get(0).getNodeId()); - - List allocated2 = alloc2Response.getAllocatedContainers(); - Assert.assertEquals(1, allocated2.size()); - Assert.assertEquals(3 * GB, allocated2.get(0).getResource().getMemory()); - Assert.assertEquals(nm1.getNodeId(), allocated2.get(0).getNodeId()); - - report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); - report_nm2 = rm.getResourceScheduler().getNodeReport(nm2.getNodeId()); - Assert.assertEquals(0, report_nm1.getAvailableResource().getMemory()); - Assert.assertEquals(2 * GB, report_nm2.getAvailableResource().getMemory()); - - Assert.assertEquals(6 * GB, report_nm1.getUsedResource().getMemory()); - Assert.assertEquals(2 * GB, report_nm2.getUsedResource().getMemory()); - - Container c1 = allocated1.get(0); - Assert.assertEquals(GB, c1.getResource().getMemory()); - ContainerStatus containerStatus = BuilderUtils.newContainerStatus( - c1.getId(), ContainerState.COMPLETE, "", 0); - nm1.containerStatus(containerStatus); - int waitCount = 0; - while (attempt1.getJustFinishedContainers().size() < 1 - && waitCount++ != 20) { - LOG.info("Waiting for containers to be finished for app 1... Tried " - + waitCount + " times already.."); - Thread.sleep(1000); - } - Assert.assertEquals(1, attempt1.getJustFinishedContainers().size()); - Assert.assertEquals(1, am1.schedule().getCompletedContainersStatuses().size()); - report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); - Assert.assertEquals(5 * GB, report_nm1.getUsedResource().getMemory()); - - rm.stop(); - } - - @Test - public void testNodeUpdateBeforeAppAttemptInit() throws Exception { - FifoScheduler scheduler = new FifoScheduler(); - MockRM rm = new MockRM(conf); - scheduler.setRMContext(rm.getRMContext()); - scheduler.init(conf); - scheduler.start(); - scheduler.reinitialize(conf, rm.getRMContext()); - - RMNode node = MockNodes.newNodeInfo(1, - Resources.createResource(1024, 4), 1, "127.0.0.1"); - scheduler.handle(new NodeAddedSchedulerEvent(node)); - - ApplicationId appId = ApplicationId.newInstance(0, 1); - scheduler.addApplication(appId, "queue1", "user1", false); - - NodeUpdateSchedulerEvent updateEvent = new NodeUpdateSchedulerEvent(node); - try { - scheduler.handle(updateEvent); - } catch (NullPointerException e) { - Assert.fail(); - } - - ApplicationAttemptId attId = ApplicationAttemptId.newInstance(appId, 1); - scheduler.addApplicationAttempt(attId, false, false); - - rm.stop(); - } - - private void testMinimumAllocation(YarnConfiguration conf, int testAlloc) - throws Exception { - MockRM rm = new MockRM(conf); - rm.start(); - - // Register node1 - MockNM nm1 = rm.registerNode("127.0.0.1:1234", 6 * GB); - - // Submit an application - RMApp app1 = rm.submitApp(testAlloc); - - // kick the scheduling - nm1.nodeHeartbeat(true); - RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); - MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); - am1.registerAppAttempt(); - SchedulerNodeReport report_nm1 = rm.getResourceScheduler().getNodeReport( - nm1.getNodeId()); - - int checkAlloc = - conf.getInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, - YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); - Assert.assertEquals(checkAlloc, report_nm1.getUsedResource().getMemory()); - - rm.stop(); - } - - @Test - public void testDefaultMinimumAllocation() throws Exception { - // Test with something lesser than default - testMinimumAllocation( - new YarnConfiguration(TestFifoScheduler.conf), - YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB / 2); - } - - @Test - public void testNonDefaultMinimumAllocation() throws Exception { - // Set custom min-alloc to test tweaking it - int allocMB = 1536; - YarnConfiguration conf = new YarnConfiguration(TestFifoScheduler.conf); - conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, allocMB); - conf.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, - allocMB * 10); - // Test for something lesser than this. - testMinimumAllocation(conf, allocMB / 2); - } - - @Test (timeout = 50000) - public void testReconnectedNode() throws Exception { - CapacitySchedulerConfiguration conf = new CapacitySchedulerConfiguration(); - conf.setQueues("default", new String[] {"default"}); - conf.setCapacity("default", 100); - FifoScheduler fs = new FifoScheduler(); - fs.init(conf); - fs.start(); - // mock rmContext to avoid NPE. - RMContext context = mock(RMContext.class); - fs.reinitialize(conf, null); - fs.setRMContext(context); - - RMNode n1 = - MockNodes.newNodeInfo(0, MockNodes.newResource(4 * GB), 1, "127.0.0.2"); - RMNode n2 = - MockNodes.newNodeInfo(0, MockNodes.newResource(2 * GB), 2, "127.0.0.3"); - - fs.handle(new NodeAddedSchedulerEvent(n1)); - fs.handle(new NodeAddedSchedulerEvent(n2)); - fs.handle(new NodeUpdateSchedulerEvent(n1)); - Assert.assertEquals(6 * GB, fs.getRootQueueMetrics().getAvailableMB()); - - // reconnect n1 with downgraded memory - n1 = - MockNodes.newNodeInfo(0, MockNodes.newResource(2 * GB), 1, "127.0.0.2"); - fs.handle(new NodeRemovedSchedulerEvent(n1)); - fs.handle(new NodeAddedSchedulerEvent(n1)); - fs.handle(new NodeUpdateSchedulerEvent(n1)); - - Assert.assertEquals(4 * GB, fs.getRootQueueMetrics().getAvailableMB()); - fs.stop(); - } - - @Test (timeout = 50000) - public void testBlackListNodes() throws Exception { - - Configuration conf = new Configuration(); - conf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, - ResourceScheduler.class); - MockRM rm = new MockRM(conf); - rm.start(); - FifoScheduler fs = (FifoScheduler) rm.getResourceScheduler(); - - int rack_num_0 = 0; - int rack_num_1 = 1; - // Add 4 nodes in 2 racks - - // host_0_0 in rack0 - String host_0_0 = "127.0.0.1"; - RMNode n1 = - MockNodes.newNodeInfo(rack_num_0, MockNodes.newResource(4 * GB), 1, host_0_0); - fs.handle(new NodeAddedSchedulerEvent(n1)); - - // host_0_1 in rack0 - String host_0_1 = "127.0.0.2"; - RMNode n2 = - MockNodes.newNodeInfo(rack_num_0, MockNodes.newResource(4 * GB), 1, host_0_1); - fs.handle(new NodeAddedSchedulerEvent(n2)); - - // host_1_0 in rack1 - String host_1_0 = "127.0.0.3"; - RMNode n3 = - MockNodes.newNodeInfo(rack_num_1, MockNodes.newResource(4 * GB), 1, host_1_0); - fs.handle(new NodeAddedSchedulerEvent(n3)); - - // host_1_1 in rack1 - String host_1_1 = "127.0.0.4"; - RMNode n4 = - MockNodes.newNodeInfo(rack_num_1, MockNodes.newResource(4 * GB), 1, host_1_1); - fs.handle(new NodeAddedSchedulerEvent(n4)); - - // Add one application - ApplicationId appId1 = BuilderUtils.newApplicationId(100, 1); - ApplicationAttemptId appAttemptId1 = BuilderUtils.newApplicationAttemptId( - appId1, 1); - SchedulerEvent appEvent = - new AppAddedSchedulerEvent(appId1, "queue", "user"); - fs.handle(appEvent); - SchedulerEvent attemptEvent = - new AppAttemptAddedSchedulerEvent(appAttemptId1, false); - fs.handle(attemptEvent); - - List emptyId = new ArrayList(); - List emptyAsk = new ArrayList(); - - // Allow rack-locality for rack_1, but blacklist host_1_0 - - // Set up resource requests - // Ask for a 1 GB container for app 1 - List ask1 = new ArrayList(); - ask1.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), - "rack1", BuilderUtils.newResource(GB, 1), 1)); - ask1.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), - ResourceRequest.ANY, BuilderUtils.newResource(GB, 1), 1)); - fs.allocate(appAttemptId1, ask1, emptyId, Collections.singletonList(host_1_0), null); - - // Trigger container assignment - fs.handle(new NodeUpdateSchedulerEvent(n3)); - - // Get the allocation for the application and verify no allocation on blacklist node - Allocation allocation1 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - - Assert.assertEquals("allocation1", 0, allocation1.getContainers().size()); - - // verify host_1_1 can get allocated as not in blacklist - fs.handle(new NodeUpdateSchedulerEvent(n4)); - Allocation allocation2 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - Assert.assertEquals("allocation2", 1, allocation2.getContainers().size()); - List containerList = allocation2.getContainers(); - for (Container container : containerList) { - Assert.assertEquals("Container is allocated on n4", - container.getNodeId(), n4.getNodeID()); - } - - // Ask for a 1 GB container again for app 1 - List ask2 = new ArrayList(); - // this time, rack0 is also in blacklist, so only host_1_1 is available to - // be assigned - ask2.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), - ResourceRequest.ANY, BuilderUtils.newResource(GB, 1), 1)); - fs.allocate(appAttemptId1, ask2, emptyId, Collections.singletonList("rack0"), null); - - // verify n1 is not qualified to be allocated - fs.handle(new NodeUpdateSchedulerEvent(n1)); - Allocation allocation3 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - Assert.assertEquals("allocation3", 0, allocation3.getContainers().size()); - - // verify n2 is not qualified to be allocated - fs.handle(new NodeUpdateSchedulerEvent(n2)); - Allocation allocation4 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - Assert.assertEquals("allocation4", 0, allocation4.getContainers().size()); - - // verify n3 is not qualified to be allocated - fs.handle(new NodeUpdateSchedulerEvent(n3)); - Allocation allocation5 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - Assert.assertEquals("allocation5", 0, allocation5.getContainers().size()); - - fs.handle(new NodeUpdateSchedulerEvent(n4)); - Allocation allocation6 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - Assert.assertEquals("allocation6", 1, allocation6.getContainers().size()); - - containerList = allocation6.getContainers(); - for (Container container : containerList) { - Assert.assertEquals("Container is allocated on n4", - container.getNodeId(), n4.getNodeID()); - } - - rm.stop(); - } - - @Test (timeout = 50000) - public void testHeadroom() throws Exception { - - Configuration conf = new Configuration(); - conf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, - ResourceScheduler.class); - MockRM rm = new MockRM(conf); - rm.start(); - FifoScheduler fs = (FifoScheduler) rm.getResourceScheduler(); - - // Add a node - RMNode n1 = - MockNodes.newNodeInfo(0, MockNodes.newResource(4 * GB), 1, "127.0.0.2"); - fs.handle(new NodeAddedSchedulerEvent(n1)); - - // Add two applications - ApplicationId appId1 = BuilderUtils.newApplicationId(100, 1); - ApplicationAttemptId appAttemptId1 = BuilderUtils.newApplicationAttemptId( - appId1, 1); - SchedulerEvent appEvent = - new AppAddedSchedulerEvent(appId1, "queue", "user"); - fs.handle(appEvent); - SchedulerEvent attemptEvent = - new AppAttemptAddedSchedulerEvent(appAttemptId1, false); - fs.handle(attemptEvent); - - ApplicationId appId2 = BuilderUtils.newApplicationId(200, 2); - ApplicationAttemptId appAttemptId2 = BuilderUtils.newApplicationAttemptId( - appId2, 1); - SchedulerEvent appEvent2 = - new AppAddedSchedulerEvent(appId2, "queue", "user"); - fs.handle(appEvent2); - SchedulerEvent attemptEvent2 = - new AppAttemptAddedSchedulerEvent(appAttemptId2, false); - fs.handle(attemptEvent2); - - List emptyId = new ArrayList(); - List emptyAsk = new ArrayList(); - - // Set up resource requests - - // Ask for a 1 GB container for app 1 - List ask1 = new ArrayList(); - ask1.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), - ResourceRequest.ANY, BuilderUtils.newResource(GB, 1), 1)); - fs.allocate(appAttemptId1, ask1, emptyId, null, null); - - // Ask for a 2 GB container for app 2 - List ask2 = new ArrayList(); - ask2.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), - ResourceRequest.ANY, BuilderUtils.newResource(2 * GB, 1), 1)); - fs.allocate(appAttemptId2, ask2, emptyId, null, null); - - // Trigger container assignment - fs.handle(new NodeUpdateSchedulerEvent(n1)); - - // Get the allocation for the applications and verify headroom - Allocation allocation1 = fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); - Assert.assertEquals("Allocation headroom", 1 * GB, - allocation1.getResourceLimit().getMemory()); - - Allocation allocation2 = fs.allocate(appAttemptId2, emptyAsk, emptyId, null, null); - Assert.assertEquals("Allocation headroom", 1 * GB, - allocation2.getResourceLimit().getMemory()); - - rm.stop(); - } - - @Test - public void testResourceOverCommit() throws Exception { - MockRM rm = new MockRM(conf); - rm.start(); - - MockNM nm1 = rm.registerNode("127.0.0.1:1234", 4 * GB); - - RMApp app1 = rm.submitApp(2048); - // kick the scheduling, 2 GB given to AM1, remaining 2GB on nm1 - nm1.nodeHeartbeat(true); - RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); - MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); - am1.registerAppAttempt(); - SchedulerNodeReport report_nm1 = rm.getResourceScheduler().getNodeReport( - nm1.getNodeId()); - // check node report, 2 GB used and 2 GB available - Assert.assertEquals(2 * GB, report_nm1.getUsedResource().getMemory()); - Assert.assertEquals(2 * GB, report_nm1.getAvailableResource().getMemory()); - - // add request for containers - am1.addRequests(new String[] { "127.0.0.1", "127.0.0.2" }, 2 * GB, 1, 1); - AllocateResponse alloc1Response = am1.schedule(); // send the request - - // kick the scheduler, 2 GB given to AM1, resource remaining 0 - nm1.nodeHeartbeat(true); - while (alloc1Response.getAllocatedContainers().size() < 1) { - LOG.info("Waiting for containers to be created for app 1..."); - Thread.sleep(1000); - alloc1Response = am1.schedule(); - } - - List allocated1 = alloc1Response.getAllocatedContainers(); - Assert.assertEquals(1, allocated1.size()); - Assert.assertEquals(2 * GB, allocated1.get(0).getResource().getMemory()); - Assert.assertEquals(nm1.getNodeId(), allocated1.get(0).getNodeId()); - - report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); - // check node report, 4 GB used and 0 GB available - Assert.assertEquals(0, report_nm1.getAvailableResource().getMemory()); - Assert.assertEquals(4 * GB, report_nm1.getUsedResource().getMemory()); - - // check container is assigned with 2 GB. - Container c1 = allocated1.get(0); - Assert.assertEquals(2 * GB, c1.getResource().getMemory()); - - // update node resource to 2 GB, so resource is over-consumed. - Map nodeResourceMap = - new HashMap(); - nodeResourceMap.put(nm1.getNodeId(), - ResourceOption.newInstance(Resource.newInstance(2 * GB, 1), -1)); - UpdateNodeResourceRequest request = - UpdateNodeResourceRequest.newInstance(nodeResourceMap); - AdminService as = rm.adminService; - as.updateNodeResource(request); - - // Now, the used resource is still 4 GB, and available resource is minus value. - report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); - Assert.assertEquals(4 * GB, report_nm1.getUsedResource().getMemory()); - Assert.assertEquals(-2 * GB, report_nm1.getAvailableResource().getMemory()); - - // Check container can complete successfully in case of resource over-commitment. - ContainerStatus containerStatus = BuilderUtils.newContainerStatus( - c1.getId(), ContainerState.COMPLETE, "", 0); - nm1.containerStatus(containerStatus); - int waitCount = 0; - while (attempt1.getJustFinishedContainers().size() < 1 - && waitCount++ != 20) { - LOG.info("Waiting for containers to be finished for app 1... Tried " - + waitCount + " times already.."); - Thread.sleep(100); - } - Assert.assertEquals(1, attempt1.getJustFinishedContainers().size()); - Assert.assertEquals(1, am1.schedule().getCompletedContainersStatuses().size()); - report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); - Assert.assertEquals(2 * GB, report_nm1.getUsedResource().getMemory()); - // As container return 2 GB back, the available resource becomes 0 again. - Assert.assertEquals(0 * GB, report_nm1.getAvailableResource().getMemory()); - rm.stop(); - } - - public static void main(String[] args) throws Exception { - TestFifoScheduler t = new TestFifoScheduler(); - t.test(); - t.testDefaultMinimumAllocation(); - t.testNonDefaultMinimumAllocation(); - t.testReconnectedNode(); - } -} diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java index 36153dec677b0..d2bde80a7e2c4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java @@ -52,8 +52,7 @@ public void setUp() throws Exception { FifoSchedulerWithMove.class); conf.set(YarnConfiguration.YARN_ADMIN_ACL, " "); conf.setBoolean(YarnConfiguration.YARN_ACL_ENABLE, true); - resourceManager = new ResourceManager(); - resourceManager.init(conf); + resourceManager = new MockRM(conf); resourceManager.getRMContext().getContainerTokenSecretManager().rollMasterKey(); resourceManager.getRMContext().getNMTokenSecretManager().rollMasterKey(); resourceManager.start(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java index 6735575aefaf6..fbf54fce00877 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java @@ -212,9 +212,8 @@ private void checkResourceUsage( public void testResourceManagerInitConfigValidation() throws Exception { Configuration conf = new YarnConfiguration(); conf.setInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, -1); - resourceManager = new ResourceManager(); try { - resourceManager.init(conf); + resourceManager = new MockRM(conf); fail("Exception is expected because the global max attempts" + " is negative."); } catch (YarnRuntimeException e) { @@ -229,9 +228,8 @@ public void testNMExpiryAndHeartbeatIntervalsValidation() throws Exception { Configuration conf = new YarnConfiguration(); conf.setLong(YarnConfiguration.RM_NM_EXPIRY_INTERVAL_MS, 1000); conf.setLong(YarnConfiguration.RM_NM_HEARTBEAT_INTERVAL_MS, 1001); - resourceManager = new ResourceManager();; try { - resourceManager.init(conf); + resourceManager = new MockRM(conf); } catch (YarnRuntimeException e) { // Exception is expected. if (!e.getMessage().startsWith("Nodemanager expiry interval should be no" diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java index 7c128481c73b8..18d7df471fbed 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java @@ -27,8 +27,10 @@ import java.io.FileOutputStream; import java.io.IOException; import java.util.ArrayList; +import java.util.Collections; import java.util.HashMap; import java.util.List; +import java.util.Set; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.IOUtils; @@ -49,13 +51,19 @@ import org.apache.hadoop.yarn.event.DrainDispatcher; import org.apache.hadoop.yarn.event.Event; import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.nodelabels.NodeLabelTestBase; import org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus; +import org.apache.hadoop.yarn.server.api.protocolrecords.NodeHeartbeatRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.NodeHeartbeatResponse; import org.apache.hadoop.yarn.server.api.protocolrecords.RegisterNodeManagerRequest; import org.apache.hadoop.yarn.server.api.protocolrecords.RegisterNodeManagerResponse; import org.apache.hadoop.yarn.server.api.records.NodeAction; +import org.apache.hadoop.yarn.server.api.records.NodeStatus; +import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NullRMNodeLabelsManager; +import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl; +import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEvent; import org.apache.hadoop.yarn.server.utils.BuilderUtils; @@ -65,7 +73,7 @@ import org.junit.Assert; import org.junit.Test; -public class TestResourceTrackerService { +public class TestResourceTrackerService extends NodeLabelTestBase { private final static File TEMP_DIR = new File(System.getProperty( "test.build.data", "/tmp"), "decommision"); @@ -304,8 +312,425 @@ public void testNodeRegistrationSuccess() throws Exception { req.setHttpPort(1234); req.setNMVersion(YarnVersionInfo.getVersion()); // trying to register a invalid node. - RegisterNodeManagerResponse response = resourceTrackerService.registerNodeManager(req); - Assert.assertEquals(NodeAction.NORMAL,response.getNodeAction()); + RegisterNodeManagerResponse response = + resourceTrackerService.registerNodeManager(req); + Assert.assertEquals(NodeAction.NORMAL, response.getNodeAction()); + } + + @Test + public void testNodeRegistrationWithLabels() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + + try { + nodeLabelsMgr.addToCluserNodeLabels(toSet("A", "B", "C")); + } catch (IOException e) { + Assert.fail("Caught Exception while intializing"); + e.printStackTrace(); + } + + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest registerReq = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + registerReq.setResource(capability); + registerReq.setNodeId(nodeId); + registerReq.setHttpPort(1234); + registerReq.setNMVersion(YarnVersionInfo.getVersion()); + registerReq.setNodeLabels(toSet("A")); + RegisterNodeManagerResponse response = + resourceTrackerService.registerNodeManager(registerReq); + + Assert.assertEquals("Action should be normal on valid Node Labels", + NodeAction.NORMAL, response.getNodeAction()); + assertCollectionEquals(nodeLabelsMgr.getNodeLabels().get(nodeId), + registerReq.getNodeLabels()); + Assert.assertTrue("Valid Node Labels were not accepted by RM", + response.getAreNodeLabelsAcceptedByRM()); + rm.stop(); + } + + @Test + public void testNodeRegistrationWithInvalidLabels() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + + try { + nodeLabelsMgr.addToCluserNodeLabels(toSet("X", "Y", "Z")); + } catch (IOException e) { + Assert.fail("Caught Exception while intializing"); + e.printStackTrace(); + } + + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest registerReq = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + registerReq.setResource(capability); + registerReq.setNodeId(nodeId); + registerReq.setHttpPort(1234); + registerReq.setNMVersion(YarnVersionInfo.getVersion()); + registerReq.setNodeLabels(toSet("A", "B", "C")); + RegisterNodeManagerResponse response = + resourceTrackerService.registerNodeManager(registerReq); + + Assert.assertEquals( + "On Invalid Node Labels action is expected to be normal", + NodeAction.NORMAL, response.getNodeAction()); + Assert.assertNull(nodeLabelsMgr.getNodeLabels().get(nodeId)); + Assert.assertNotNull(response.getDiagnosticsMessage()); + Assert.assertFalse("Node Labels should not accepted by RM If Invalid", + response.getAreNodeLabelsAcceptedByRM()); + + if (rm != null) { + rm.stop(); + } + } + + @Test + public void testNodeRegistrationWithInvalidLabelsSyntax() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + + try { + nodeLabelsMgr.addToCluserNodeLabels(toSet("X", "Y", "Z")); + } catch (IOException e) { + Assert.fail("Caught Exception while intializing"); + e.printStackTrace(); + } + + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest req = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + req.setResource(capability); + req.setNodeId(nodeId); + req.setHttpPort(1234); + req.setNMVersion(YarnVersionInfo.getVersion()); + req.setNodeLabels(toSet("#Y")); + RegisterNodeManagerResponse response = + resourceTrackerService.registerNodeManager(req); + + Assert.assertEquals( + "On Invalid Node Labels action is expected to be normal", + NodeAction.NORMAL, response.getNodeAction()); + Assert.assertNull(nodeLabelsMgr.getNodeLabels().get(nodeId)); + Assert.assertNotNull(response.getDiagnosticsMessage()); + Assert.assertFalse("Node Labels should not accepted by RM If Invalid", + response.getAreNodeLabelsAcceptedByRM()); + + if (rm != null) { + rm.stop(); + } + } + + @Test + public void testNodeRegistrationWithCentralLabelConfig() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DEFAULT_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + try { + nodeLabelsMgr.addToCluserNodeLabels(toSet("A", "B", "C")); + } catch (IOException e) { + Assert.fail("Caught Exception while intializing"); + e.printStackTrace(); + } + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest req = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + req.setResource(capability); + req.setNodeId(nodeId); + req.setHttpPort(1234); + req.setNMVersion(YarnVersionInfo.getVersion()); + req.setNodeLabels(toSet("A")); + RegisterNodeManagerResponse response = + resourceTrackerService.registerNodeManager(req); + // registered to RM with central label config + Assert.assertEquals(NodeAction.NORMAL, response.getNodeAction()); + Assert.assertNull(nodeLabelsMgr.getNodeLabels().get(nodeId)); + Assert + .assertFalse( + "Node Labels should not accepted by RM If its configured with Central configuration", + response.getAreNodeLabelsAcceptedByRM()); + if (rm != null) { + rm.stop(); + } + } + + @SuppressWarnings("unchecked") + private NodeStatus getNodeStatusObject(NodeId nodeId) { + NodeStatus status = Records.newRecord(NodeStatus.class); + status.setNodeId(nodeId); + status.setResponseId(0); + status.setContainersStatuses(Collections.EMPTY_LIST); + status.setKeepAliveApplications(Collections.EMPTY_LIST); + return status; + } + + @Test + public void testNodeHeartBeatWithLabels() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + // adding valid labels + try { + nodeLabelsMgr.addToCluserNodeLabels(toSet("A", "B", "C")); + } catch (IOException e) { + Assert.fail("Caught Exception while intializing"); + e.printStackTrace(); + } + + // Registering of labels and other required info to RM + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest registerReq = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + registerReq.setResource(capability); + registerReq.setNodeId(nodeId); + registerReq.setHttpPort(1234); + registerReq.setNMVersion(YarnVersionInfo.getVersion()); + registerReq.setNodeLabels(toSet("A")); // Node register label + RegisterNodeManagerResponse registerResponse = + resourceTrackerService.registerNodeManager(registerReq); + + // modification of labels during heartbeat + NodeHeartbeatRequest heartbeatReq = + Records.newRecord(NodeHeartbeatRequest.class); + heartbeatReq.setNodeLabels(toSet("B")); // Node heartbeat label update + NodeStatus nodeStatusObject = getNodeStatusObject(nodeId); + heartbeatReq.setNodeStatus(nodeStatusObject); + heartbeatReq.setLastKnownNMTokenMasterKey(registerResponse + .getNMTokenMasterKey()); + heartbeatReq.setLastKnownContainerTokenMasterKey(registerResponse + .getContainerTokenMasterKey()); + NodeHeartbeatResponse nodeHeartbeatResponse = + resourceTrackerService.nodeHeartbeat(heartbeatReq); + + Assert.assertEquals("InValid Node Labels were not accepted by RM", + NodeAction.NORMAL, nodeHeartbeatResponse.getNodeAction()); + assertCollectionEquals(nodeLabelsMgr.getNodeLabels().get(nodeId), + heartbeatReq.getNodeLabels()); + Assert.assertTrue("Valid Node Labels were not accepted by RM", + nodeHeartbeatResponse.getAreNodeLabelsAcceptedByRM()); + + // After modification of labels next heartbeat sends null informing no update + Set oldLabels = nodeLabelsMgr.getNodeLabels().get(nodeId); + int responseId = nodeStatusObject.getResponseId(); + heartbeatReq = + Records.newRecord(NodeHeartbeatRequest.class); + heartbeatReq.setNodeLabels(null); // Node heartbeat label update + nodeStatusObject = getNodeStatusObject(nodeId); + nodeStatusObject.setResponseId(responseId+2); + heartbeatReq.setNodeStatus(nodeStatusObject); + heartbeatReq.setLastKnownNMTokenMasterKey(registerResponse + .getNMTokenMasterKey()); + heartbeatReq.setLastKnownContainerTokenMasterKey(registerResponse + .getContainerTokenMasterKey()); + nodeHeartbeatResponse = resourceTrackerService.nodeHeartbeat(heartbeatReq); + + Assert.assertEquals("InValid Node Labels were not accepted by RM", + NodeAction.NORMAL, nodeHeartbeatResponse.getNodeAction()); + assertCollectionEquals(nodeLabelsMgr.getNodeLabels().get(nodeId), + oldLabels); + Assert.assertFalse("Node Labels should not accepted by RM", + nodeHeartbeatResponse.getAreNodeLabelsAcceptedByRM()); + rm.stop(); + } + + @Test + public void testNodeHeartBeatWithInvalidLabels() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DISTRIBUTED_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + + try { + nodeLabelsMgr.addToCluserNodeLabels(toSet("A", "B", "C")); + } catch (IOException e) { + Assert.fail("Caught Exception while intializing"); + e.printStackTrace(); + } + + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest registerReq = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + registerReq.setResource(capability); + registerReq.setNodeId(nodeId); + registerReq.setHttpPort(1234); + registerReq.setNMVersion(YarnVersionInfo.getVersion()); + registerReq.setNodeLabels(toSet("A")); + RegisterNodeManagerResponse registerResponse = + resourceTrackerService.registerNodeManager(registerReq); + + NodeHeartbeatRequest heartbeatReq = + Records.newRecord(NodeHeartbeatRequest.class); + heartbeatReq.setNodeLabels(toSet("B", "#C")); // Invalid heart beat labels + heartbeatReq.setNodeStatus(getNodeStatusObject(nodeId)); + heartbeatReq.setLastKnownNMTokenMasterKey(registerResponse + .getNMTokenMasterKey()); + heartbeatReq.setLastKnownContainerTokenMasterKey(registerResponse + .getContainerTokenMasterKey()); + NodeHeartbeatResponse nodeHeartbeatResponse = + resourceTrackerService.nodeHeartbeat(heartbeatReq); + + // response should be NORMAL when RM heartbeat labels are rejected + Assert.assertEquals("Response should be NORMAL when RM heartbeat labels" + + " are rejected", NodeAction.NORMAL, + nodeHeartbeatResponse.getNodeAction()); + Assert.assertFalse(nodeHeartbeatResponse.getAreNodeLabelsAcceptedByRM()); + Assert.assertNotNull(nodeHeartbeatResponse.getDiagnosticsMessage()); + rm.stop(); + } + + @Test + public void testNodeHeartbeatWithCentralLabelConfig() throws Exception { + writeToHostsFile("host2"); + Configuration conf = new Configuration(); + conf.set(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH, + hostFile.getAbsolutePath()); + conf.set(YarnConfiguration.NODELABEL_CONFIGURATION_TYPE, + YarnConfiguration.DEFAULT_NODELABEL_CONFIGURATION_TYPE); + + final RMNodeLabelsManager nodeLabelsMgr = new NullRMNodeLabelsManager(); + + rm = new MockRM(conf) { + @Override + protected RMNodeLabelsManager createNodeLabelManager() { + return nodeLabelsMgr; + } + }; + rm.start(); + + ResourceTrackerService resourceTrackerService = + rm.getResourceTrackerService(); + RegisterNodeManagerRequest req = + Records.newRecord(RegisterNodeManagerRequest.class); + NodeId nodeId = NodeId.newInstance("host2", 1234); + Resource capability = BuilderUtils.newResource(1024, 1); + req.setResource(capability); + req.setNodeId(nodeId); + req.setHttpPort(1234); + req.setNMVersion(YarnVersionInfo.getVersion()); + req.setNodeLabels(toSet("A", "B", "C")); + RegisterNodeManagerResponse registerResponse = + resourceTrackerService.registerNodeManager(req); + + NodeHeartbeatRequest heartbeatReq = + Records.newRecord(NodeHeartbeatRequest.class); + heartbeatReq.setNodeLabels(toSet("B")); // Valid heart beat labels + heartbeatReq.setNodeStatus(getNodeStatusObject(nodeId)); + heartbeatReq.setLastKnownNMTokenMasterKey(registerResponse + .getNMTokenMasterKey()); + heartbeatReq.setLastKnownContainerTokenMasterKey(registerResponse + .getContainerTokenMasterKey()); + NodeHeartbeatResponse nodeHeartbeatResponse = + resourceTrackerService.nodeHeartbeat(heartbeatReq); + + // response should be ok but the RMacceptNodeLabelsUpdate should be false + Assert.assertEquals(NodeAction.NORMAL, + nodeHeartbeatResponse.getNodeAction()); + // no change in the labels, + Assert.assertNull(nodeLabelsMgr.getNodeLabels().get(nodeId)); + // heartbeat labels rejected + Assert.assertFalse("Invalid Node Labels should not accepted by RM", + nodeHeartbeatResponse.getAreNodeLabelsAcceptedByRM()); + if (rm != null) { + rm.stop(); + } } @Test @@ -623,7 +1048,7 @@ protected Dispatcher createDispatcher() { dispatcher.await(); Assert.assertTrue(NodeAction.NORMAL.equals(response.getNodeAction())); Assert.assertEquals(5120 + 10240, metrics.getAvailableMB()); - + // reconnect of node with changed capability and running applications List runningApps = new ArrayList(); runningApps.add(ApplicationId.newInstance(1, 0)); @@ -633,6 +1058,20 @@ protected Dispatcher createDispatcher() { dispatcher.await(); Assert.assertTrue(NodeAction.NORMAL.equals(response.getNodeAction())); Assert.assertEquals(5120 + 15360, metrics.getAvailableMB()); + + // reconnect healthy node changing http port + nm1 = new MockNM("host1:1234", 5120, rm.getResourceTrackerService()); + nm1.setHttpPort(3); + nm1.registerNode(); + dispatcher.await(); + response = nm1.nodeHeartbeat(true); + response = nm1.nodeHeartbeat(true); + dispatcher.await(); + RMNode rmNode = rm.getRMContext().getRMNodes().get(nm1.getNodeId()); + Assert.assertEquals(3, rmNode.getHttpPort()); + Assert.assertEquals(5120, rmNode.getTotalCapability().getMemory()); + Assert.assertEquals(5120 + 15360, metrics.getAvailableMB()); + } private void writeToHostsFile(String... hosts) throws IOException { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java index 9f02721399a33..7ed3835bdb73a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java @@ -38,7 +38,6 @@ import org.apache.hadoop.yarn.api.records.timeline.TimelineEvent; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer; -import org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebApp; import org.apache.hadoop.yarn.server.metrics.AppAttemptMetricsConstants; import org.apache.hadoop.yarn.server.metrics.ApplicationMetricsConstants; import org.apache.hadoop.yarn.server.metrics.ContainerMetricsConstants; @@ -386,6 +385,10 @@ private static RMContainer createRMContainer(ContainerId containerId) { when(container.getDiagnosticsInfo()).thenReturn("test diagnostics info"); when(container.getContainerExitStatus()).thenReturn(-1); when(container.getContainerState()).thenReturn(ContainerState.COMPLETE); + Container mockContainer = mock(Container.class); + when(container.getContainer()).thenReturn(mockContainer); + when(mockContainer.getNodeHttpAddress()) + .thenReturn("http://localhost:1234"); return container; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/TestSchedulingMonitor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/TestSchedulingMonitor.java index 5bf61e347d0bd..9ec17da62b0d4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/TestSchedulingMonitor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/TestSchedulingMonitor.java @@ -20,6 +20,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.server.resourcemanager.MockRM; import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; import org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy; import org.junit.Test; @@ -35,7 +36,7 @@ public void testRMStarts() { conf.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES, ProportionalCapacityPreemptionPolicy.class.getCanonicalName()); - ResourceManager rm = new ResourceManager(); + ResourceManager rm = new MockRM(); try { rm.init(conf); } catch (Exception e) { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java index 696b9bb77cf7a..8f5237e1a3f4a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java @@ -531,6 +531,30 @@ public void testPerQueueDisablePreemptionRootDisablesAll() { verify(mDisp, never()).handle(argThat(new IsPreemptionRequestFor(appF))); // queueI } + @Test + public void testPerQueueDisablePreemptionOverAbsMaxCapacity() { + int[][] qData = new int[][] { + // / A D + // B C E F + {1000, 725, 360, 365, 275, 17, 258 }, // absCap + {1000,1000,1000,1000, 550, 109,1000 }, // absMaxCap + {1000, 741, 396, 345, 259, 110, 149 }, // used + { 40, 20, 0, 20, 20, 20, 0 }, // pending + { 0, 0, 0, 0, 0, 0, 0 }, // reserved + // appA appB appC appD + { 4, 2, 1, 1, 2, 1, 1 }, // apps + { -1, -1, 1, 1, -1, 1, 1 }, // req granulrity + { 2, 2, 0, 0, 2, 0, 0 }, // subqueues + }; + // QueueE inherits non-preemption from QueueD + schedConf.setPreemptionDisabled("root.queueD", true); + ProportionalCapacityPreemptionPolicy policy = buildPolicy(qData); + policy.editSchedule(); + // appC is running on QueueE. QueueE is over absMaxCap, but is not + // preemptable. Therefore, appC resources should not be preempted. + verify(mDisp, never()).handle(argThat(new IsPreemptionRequestFor(appC))); + } + @Test public void testOverCapacityImbalance() { int[][] qData = new int[][]{ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NullRMNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NullRMNodeLabelsManager.java index b1be525ae8574..14d96a04830eb 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NullRMNodeLabelsManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/NullRMNodeLabelsManager.java @@ -20,11 +20,13 @@ import java.io.IOException; import java.util.Collection; +import java.util.List; import java.util.Map; import java.util.Set; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.yarn.api.records.NodeId; +import org.apache.hadoop.yarn.api.records.NodeLabel; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.nodelabels.NodeLabelsStore; @@ -63,6 +65,12 @@ public void storeNewClusterNodeLabels(Set label) throws IOException { public void close() throws IOException { // do nothing } + + @Override + public void updateNodeLabels(List updatedNodeLabels) + throws IOException { + // do nothing + } }; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java index 8a37c2492de04..ace874253381b 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/nodelabels/TestRMNodeLabelsManager.java @@ -29,7 +29,7 @@ import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager; -import org.apache.hadoop.yarn.nodelabels.NodeLabel; +import org.apache.hadoop.yarn.nodelabels.RMNodeLabel; import org.apache.hadoop.yarn.nodelabels.NodeLabelTestBase; import org.apache.hadoop.yarn.util.resource.Resources; import org.junit.After; @@ -476,8 +476,8 @@ public void testGetLabelsOnNodesWhenNodeActiveDeactive() throws Exception { } - private void checkNodeLabelInfo(List infos, String labelName, int activeNMs, int memory) { - for (NodeLabel info : infos) { + private void checkNodeLabelInfo(List infos, String labelName, int activeNMs, int memory) { + for (RMNodeLabel info : infos) { if (info.getLabelName().equals(labelName)) { Assert.assertEquals(activeNMs, info.getNumActiveNMs()); Assert.assertEquals(memory, info.getResource().getMemory()); @@ -499,7 +499,7 @@ public void testPullRMNodeLabelsInfo() throws IOException { toNodeId("n2"), toSet("x"), toNodeId("n3"), toSet("y"))); // x, y, z and "" - List infos = mgr.pullRMNodeLabelsInfo(); + List infos = mgr.pullRMNodeLabelsInfo(); Assert.assertEquals(4, infos.size()); checkNodeLabelInfo(infos, RMNodeLabelsManager.NO_LABEL, 2, 20); checkNodeLabelInfo(infos, "x", 2, 20); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java index 675d73c1fbe71..c5f557855b491 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestFSRMStateStore.java @@ -100,12 +100,17 @@ public RMStateStore getRMStateStore() throws Exception { workingDirPathURI.toString()); conf.set(YarnConfiguration.FS_RM_STATE_STORE_RETRY_POLICY_SPEC, "100,6000"); - conf.setInt(YarnConfiguration.FS_RM_STATE_STORE_NUM_RETRIES, 5); + conf.setInt(YarnConfiguration.FS_RM_STATE_STORE_NUM_RETRIES, 8); conf.setLong(YarnConfiguration.FS_RM_STATE_STORE_RETRY_INTERVAL_MS, 900L); this.store = new TestFileSystemRMStore(conf); - Assert.assertEquals(store.getNumRetries(), 5); + Assert.assertEquals(store.getNumRetries(), 8); Assert.assertEquals(store.getRetryInterval(), 900L); + Assert.assertTrue(store.fs.getConf() == store.fsConf); + FileSystem previousFs = store.fs; + store.startInternal(); + Assert.assertTrue(store.fs != previousFs); + Assert.assertTrue(store.fs.getConf() == store.fsConf); return store; } @@ -277,12 +282,7 @@ public void run() { ApplicationStateData.newInstance(111, 111, "user", null, RMAppState.ACCEPTED, "diagnostics", 333)); } catch (Exception e) { - // TODO 0 datanode exception will not be retried by dfs client, fix - // that separately. - if (!e.getMessage().contains("could only be replicated" + - " to 0 nodes instead of minReplication (=1)")) { - assertionFailedInThread.set(true); - } + assertionFailedInThread.set(true); e.printStackTrace(); } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java index bbbf5a12ca369..608adbee781f4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java @@ -18,6 +18,7 @@ package org.apache.hadoop.yarn.server.resourcemanager.recovery; +import org.apache.hadoop.yarn.server.resourcemanager.MockRM; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import static org.mockito.Mockito.mock; @@ -224,8 +225,7 @@ public void testFencing() throws Exception { Configuration conf1 = createHARMConf("rm1,rm2", "rm1", 1234); conf1.setBoolean(YarnConfiguration.AUTO_FAILOVER_ENABLED, false); - ResourceManager rm1 = new ResourceManager(); - rm1.init(conf1); + ResourceManager rm1 = new MockRM(conf1); rm1.start(); rm1.getRMContext().getRMAdminService().transitionToActive(req); assertEquals("RM with ZKStore didn't start", @@ -236,8 +236,7 @@ public void testFencing() throws Exception { Configuration conf2 = createHARMConf("rm1,rm2", "rm2", 5678); conf2.setBoolean(YarnConfiguration.AUTO_FAILOVER_ENABLED, false); - ResourceManager rm2 = new ResourceManager(); - rm2.init(conf2); + ResourceManager rm2 = new MockRM(conf2); rm2.start(); rm2.getRMContext().getRMAdminService().transitionToActive(req); assertEquals("RM with ZKStore didn't start", diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStoreZKClientConnections.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStoreZKClientConnections.java index 8dc3628e2a62d..62dc5efd394d7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStoreZKClientConnections.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStoreZKClientConnections.java @@ -71,6 +71,7 @@ class TestZKClient { ZKRMStateStore store; boolean forExpire = false; + TestForwardingWatcher oldWatcher; TestForwardingWatcher watcher; CyclicBarrier syncBarrier = new CyclicBarrier(2); @@ -86,35 +87,36 @@ public TestZKRMStateStore(Configuration conf, String workingZnode) @Override public ZooKeeper getNewZooKeeper() throws IOException, InterruptedException { + oldWatcher = watcher; + watcher = new TestForwardingWatcher(); return createClient(watcher, hostPort, ZK_TIMEOUT_MS); } @Override - public synchronized void processWatchEvent(WatchedEvent event) - throws Exception { + public synchronized void processWatchEvent(ZooKeeper zk, + WatchedEvent event) throws Exception { if (forExpire) { // a hack... couldn't find a way to trigger expired event. WatchedEvent expriredEvent = new WatchedEvent( Watcher.Event.EventType.None, Watcher.Event.KeeperState.Expired, null); - super.processWatchEvent(expriredEvent); + super.processWatchEvent(zk, expriredEvent); forExpire = false; syncBarrier.await(); } else { - super.processWatchEvent(event); + super.processWatchEvent(zk, event); } } } private class TestForwardingWatcher extends ClientBaseWithFixes.CountdownWatcher { - public void process(WatchedEvent event) { super.process(event); try { if (store != null) { - store.processWatchEvent(event); + store.processWatchEvent(client, event); } } catch (Throwable t) { LOG.error("Failed to process watcher event " + event + ": " @@ -127,7 +129,6 @@ public RMStateStore getRMStateStore(Configuration conf) throws Exception { String workingZnode = "/Test"; conf.set(YarnConfiguration.RM_ZK_ADDRESS, hostPort); conf.set(YarnConfiguration.ZK_RM_STATE_STORE_PARENT_PATH, workingZnode); - watcher = new TestForwardingWatcher(); this.store = new TestZKRMStateStore(conf, workingZnode); return this.store; } @@ -239,6 +240,24 @@ public void testZKSessionTimeout() throws Exception { LOG.error(error, e); fail(error); } + + // send Disconnected event from old client session to ZKRMStateStore + // check the current client session is not affected. + Assert.assertTrue(zkClientTester.oldWatcher != null); + WatchedEvent disconnectedEvent = new WatchedEvent( + Watcher.Event.EventType.None, + Watcher.Event.KeeperState.Disconnected, null); + zkClientTester.oldWatcher.process(disconnectedEvent); + Assert.assertTrue(store.zkClient != null); + + zkClientTester.watcher.process(disconnectedEvent); + Assert.assertTrue(store.zkClient == null); + WatchedEvent connectedEvent = new WatchedEvent( + Watcher.Event.EventType.None, + Watcher.Event.KeeperState.SyncConnected, null); + zkClientTester.watcher.process(connectedEvent); + Assert.assertTrue(store.zkClient != null); + Assert.assertTrue(store.zkClient == store.activeZkClient); } @Test(timeout = 20000) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java index 8cad05725ba1a..1ca5c97a411a5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java @@ -611,7 +611,7 @@ public void testHeadroom() throws Exception { app_0_0.updateResourceRequests(app_0_0_requests); // Schedule to compute - queue.assignContainers(clusterResource, node_0, false, new ResourceLimits( + queue.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); Resource expectedHeadroom = Resources.createResource(10*16*GB, 1); assertEquals(expectedHeadroom, app_0_0.getHeadroom()); @@ -631,7 +631,7 @@ public void testHeadroom() throws Exception { app_0_1.updateResourceRequests(app_0_1_requests); // Schedule to compute - queue.assignContainers(clusterResource, node_0, false, new ResourceLimits( + queue.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); // Schedule to compute assertEquals(expectedHeadroom, app_0_0.getHeadroom()); assertEquals(expectedHeadroom, app_0_1.getHeadroom());// no change @@ -651,7 +651,7 @@ public void testHeadroom() throws Exception { app_1_0.updateResourceRequests(app_1_0_requests); // Schedule to compute - queue.assignContainers(clusterResource, node_0, false, new ResourceLimits( + queue.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); // Schedule to compute expectedHeadroom = Resources.createResource(10*16*GB / 2, 1); // changes assertEquals(expectedHeadroom, app_0_0.getHeadroom()); @@ -660,7 +660,7 @@ public void testHeadroom() throws Exception { // Now reduce cluster size and check for the smaller headroom clusterResource = Resources.createResource(90*16*GB); - queue.assignContainers(clusterResource, node_0, false, new ResourceLimits( + queue.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); // Schedule to compute expectedHeadroom = Resources.createResource(9*16*GB / 2, 1); // changes assertEquals(expectedHeadroom, app_0_0.getHeadroom()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java index 83ab1046288d5..aaa615dc42157 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java @@ -29,11 +29,13 @@ import java.net.InetSocketAddress; import java.security.PrivilegedAction; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.Comparator; import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Set; import java.util.concurrent.BrokenBarrierException; import java.util.concurrent.CyclicBarrier; @@ -89,15 +91,18 @@ import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.recovery.MemoryRMStateStore; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppMetrics; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppState; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptState; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerState; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.Allocation; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt; @@ -108,6 +113,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAddedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAttemptAddedSchedulerEvent; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAttemptRemovedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeAddedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeRemovedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeUpdateSchedulerEvent; @@ -128,6 +134,10 @@ import org.junit.Test; import org.mockito.Mockito; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Sets; + public class TestCapacityScheduler { private static final Log LOG = LogFactory.getLog(TestCapacityScheduler.class); @@ -634,6 +644,18 @@ public void testBlackListNodes() throws Exception { ApplicationId appId = BuilderUtils.newApplicationId(100, 1); ApplicationAttemptId appAttemptId = BuilderUtils.newApplicationAttemptId( appId, 1); + + RMAppAttemptMetrics attemptMetric = + new RMAppAttemptMetrics(appAttemptId, rm.getRMContext()); + RMAppImpl app = mock(RMAppImpl.class); + when(app.getApplicationId()).thenReturn(appId); + RMAppAttemptImpl attempt = mock(RMAppAttemptImpl.class); + when(attempt.getAppAttemptId()).thenReturn(appAttemptId); + when(attempt.getRMAppAttemptMetrics()).thenReturn(attemptMetric); + when(app.getCurrentAppAttempt()).thenReturn(attempt); + + rm.getRMContext().getRMApps().put(appId, app); + SchedulerEvent addAppEvent = new AppAddedSchedulerEvent(appId, "default", "user"); cs.handle(addAppEvent); @@ -2483,6 +2505,311 @@ public void testHierarchyQueuesCurrentLimits() throws Exception { Assert.assertEquals(30 * GB, am1.doHeartbeat().getAvailableResources().getMemory()); } + + @Test + public void testParentQueueMaxCapsAreRespected() throws Exception { + /* + * Queue tree: + * Root + * / \ + * A B + * / \ + * A1 A2 + */ + CapacitySchedulerConfiguration csConf = new CapacitySchedulerConfiguration(); + csConf.setQueues(CapacitySchedulerConfiguration.ROOT, new String[] {"a", "b"}); + csConf.setCapacity(A, 50); + csConf.setMaximumCapacity(A, 50); + csConf.setCapacity(B, 50); + + // Define 2nd-level queues + csConf.setQueues(A, new String[] {"a1", "a2"}); + csConf.setCapacity(A1, 50); + csConf.setUserLimitFactor(A1, 100.0f); + csConf.setCapacity(A2, 50); + csConf.setUserLimitFactor(A2, 100.0f); + csConf.setCapacity(B1, B1_CAPACITY); + csConf.setUserLimitFactor(B1, 100.0f); + + YarnConfiguration conf = new YarnConfiguration(csConf); + conf.setBoolean(CapacitySchedulerConfiguration.ENABLE_USER_METRICS, true); + + MemoryRMStateStore memStore = new MemoryRMStateStore(); + memStore.init(conf); + MockRM rm1 = new MockRM(conf, memStore); + rm1.start(); + MockNM nm1 = + new MockNM("127.0.0.1:1234", 24 * GB, rm1.getResourceTrackerService()); + nm1.registerNode(); + + // Launch app1 in a1, resource usage is 1GB (am) + 4GB * 2 = 9GB + RMApp app1 = rm1.submitApp(1 * GB, "app", "user", null, "a1"); + MockAM am1 = MockRM.launchAndRegisterAM(app1, rm1, nm1); + waitContainerAllocated(am1, 4 * GB, 2, 2, rm1, nm1); + + // Try to launch app2 in a2, asked 2GB, should success + RMApp app2 = rm1.submitApp(2 * GB, "app", "user", null, "a2"); + MockAM am2 = MockRM.launchAndRegisterAM(app2, rm1, nm1); + try { + // Try to allocate a container, a's usage=11G/max=12 + // a1's usage=9G/max=12 + // a2's usage=2G/max=12 + // In this case, if a2 asked 2G, should fail. + waitContainerAllocated(am2, 2 * GB, 1, 2, rm1, nm1); + } catch (AssertionError failure) { + // Expected, return; + return; + } + Assert.fail("Shouldn't successfully allocate containers for am2, " + + "queue-a's max capacity will be violated if container allocated"); + } + + @SuppressWarnings("unchecked") + private Set toSet(E... elements) { + Set set = Sets.newHashSet(elements); + return set; + } + + @Test + public void testQueueHierarchyPendingResourceUpdate() throws Exception { + Configuration conf = + TestUtils.getConfigurationWithQueueLabels(new Configuration(false)); + conf.setBoolean(YarnConfiguration.NODE_LABELS_ENABLED, true); + + final RMNodeLabelsManager mgr = new NullRMNodeLabelsManager(); + mgr.init(conf); + mgr.addToCluserNodeLabels(ImmutableSet.of("x", "y")); + mgr.addLabelsToNode(ImmutableMap.of(NodeId.newInstance("h1", 0), toSet("x"))); + + MemoryRMStateStore memStore = new MemoryRMStateStore(); + memStore.init(conf); + MockRM rm = new MockRM(conf, memStore) { + protected RMNodeLabelsManager createNodeLabelManager() { + return mgr; + } + }; + + rm.start(); + MockNM nm1 = // label = x + new MockNM("h1:1234", 200 * GB, rm.getResourceTrackerService()); + nm1.registerNode(); + + MockNM nm2 = // label = "" + new MockNM("h2:1234", 200 * GB, rm.getResourceTrackerService()); + nm2.registerNode(); + + // Launch app1 in queue=a1 + RMApp app1 = rm.submitApp(1 * GB, "app", "user", null, "a1"); + MockAM am1 = MockRM.launchAndRegisterAM(app1, rm, nm2); + + // Launch app2 in queue=b1 + RMApp app2 = rm.submitApp(8 * GB, "app", "user", null, "b1"); + MockAM am2 = MockRM.launchAndRegisterAM(app2, rm, nm2); + + // am1 asks for 8 * 1GB container for no label + am1.allocate(Arrays.asList(ResourceRequest.newInstance( + Priority.newInstance(1), "*", Resources.createResource(1 * GB), 8)), + null); + + checkPendingResource(rm, "a1", 8 * GB, null); + checkPendingResource(rm, "a", 8 * GB, null); + checkPendingResource(rm, "root", 8 * GB, null); + + // am2 asks for 8 * 1GB container for no label + am2.allocate(Arrays.asList(ResourceRequest.newInstance( + Priority.newInstance(1), "*", Resources.createResource(1 * GB), 8)), + null); + + checkPendingResource(rm, "a1", 8 * GB, null); + checkPendingResource(rm, "a", 8 * GB, null); + checkPendingResource(rm, "b1", 8 * GB, null); + checkPendingResource(rm, "b", 8 * GB, null); + // root = a + b + checkPendingResource(rm, "root", 16 * GB, null); + + // am2 asks for 8 * 1GB container in another priority for no label + am2.allocate(Arrays.asList(ResourceRequest.newInstance( + Priority.newInstance(2), "*", Resources.createResource(1 * GB), 8)), + null); + + checkPendingResource(rm, "a1", 8 * GB, null); + checkPendingResource(rm, "a", 8 * GB, null); + checkPendingResource(rm, "b1", 16 * GB, null); + checkPendingResource(rm, "b", 16 * GB, null); + // root = a + b + checkPendingResource(rm, "root", 24 * GB, null); + + // am1 asks 4 GB resource instead of 8 * GB for priority=1 + am1.allocate(Arrays.asList(ResourceRequest.newInstance( + Priority.newInstance(1), "*", Resources.createResource(4 * GB), 1)), + null); + + checkPendingResource(rm, "a1", 4 * GB, null); + checkPendingResource(rm, "a", 4 * GB, null); + checkPendingResource(rm, "b1", 16 * GB, null); + checkPendingResource(rm, "b", 16 * GB, null); + // root = a + b + checkPendingResource(rm, "root", 20 * GB, null); + + // am1 asks 8 * GB resource which label=x + am1.allocate(Arrays.asList(ResourceRequest.newInstance( + Priority.newInstance(2), "*", Resources.createResource(8 * GB), 1, + true, "x")), null); + + checkPendingResource(rm, "a1", 4 * GB, null); + checkPendingResource(rm, "a", 4 * GB, null); + checkPendingResource(rm, "a1", 8 * GB, "x"); + checkPendingResource(rm, "a", 8 * GB, "x"); + checkPendingResource(rm, "b1", 16 * GB, null); + checkPendingResource(rm, "b", 16 * GB, null); + // root = a + b + checkPendingResource(rm, "root", 20 * GB, null); + checkPendingResource(rm, "root", 8 * GB, "x"); + + // some containers allocated for am1, pending resource should decrease + ContainerId containerId = + ContainerId.newContainerId(am1.getApplicationAttemptId(), 2); + Assert.assertTrue(rm.waitForState(nm1, containerId, + RMContainerState.ALLOCATED, 10 * 1000)); + containerId = ContainerId.newContainerId(am1.getApplicationAttemptId(), 3); + Assert.assertTrue(rm.waitForState(nm2, containerId, + RMContainerState.ALLOCATED, 10 * 1000)); + + checkPendingResource(rm, "a1", 0 * GB, null); + checkPendingResource(rm, "a", 0 * GB, null); + checkPendingResource(rm, "a1", 0 * GB, "x"); + checkPendingResource(rm, "a", 0 * GB, "x"); + // some containers could be allocated for am2 when we allocating containers + // for am1, just check if pending resource of b1/b/root > 0 + checkPendingResourceGreaterThanZero(rm, "b1", null); + checkPendingResourceGreaterThanZero(rm, "b", null); + // root = a + b + checkPendingResourceGreaterThanZero(rm, "root", null); + checkPendingResource(rm, "root", 0 * GB, "x"); + + // complete am2, pending resource should be 0 now + AppAttemptRemovedSchedulerEvent appRemovedEvent = + new AppAttemptRemovedSchedulerEvent( + am2.getApplicationAttemptId(), RMAppAttemptState.FINISHED, false); + rm.getResourceScheduler().handle(appRemovedEvent); + + checkPendingResource(rm, "a1", 0 * GB, null); + checkPendingResource(rm, "a", 0 * GB, null); + checkPendingResource(rm, "a1", 0 * GB, "x"); + checkPendingResource(rm, "a", 0 * GB, "x"); + checkPendingResource(rm, "b1", 0 * GB, null); + checkPendingResource(rm, "b", 0 * GB, null); + checkPendingResource(rm, "root", 0 * GB, null); + checkPendingResource(rm, "root", 0 * GB, "x"); + } + + private void checkPendingResource(MockRM rm, String queueName, int memory, + String label) { + CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler(); + CSQueue queue = cs.getQueue(queueName); + Assert.assertEquals( + memory, + queue.getQueueResourceUsage() + .getPending(label == null ? RMNodeLabelsManager.NO_LABEL : label) + .getMemory()); + } + + private void checkPendingResourceGreaterThanZero(MockRM rm, String queueName, + String label) { + CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler(); + CSQueue queue = cs.getQueue(queueName); + Assert.assertTrue(queue.getQueueResourceUsage() + .getPending(label == null ? RMNodeLabelsManager.NO_LABEL : label) + .getMemory() > 0); + } + + // Test verifies AM Used resource for LeafQueue when AM ResourceRequest is + // lesser than minimumAllocation + @Test(timeout = 30000) + public void testAMUsedResource() throws Exception { + MockRM rm = setUpMove(); + Configuration conf = rm.getConfig(); + int minAllocMb = + conf.getInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); + int amMemory = 50; + assertTrue("AM memory is greater than or equql to minAllocation", + amMemory < minAllocMb); + Resource minAllocResource = Resource.newInstance(minAllocMb, 1); + String queueName = "a1"; + RMApp rmApp = rm.submitApp(amMemory, "app-1", "user_0", null, queueName); + + assertEquals("RMApp does not containes minimum allocation", + minAllocResource, rmApp.getAMResourceRequest().getCapability()); + + ResourceScheduler scheduler = rm.getRMContext().getScheduler(); + LeafQueue queueA = + (LeafQueue) ((CapacityScheduler) scheduler).getQueue(queueName); + assertEquals("Minimum Resource for AM is incorrect", minAllocResource, + queueA.getUser("user_0").getResourceUsage().getAMUsed()); + rm.stop(); + } + + // Verifies headroom passed to ApplicationMaster has been updated in + // RMAppAttemptMetrics + @Test + public void testApplicationHeadRoom() throws Exception { + Configuration conf = new Configuration(); + conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class, + ResourceScheduler.class); + MockRM rm = new MockRM(conf); + rm.start(); + CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler(); + + ApplicationId appId = BuilderUtils.newApplicationId(100, 1); + ApplicationAttemptId appAttemptId = + BuilderUtils.newApplicationAttemptId(appId, 1); + + RMAppAttemptMetrics attemptMetric = + new RMAppAttemptMetrics(appAttemptId, rm.getRMContext()); + RMAppImpl app = mock(RMAppImpl.class); + when(app.getApplicationId()).thenReturn(appId); + RMAppAttemptImpl attempt = mock(RMAppAttemptImpl.class); + when(attempt.getAppAttemptId()).thenReturn(appAttemptId); + when(attempt.getRMAppAttemptMetrics()).thenReturn(attemptMetric); + when(app.getCurrentAppAttempt()).thenReturn(attempt); + + rm.getRMContext().getRMApps().put(appId, app); + + SchedulerEvent addAppEvent = + new AppAddedSchedulerEvent(appId, "default", "user"); + cs.handle(addAppEvent); + SchedulerEvent addAttemptEvent = + new AppAttemptAddedSchedulerEvent(appAttemptId, false); + cs.handle(addAttemptEvent); + + Allocation allocate = + cs.allocate(appAttemptId, Collections. emptyList(), + Collections. emptyList(), null, null); + + Assert.assertNotNull(attempt); + + Assert + .assertEquals(Resource.newInstance(0, 0), allocate.getResourceLimit()); + Assert.assertEquals(Resource.newInstance(0, 0), + attemptMetric.getApplicationAttemptHeadroom()); + + // Add a node to cluster + Resource newResource = Resource.newInstance(4 * GB, 1); + RMNode node = MockNodes.newNodeInfo(0, newResource, 1, "127.0.0.1"); + cs.handle(new NodeAddedSchedulerEvent(node)); + + allocate = + cs.allocate(appAttemptId, Collections. emptyList(), + Collections. emptyList(), null, null); + + // All resources should be sent as headroom + Assert.assertEquals(newResource, allocate.getResourceLimit()); + Assert.assertEquals(newResource, + attemptMetric.getApplicationAttemptHeadroom()); + + rm.stop(); + } private void setMaxAllocMb(Configuration conf, int maxAllocMb) { conf.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java index 7edb17d0e7e50..23b31faeb8f7d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java @@ -20,7 +20,6 @@ import static org.junit.Assert.assertEquals; import static org.mockito.Matchers.any; -import static org.mockito.Matchers.anyBoolean; import static org.mockito.Matchers.eq; import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.doNothing; @@ -145,7 +144,7 @@ public CSAssignment answer(InvocationOnMock invocation) throws Throwable { if (allocation > 0) { doReturn(new CSAssignment(Resources.none(), type)). when(queue) - .assignContainers(eq(clusterResource), eq(node), anyBoolean(), + .assignContainers(eq(clusterResource), eq(node), any(ResourceLimits.class)); // Mock the node's resource availability @@ -157,7 +156,7 @@ public CSAssignment answer(InvocationOnMock invocation) throws Throwable { return new CSAssignment(allocatedResource, type); } }). - when(queue).assignContainers(eq(clusterResource), eq(node), anyBoolean(), + when(queue).assignContainers(eq(clusterResource), eq(node), any(ResourceLimits.class)); doNothing().when(node).releaseContainer(any(Container.class)); } @@ -248,7 +247,8 @@ public void testSortedQueues() throws Exception { // Stub an App and its containerCompleted FiCaSchedulerApp app_0 = getMockApplication(0,user_0); doReturn(true).when(app_0).containerCompleted(any(RMContainer.class), - any(ContainerStatus.class),any(RMContainerEventType.class)); + any(ContainerStatus.class), any(RMContainerEventType.class), + any(String.class)); Priority priority = TestUtils.createMockPriority(1); ContainerAllocationExpirer expirer = @@ -274,7 +274,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 0*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); for(int i=0; i < 2; i++) { @@ -282,7 +282,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 1*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); } for(int i=0; i < 3; i++) @@ -291,7 +291,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 0*GB); stubQueueAllocation(c, clusterResource, node_0, 1*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); } for(int i=0; i < 4; i++) @@ -300,7 +300,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 0*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 1*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); } verifyQueueMetrics(a, 1*GB, clusterResource); @@ -334,7 +334,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 0*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); } verifyQueueMetrics(a, 3*GB, clusterResource); @@ -362,7 +362,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 1*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); verifyQueueMetrics(a, 2*GB, clusterResource); verifyQueueMetrics(b, 3*GB, clusterResource); @@ -389,7 +389,7 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 0*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); verifyQueueMetrics(a, 3*GB, clusterResource); verifyQueueMetrics(b, 2*GB, clusterResource); @@ -404,13 +404,13 @@ public void testSortedQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 1*GB); stubQueueAllocation(c, clusterResource, node_0, 0*GB); stubQueueAllocation(d, clusterResource, node_0, 1*GB); - root.assignContainers(clusterResource, node_0, false, new ResourceLimits( + root.assignContainers(clusterResource, node_0, new ResourceLimits( clusterResource)); InOrder allocationOrder = inOrder(d,b); allocationOrder.verify(d).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), any(ResourceLimits.class)); + any(FiCaSchedulerNode.class), any(ResourceLimits.class)); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), any(ResourceLimits.class)); + any(FiCaSchedulerNode.class), any(ResourceLimits.class)); verifyQueueMetrics(a, 3*GB, clusterResource); verifyQueueMetrics(b, 2*GB, clusterResource); verifyQueueMetrics(c, 3*GB, clusterResource); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java index 169517dd37246..03b8f5c1fe195 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java @@ -44,7 +44,6 @@ import org.apache.hadoop.yarn.server.resourcemanager.RMContextImpl; import org.apache.hadoop.yarn.server.resourcemanager.RMSecretManagerService; import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; -import org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler; import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NullRMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; @@ -68,7 +67,8 @@ public class TestContainerAllocation { - private static final Log LOG = LogFactory.getLog(TestFifoScheduler.class); + private static final Log LOG = LogFactory + .getLog(TestContainerAllocation.class); private final int GB = 1024; @@ -230,12 +230,18 @@ public void testLogAggregationContextPassedIntoContainerToken() // create a not-null LogAggregationContext LogAggregationContext logAggregationContext = LogAggregationContext.newInstance( - "includePattern", "excludePattern"); + "includePattern", "excludePattern", + "rolledLogsIncludePattern", + "rolledLogsExcludePattern"); LogAggregationContext returned = getLogAggregationContextFromContainerToken(rm1, nm2, logAggregationContext); Assert.assertEquals("includePattern", returned.getIncludePattern()); Assert.assertEquals("excludePattern", returned.getExcludePattern()); + Assert.assertEquals("rolledLogsIncludePattern", + returned.getRolledLogsIncludePattern()); + Assert.assertEquals("rolledLogsExcludePattern", + returned.getRolledLogsExcludePattern()); rm1.stop(); } @@ -322,19 +328,6 @@ protected RMSecretManagerService createRMSecretManagerService() { MockRM.launchAndRegisterAM(app1, rm1, nm1); } - private Configuration getConfigurationWithDefaultQueueLabels( - Configuration config) { - final String A = CapacitySchedulerConfiguration.ROOT + ".a"; - final String B = CapacitySchedulerConfiguration.ROOT + ".b"; - - CapacitySchedulerConfiguration conf = - (CapacitySchedulerConfiguration) getConfigurationWithQueueLabels(config); - new CapacitySchedulerConfiguration(config); - conf.setDefaultNodeLabelExpression(A, "x"); - conf.setDefaultNodeLabelExpression(B, "y"); - return conf; - } - private Configuration getConfigurationWithQueueLabels(Configuration config) { CapacitySchedulerConfiguration conf = new CapacitySchedulerConfiguration(config); @@ -400,57 +393,6 @@ private Set toSet(E... elements) { return set; } - private Configuration getComplexConfigurationWithQueueLabels( - Configuration config) { - CapacitySchedulerConfiguration conf = - new CapacitySchedulerConfiguration(config); - - // Define top-level queues - conf.setQueues(CapacitySchedulerConfiguration.ROOT, new String[] {"a", "b"}); - conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "x", 100); - conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "y", 100); - conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "z", 100); - - final String A = CapacitySchedulerConfiguration.ROOT + ".a"; - conf.setCapacity(A, 10); - conf.setMaximumCapacity(A, 10); - conf.setAccessibleNodeLabels(A, toSet("x", "y")); - conf.setCapacityByLabel(A, "x", 100); - conf.setCapacityByLabel(A, "y", 50); - - final String B = CapacitySchedulerConfiguration.ROOT + ".b"; - conf.setCapacity(B, 90); - conf.setMaximumCapacity(B, 100); - conf.setAccessibleNodeLabels(B, toSet("y", "z")); - conf.setCapacityByLabel(B, "y", 50); - conf.setCapacityByLabel(B, "z", 100); - - // Define 2nd-level queues - final String A1 = A + ".a1"; - conf.setQueues(A, new String[] {"a1"}); - conf.setCapacity(A1, 100); - conf.setMaximumCapacity(A1, 100); - conf.setAccessibleNodeLabels(A1, toSet("x", "y")); - conf.setDefaultNodeLabelExpression(A1, "x"); - conf.setCapacityByLabel(A1, "x", 100); - conf.setCapacityByLabel(A1, "y", 100); - - conf.setQueues(B, new String[] {"b1", "b2"}); - final String B1 = B + ".b1"; - conf.setCapacity(B1, 50); - conf.setMaximumCapacity(B1, 50); - conf.setAccessibleNodeLabels(B1, RMNodeLabelsManager.EMPTY_STRING_SET); - - final String B2 = B + ".b2"; - conf.setCapacity(B2, 50); - conf.setMaximumCapacity(B2, 50); - conf.setAccessibleNodeLabels(B2, toSet("y", "z")); - conf.setCapacityByLabel(B2, "y", 100); - conf.setCapacityByLabel(B2, "z", 100); - - return conf; - } - @Test (timeout = 300000) public void testContainerAllocationWithSingleUserLimits() throws Exception { final RMNodeLabelsManager mgr = new NullRMNodeLabelsManager(); @@ -462,7 +404,7 @@ public void testContainerAllocationWithSingleUserLimits() throws Exception { NodeId.newInstance("h2", 0), toSet("y"))); // inject node label manager - MockRM rm1 = new MockRM(getConfigurationWithDefaultQueueLabels(conf)) { + MockRM rm1 = new MockRM(TestUtils.getConfigurationWithDefaultQueueLabels(conf)) { @Override public RMNodeLabelsManager createNodeLabelManager() { return mgr; @@ -548,7 +490,7 @@ public void testContainerAllocateWithComplexLabels() throws Exception { RMNodeLabelsManager.EMPTY_STRING_SET)); // inject node label manager - MockRM rm1 = new MockRM(getComplexConfigurationWithQueueLabels(conf)) { + MockRM rm1 = new MockRM(TestUtils.getComplexConfigurationWithQueueLabels(conf)) { @Override public RMNodeLabelsManager createNodeLabelManager() { return mgr; @@ -705,7 +647,7 @@ public void testContainerAllocateWithDefaultQueueLabels() throws Exception { NodeId.newInstance("h2", 0), toSet("y"))); // inject node label manager - MockRM rm1 = new MockRM(getConfigurationWithDefaultQueueLabels(conf)) { + MockRM rm1 = new MockRM(TestUtils.getConfigurationWithDefaultQueueLabels(conf)) { @Override public RMNodeLabelsManager createNodeLabelManager() { return mgr; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java index a5a2e5fe0ddd4..972cabbf2cc2c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java @@ -350,8 +350,8 @@ public void testSingleQueueOneUserMetrics() throws Exception { // Start testing... // Only 1 container - a.assignContainers(clusterResource, node_0, false, - new ResourceLimits(clusterResource)); + a.assignContainers(clusterResource, node_0, new ResourceLimits( + clusterResource)); assertEquals( (int)(node_0.getTotalResource().getMemory() * a.getCapacity()) - (1*GB), a.getMetrics().getAvailableMB()); @@ -486,7 +486,7 @@ public void testSingleQueueWithOneUser() throws Exception { // Start testing... // Only 1 container - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(1*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -497,7 +497,7 @@ public void testSingleQueueWithOneUser() throws Exception { // Also 2nd -> minCapacity = 1024 since (.1 * 8G) < minAlloc, also // you can get one container more than user-limit - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -506,7 +506,7 @@ public void testSingleQueueWithOneUser() throws Exception { assertEquals(2*GB, a.getMetrics().getAllocatedMB()); // Can't allocate 3rd due to user-limit - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -516,7 +516,7 @@ public void testSingleQueueWithOneUser() throws Exception { // Bump up user-limit-factor, now allocate should work a.setUserLimitFactor(10); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(3*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); @@ -525,7 +525,7 @@ public void testSingleQueueWithOneUser() throws Exception { assertEquals(3*GB, a.getMetrics().getAllocatedMB()); // One more should work, for app_1, due to user-limit-factor - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(4*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); @@ -536,8 +536,8 @@ public void testSingleQueueWithOneUser() throws Exception { // Test max-capacity // Now - no more allocs since we are at max-cap a.setMaxCapacity(0.5f); - a.assignContainers(clusterResource, node_0, false, - new ResourceLimits(clusterResource)); + a.assignContainers(clusterResource, node_0, new ResourceLimits( + clusterResource)); assertEquals(4*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); assertEquals(1*GB, app_1.getCurrentConsumption().getMemory()); @@ -652,21 +652,21 @@ public void testUserLimits() throws Exception { // recordFactory))); // 1 container to user_0 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); assertEquals(0*GB, app_1.getCurrentConsumption().getMemory()); // Again one to user_0 since he hasn't exceeded user limit yet - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(3*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); assertEquals(1*GB, app_1.getCurrentConsumption().getMemory()); // One more to user_0 since he is the only active user - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(4*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -718,7 +718,7 @@ public void testComputeUserLimitAndSetHeadroom(){ assertEquals("There should only be 1 active user!", 1, qb.getActiveUsersManager().getNumActiveUsers()); //get headroom - qb.assignContainers(clusterResource, node_0, false, + qb.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); qb.computeUserLimitAndSetHeadroom(app_0, clusterResource, app_0 .getResourceRequest(u0Priority, ResourceRequest.ANY).getCapability(), @@ -738,7 +738,7 @@ public void testComputeUserLimitAndSetHeadroom(){ TestUtils.createResourceRequest(ResourceRequest.ANY, 4*GB, 1, true, u1Priority, recordFactory))); qb.submitApplicationAttempt(app_2, user_1); - qb.assignContainers(clusterResource, node_1, false, + qb.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); qb.computeUserLimitAndSetHeadroom(app_0, clusterResource, app_0 .getResourceRequest(u0Priority, ResourceRequest.ANY).getCapability(), @@ -781,9 +781,9 @@ public void testComputeUserLimitAndSetHeadroom(){ u1Priority, recordFactory))); qb.submitApplicationAttempt(app_1, user_0); qb.submitApplicationAttempt(app_3, user_1); - qb.assignContainers(clusterResource, node_0, false, + qb.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); - qb.assignContainers(clusterResource, node_0, false, + qb.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); qb.computeUserLimitAndSetHeadroom(app_3, clusterResource, app_3 .getResourceRequest(u1Priority, ResourceRequest.ANY).getCapability(), @@ -802,7 +802,7 @@ public void testComputeUserLimitAndSetHeadroom(){ app_4.updateResourceRequests(Collections.singletonList( TestUtils.createResourceRequest(ResourceRequest.ANY, 6*GB, 1, true, u0Priority, recordFactory))); - qb.assignContainers(clusterResource, node_1, false, + qb.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); qb.computeUserLimitAndSetHeadroom(app_4, clusterResource, app_4 .getResourceRequest(u0Priority, ResourceRequest.ANY).getCapability(), @@ -875,7 +875,7 @@ public void testUserHeadroomMultiApp() throws Exception { TestUtils.createResourceRequest(ResourceRequest.ANY, 1*GB, 1, true, priority, recordFactory))); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(1*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -892,7 +892,7 @@ public void testUserHeadroomMultiApp() throws Exception { TestUtils.createResourceRequest(ResourceRequest.ANY, 1*GB, 2, true, priority, recordFactory))); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -981,7 +981,7 @@ public void testHeadroomWithMaxCap() throws Exception { 1, a.getActiveUsersManager().getNumActiveUsers()); // 1 container to user_0 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -992,7 +992,7 @@ public void testHeadroomWithMaxCap() throws Exception { // the application is not yet active // Again one to user_0 since he hasn't exceeded user limit yet - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(3*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1009,7 +1009,7 @@ public void testHeadroomWithMaxCap() throws Exception { // No more to user_0 since he is already over user-limit // and no more containers to queue since it's already at max-cap - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(3*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1023,7 +1023,7 @@ public void testHeadroomWithMaxCap() throws Exception { TestUtils.createResourceRequest(ResourceRequest.ANY, 1*GB, 0, true, priority, recordFactory))); assertEquals(1, a.getActiveUsersManager().getNumActiveUsers()); - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(0*GB, app_2.getHeadroom().getMemory()); // hit queue max-cap } @@ -1094,7 +1094,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { */ // Only 1 container - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(1*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1102,7 +1102,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { // Also 2nd -> minCapacity = 1024 since (.1 * 8G) < minAlloc, also // you can get one container more than user-limit - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1110,7 +1110,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { // Can't allocate 3rd due to user-limit a.setUserLimit(25); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1129,7 +1129,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { // Now allocations should goto app_2 since // user_0 is at limit inspite of high user-limit-factor a.setUserLimitFactor(10); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1139,7 +1139,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { // Now allocations should goto app_0 since // user_0 is at user-limit not above it - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(6*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); @@ -1150,7 +1150,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { // Test max-capacity // Now - no more allocs since we are at max-cap a.setMaxCapacity(0.5f); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(6*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); @@ -1162,7 +1162,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { // Now, allocations should goto app_3 since it's under user-limit a.setMaxCapacity(1.0f); a.setUserLimitFactor(1); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(7*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); @@ -1171,7 +1171,7 @@ public void testSingleQueueWithMultipleUsers() throws Exception { assertEquals(1*GB, app_3.getCurrentConsumption().getMemory()); // Now we should assign to app_3 again since user_2 is under user-limit - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(8*GB, a.getUsedResources().getMemory()); assertEquals(3*GB, app_0.getCurrentConsumption().getMemory()); @@ -1271,7 +1271,7 @@ public void testReservation() throws Exception { // Start testing... // Only 1 container - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(1*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1282,7 +1282,7 @@ public void testReservation() throws Exception { // Also 2nd -> minCapacity = 1024 since (.1 * 8G) < minAlloc, also // you can get one container more than user-limit - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1291,7 +1291,7 @@ public void testReservation() throws Exception { assertEquals(2*GB, a.getMetrics().getAllocatedMB()); // Now, reservation should kick in for app_1 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(6*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1308,7 +1308,7 @@ public void testReservation() throws Exception { ContainerState.COMPLETE, "", ContainerExitStatus.KILLED_BY_RESOURCEMANAGER), RMContainerEventType.KILL, null, true); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1325,7 +1325,7 @@ public void testReservation() throws Exception { ContainerState.COMPLETE, "", ContainerExitStatus.KILLED_BY_RESOURCEMANAGER), RMContainerEventType.KILL, null, true); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(4*GB, a.getUsedResources().getMemory()); assertEquals(0*GB, app_0.getCurrentConsumption().getMemory()); @@ -1393,7 +1393,7 @@ public void testStolenReservedContainer() throws Exception { // Start testing... - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1403,7 +1403,7 @@ public void testStolenReservedContainer() throws Exception { assertEquals(0*GB, a.getMetrics().getAvailableMB()); // Now, reservation should kick in for app_1 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(6*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1417,7 +1417,7 @@ public void testStolenReservedContainer() throws Exception { // We do not need locality delay here doReturn(-1).when(a).getNodeLocalityDelay(); - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(10*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1434,7 +1434,7 @@ public void testStolenReservedContainer() throws Exception { ContainerState.COMPLETE, "", ContainerExitStatus.KILLED_BY_RESOURCEMANAGER), RMContainerEventType.KILL, null, true); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(8*GB, a.getUsedResources().getMemory()); assertEquals(0*GB, app_0.getCurrentConsumption().getMemory()); @@ -1503,7 +1503,7 @@ public void testReservationExchange() throws Exception { // Start testing... // Only 1 container - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(1*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1511,14 +1511,14 @@ public void testReservationExchange() throws Exception { // Also 2nd -> minCapacity = 1024 since (.1 * 8G) < minAlloc, also // you can get one container more than user-limit - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); assertEquals(0*GB, app_1.getCurrentConsumption().getMemory()); // Now, reservation should kick in for app_1 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(6*GB, a.getUsedResources().getMemory()); assertEquals(2*GB, app_0.getCurrentConsumption().getMemory()); @@ -1533,7 +1533,7 @@ public void testReservationExchange() throws Exception { ContainerState.COMPLETE, "", ContainerExitStatus.KILLED_BY_RESOURCEMANAGER), RMContainerEventType.KILL, null, true); - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1543,7 +1543,7 @@ public void testReservationExchange() throws Exception { assertEquals(1, app_1.getReReservations(priority)); // Re-reserve - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1553,7 +1553,7 @@ public void testReservationExchange() throws Exception { assertEquals(2, app_1.getReReservations(priority)); // Try to schedule on node_1 now, should *move* the reservation - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(9*GB, a.getUsedResources().getMemory()); assertEquals(1*GB, app_0.getCurrentConsumption().getMemory()); @@ -1571,7 +1571,7 @@ public void testReservationExchange() throws Exception { ContainerState.COMPLETE, "", ContainerExitStatus.KILLED_BY_RESOURCEMANAGER), RMContainerEventType.KILL, null, true); - CSAssignment assignment = a.assignContainers(clusterResource, node_0, false, + CSAssignment assignment = a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(8*GB, a.getUsedResources().getMemory()); assertEquals(0*GB, app_0.getCurrentConsumption().getMemory()); @@ -1643,7 +1643,7 @@ public void testLocalityScheduling() throws Exception { CSAssignment assignment = null; // Start with off switch, shouldn't allocate due to delay scheduling - assignment = a.assignContainers(clusterResource, node_2, false, + assignment = a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_2), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1652,7 +1652,7 @@ public void testLocalityScheduling() throws Exception { assertEquals(NodeType.NODE_LOCAL, assignment.getType()); // None->NODE_LOCAL // Another off switch, shouldn't allocate due to delay scheduling - assignment = a.assignContainers(clusterResource, node_2, false, + assignment = a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_2), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1661,7 +1661,7 @@ public void testLocalityScheduling() throws Exception { assertEquals(NodeType.NODE_LOCAL, assignment.getType()); // None->NODE_LOCAL // Another off switch, shouldn't allocate due to delay scheduling - assignment = a.assignContainers(clusterResource, node_2, false, + assignment = a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_2), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1671,7 +1671,7 @@ public void testLocalityScheduling() throws Exception { // Another off switch, now we should allocate // since missedOpportunities=3 and reqdContainers=3 - assignment = a.assignContainers(clusterResource, node_2, false, + assignment = a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.OFF_SWITCH), eq(node_2), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1680,7 +1680,7 @@ public void testLocalityScheduling() throws Exception { assertEquals(NodeType.OFF_SWITCH, assignment.getType()); // NODE_LOCAL - node_0 - assignment = a.assignContainers(clusterResource, node_0, false, + assignment = a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.NODE_LOCAL), eq(node_0), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1689,7 +1689,7 @@ public void testLocalityScheduling() throws Exception { assertEquals(NodeType.NODE_LOCAL, assignment.getType()); // NODE_LOCAL - node_1 - assignment = a.assignContainers(clusterResource, node_1, false, + assignment = a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.NODE_LOCAL), eq(node_1), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1718,14 +1718,14 @@ public void testLocalityScheduling() throws Exception { doReturn(1).when(a).getNodeLocalityDelay(); // Shouldn't assign RACK_LOCAL yet - assignment = a.assignContainers(clusterResource, node_3, false, + assignment = a.assignContainers(clusterResource, node_3, new ResourceLimits(clusterResource)); assertEquals(1, app_0.getSchedulingOpportunities(priority)); assertEquals(2, app_0.getTotalRequiredResources(priority)); assertEquals(NodeType.NODE_LOCAL, assignment.getType()); // None->NODE_LOCAL // Should assign RACK_LOCAL now - assignment = a.assignContainers(clusterResource, node_3, false, + assignment = a.assignContainers(clusterResource, node_3, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.RACK_LOCAL), eq(node_3), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1807,7 +1807,7 @@ public void testApplicationPriorityScheduling() throws Exception { // Start with off switch, shouldn't allocate P1 due to delay scheduling // thus, no P2 either! - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_2), eq(priority_1), any(ResourceRequest.class), any(Container.class)); @@ -1820,7 +1820,7 @@ public void testApplicationPriorityScheduling() throws Exception { // Another off-switch, shouldn't allocate P1 due to delay scheduling // thus, no P2 either! - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_2), eq(priority_1), any(ResourceRequest.class), any(Container.class)); @@ -1832,7 +1832,7 @@ public void testApplicationPriorityScheduling() throws Exception { assertEquals(1, app_0.getTotalRequiredResources(priority_2)); // Another off-switch, shouldn't allocate OFF_SWITCH P1 - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.OFF_SWITCH), eq(node_2), eq(priority_1), any(ResourceRequest.class), any(Container.class)); @@ -1844,7 +1844,7 @@ public void testApplicationPriorityScheduling() throws Exception { assertEquals(1, app_0.getTotalRequiredResources(priority_2)); // Now, DATA_LOCAL for P1 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.NODE_LOCAL), eq(node_0), eq(priority_1), any(ResourceRequest.class), any(Container.class)); @@ -1856,7 +1856,7 @@ public void testApplicationPriorityScheduling() throws Exception { assertEquals(1, app_0.getTotalRequiredResources(priority_2)); // Now, OFF_SWITCH for P2 - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_1), eq(priority_1), any(ResourceRequest.class), any(Container.class)); @@ -1933,7 +1933,7 @@ public void testSchedulingConstraints() throws Exception { app_0.updateResourceRequests(app_0_requests_0); // NODE_LOCAL - node_0_1 - a.assignContainers(clusterResource, node_0_0, false, + a.assignContainers(clusterResource, node_0_0, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.NODE_LOCAL), eq(node_0_0), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1942,7 +1942,7 @@ public void testSchedulingConstraints() throws Exception { // No allocation on node_1_0 even though it's node/rack local since // required(ANY) == 0 - a.assignContainers(clusterResource, node_1_0, false, + a.assignContainers(clusterResource, node_1_0, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_1_0), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1959,7 +1959,7 @@ public void testSchedulingConstraints() throws Exception { // No allocation on node_0_1 even though it's node/rack local since // required(rack_1) == 0 - a.assignContainers(clusterResource, node_0_1, false, + a.assignContainers(clusterResource, node_0_1, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_1_0), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -1967,7 +1967,7 @@ public void testSchedulingConstraints() throws Exception { assertEquals(1, app_0.getTotalRequiredResources(priority)); // NODE_LOCAL - node_1 - a.assignContainers(clusterResource, node_1_0, false, + a.assignContainers(clusterResource, node_1_0, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.NODE_LOCAL), eq(node_1_0), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2220,7 +2220,7 @@ public void testLocalityConstraints() throws Exception { // node_0_1 // Shouldn't allocate since RR(rack_0) = null && RR(ANY) = relax: false - a.assignContainers(clusterResource, node_0_1, false, + a.assignContainers(clusterResource, node_0_1, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_0_1), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2243,7 +2243,7 @@ public void testLocalityConstraints() throws Exception { // node_1_1 // Shouldn't allocate since RR(rack_1) = relax: false - a.assignContainers(clusterResource, node_1_1, false, + a.assignContainers(clusterResource, node_1_1, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_0_1), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2274,7 +2274,7 @@ public void testLocalityConstraints() throws Exception { // node_1_1 // Shouldn't allocate since node_1_1 is blacklisted - a.assignContainers(clusterResource, node_1_1, false, + a.assignContainers(clusterResource, node_1_1, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_1_1), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2303,7 +2303,7 @@ public void testLocalityConstraints() throws Exception { // node_1_1 // Shouldn't allocate since rack_1 is blacklisted - a.assignContainers(clusterResource, node_1_1, false, + a.assignContainers(clusterResource, node_1_1, new ResourceLimits(clusterResource)); verify(app_0, never()).allocate(any(NodeType.class), eq(node_1_1), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2330,7 +2330,7 @@ public void testLocalityConstraints() throws Exception { // Blacklist: < host_0_0 > <---- // Now, should allocate since RR(rack_1) = relax: true - a.assignContainers(clusterResource, node_1_1, false, + a.assignContainers(clusterResource, node_1_1, new ResourceLimits(clusterResource)); verify(app_0,never()).allocate(eq(NodeType.RACK_LOCAL), eq(node_1_1), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2361,7 +2361,7 @@ public void testLocalityConstraints() throws Exception { // host_1_0: 8G // host_1_1: 7G - a.assignContainers(clusterResource, node_1_0, false, + a.assignContainers(clusterResource, node_1_0, new ResourceLimits(clusterResource)); verify(app_0).allocate(eq(NodeType.NODE_LOCAL), eq(node_1_0), any(Priority.class), any(ResourceRequest.class), any(Container.class)); @@ -2444,7 +2444,7 @@ public void testAllocateContainerOnNodeWithoutOffSwitchSpecified() recordFactory))); try { - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); } catch (NullPointerException e) { Assert.fail("NPE when allocating container on node but " diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java index 4f8938607e74a..7da1c97fec0ef 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java @@ -156,7 +156,7 @@ public CSAssignment answer(InvocationOnMock invocation) throws Throwable { // Next call - nothing if (allocation > 0) { doReturn(new CSAssignment(Resources.none(), type)).when(queue) - .assignContainers(eq(clusterResource), eq(node), eq(false), + .assignContainers(eq(clusterResource), eq(node), any(ResourceLimits.class)); // Mock the node's resource availability @@ -167,8 +167,7 @@ public CSAssignment answer(InvocationOnMock invocation) throws Throwable { return new CSAssignment(allocatedResource, type); } - }). -when(queue).assignContainers(eq(clusterResource), eq(node), eq(false), + }).when(queue).assignContainers(eq(clusterResource), eq(node), any(ResourceLimits.class)); } @@ -232,7 +231,7 @@ public void testSingleLevelQueues() throws Exception { // Simulate B returning a container on node_0 stubQueueAllocation(a, clusterResource, node_0, 0*GB); stubQueueAllocation(b, clusterResource, node_0, 1*GB); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); verifyQueueMetrics(a, 0*GB, clusterResource); verifyQueueMetrics(b, 1*GB, clusterResource); @@ -240,13 +239,13 @@ public void testSingleLevelQueues() throws Exception { // Now, A should get the scheduling opportunity since A=0G/6G, B=1G/14G stubQueueAllocation(a, clusterResource, node_1, 2*GB); stubQueueAllocation(b, clusterResource, node_1, 1*GB); - root.assignContainers(clusterResource, node_1, false, + root.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); InOrder allocationOrder = inOrder(a, b); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 2*GB, clusterResource); verifyQueueMetrics(b, 2*GB, clusterResource); @@ -254,13 +253,13 @@ public void testSingleLevelQueues() throws Exception { // since A has 2/6G while B has 2/14G stubQueueAllocation(a, clusterResource, node_0, 1*GB); stubQueueAllocation(b, clusterResource, node_0, 2*GB); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); allocationOrder = inOrder(b, a); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 3*GB, clusterResource); verifyQueueMetrics(b, 4*GB, clusterResource); @@ -268,13 +267,13 @@ public void testSingleLevelQueues() throws Exception { // since A has 3/6G while B has 4/14G stubQueueAllocation(a, clusterResource, node_0, 0*GB); stubQueueAllocation(b, clusterResource, node_0, 4*GB); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); allocationOrder = inOrder(b, a); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 3*GB, clusterResource); verifyQueueMetrics(b, 8*GB, clusterResource); @@ -282,13 +281,13 @@ public void testSingleLevelQueues() throws Exception { // since A has 3/6G while B has 8/14G stubQueueAllocation(a, clusterResource, node_1, 1*GB); stubQueueAllocation(b, clusterResource, node_1, 1*GB); - root.assignContainers(clusterResource, node_1, false, + root.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); allocationOrder = inOrder(a, b); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 4*GB, clusterResource); verifyQueueMetrics(b, 9*GB, clusterResource); } @@ -405,6 +404,22 @@ private void setupMultiLevelQueues(CapacitySchedulerConfiguration conf) { @Test public void testMultiLevelQueues() throws Exception { + /* + * Structure of queue: + * Root + * ____________ + * / | \ \ + * A B C D + * / | / | \ \ + * A1 A2 B1 B2 B3 C1 + * \ + * C11 + * \ + * C111 + * \ + * C1111 + */ + // Setup queue configs setupMultiLevelQueues(csConf); @@ -449,7 +464,7 @@ public void testMultiLevelQueues() throws Exception { stubQueueAllocation(b, clusterResource, node_0, 0*GB); stubQueueAllocation(c, clusterResource, node_0, 1*GB); stubQueueAllocation(d, clusterResource, node_0, 0*GB); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); verifyQueueMetrics(a, 0*GB, clusterResource); verifyQueueMetrics(b, 0*GB, clusterResource); @@ -462,7 +477,7 @@ public void testMultiLevelQueues() throws Exception { stubQueueAllocation(a, clusterResource, node_1, 0*GB); stubQueueAllocation(b2, clusterResource, node_1, 4*GB); stubQueueAllocation(c, clusterResource, node_1, 0*GB); - root.assignContainers(clusterResource, node_1, false, + root.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); verifyQueueMetrics(a, 0*GB, clusterResource); verifyQueueMetrics(b, 4*GB, clusterResource); @@ -474,15 +489,15 @@ public void testMultiLevelQueues() throws Exception { stubQueueAllocation(a1, clusterResource, node_0, 1*GB); stubQueueAllocation(b3, clusterResource, node_0, 2*GB); stubQueueAllocation(c, clusterResource, node_0, 2*GB); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); InOrder allocationOrder = inOrder(a, c, b); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(c).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 1*GB, clusterResource); verifyQueueMetrics(b, 6*GB, clusterResource); verifyQueueMetrics(c, 3*GB, clusterResource); @@ -501,17 +516,17 @@ public void testMultiLevelQueues() throws Exception { stubQueueAllocation(b3, clusterResource, node_2, 1*GB); stubQueueAllocation(b1, clusterResource, node_2, 1*GB); stubQueueAllocation(c, clusterResource, node_2, 1*GB); - root.assignContainers(clusterResource, node_2, false, + root.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); allocationOrder = inOrder(a, a2, a1, b, c); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(a2).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(c).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 3*GB, clusterResource); verifyQueueMetrics(b, 8*GB, clusterResource); verifyQueueMetrics(c, 4*GB, clusterResource); @@ -611,7 +626,7 @@ public void testOffSwitchScheduling() throws Exception { // Simulate B returning a container on node_0 stubQueueAllocation(a, clusterResource, node_0, 0*GB, NodeType.OFF_SWITCH); stubQueueAllocation(b, clusterResource, node_0, 1*GB, NodeType.OFF_SWITCH); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); verifyQueueMetrics(a, 0*GB, clusterResource); verifyQueueMetrics(b, 1*GB, clusterResource); @@ -620,13 +635,13 @@ public void testOffSwitchScheduling() throws Exception { // also, B gets a scheduling opportunity since A allocates RACK_LOCAL stubQueueAllocation(a, clusterResource, node_1, 2*GB, NodeType.RACK_LOCAL); stubQueueAllocation(b, clusterResource, node_1, 1*GB, NodeType.OFF_SWITCH); - root.assignContainers(clusterResource, node_1, false, + root.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); InOrder allocationOrder = inOrder(a, b); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 2*GB, clusterResource); verifyQueueMetrics(b, 2*GB, clusterResource); @@ -635,13 +650,13 @@ public void testOffSwitchScheduling() throws Exception { // However, since B returns off-switch, A won't get an opportunity stubQueueAllocation(a, clusterResource, node_0, 1*GB, NodeType.NODE_LOCAL); stubQueueAllocation(b, clusterResource, node_0, 2*GB, NodeType.OFF_SWITCH); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); allocationOrder = inOrder(b, a); allocationOrder.verify(b).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(a).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(a, 2*GB, clusterResource); verifyQueueMetrics(b, 4*GB, clusterResource); @@ -680,7 +695,7 @@ public void testOffSwitchSchedulingMultiLevelQueues() throws Exception { // Simulate B3 returning a container on node_0 stubQueueAllocation(b2, clusterResource, node_0, 0*GB, NodeType.OFF_SWITCH); stubQueueAllocation(b3, clusterResource, node_0, 1*GB, NodeType.OFF_SWITCH); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); verifyQueueMetrics(b2, 0*GB, clusterResource); verifyQueueMetrics(b3, 1*GB, clusterResource); @@ -689,13 +704,13 @@ public void testOffSwitchSchedulingMultiLevelQueues() throws Exception { // also, B3 gets a scheduling opportunity since B2 allocates RACK_LOCAL stubQueueAllocation(b2, clusterResource, node_1, 1*GB, NodeType.RACK_LOCAL); stubQueueAllocation(b3, clusterResource, node_1, 1*GB, NodeType.OFF_SWITCH); - root.assignContainers(clusterResource, node_1, false, + root.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); InOrder allocationOrder = inOrder(b2, b3); allocationOrder.verify(b2).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(b3).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(b2, 1*GB, clusterResource); verifyQueueMetrics(b3, 2*GB, clusterResource); @@ -704,13 +719,13 @@ public void testOffSwitchSchedulingMultiLevelQueues() throws Exception { // However, since B3 returns off-switch, B2 won't get an opportunity stubQueueAllocation(b2, clusterResource, node_0, 1*GB, NodeType.NODE_LOCAL); stubQueueAllocation(b3, clusterResource, node_0, 1*GB, NodeType.OFF_SWITCH); - root.assignContainers(clusterResource, node_0, false, + root.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); allocationOrder = inOrder(b3, b2); allocationOrder.verify(b3).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); allocationOrder.verify(b2).assignContainers(eq(clusterResource), - any(FiCaSchedulerNode.class), anyBoolean(), anyResourceLimits()); + any(FiCaSchedulerNode.class), anyResourceLimits()); verifyQueueMetrics(b2, 1*GB, clusterResource); verifyQueueMetrics(b3, 3*GB, clusterResource); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java index 4c6b25f1f1e1e..e8a8243203365 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java @@ -52,6 +52,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.RMContext; import org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter; import org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer; import org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerEventType; @@ -218,6 +219,7 @@ public void testReservation() throws Exception { .getMockApplicationAttemptId(0, 0); FiCaSchedulerApp app_0 = new FiCaSchedulerApp(appAttemptId_0, user_0, a, mock(ActiveUsersManager.class), spyRMContext); + rmContext.getRMApps().put(app_0.getApplicationId(), mock(RMApp.class)); a.submitApplicationAttempt(app_0, user_0); @@ -263,7 +265,7 @@ public void testReservation() throws Exception { // Start testing... // Only AM - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2 * GB, a.getUsedResources().getMemory()); assertEquals(2 * GB, app_0.getCurrentConsumption().getMemory()); @@ -275,7 +277,7 @@ public void testReservation() throws Exception { assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // Only 1 map - simulating reduce - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5 * GB, a.getUsedResources().getMemory()); assertEquals(5 * GB, app_0.getCurrentConsumption().getMemory()); @@ -287,7 +289,7 @@ public void testReservation() throws Exception { assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // Only 1 map to other node - simulating reduce - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -302,7 +304,7 @@ public void testReservation() throws Exception { assertEquals(2, app_0.getTotalRequiredResources(priorityReduce)); // try to assign reducer (5G on node 0 and should reserve) - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -318,7 +320,7 @@ public void testReservation() throws Exception { assertEquals(2, app_0.getTotalRequiredResources(priorityReduce)); // assign reducer to node 2 - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); assertEquals(18 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); @@ -335,7 +337,7 @@ public void testReservation() throws Exception { // node_1 heartbeat and unreserves from node_0 in order to allocate // on node_1 - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(18 * GB, a.getUsedResources().getMemory()); assertEquals(18 * GB, app_0.getCurrentConsumption().getMemory()); @@ -373,6 +375,7 @@ public void testReservationNoContinueLook() throws Exception { .getMockApplicationAttemptId(0, 0); FiCaSchedulerApp app_0 = new FiCaSchedulerApp(appAttemptId_0, user_0, a, mock(ActiveUsersManager.class), spyRMContext); + rmContext.getRMApps().put(app_0.getApplicationId(), mock(RMApp.class)); a.submitApplicationAttempt(app_0, user_0); @@ -418,7 +421,7 @@ public void testReservationNoContinueLook() throws Exception { // Start testing... // Only AM - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2 * GB, a.getUsedResources().getMemory()); assertEquals(2 * GB, app_0.getCurrentConsumption().getMemory()); @@ -430,7 +433,7 @@ public void testReservationNoContinueLook() throws Exception { assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // Only 1 map - simulating reduce - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5 * GB, a.getUsedResources().getMemory()); assertEquals(5 * GB, app_0.getCurrentConsumption().getMemory()); @@ -442,7 +445,7 @@ public void testReservationNoContinueLook() throws Exception { assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // Only 1 map to other node - simulating reduce - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -457,7 +460,7 @@ public void testReservationNoContinueLook() throws Exception { assertEquals(2, app_0.getTotalRequiredResources(priorityReduce)); // try to assign reducer (5G on node 0 and should reserve) - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -473,7 +476,7 @@ public void testReservationNoContinueLook() throws Exception { assertEquals(2, app_0.getTotalRequiredResources(priorityReduce)); // assign reducer to node 2 - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); assertEquals(18 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); @@ -490,7 +493,7 @@ public void testReservationNoContinueLook() throws Exception { // node_1 heartbeat and won't unreserve from node_0, potentially stuck // if AM doesn't handle - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(18 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); @@ -524,6 +527,7 @@ public void testAssignContainersNeedToUnreserve() throws Exception { .getMockApplicationAttemptId(0, 0); FiCaSchedulerApp app_0 = new FiCaSchedulerApp(appAttemptId_0, user_0, a, mock(ActiveUsersManager.class), spyRMContext); + rmContext.getRMApps().put(app_0.getApplicationId(), mock(RMApp.class)); a.submitApplicationAttempt(app_0, user_0); @@ -565,7 +569,7 @@ public void testAssignContainersNeedToUnreserve() throws Exception { // Start testing... // Only AM - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2 * GB, a.getUsedResources().getMemory()); assertEquals(2 * GB, app_0.getCurrentConsumption().getMemory()); @@ -576,7 +580,7 @@ public void testAssignContainersNeedToUnreserve() throws Exception { assertEquals(0 * GB, node_1.getUsedResource().getMemory()); // Only 1 map - simulating reduce - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5 * GB, a.getUsedResources().getMemory()); assertEquals(5 * GB, app_0.getCurrentConsumption().getMemory()); @@ -587,7 +591,7 @@ public void testAssignContainersNeedToUnreserve() throws Exception { assertEquals(0 * GB, node_1.getUsedResource().getMemory()); // Only 1 map to other node - simulating reduce - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -601,7 +605,7 @@ public void testAssignContainersNeedToUnreserve() throws Exception { assertEquals(2, app_0.getTotalRequiredResources(priorityReduce)); // try to assign reducer (5G on node 0 and should reserve) - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -616,7 +620,7 @@ public void testAssignContainersNeedToUnreserve() throws Exception { assertEquals(2, app_0.getTotalRequiredResources(priorityReduce)); // could allocate but told need to unreserve first - a.assignContainers(clusterResource, node_1, true, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); @@ -648,6 +652,8 @@ public void testGetAppToUnreserve() throws Exception { String host_1 = "host_1"; FiCaSchedulerNode node_1 = TestUtils.getMockNode(host_1, DEFAULT_RACK, 0, 8 * GB); + + Resource clusterResource = Resources.createResource(2 * 8 * GB); // Setup resource-requests Priority priorityMap = TestUtils.createMockPriority(5); @@ -677,23 +683,28 @@ public void testGetAppToUnreserve() throws Exception { node_0.getNodeID(), "user", rmContext); // no reserved containers - NodeId unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability); + NodeId unreserveId = + app_0.getNodeIdToUnreserve(priorityMap, capability, + cs.getResourceCalculator(), clusterResource); assertEquals(null, unreserveId); // no reserved containers - reserve then unreserve app_0.reserve(node_0, priorityMap, rmContainer_1, container_1); app_0.unreserve(node_0, priorityMap); - unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability); + unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability, + cs.getResourceCalculator(), clusterResource); assertEquals(null, unreserveId); // no container large enough is reserved app_0.reserve(node_0, priorityMap, rmContainer_1, container_1); - unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability); + unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability, + cs.getResourceCalculator(), clusterResource); assertEquals(null, unreserveId); // reserve one that is now large enough app_0.reserve(node_1, priorityMap, rmContainer, container); - unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability); + unreserveId = app_0.getNodeIdToUnreserve(priorityMap, capability, + cs.getResourceCalculator(), clusterResource); assertEquals(node_1.getNodeID(), unreserveId); } @@ -737,14 +748,14 @@ public void testFindNodeToUnreserve() throws Exception { // nothing reserved boolean res = a.findNodeToUnreserve(csContext.getClusterResource(), - node_1, app_0, priorityMap, capability); + node_1, app_0, priorityMap, capability, capability); assertFalse(res); // reserved but scheduler doesn't know about that node. app_0.reserve(node_1, priorityMap, rmContainer, container); node_1.reserveResource(app_0, priorityMap, rmContainer); res = a.findNodeToUnreserve(csContext.getClusterResource(), node_1, app_0, - priorityMap, capability); + priorityMap, capability, capability); assertFalse(res); } @@ -765,6 +776,7 @@ public void testAssignToQueue() throws Exception { .getMockApplicationAttemptId(0, 0); FiCaSchedulerApp app_0 = new FiCaSchedulerApp(appAttemptId_0, user_0, a, mock(ActiveUsersManager.class), spyRMContext); + rmContext.getRMApps().put(app_0.getApplicationId(), mock(RMApp.class)); a.submitApplicationAttempt(app_0, user_0); @@ -810,7 +822,7 @@ public void testAssignToQueue() throws Exception { // Start testing... // Only AM - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2 * GB, a.getUsedResources().getMemory()); assertEquals(2 * GB, app_0.getCurrentConsumption().getMemory()); @@ -821,7 +833,7 @@ public void testAssignToQueue() throws Exception { assertEquals(0 * GB, node_1.getUsedResource().getMemory()); // Only 1 map - simulating reduce - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5 * GB, a.getUsedResources().getMemory()); assertEquals(5 * GB, app_0.getCurrentConsumption().getMemory()); @@ -832,7 +844,7 @@ public void testAssignToQueue() throws Exception { assertEquals(0 * GB, node_1.getUsedResource().getMemory()); // Only 1 map to other node - simulating reduce - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -847,14 +859,15 @@ public void testAssignToQueue() throws Exception { // absoluteMaxCapacity Resource capability = Resources.createResource(32 * GB, 0); boolean res = - a.canAssignToThisQueue(clusterResource, capability, - CommonNodeLabelsManager.EMPTY_STRING_SET, app_0, true); + a.canAssignToThisQueue(clusterResource, + CommonNodeLabelsManager.EMPTY_STRING_SET, new ResourceLimits( + clusterResource), capability, Resources.none()); assertFalse(res); // now add in reservations and make sure it continues if config set // allocate to queue so that the potential new capacity is greater then // absoluteMaxCapacity - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -867,14 +880,17 @@ public void testAssignToQueue() throws Exception { capability = Resources.createResource(5 * GB, 0); res = - a.canAssignToThisQueue(clusterResource, capability, - CommonNodeLabelsManager.EMPTY_STRING_SET, app_0, true); + a.canAssignToThisQueue(clusterResource, + CommonNodeLabelsManager.EMPTY_STRING_SET, new ResourceLimits( + clusterResource), capability, Resources + .createResource(5 * GB)); assertTrue(res); // tell to not check reservations res = - a.canAssignToThisQueue(clusterResource, capability, - CommonNodeLabelsManager.EMPTY_STRING_SET, app_0, false); + a.canAssignToThisQueue(clusterResource, + CommonNodeLabelsManager.EMPTY_STRING_SET, new ResourceLimits( + clusterResource), capability, Resources.none()); assertFalse(res); refreshQueuesTurnOffReservationsContLook(a, csConf); @@ -882,13 +898,16 @@ public void testAssignToQueue() throws Exception { // should return false no matter what checkReservations is passed // in since feature is off res = - a.canAssignToThisQueue(clusterResource, capability, - CommonNodeLabelsManager.EMPTY_STRING_SET, app_0, false); + a.canAssignToThisQueue(clusterResource, + CommonNodeLabelsManager.EMPTY_STRING_SET, new ResourceLimits( + clusterResource), capability, Resources.none()); assertFalse(res); res = - a.canAssignToThisQueue(clusterResource, capability, - CommonNodeLabelsManager.EMPTY_STRING_SET, app_0, true); + a.canAssignToThisQueue(clusterResource, + CommonNodeLabelsManager.EMPTY_STRING_SET, new ResourceLimits( + clusterResource), capability, Resources + .createResource(5 * GB)); assertFalse(res); } @@ -943,7 +962,7 @@ public void testAssignToUser() throws Exception { .getMockApplicationAttemptId(0, 0); FiCaSchedulerApp app_0 = new FiCaSchedulerApp(appAttemptId_0, user_0, a, mock(ActiveUsersManager.class), spyRMContext); - + rmContext.getRMApps().put(app_0.getApplicationId(), mock(RMApp.class)); a.submitApplicationAttempt(app_0, user_0); final ApplicationAttemptId appAttemptId_1 = TestUtils @@ -979,16 +998,16 @@ public void testAssignToUser() throws Exception { app_0.updateResourceRequests(Collections.singletonList(TestUtils .createResourceRequest(ResourceRequest.ANY, 2 * GB, 1, true, priorityAM, recordFactory))); - app_0.updateResourceRequests(Collections.singletonList(TestUtils - .createResourceRequest(ResourceRequest.ANY, 5 * GB, 2, true, - priorityReduce, recordFactory))); app_0.updateResourceRequests(Collections.singletonList(TestUtils .createResourceRequest(ResourceRequest.ANY, 3 * GB, 2, true, priorityMap, recordFactory))); + app_0.updateResourceRequests(Collections.singletonList(TestUtils + .createResourceRequest(ResourceRequest.ANY, 5 * GB, 2, true, + priorityReduce, recordFactory))); // Start testing... // Only AM - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2 * GB, a.getUsedResources().getMemory()); assertEquals(2 * GB, app_0.getCurrentConsumption().getMemory()); @@ -999,7 +1018,7 @@ public void testAssignToUser() throws Exception { assertEquals(0 * GB, node_1.getUsedResource().getMemory()); // Only 1 map - simulating reduce - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5 * GB, a.getUsedResources().getMemory()); assertEquals(5 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1010,7 +1029,7 @@ public void testAssignToUser() throws Exception { assertEquals(0 * GB, node_1.getUsedResource().getMemory()); // Only 1 map to other node - simulating reduce - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1024,7 +1043,7 @@ public void testAssignToUser() throws Exception { // now add in reservations and make sure it continues if config set // allocate to queue so that the potential new capacity is greater then // absoluteMaxCapacity - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1039,19 +1058,19 @@ public void testAssignToUser() throws Exception { // set limit so subtrace reservations it can continue Resource limit = Resources.createResource(12 * GB, 0); - boolean res = a.assignToUser(clusterResource, user_0, limit, app_0, + boolean res = a.canAssignToUser(clusterResource, user_0, limit, app_0, true, null); assertTrue(res); // tell it not to check for reservations and should fail as already over // limit - res = a.assignToUser(clusterResource, user_0, limit, app_0, false, null); + res = a.canAssignToUser(clusterResource, user_0, limit, app_0, false, null); assertFalse(res); refreshQueuesTurnOffReservationsContLook(a, csConf); // should now return false since feature off - res = a.assignToUser(clusterResource, user_0, limit, app_0, true, null); + res = a.canAssignToUser(clusterResource, user_0, limit, app_0, true, null); assertFalse(res); } @@ -1073,6 +1092,7 @@ public void testReservationsNoneAvailable() throws Exception { .getMockApplicationAttemptId(0, 0); FiCaSchedulerApp app_0 = new FiCaSchedulerApp(appAttemptId_0, user_0, a, mock(ActiveUsersManager.class), spyRMContext); + rmContext.getRMApps().put(app_0.getApplicationId(), mock(RMApp.class)); a.submitApplicationAttempt(app_0, user_0); @@ -1110,19 +1130,19 @@ public void testReservationsNoneAvailable() throws Exception { app_0.updateResourceRequests(Collections.singletonList(TestUtils .createResourceRequest(ResourceRequest.ANY, 2 * GB, 1, true, priorityAM, recordFactory))); - app_0.updateResourceRequests(Collections.singletonList(TestUtils - .createResourceRequest(ResourceRequest.ANY, 5 * GB, 1, true, - priorityReduce, recordFactory))); app_0.updateResourceRequests(Collections.singletonList(TestUtils .createResourceRequest(ResourceRequest.ANY, 3 * GB, 2, true, priorityMap, recordFactory))); + app_0.updateResourceRequests(Collections.singletonList(TestUtils + .createResourceRequest(ResourceRequest.ANY, 5 * GB, 1, true, + priorityReduce, recordFactory))); app_0.updateResourceRequests(Collections.singletonList(TestUtils .createResourceRequest(ResourceRequest.ANY, 8 * GB, 2, true, priorityLast, recordFactory))); // Start testing... // Only AM - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(2 * GB, a.getUsedResources().getMemory()); assertEquals(2 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1134,7 +1154,7 @@ public void testReservationsNoneAvailable() throws Exception { assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // Only 1 map - simulating reduce - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(5 * GB, a.getUsedResources().getMemory()); assertEquals(5 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1146,7 +1166,7 @@ public void testReservationsNoneAvailable() throws Exception { assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // Only 1 map to other node - simulating reduce - a.assignContainers(clusterResource, node_1, false, + a.assignContainers(clusterResource, node_1, new ResourceLimits(clusterResource)); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1158,38 +1178,41 @@ public void testReservationsNoneAvailable() throws Exception { assertEquals(3 * GB, node_1.getUsedResource().getMemory()); assertEquals(0 * GB, node_2.getUsedResource().getMemory()); - // try to assign reducer (5G on node 0), but tell it - // it has to unreserve. No room to allocate and shouldn't reserve - // since nothing currently reserved. - a.assignContainers(clusterResource, node_0, true, - new ResourceLimits(clusterResource)); + // try to assign reducer (5G on node 0), but tell it's resource limits < + // used (8G) + required (5G). It will not reserved since it has to unreserve + // some resource. Even with continous reservation looking, we don't allow + // unreserve resource to reserve container. + a.assignContainers(clusterResource, node_0, + new ResourceLimits(Resources.createResource(10 * GB))); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); assertEquals(0 * GB, a.getMetrics().getReservedMB()); assertEquals(8 * GB, a.getMetrics().getAllocatedMB()); assertEquals(16 * GB, a.getMetrics().getAvailableMB()); - assertEquals(16 * GB, app_0.getHeadroom().getMemory()); + // app_0's headroom = limit (10G) - used (8G) = 2G + assertEquals(2 * GB, app_0.getHeadroom().getMemory()); assertEquals(5 * GB, node_0.getUsedResource().getMemory()); assertEquals(3 * GB, node_1.getUsedResource().getMemory()); assertEquals(0 * GB, node_2.getUsedResource().getMemory()); - // try to assign reducer (5G on node 2), but tell it - // it has to unreserve. Has room but shouldn't reserve - // since nothing currently reserved. - a.assignContainers(clusterResource, node_2, true, - new ResourceLimits(clusterResource)); + // try to assign reducer (5G on node 0), but tell it's resource limits < + // used (8G) + required (5G). It will not reserved since it has to unreserve + // some resource. Unfortunately, there's nothing to unreserve. + a.assignContainers(clusterResource, node_2, + new ResourceLimits(Resources.createResource(10 * GB))); assertEquals(8 * GB, a.getUsedResources().getMemory()); assertEquals(8 * GB, app_0.getCurrentConsumption().getMemory()); assertEquals(0 * GB, a.getMetrics().getReservedMB()); assertEquals(8 * GB, a.getMetrics().getAllocatedMB()); assertEquals(16 * GB, a.getMetrics().getAvailableMB()); - assertEquals(16 * GB, app_0.getHeadroom().getMemory()); + // app_0's headroom = limit (10G) - used (8G) = 2G + assertEquals(2 * GB, app_0.getHeadroom().getMemory()); assertEquals(5 * GB, node_0.getUsedResource().getMemory()); assertEquals(3 * GB, node_1.getUsedResource().getMemory()); assertEquals(0 * GB, node_2.getUsedResource().getMemory()); // let it assign 5G to node_2 - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); assertEquals(13 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1202,7 +1225,7 @@ public void testReservationsNoneAvailable() throws Exception { assertEquals(5 * GB, node_2.getUsedResource().getMemory()); // reserve 8G node_0 - a.assignContainers(clusterResource, node_0, false, + a.assignContainers(clusterResource, node_0, new ResourceLimits(clusterResource)); assertEquals(21 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); @@ -1217,7 +1240,7 @@ public void testReservationsNoneAvailable() throws Exception { // try to assign (8G on node 2). No room to allocate, // continued to try due to having reservation above, // but hits queue limits so can't reserve anymore. - a.assignContainers(clusterResource, node_2, false, + a.assignContainers(clusterResource, node_2, new ResourceLimits(clusterResource)); assertEquals(21 * GB, a.getUsedResources().getMemory()); assertEquals(13 * GB, app_0.getCurrentConsumption().getMemory()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java index 9e352a70ccbdf..62135b91df4d7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java @@ -60,6 +60,8 @@ import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; +import com.google.common.collect.Sets; + public class TestUtils { private static final Log LOG = LogFactory.getLog(TestUtils.class); @@ -216,4 +218,131 @@ public static Container getMockContainer( when(container.getPriority()).thenReturn(priority); return container; } + + @SuppressWarnings("unchecked") + private static Set toSet(E... elements) { + Set set = Sets.newHashSet(elements); + return set; + } + + /** + * Get a queue structure: + * 
        +   *             Root
+   *            /  |  \
+   *           a   b   c
+   *           |   |   |
+   *           a1  b1  c1
+   *          (x)  (y)
+   *
        + */ + public static Configuration getConfigurationWithQueueLabels(Configuration config) { + CapacitySchedulerConfiguration conf = + new CapacitySchedulerConfiguration(config); + + // Define top-level queues + conf.setQueues(CapacitySchedulerConfiguration.ROOT, new String[] {"a", "b", "c"}); + conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "x", 100); + conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "y", 100); + + final String A = CapacitySchedulerConfiguration.ROOT + ".a"; + conf.setCapacity(A, 10); + conf.setMaximumCapacity(A, 15); + conf.setAccessibleNodeLabels(A, toSet("x")); + conf.setCapacityByLabel(A, "x", 100); + + final String B = CapacitySchedulerConfiguration.ROOT + ".b"; + conf.setCapacity(B, 20); + conf.setAccessibleNodeLabels(B, toSet("y")); + conf.setCapacityByLabel(B, "y", 100); + + final String C = CapacitySchedulerConfiguration.ROOT + ".c"; + conf.setCapacity(C, 70); + conf.setMaximumCapacity(C, 70); + conf.setAccessibleNodeLabels(C, RMNodeLabelsManager.EMPTY_STRING_SET); + + // Define 2nd-level queues + final String A1 = A + ".a1"; + conf.setQueues(A, new String[] {"a1"}); + conf.setCapacity(A1, 100); + conf.setMaximumCapacity(A1, 100); + conf.setCapacityByLabel(A1, "x", 100); + + final String B1 = B + ".b1"; + conf.setQueues(B, new String[] {"b1"}); + conf.setCapacity(B1, 100); + conf.setMaximumCapacity(B1, 100); + conf.setCapacityByLabel(B1, "y", 100); + + final String C1 = C + ".c1"; + conf.setQueues(C, new String[] {"c1"}); + conf.setCapacity(C1, 100); + conf.setMaximumCapacity(C1, 100); + + return conf; + } + + public static Configuration getComplexConfigurationWithQueueLabels( + Configuration config) { + CapacitySchedulerConfiguration conf = + new CapacitySchedulerConfiguration(config); + + // Define top-level queues + conf.setQueues(CapacitySchedulerConfiguration.ROOT, new String[] {"a", "b"}); + conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "x", 100); + conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "y", 100); + conf.setCapacityByLabel(CapacitySchedulerConfiguration.ROOT, "z", 100); + + final String A = CapacitySchedulerConfiguration.ROOT + ".a"; + conf.setCapacity(A, 10); + conf.setMaximumCapacity(A, 10); + conf.setAccessibleNodeLabels(A, toSet("x", "y")); + conf.setCapacityByLabel(A, "x", 100); + conf.setCapacityByLabel(A, "y", 50); + + final String B = CapacitySchedulerConfiguration.ROOT + ".b"; + conf.setCapacity(B, 90); + conf.setMaximumCapacity(B, 100); + conf.setAccessibleNodeLabels(B, toSet("y", "z")); + conf.setCapacityByLabel(B, "y", 50); + conf.setCapacityByLabel(B, "z", 100); + + // Define 2nd-level queues + final String A1 = A + ".a1"; + conf.setQueues(A, new String[] {"a1"}); + conf.setCapacity(A1, 100); + conf.setMaximumCapacity(A1, 100); + conf.setAccessibleNodeLabels(A1, toSet("x", "y")); + conf.setDefaultNodeLabelExpression(A1, "x"); + conf.setCapacityByLabel(A1, "x", 100); + conf.setCapacityByLabel(A1, "y", 100); + + conf.setQueues(B, new String[] {"b1", "b2"}); + final String B1 = B + ".b1"; + conf.setCapacity(B1, 50); + conf.setMaximumCapacity(B1, 50); + conf.setAccessibleNodeLabels(B1, RMNodeLabelsManager.EMPTY_STRING_SET); + + final String B2 = B + ".b2"; + conf.setCapacity(B2, 50); + conf.setMaximumCapacity(B2, 50); + conf.setAccessibleNodeLabels(B2, toSet("y", "z")); + conf.setCapacityByLabel(B2, "y", 100); + conf.setCapacityByLabel(B2, "z", 100); + + return conf; + } + + public static Configuration getConfigurationWithDefaultQueueLabels( + Configuration config) { + final String A = CapacitySchedulerConfiguration.ROOT + ".a"; + final String B = CapacitySchedulerConfiguration.ROOT + ".b"; + + CapacitySchedulerConfiguration conf = + (CapacitySchedulerConfiguration) getConfigurationWithQueueLabels(config); + new CapacitySchedulerConfiguration(config); + conf.setDefaultNodeLabelExpression(A, "x"); + conf.setDefaultNodeLabelExpression(B, "y"); + return conf; + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java index 8656175a7f483..0a9c3895565c8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java @@ -40,6 +40,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAddedSchedulerEvent; @@ -157,7 +158,7 @@ protected ApplicationAttemptId createSchedulingRequest( ResourceRequest request = createResourceRequest(memory, vcores, ResourceRequest.ANY, priority, numContainers, true); ask.add(request); - scheduler.allocate(id, ask, new ArrayList(), null, null); + RMApp rmApp = mock(RMApp.class); RMAppAttempt rmAppAttempt = mock(RMAppAttempt.class); when(rmApp.getCurrentAppAttempt()).thenReturn(rmAppAttempt); @@ -165,6 +166,8 @@ protected ApplicationAttemptId createSchedulingRequest( new RMAppAttemptMetrics(id, resourceManager.getRMContext())); resourceManager.getRMContext().getRMApps() .put(id.getApplicationId(), rmApp); + + scheduler.allocate(id, ask, new ArrayList(), null, null); return id; } @@ -178,7 +181,7 @@ protected ApplicationAttemptId createSchedulingRequest(String queueId, if (scheduler.getSchedulerApplications().containsKey(id.getApplicationId())) { scheduler.addApplicationAttempt(id, false, false); } - scheduler.allocate(id, ask, new ArrayList(), null, null); + RMApp rmApp = mock(RMApp.class); RMAppAttempt rmAppAttempt = mock(RMAppAttempt.class); when(rmApp.getCurrentAppAttempt()).thenReturn(rmAppAttempt); @@ -186,6 +189,8 @@ protected ApplicationAttemptId createSchedulingRequest(String queueId, new RMAppAttemptMetrics(id,resourceManager.getRMContext())); resourceManager.getRMContext().getRMApps() .put(id.getApplicationId(), rmApp); + + scheduler.allocate(id, ask, new ArrayList(), null, null); return id; } @@ -225,4 +230,17 @@ protected void createApplicationWithAMResource(ApplicationAttemptId attId, new AppAttemptAddedSchedulerEvent(attId, false); scheduler.handle(attempAddedEvent); } + + protected RMApp createMockRMApp(ApplicationAttemptId attemptId) { + RMApp app = mock(RMAppImpl.class); + when(app.getApplicationId()).thenReturn(attemptId.getApplicationId()); + RMAppAttemptImpl attempt = mock(RMAppAttemptImpl.class); + when(attempt.getAppAttemptId()).thenReturn(attemptId); + RMAppAttemptMetrics attemptMetric = mock(RMAppAttemptMetrics.class); + when(attempt.getRMAppAttemptMetrics()).thenReturn(attemptMetric); + when(app.getCurrentAppAttempt()).thenReturn(attempt); + resourceManager.getRMContext().getRMApps() + .put(attemptId.getApplicationId(), app); + return app; + } } \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java index 3c166a5edcb59..b09573cd8cd5e 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java @@ -550,7 +550,30 @@ public void testQueueNameContainingPeriods() throws Exception { allocLoader.setReloadListener(confHolder); allocLoader.reloadAllocations(); } - + + /** + * Verify that you can't have the queue name with whitespace only in the + * allocations file. + */ + @Test (expected = AllocationConfigurationException.class) + public void testQueueNameContainingOnlyWhitespace() throws Exception { + Configuration conf = new Configuration(); + conf.set(FairSchedulerConfiguration.ALLOCATION_FILE, ALLOC_FILE); + + PrintWriter out = new PrintWriter(new FileWriter(ALLOC_FILE)); + out.println(""); + out.println(""); + out.println(""); + out.println(""); + out.println(""); + out.close(); + + AllocationFileLoaderService allocLoader = new AllocationFileLoaderService(); + allocLoader.init(conf); + ReloadListener confHolder = new ReloadListener(); + allocLoader.setReloadListener(confHolder); + allocLoader.reloadAllocations(); + } @Test public void testReservableQueue() throws Exception { diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java index c7a0e556f8d4c..a72e3938f4a8f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java @@ -27,6 +27,7 @@ import org.apache.hadoop.yarn.api.records.ResourceRequest; import org.apache.hadoop.yarn.server.resourcemanager.MockNodes; import org.apache.hadoop.yarn.server.resourcemanager.MockRM; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeAddedSchedulerEvent; @@ -96,6 +97,8 @@ public void testSchedulingDelay() throws InterruptedException { // and ANY requests ApplicationAttemptId appAttemptId = createAppAttemptId(this.APP_ID++, this.ATTEMPT_ID++); + createMockRMApp(appAttemptId); + scheduler.addApplication(appAttemptId.getApplicationId(), "queue11", "user11", false); scheduler.addApplicationAttempt(appAttemptId, false, false); List ask = new ArrayList<>(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java index c29dbfc149a34..ff215cdfbed12 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java @@ -117,8 +117,7 @@ public class TestFairScheduler extends FairSchedulerTestBase { public void setUp() throws IOException { scheduler = new FairScheduler(); conf = createConfiguration(); - resourceManager = new ResourceManager(); - resourceManager.init(conf); + resourceManager = new MockRM(conf); // TODO: This test should really be using MockRM. For now starting stuff // that is needed at a bare minimum. @@ -1379,12 +1378,16 @@ public void testQueueDemandCalculation() throws Exception { scheduler.reinitialize(conf, resourceManager.getRMContext()); ApplicationAttemptId id11 = createAppAttemptId(1, 1); + createMockRMApp(id11); scheduler.addApplication(id11.getApplicationId(), "root.queue1", "user1", false); scheduler.addApplicationAttempt(id11, false, false); ApplicationAttemptId id21 = createAppAttemptId(2, 1); + createMockRMApp(id21); scheduler.addApplication(id21.getApplicationId(), "root.queue2", "user1", false); scheduler.addApplicationAttempt(id21, false, false); ApplicationAttemptId id22 = createAppAttemptId(2, 2); + createMockRMApp(id22); + scheduler.addApplication(id22.getApplicationId(), "root.queue2", "user1", false); scheduler.addApplicationAttempt(id22, false, false); @@ -2288,7 +2291,315 @@ public void testUserMaxRunningApps() throws Exception { // Request should be fulfilled assertEquals(2, scheduler.getSchedulerApp(attId1).getLiveContainers().size()); } - + + @Test (timeout = 5000) + public void testIncreaseQueueMaxRunningAppsOnTheFly() throws Exception { + String allocBefore = "" + + "" + + "" + + "" + + "1" + + "" + + "" + + ""; + + String allocAfter = "" + + "" + + "" + + "" + + "3" + + "" + + "" + + ""; + + testIncreaseQueueSettingOnTheFlyInternal(allocBefore, allocAfter); + } + + @Test (timeout = 5000) + public void testIncreaseUserMaxRunningAppsOnTheFly() throws Exception { + String allocBefore = ""+ + ""+ + ""+ + ""+ + "10"+ + ""+ + ""+ + ""+ + "1"+ + ""+ + ""; + + String allocAfter = ""+ + ""+ + ""+ + ""+ + "10"+ + ""+ + ""+ + ""+ + "3"+ + ""+ + ""; + + testIncreaseQueueSettingOnTheFlyInternal(allocBefore, allocAfter); + } + + private void testIncreaseQueueSettingOnTheFlyInternal(String allocBefore, + String allocAfter) throws Exception { + // Set max running apps + conf.set(FairSchedulerConfiguration.ALLOCATION_FILE, ALLOC_FILE); + + PrintWriter out = new PrintWriter(new FileWriter(ALLOC_FILE)); + out.println(allocBefore); + out.close(); + + scheduler.init(conf); + scheduler.start(); + scheduler.reinitialize(conf, resourceManager.getRMContext()); + + // Add a node + RMNode node1 = + MockNodes + .newNodeInfo(1, Resources.createResource(8192, 8), 1, "127.0.0.1"); + NodeAddedSchedulerEvent nodeEvent1 = new NodeAddedSchedulerEvent(node1); + scheduler.handle(nodeEvent1); + + // Request for app 1 + ApplicationAttemptId attId1 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + NodeUpdateSchedulerEvent updateEvent = new NodeUpdateSchedulerEvent(node1); + scheduler.handle(updateEvent); + + // App 1 should be running + assertEquals(1, scheduler.getSchedulerApp(attId1).getLiveContainers().size()); + + ApplicationAttemptId attId2 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + scheduler.handle(updateEvent); + + ApplicationAttemptId attId3 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + scheduler.handle(updateEvent); + + ApplicationAttemptId attId4 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 2 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId2).getLiveContainers().size()); + // App 3 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId3).getLiveContainers().size()); + // App 4 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + + out = new PrintWriter(new FileWriter(ALLOC_FILE)); + out.println(allocAfter); + out.close(); + scheduler.reinitialize(conf, resourceManager.getRMContext()); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 2 should be running + assertEquals(1, scheduler.getSchedulerApp(attId2).getLiveContainers().size()); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 3 should be running + assertEquals(1, scheduler.getSchedulerApp(attId3).getLiveContainers().size()); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 4 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + + // Now remove app 1 + AppAttemptRemovedSchedulerEvent appRemovedEvent1 = new AppAttemptRemovedSchedulerEvent( + attId1, RMAppAttemptState.FINISHED, false); + + scheduler.handle(appRemovedEvent1); + scheduler.update(); + scheduler.handle(updateEvent); + + // App 4 should be running + assertEquals(1, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + } + + @Test (timeout = 5000) + public void testDecreaseQueueMaxRunningAppsOnTheFly() throws Exception { + String allocBefore = "" + + "" + + "" + + "" + + "3" + + "" + + "" + + ""; + + String allocAfter = "" + + "" + + "" + + "" + + "1" + + "" + + "" + + ""; + + testDecreaseQueueSettingOnTheFlyInternal(allocBefore, allocAfter); + } + + @Test (timeout = 5000) + public void testDecreaseUserMaxRunningAppsOnTheFly() throws Exception { + String allocBefore = ""+ + ""+ + ""+ + ""+ + "10"+ + ""+ + ""+ + ""+ + "3"+ + ""+ + ""; + + String allocAfter = ""+ + ""+ + ""+ + ""+ + "10"+ + ""+ + ""+ + ""+ + "1"+ + ""+ + ""; + + testDecreaseQueueSettingOnTheFlyInternal(allocBefore, allocAfter); + } + + private void testDecreaseQueueSettingOnTheFlyInternal(String allocBefore, + String allocAfter) throws Exception { + // Set max running apps + conf.set(FairSchedulerConfiguration.ALLOCATION_FILE, ALLOC_FILE); + + PrintWriter out = new PrintWriter(new FileWriter(ALLOC_FILE)); + out.println(allocBefore); + out.close(); + + scheduler.init(conf); + scheduler.start(); + scheduler.reinitialize(conf, resourceManager.getRMContext()); + + // Add a node + RMNode node1 = + MockNodes + .newNodeInfo(1, Resources.createResource(8192, 8), 1, "127.0.0.1"); + NodeAddedSchedulerEvent nodeEvent1 = new NodeAddedSchedulerEvent(node1); + scheduler.handle(nodeEvent1); + + // Request for app 1 + ApplicationAttemptId attId1 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + NodeUpdateSchedulerEvent updateEvent = new NodeUpdateSchedulerEvent(node1); + scheduler.handle(updateEvent); + + // App 1 should be running + assertEquals(1, scheduler.getSchedulerApp(attId1).getLiveContainers().size()); + + ApplicationAttemptId attId2 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + scheduler.handle(updateEvent); + + ApplicationAttemptId attId3 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + scheduler.handle(updateEvent); + + ApplicationAttemptId attId4 = createSchedulingRequest(1024, "queue1", + "user1", 1); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 2 should be running + assertEquals(1, scheduler.getSchedulerApp(attId2).getLiveContainers().size()); + // App 3 should be running + assertEquals(1, scheduler.getSchedulerApp(attId3).getLiveContainers().size()); + // App 4 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + + out = new PrintWriter(new FileWriter(ALLOC_FILE)); + out.println(allocAfter); + out.close(); + scheduler.reinitialize(conf, resourceManager.getRMContext()); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 2 should still be running + assertEquals(1, scheduler.getSchedulerApp(attId2).getLiveContainers().size()); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 3 should still be running + assertEquals(1, scheduler.getSchedulerApp(attId3).getLiveContainers().size()); + + scheduler.update(); + scheduler.handle(updateEvent); + + // App 4 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + + // Now remove app 1 + AppAttemptRemovedSchedulerEvent appRemovedEvent1 = new AppAttemptRemovedSchedulerEvent( + attId1, RMAppAttemptState.FINISHED, false); + + scheduler.handle(appRemovedEvent1); + scheduler.update(); + scheduler.handle(updateEvent); + + // App 4 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + + // Now remove app 2 + appRemovedEvent1 = new AppAttemptRemovedSchedulerEvent( + attId2, RMAppAttemptState.FINISHED, false); + + scheduler.handle(appRemovedEvent1); + scheduler.update(); + scheduler.handle(updateEvent); + + // App 4 should not be running + assertEquals(0, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + + // Now remove app 3 + appRemovedEvent1 = new AppAttemptRemovedSchedulerEvent( + attId3, RMAppAttemptState.FINISHED, false); + + scheduler.handle(appRemovedEvent1); + scheduler.update(); + scheduler.handle(updateEvent); + + // App 4 should be running now + assertEquals(1, scheduler.getSchedulerApp(attId4).getLiveContainers().size()); + } + @Test (timeout = 5000) public void testReservationWhileMultiplePriorities() throws IOException { scheduler.init(conf); @@ -2409,9 +2720,13 @@ public void testMultipleNodesSingleRackRequest() throws Exception { NodeAddedSchedulerEvent nodeEvent2 = new NodeAddedSchedulerEvent(node2); scheduler.handle(nodeEvent2); - ApplicationAttemptId appId = createAppAttemptId(this.APP_ID++, this.ATTEMPT_ID++); - scheduler.addApplication(appId.getApplicationId(), "queue1", "user1", false); - scheduler.addApplicationAttempt(appId, false, false); + ApplicationAttemptId attemptId = + createAppAttemptId(this.APP_ID++, this.ATTEMPT_ID++); + createMockRMApp(attemptId); + + scheduler.addApplication(attemptId.getApplicationId(), "queue1", "user1", + false); + scheduler.addApplicationAttempt(attemptId, false, false); // 1 request with 2 nodes on the same rack. another request with 1 node on // a different rack @@ -2423,21 +2738,24 @@ public void testMultipleNodesSingleRackRequest() throws Exception { asks.add(createResourceRequest(1024, node3.getRackName(), 1, 1, true)); asks.add(createResourceRequest(1024, ResourceRequest.ANY, 1, 2, true)); - scheduler.allocate(appId, asks, new ArrayList(), null, null); + scheduler.allocate(attemptId, asks, new ArrayList(), null, + null); // node 1 checks in scheduler.update(); NodeUpdateSchedulerEvent updateEvent1 = new NodeUpdateSchedulerEvent(node1); scheduler.handle(updateEvent1); // should assign node local - assertEquals(1, scheduler.getSchedulerApp(appId).getLiveContainers().size()); + assertEquals(1, scheduler.getSchedulerApp(attemptId).getLiveContainers() + .size()); // node 2 checks in scheduler.update(); NodeUpdateSchedulerEvent updateEvent2 = new NodeUpdateSchedulerEvent(node2); scheduler.handle(updateEvent2); // should assign rack local - assertEquals(2, scheduler.getSchedulerApp(appId).getLiveContainers().size()); + assertEquals(2, scheduler.getSchedulerApp(attemptId).getLiveContainers() + .size()); } @Test (timeout = 5000) @@ -3547,6 +3865,8 @@ public void testContinuousScheduling() throws Exception { // send application request ApplicationAttemptId appAttemptId = createAppAttemptId(this.APP_ID++, this.ATTEMPT_ID++); + createMockRMApp(appAttemptId); + scheduler.addApplication(appAttemptId.getApplicationId(), "queue11", "user11", false); scheduler.addApplicationAttempt(appAttemptId, false, false); List ask = new ArrayList(); @@ -4123,4 +4443,82 @@ public void testPerfMetricsInited() { assertEquals("Incorrect number of perf metrics", 1, collector.getRecords().size()); } + + @Test + public void testQueueNameWithTrailingSpace() throws Exception { + scheduler.init(conf); + scheduler.start(); + scheduler.reinitialize(conf, resourceManager.getRMContext()); + + // only default queue + assertEquals(1, scheduler.getQueueManager().getLeafQueues().size()); + + // submit app with queue name "A" + ApplicationAttemptId appAttemptId1 = createAppAttemptId(1, 1); + AppAddedSchedulerEvent appAddedEvent1 = new AppAddedSchedulerEvent( + appAttemptId1.getApplicationId(), "A", "user1"); + scheduler.handle(appAddedEvent1); + // submission accepted + assertEquals(2, scheduler.getQueueManager().getLeafQueues().size()); + assertNotNull(scheduler.getSchedulerApplications().get(appAttemptId1. + getApplicationId())); + + AppAttemptAddedSchedulerEvent attempAddedEvent = + new AppAttemptAddedSchedulerEvent(appAttemptId1, false); + scheduler.handle(attempAddedEvent); + // That queue should have one app + assertEquals(1, scheduler.getQueueManager().getLeafQueue("A", true) + .getNumRunnableApps()); + assertNotNull(scheduler.getSchedulerApp(appAttemptId1)); + + // submit app with queue name "A " + ApplicationAttemptId appAttemptId2 = createAppAttemptId(2, 1); + AppAddedSchedulerEvent appAddedEvent2 = new AppAddedSchedulerEvent( + appAttemptId2.getApplicationId(), "A ", "user1"); + scheduler.handle(appAddedEvent2); + // submission rejected + assertEquals(2, scheduler.getQueueManager().getLeafQueues().size()); + assertNull(scheduler.getSchedulerApplications().get(appAttemptId2. + getApplicationId())); + assertNull(scheduler.getSchedulerApp(appAttemptId2)); + + // submit app with queue name "B.C" + ApplicationAttemptId appAttemptId3 = createAppAttemptId(3, 1); + AppAddedSchedulerEvent appAddedEvent3 = new AppAddedSchedulerEvent( + appAttemptId3.getApplicationId(), "B.C", "user1"); + scheduler.handle(appAddedEvent3); + // submission accepted + assertEquals(3, scheduler.getQueueManager().getLeafQueues().size()); + assertNotNull(scheduler.getSchedulerApplications().get(appAttemptId3. + getApplicationId())); + + attempAddedEvent = + new AppAttemptAddedSchedulerEvent(appAttemptId3, false); + scheduler.handle(attempAddedEvent); + // That queue should have one app + assertEquals(1, scheduler.getQueueManager().getLeafQueue("B.C", true) + .getNumRunnableApps()); + assertNotNull(scheduler.getSchedulerApp(appAttemptId3)); + } + + @Test + public void testEmptyQueueNameInConfigFile() throws IOException { + conf.set(FairSchedulerConfiguration.ALLOCATION_FILE, ALLOC_FILE); + // set empty queue name + PrintWriter out = new PrintWriter(new FileWriter(ALLOC_FILE)); + out.println(""); + out.println(""); + out.println(""); + out.println(""); + out.println(""); + out.close(); + try { + scheduler.init(conf); + Assert.fail("scheduler init should fail because" + + " empty queue name."); + } catch (Exception e) { + Assert.assertTrue(e.getMessage().contains( + "Failed to initialize FairScheduler")); + } + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java index 311b53182df60..a8741d2f7f10f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java @@ -21,6 +21,7 @@ import java.io.File; import java.io.IOException; +import org.apache.hadoop.yarn.server.resourcemanager.MockRM; import org.junit.Assert; import org.apache.hadoop.conf.Configuration; @@ -48,8 +49,7 @@ public void setUp() throws IOException { // All tests assume only one assignment per node update conf.set(FairSchedulerConfiguration.ASSIGN_MULTIPLE, "false"); - resourceManager = new ResourceManager(); - resourceManager.init(conf); + resourceManager = new MockRM(conf); ((AsyncDispatcher)resourceManager.getRMContext().getDispatcher()).start(); scheduler.init(conf); scheduler.start(); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java index ef0ec7e2d788b..b3ed542b53d47 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java @@ -123,7 +123,18 @@ public void testReloadTurnsLeafToParentWithNoLeaf() { assertTrue(queueManager.getParentQueue("root.queue1", false) .getChildQueues().isEmpty()); } - + + @Test + public void testCheckQueueNodeName() { + assertFalse(queueManager.isQueueNameValid("")); + assertFalse(queueManager.isQueueNameValid(" ")); + assertFalse(queueManager.isQueueNameValid(" a")); + assertFalse(queueManager.isQueueNameValid("a ")); + assertFalse(queueManager.isQueueNameValid(" a ")); + assertTrue(queueManager.isQueueNameValid("a b")); + assertTrue(queueManager.isQueueNameValid("a")); + } + private void updateConfiguredLeafQueues(QueueManager queueMgr, String... confLeafQueues) { AllocationConfiguration allocConf = new AllocationConfiguration(conf); allocConf.configuredQueues.get(FSQueueType.LEAF).addAll(Sets.newHashSet(confLeafQueues)); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java index 3918bf70ba3ca..77eebdf74665f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java @@ -20,12 +20,15 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; import java.io.IOException; import java.lang.reflect.Method; import java.util.ArrayList; import java.util.Collections; +import java.util.HashMap; import java.util.List; import java.util.Map; @@ -33,9 +36,13 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.net.NetworkTopology; +import org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.Container; import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.api.records.ContainerState; +import org.apache.hadoop.yarn.api.records.ContainerStatus; import org.apache.hadoop.yarn.api.records.NodeId; import org.apache.hadoop.yarn.api.records.Priority; import org.apache.hadoop.yarn.api.records.QueueInfo; @@ -46,9 +53,13 @@ import org.apache.hadoop.yarn.event.AsyncDispatcher; import org.apache.hadoop.yarn.event.InlineDispatcher; import org.apache.hadoop.yarn.exceptions.YarnException; +import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; import org.apache.hadoop.yarn.factories.RecordFactory; import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider; +import org.apache.hadoop.yarn.server.api.protocolrecords.UpdateNodeResourceRequest; import org.apache.hadoop.yarn.server.resourcemanager.Application; +import org.apache.hadoop.yarn.server.resourcemanager.MockAM; +import org.apache.hadoop.yarn.server.resourcemanager.MockNM; import org.apache.hadoop.yarn.server.resourcemanager.MockNodes; import org.apache.hadoop.yarn.server.resourcemanager.MockRM; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; @@ -57,18 +68,27 @@ import org.apache.hadoop.yarn.server.resourcemanager.Task; import org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter; import org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttempt; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl; +import org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics; import org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.Allocation; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNodeReport; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAddedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAttemptAddedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeAddedSchedulerEvent; +import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeRemovedSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeResourceUpdateSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.NodeUpdateSchedulerEvent; import org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEvent; @@ -76,6 +96,9 @@ import org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager; import org.apache.hadoop.yarn.server.utils.BuilderUtils; import org.apache.hadoop.yarn.util.resource.Resources; +import org.apache.log4j.Level; +import org.apache.log4j.LogManager; +import org.apache.log4j.Logger; import org.junit.After; import org.junit.Assert; import org.junit.Before; @@ -86,17 +109,17 @@ public class TestFifoScheduler { private final int GB = 1024; private ResourceManager resourceManager = null; + private static Configuration conf; private static final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null); @Before public void setUp() throws Exception { - resourceManager = new ResourceManager(); - Configuration conf = new Configuration(); - conf.setClass(YarnConfiguration.RM_SCHEDULER, + conf = new Configuration(); + conf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, ResourceScheduler.class); - resourceManager.init(conf); + resourceManager = new MockRM(conf); } @After @@ -212,9 +235,12 @@ public void testNodeLocalAssignment() throws Exception { int _appAttemptId = 1; ApplicationAttemptId appAttemptId = createAppAttemptId(_appId, _appAttemptId); + + createMockRMApp(appAttemptId, rmContext); + AppAddedSchedulerEvent appEvent = new AppAddedSchedulerEvent(appAttemptId.getApplicationId(), "queue1", - "user1"); + "user1"); scheduler.handle(appEvent); AppAttemptAddedSchedulerEvent attemptEvent = new AppAttemptAddedSchedulerEvent(appAttemptId, false); @@ -310,6 +336,8 @@ public Map getNodes(){ int _appAttemptId = 1; ApplicationAttemptId appAttemptId = createAppAttemptId(_appId, _appAttemptId); + createMockRMApp(appAttemptId, rmContext); + AppAddedSchedulerEvent appEvent = new AppAddedSchedulerEvent(appAttemptId.getApplicationId(), "queue1", "user1"); @@ -557,43 +585,6 @@ public void testFifoScheduler() throws Exception { LOG.info("--- END: testFifoScheduler ---"); } - @Test - public void testBlackListNodes() throws Exception { - Configuration conf = new Configuration(); - conf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, - ResourceScheduler.class); - MockRM rm = new MockRM(conf); - rm.start(); - FifoScheduler fs = (FifoScheduler) rm.getResourceScheduler(); - - String host = "127.0.0.1"; - RMNode node = - MockNodes.newNodeInfo(0, MockNodes.newResource(4 * GB), 1, host); - fs.handle(new NodeAddedSchedulerEvent(node)); - - ApplicationId appId = BuilderUtils.newApplicationId(100, 1); - ApplicationAttemptId appAttemptId = BuilderUtils.newApplicationAttemptId( - appId, 1); - SchedulerEvent appEvent = - new AppAddedSchedulerEvent(appId, "default", - "user"); - fs.handle(appEvent); - SchedulerEvent attemptEvent = - new AppAttemptAddedSchedulerEvent(appAttemptId, false); - fs.handle(attemptEvent); - - // Verify the blacklist can be updated independent of requesting containers - fs.allocate(appAttemptId, Collections.emptyList(), - Collections.emptyList(), - Collections.singletonList(host), null); - Assert.assertTrue(fs.getApplicationAttempt(appAttemptId).isBlacklisted(host)); - fs.allocate(appAttemptId, Collections.emptyList(), - Collections.emptyList(), null, - Collections.singletonList(host)); - Assert.assertFalse(fs.getApplicationAttempt(appAttemptId).isBlacklisted(host)); - rm.stop(); - } - @Test public void testGetAppsInQueue() throws Exception { Application application_0 = new Application("user_0", resourceManager); @@ -626,6 +617,542 @@ public void testAddAndRemoveAppFromFiFoScheduler() throws Exception { fs.getSchedulerApplications(), fs, "queue"); } + @Test(timeout = 30000) + public void testConfValidation() throws Exception { + FifoScheduler scheduler = new FifoScheduler(); + Configuration conf = new YarnConfiguration(); + conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 2048); + conf.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 1024); + try { + scheduler.serviceInit(conf); + fail("Exception is expected because the min memory allocation is" + + " larger than the max memory allocation."); + } catch (YarnRuntimeException e) { + // Exception is expected. + assertTrue("The thrown exception is not the expected one.", e + .getMessage().startsWith("Invalid resource scheduler memory")); + } + } + + @Test(timeout = 60000) + public void testAllocateContainerOnNodeWithoutOffSwitchSpecified() + throws Exception { + Logger rootLogger = LogManager.getRootLogger(); + rootLogger.setLevel(Level.DEBUG); + + MockRM rm = new MockRM(conf); + rm.start(); + MockNM nm1 = rm.registerNode("127.0.0.1:1234", 6 * GB); + + RMApp app1 = rm.submitApp(2048); + // kick the scheduling, 2 GB given to AM1, remaining 4GB on nm1 + nm1.nodeHeartbeat(true); + RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); + MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); + am1.registerAppAttempt(); + + // add request for containers + List requests = new ArrayList(); + requests.add(am1.createResourceReq("127.0.0.1", 1 * GB, 1, 1)); + requests.add(am1.createResourceReq("/default-rack", 1 * GB, 1, 1)); + am1.allocate(requests, null); // send the request + + try { + // kick the schedule + nm1.nodeHeartbeat(true); + } catch (NullPointerException e) { + Assert.fail("NPE when allocating container on node but " + + "forget to set off-switch request should be handled"); + } + rm.stop(); + } + + @Test(timeout = 60000) + public void testFifoScheduling() throws Exception { + Logger rootLogger = LogManager.getRootLogger(); + rootLogger.setLevel(Level.DEBUG); + MockRM rm = new MockRM(conf); + rm.start(); + MockNM nm1 = rm.registerNode("127.0.0.1:1234", 6 * GB); + MockNM nm2 = rm.registerNode("127.0.0.2:5678", 4 * GB); + + RMApp app1 = rm.submitApp(2048); + // kick the scheduling, 2 GB given to AM1, remaining 4GB on nm1 + nm1.nodeHeartbeat(true); + RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); + MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); + am1.registerAppAttempt(); + SchedulerNodeReport report_nm1 = + rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + Assert.assertEquals(2 * GB, report_nm1.getUsedResource().getMemory()); + + RMApp app2 = rm.submitApp(2048); + // kick the scheduling, 2GB given to AM, remaining 2 GB on nm2 + nm2.nodeHeartbeat(true); + RMAppAttempt attempt2 = app2.getCurrentAppAttempt(); + MockAM am2 = rm.sendAMLaunched(attempt2.getAppAttemptId()); + am2.registerAppAttempt(); + SchedulerNodeReport report_nm2 = + rm.getResourceScheduler().getNodeReport(nm2.getNodeId()); + Assert.assertEquals(2 * GB, report_nm2.getUsedResource().getMemory()); + + // add request for containers + am1.addRequests(new String[] { "127.0.0.1", "127.0.0.2" }, GB, 1, 1); + AllocateResponse alloc1Response = am1.schedule(); // send the request + // add request for containers + am2.addRequests(new String[] { "127.0.0.1", "127.0.0.2" }, 3 * GB, 0, 1); + AllocateResponse alloc2Response = am2.schedule(); // send the request + + // kick the scheduler, 1 GB and 3 GB given to AM1 and AM2, remaining 0 + nm1.nodeHeartbeat(true); + while (alloc1Response.getAllocatedContainers().size() < 1) { + LOG.info("Waiting for containers to be created for app 1..."); + Thread.sleep(1000); + alloc1Response = am1.schedule(); + } + while (alloc2Response.getAllocatedContainers().size() < 1) { + LOG.info("Waiting for containers to be created for app 2..."); + Thread.sleep(1000); + alloc2Response = am2.schedule(); + } + // kick the scheduler, nothing given remaining 2 GB. + nm2.nodeHeartbeat(true); + + List allocated1 = alloc1Response.getAllocatedContainers(); + Assert.assertEquals(1, allocated1.size()); + Assert.assertEquals(1 * GB, allocated1.get(0).getResource().getMemory()); + Assert.assertEquals(nm1.getNodeId(), allocated1.get(0).getNodeId()); + + List allocated2 = alloc2Response.getAllocatedContainers(); + Assert.assertEquals(1, allocated2.size()); + Assert.assertEquals(3 * GB, allocated2.get(0).getResource().getMemory()); + Assert.assertEquals(nm1.getNodeId(), allocated2.get(0).getNodeId()); + + report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + report_nm2 = rm.getResourceScheduler().getNodeReport(nm2.getNodeId()); + Assert.assertEquals(0, report_nm1.getAvailableResource().getMemory()); + Assert.assertEquals(2 * GB, report_nm2.getAvailableResource().getMemory()); + + Assert.assertEquals(6 * GB, report_nm1.getUsedResource().getMemory()); + Assert.assertEquals(2 * GB, report_nm2.getUsedResource().getMemory()); + + Container c1 = allocated1.get(0); + Assert.assertEquals(GB, c1.getResource().getMemory()); + ContainerStatus containerStatus = + BuilderUtils.newContainerStatus(c1.getId(), ContainerState.COMPLETE, + "", 0); + nm1.containerStatus(containerStatus); + int waitCount = 0; + while (attempt1.getJustFinishedContainers().size() < 1 && waitCount++ != 20) { + LOG.info("Waiting for containers to be finished for app 1... Tried " + + waitCount + " times already.."); + Thread.sleep(1000); + } + Assert.assertEquals(1, attempt1.getJustFinishedContainers().size()); + Assert.assertEquals(1, am1.schedule().getCompletedContainersStatuses() + .size()); + report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + Assert.assertEquals(5 * GB, report_nm1.getUsedResource().getMemory()); + + rm.stop(); + } + + @Test(timeout = 60000) + public void testNodeUpdateBeforeAppAttemptInit() throws Exception { + FifoScheduler scheduler = new FifoScheduler(); + MockRM rm = new MockRM(conf); + scheduler.setRMContext(rm.getRMContext()); + scheduler.init(conf); + scheduler.start(); + scheduler.reinitialize(conf, rm.getRMContext()); + + RMNode node = + MockNodes.newNodeInfo(1, Resources.createResource(1024, 4), 1, + "127.0.0.1"); + scheduler.handle(new NodeAddedSchedulerEvent(node)); + + ApplicationId appId = ApplicationId.newInstance(0, 1); + scheduler.addApplication(appId, "queue1", "user1", false); + + NodeUpdateSchedulerEvent updateEvent = new NodeUpdateSchedulerEvent(node); + try { + scheduler.handle(updateEvent); + } catch (NullPointerException e) { + Assert.fail(); + } + + ApplicationAttemptId attId = ApplicationAttemptId.newInstance(appId, 1); + scheduler.addApplicationAttempt(attId, false, false); + + rm.stop(); + } + + private void testMinimumAllocation(YarnConfiguration conf, int testAlloc) + throws Exception { + MockRM rm = new MockRM(conf); + rm.start(); + + // Register node1 + MockNM nm1 = rm.registerNode("127.0.0.1:1234", 6 * GB); + + // Submit an application + RMApp app1 = rm.submitApp(testAlloc); + + // kick the scheduling + nm1.nodeHeartbeat(true); + RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); + MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); + am1.registerAppAttempt(); + SchedulerNodeReport report_nm1 = + rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + + int checkAlloc = + conf.getInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); + Assert.assertEquals(checkAlloc, report_nm1.getUsedResource().getMemory()); + + rm.stop(); + } + + @Test(timeout = 60000) + public void testDefaultMinimumAllocation() throws Exception { + // Test with something lesser than default + testMinimumAllocation(new YarnConfiguration(TestFifoScheduler.conf), + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB / 2); + } + + @Test(timeout = 60000) + public void testNonDefaultMinimumAllocation() throws Exception { + // Set custom min-alloc to test tweaking it + int allocMB = 1536; + YarnConfiguration conf = new YarnConfiguration(TestFifoScheduler.conf); + conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, allocMB); + conf.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, + allocMB * 10); + // Test for something lesser than this. + testMinimumAllocation(conf, allocMB / 2); + } + + @Test(timeout = 50000) + public void testReconnectedNode() throws Exception { + CapacitySchedulerConfiguration conf = new CapacitySchedulerConfiguration(); + conf.setQueues("default", new String[] { "default" }); + conf.setCapacity("default", 100); + FifoScheduler fs = new FifoScheduler(); + fs.init(conf); + fs.start(); + // mock rmContext to avoid NPE. + RMContext context = mock(RMContext.class); + fs.reinitialize(conf, null); + fs.setRMContext(context); + + RMNode n1 = + MockNodes.newNodeInfo(0, MockNodes.newResource(4 * GB), 1, "127.0.0.2"); + RMNode n2 = + MockNodes.newNodeInfo(0, MockNodes.newResource(2 * GB), 2, "127.0.0.3"); + + fs.handle(new NodeAddedSchedulerEvent(n1)); + fs.handle(new NodeAddedSchedulerEvent(n2)); + fs.handle(new NodeUpdateSchedulerEvent(n1)); + Assert.assertEquals(6 * GB, fs.getRootQueueMetrics().getAvailableMB()); + + // reconnect n1 with downgraded memory + n1 = + MockNodes.newNodeInfo(0, MockNodes.newResource(2 * GB), 1, "127.0.0.2"); + fs.handle(new NodeRemovedSchedulerEvent(n1)); + fs.handle(new NodeAddedSchedulerEvent(n1)); + fs.handle(new NodeUpdateSchedulerEvent(n1)); + + Assert.assertEquals(4 * GB, fs.getRootQueueMetrics().getAvailableMB()); + fs.stop(); + } + + @Test(timeout = 50000) + public void testBlackListNodes() throws Exception { + + Configuration conf = new Configuration(); + conf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, + ResourceScheduler.class); + MockRM rm = new MockRM(conf); + rm.start(); + FifoScheduler fs = (FifoScheduler) rm.getResourceScheduler(); + + int rack_num_0 = 0; + int rack_num_1 = 1; + // Add 4 nodes in 2 racks + + // host_0_0 in rack0 + String host_0_0 = "127.0.0.1"; + RMNode n1 = + MockNodes.newNodeInfo(rack_num_0, MockNodes.newResource(4 * GB), 1, + host_0_0); + fs.handle(new NodeAddedSchedulerEvent(n1)); + + // host_0_1 in rack0 + String host_0_1 = "127.0.0.2"; + RMNode n2 = + MockNodes.newNodeInfo(rack_num_0, MockNodes.newResource(4 * GB), 1, + host_0_1); + fs.handle(new NodeAddedSchedulerEvent(n2)); + + // host_1_0 in rack1 + String host_1_0 = "127.0.0.3"; + RMNode n3 = + MockNodes.newNodeInfo(rack_num_1, MockNodes.newResource(4 * GB), 1, + host_1_0); + fs.handle(new NodeAddedSchedulerEvent(n3)); + + // host_1_1 in rack1 + String host_1_1 = "127.0.0.4"; + RMNode n4 = + MockNodes.newNodeInfo(rack_num_1, MockNodes.newResource(4 * GB), 1, + host_1_1); + fs.handle(new NodeAddedSchedulerEvent(n4)); + + // Add one application + ApplicationId appId1 = BuilderUtils.newApplicationId(100, 1); + ApplicationAttemptId appAttemptId1 = + BuilderUtils.newApplicationAttemptId(appId1, 1); + createMockRMApp(appAttemptId1, rm.getRMContext()); + + SchedulerEvent appEvent = + new AppAddedSchedulerEvent(appId1, "queue", "user"); + fs.handle(appEvent); + SchedulerEvent attemptEvent = + new AppAttemptAddedSchedulerEvent(appAttemptId1, false); + fs.handle(attemptEvent); + + List emptyId = new ArrayList(); + List emptyAsk = new ArrayList(); + + // Allow rack-locality for rack_1, but blacklist host_1_0 + + // Set up resource requests + // Ask for a 1 GB container for app 1 + List ask1 = new ArrayList(); + ask1.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), + "rack1", BuilderUtils.newResource(GB, 1), 1)); + ask1.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), + ResourceRequest.ANY, BuilderUtils.newResource(GB, 1), 1)); + fs.allocate(appAttemptId1, ask1, emptyId, + Collections.singletonList(host_1_0), null); + + // Trigger container assignment + fs.handle(new NodeUpdateSchedulerEvent(n3)); + + // Get the allocation for the application and verify no allocation on + // blacklist node + Allocation allocation1 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + + Assert.assertEquals("allocation1", 0, allocation1.getContainers().size()); + + // verify host_1_1 can get allocated as not in blacklist + fs.handle(new NodeUpdateSchedulerEvent(n4)); + Allocation allocation2 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + Assert.assertEquals("allocation2", 1, allocation2.getContainers().size()); + List containerList = allocation2.getContainers(); + for (Container container : containerList) { + Assert.assertEquals("Container is allocated on n4", + container.getNodeId(), n4.getNodeID()); + } + + // Ask for a 1 GB container again for app 1 + List ask2 = new ArrayList(); + // this time, rack0 is also in blacklist, so only host_1_1 is available to + // be assigned + ask2.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), + ResourceRequest.ANY, BuilderUtils.newResource(GB, 1), 1)); + fs.allocate(appAttemptId1, ask2, emptyId, + Collections.singletonList("rack0"), null); + + // verify n1 is not qualified to be allocated + fs.handle(new NodeUpdateSchedulerEvent(n1)); + Allocation allocation3 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + Assert.assertEquals("allocation3", 0, allocation3.getContainers().size()); + + // verify n2 is not qualified to be allocated + fs.handle(new NodeUpdateSchedulerEvent(n2)); + Allocation allocation4 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + Assert.assertEquals("allocation4", 0, allocation4.getContainers().size()); + + // verify n3 is not qualified to be allocated + fs.handle(new NodeUpdateSchedulerEvent(n3)); + Allocation allocation5 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + Assert.assertEquals("allocation5", 0, allocation5.getContainers().size()); + + fs.handle(new NodeUpdateSchedulerEvent(n4)); + Allocation allocation6 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + Assert.assertEquals("allocation6", 1, allocation6.getContainers().size()); + + containerList = allocation6.getContainers(); + for (Container container : containerList) { + Assert.assertEquals("Container is allocated on n4", + container.getNodeId(), n4.getNodeID()); + } + + rm.stop(); + } + + @Test(timeout = 50000) + public void testHeadroom() throws Exception { + + Configuration conf = new Configuration(); + conf.setClass(YarnConfiguration.RM_SCHEDULER, FifoScheduler.class, + ResourceScheduler.class); + MockRM rm = new MockRM(conf); + rm.start(); + FifoScheduler fs = (FifoScheduler) rm.getResourceScheduler(); + + // Add a node + RMNode n1 = + MockNodes.newNodeInfo(0, MockNodes.newResource(4 * GB), 1, "127.0.0.2"); + fs.handle(new NodeAddedSchedulerEvent(n1)); + + // Add two applications + ApplicationId appId1 = BuilderUtils.newApplicationId(100, 1); + ApplicationAttemptId appAttemptId1 = + BuilderUtils.newApplicationAttemptId(appId1, 1); + createMockRMApp(appAttemptId1, rm.getRMContext()); + SchedulerEvent appEvent = + new AppAddedSchedulerEvent(appId1, "queue", "user"); + fs.handle(appEvent); + SchedulerEvent attemptEvent = + new AppAttemptAddedSchedulerEvent(appAttemptId1, false); + fs.handle(attemptEvent); + + ApplicationId appId2 = BuilderUtils.newApplicationId(200, 2); + ApplicationAttemptId appAttemptId2 = + BuilderUtils.newApplicationAttemptId(appId2, 1); + createMockRMApp(appAttemptId2, rm.getRMContext()); + SchedulerEvent appEvent2 = + new AppAddedSchedulerEvent(appId2, "queue", "user"); + fs.handle(appEvent2); + SchedulerEvent attemptEvent2 = + new AppAttemptAddedSchedulerEvent(appAttemptId2, false); + fs.handle(attemptEvent2); + + List emptyId = new ArrayList(); + List emptyAsk = new ArrayList(); + + // Set up resource requests + + // Ask for a 1 GB container for app 1 + List ask1 = new ArrayList(); + ask1.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), + ResourceRequest.ANY, BuilderUtils.newResource(GB, 1), 1)); + fs.allocate(appAttemptId1, ask1, emptyId, null, null); + + // Ask for a 2 GB container for app 2 + List ask2 = new ArrayList(); + ask2.add(BuilderUtils.newResourceRequest(BuilderUtils.newPriority(0), + ResourceRequest.ANY, BuilderUtils.newResource(2 * GB, 1), 1)); + fs.allocate(appAttemptId2, ask2, emptyId, null, null); + + // Trigger container assignment + fs.handle(new NodeUpdateSchedulerEvent(n1)); + + // Get the allocation for the applications and verify headroom + Allocation allocation1 = + fs.allocate(appAttemptId1, emptyAsk, emptyId, null, null); + Assert.assertEquals("Allocation headroom", 1 * GB, allocation1 + .getResourceLimit().getMemory()); + + Allocation allocation2 = + fs.allocate(appAttemptId2, emptyAsk, emptyId, null, null); + Assert.assertEquals("Allocation headroom", 1 * GB, allocation2 + .getResourceLimit().getMemory()); + + rm.stop(); + } + + @Test(timeout = 60000) + public void testResourceOverCommit() throws Exception { + MockRM rm = new MockRM(conf); + rm.start(); + + MockNM nm1 = rm.registerNode("127.0.0.1:1234", 4 * GB); + + RMApp app1 = rm.submitApp(2048); + // kick the scheduling, 2 GB given to AM1, remaining 2GB on nm1 + nm1.nodeHeartbeat(true); + RMAppAttempt attempt1 = app1.getCurrentAppAttempt(); + MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId()); + am1.registerAppAttempt(); + SchedulerNodeReport report_nm1 = + rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + // check node report, 2 GB used and 2 GB available + Assert.assertEquals(2 * GB, report_nm1.getUsedResource().getMemory()); + Assert.assertEquals(2 * GB, report_nm1.getAvailableResource().getMemory()); + + // add request for containers + am1.addRequests(new String[] { "127.0.0.1", "127.0.0.2" }, 2 * GB, 1, 1); + AllocateResponse alloc1Response = am1.schedule(); // send the request + + // kick the scheduler, 2 GB given to AM1, resource remaining 0 + nm1.nodeHeartbeat(true); + while (alloc1Response.getAllocatedContainers().size() < 1) { + LOG.info("Waiting for containers to be created for app 1..."); + Thread.sleep(1000); + alloc1Response = am1.schedule(); + } + + List allocated1 = alloc1Response.getAllocatedContainers(); + Assert.assertEquals(1, allocated1.size()); + Assert.assertEquals(2 * GB, allocated1.get(0).getResource().getMemory()); + Assert.assertEquals(nm1.getNodeId(), allocated1.get(0).getNodeId()); + + report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + // check node report, 4 GB used and 0 GB available + Assert.assertEquals(0, report_nm1.getAvailableResource().getMemory()); + Assert.assertEquals(4 * GB, report_nm1.getUsedResource().getMemory()); + + // check container is assigned with 2 GB. + Container c1 = allocated1.get(0); + Assert.assertEquals(2 * GB, c1.getResource().getMemory()); + + // update node resource to 2 GB, so resource is over-consumed. + Map nodeResourceMap = + new HashMap(); + nodeResourceMap.put(nm1.getNodeId(), + ResourceOption.newInstance(Resource.newInstance(2 * GB, 1), -1)); + UpdateNodeResourceRequest request = + UpdateNodeResourceRequest.newInstance(nodeResourceMap); + rm.getAdminService().updateNodeResource(request); + + // Now, the used resource is still 4 GB, and available resource is minus + // value. + report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + Assert.assertEquals(4 * GB, report_nm1.getUsedResource().getMemory()); + Assert.assertEquals(-2 * GB, report_nm1.getAvailableResource().getMemory()); + + // Check container can complete successfully in case of resource + // over-commitment. + ContainerStatus containerStatus = + BuilderUtils.newContainerStatus(c1.getId(), ContainerState.COMPLETE, + "", 0); + nm1.containerStatus(containerStatus); + int waitCount = 0; + while (attempt1.getJustFinishedContainers().size() < 1 && waitCount++ != 20) { + LOG.info("Waiting for containers to be finished for app 1... Tried " + + waitCount + " times already.."); + Thread.sleep(100); + } + Assert.assertEquals(1, attempt1.getJustFinishedContainers().size()); + Assert.assertEquals(1, am1.schedule().getCompletedContainersStatuses() + .size()); + report_nm1 = rm.getResourceScheduler().getNodeReport(nm1.getNodeId()); + Assert.assertEquals(2 * GB, report_nm1.getUsedResource().getMemory()); + // As container return 2 GB back, the available resource becomes 0 again. + Assert.assertEquals(0 * GB, report_nm1.getAvailableResource().getMemory()); + rm.stop(); + } + private void checkApplicationResourceUsage(int expected, Application application) { Assert.assertEquals(expected, application.getUsedResources().getMemory()); @@ -643,4 +1170,18 @@ public static void main(String[] arg) throws Exception { t.testFifoScheduler(); t.tearDown(); } + + private RMAppImpl createMockRMApp(ApplicationAttemptId attemptId, + RMContext context) { + RMAppImpl app = mock(RMAppImpl.class); + when(app.getApplicationId()).thenReturn(attemptId.getApplicationId()); + RMAppAttemptImpl attempt = mock(RMAppAttemptImpl.class); + when(attempt.getAppAttemptId()).thenReturn(attemptId); + RMAppAttemptMetrics attemptMetric = mock(RMAppAttemptMetrics.class); + when(attempt.getRMAppAttemptMetrics()).thenReturn(attemptMetric); + when(app.getCurrentAppAttempt()).thenReturn(attempt); + context.getRMApps().putIfAbsent(attemptId.getApplicationId(), app); + return app; + } + } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java index 5d31404b4cca6..99a506afaaa59 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java @@ -287,9 +287,16 @@ public String toString() { * exception */ static class MyFS extends DistributedFileSystem { - - public MyFS() {} - public void close() {} + private static AtomicInteger instanceCounter = new AtomicInteger(); + public MyFS() { + instanceCounter.incrementAndGet(); + } + public void close() { + instanceCounter.decrementAndGet(); + } + public static int getInstanceCounter() { + return instanceCounter.get(); + } @Override public void initialize(URI uri, Configuration conf) throws IOException {} @@ -299,6 +306,11 @@ public MyToken getDelegationToken(String renewer) throws IOException { LOG.info("Called MYDFS.getdelegationtoken " + result); return result; } + + public Token[] addDelegationTokens( + final String renewer, Credentials credentials) throws IOException { + return new Token[0]; + } } /** @@ -1022,4 +1034,16 @@ public void testAppSubmissionWithPreviousToken() throws Exception{ // app2 completes, app1 is still running, check the token is not cancelled Assert.assertFalse(Renewer.cancelled); } + + // Test FileSystem memory leak in obtainSystemTokensForUser. + @Test + public void testFSLeakInObtainSystemTokensForUser() throws Exception{ + Credentials credentials = new Credentials(); + String user = "test"; + int oldCounter = MyFS.getInstanceCounter(); + delegationTokenRenewer.obtainSystemTokensForUser(user, credentials); + delegationTokenRenewer.obtainSystemTokensForUser(user, credentials); + delegationTokenRenewer.obtainSystemTokensForUser(user, credentials); + Assert.assertEquals(oldCounter, MyFS.getInstanceCounter()); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java index 9732c193b51df..8c7b14d859a1f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestAppPage.java @@ -23,6 +23,7 @@ import java.io.IOException; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.FinalApplicationStatus; import org.apache.hadoop.yarn.api.records.Resource; @@ -32,6 +33,7 @@ import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppMetrics; import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppState; +import org.apache.hadoop.yarn.server.webapp.AppBlock; import org.apache.hadoop.yarn.webapp.YarnWebParams; import org.apache.hadoop.yarn.webapp.test.WebAppTests; import org.junit.Test; @@ -75,8 +77,10 @@ public void testAppBlockRenderWithNullCurrentAppAttempt() throws Exception { @Override public void configure(Binder binder) { try { - binder.bind(ResourceManager.class).toInstance( - TestRMWebApp.mockRm(rmContext)); + ResourceManager rm = TestRMWebApp.mockRm(rmContext); + binder.bind(ResourceManager.class).toInstance(rm); + binder.bind(ApplicationBaseProtocol.class).toInstance( + rm.getClientRMService()); } catch (IOException e) { throw new IllegalStateException(e); } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java index 62713cfc7c183..4e10a2b5413e8 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java @@ -48,7 +48,7 @@ public class TestNodesPage { // Number of Actual Table Headers for NodesPage.NodesBlock might change in // future. In that case this value should be adjusted to the new value. - final int numberOfThInMetricsTable = 16; + final int numberOfThInMetricsTable = 20; final int numberOfActualTableHeaders = 13; private Injector injector; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java index fb1e61de0547f..481a53b622118 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java @@ -21,19 +21,30 @@ import static org.apache.hadoop.yarn.server.resourcemanager.MockNodes.newResource; import static org.apache.hadoop.yarn.webapp.Params.TITLE; import static org.junit.Assert.assertEquals; +import static org.mockito.Matchers.any; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; import java.io.IOException; +import java.util.ArrayList; import java.util.List; import java.util.concurrent.ConcurrentMap; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest; +import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsResponse; +import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.ApplicationReport; +import org.apache.hadoop.yarn.api.records.ApplicationResourceUsageReport; import org.apache.hadoop.yarn.api.records.NodeId; import org.apache.hadoop.yarn.api.records.NodeState; +import org.apache.hadoop.yarn.api.records.Token; import org.apache.hadoop.yarn.api.records.YarnApplicationState; import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.exceptions.YarnException; +import org.apache.hadoop.yarn.server.resourcemanager.ClientRMService; import org.apache.hadoop.yarn.server.resourcemanager.MockNodes; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; import org.apache.hadoop.yarn.server.resourcemanager.RMContextImpl; @@ -54,6 +65,7 @@ import org.apache.hadoop.yarn.webapp.WebApps; import org.apache.hadoop.yarn.webapp.YarnWebParams; import org.apache.hadoop.yarn.webapp.test.WebAppTests; +import org.junit.Assert; import org.junit.Test; import com.google.common.collect.Maps; @@ -87,7 +99,10 @@ public void configure(Binder binder) { @Override public void configure(Binder binder) { try { - binder.bind(ResourceManager.class).toInstance(mockRm(3, 1, 2, 8*GiB)); + ResourceManager mockRm = mockRm(3, 1, 2, 8*GiB); + binder.bind(ResourceManager.class).toInstance(mockRm); + binder.bind(ApplicationBaseProtocol.class) + .toInstance(mockRm.getClientRMService()); } catch (IOException e) { throw new IllegalStateException(e); } @@ -194,9 +209,11 @@ public static ResourceManager mockRm(RMContext rmContext) throws IOException { ResourceManager rm = mock(ResourceManager.class); ResourceScheduler rs = mockCapacityScheduler(); ApplicationACLsManager aclMgr = mockAppACLsManager(); + ClientRMService clientRMService = mockClientRMService(rmContext); when(rm.getResourceScheduler()).thenReturn(rs); when(rm.getRMContext()).thenReturn(rmContext); when(rm.getApplicationACLsManager()).thenReturn(aclMgr); + when(rm.getClientRMService()).thenReturn(clientRMService); return rm; } @@ -222,6 +239,35 @@ public static ApplicationACLsManager mockAppACLsManager() { return new ApplicationACLsManager(conf); } + public static ClientRMService mockClientRMService(RMContext rmContext) { + ClientRMService clientRMService = mock(ClientRMService.class); + List appReports = new ArrayList(); + for (RMApp app : rmContext.getRMApps().values()) { + ApplicationReport appReport = + ApplicationReport.newInstance( + app.getApplicationId(), (ApplicationAttemptId) null, + app.getUser(), app.getQueue(), + app.getName(), (String) null, 0, (Token) null, + app.createApplicationState(), + app.getDiagnostics().toString(), (String) null, + app.getStartTime(), app.getFinishTime(), + app.getFinalApplicationStatus(), + (ApplicationResourceUsageReport) null, app.getTrackingUrl(), + app.getProgress(), app.getApplicationType(), (Token) null); + appReports.add(appReport); + } + GetApplicationsResponse response = mock(GetApplicationsResponse.class); + when(response.getApplicationList()).thenReturn(appReports); + try { + when(clientRMService.getApplications(any(GetApplicationsRequest.class))) + .thenReturn(response); + } catch (YarnException e) { + Assert.fail("Exception is not expteced."); + } + return clientRMService; + } + + static void setupQueueConfiguration(CapacitySchedulerConfiguration conf) { // Define top-level queues conf.setQueues(CapacitySchedulerConfiguration.ROOT, new String[] {"a", "b", "c"}); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java index b850a5e70b968..06fa0d4f3516d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebAppFairScheduler.java @@ -24,10 +24,12 @@ import com.google.inject.Module; import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.yarn.api.ApplicationBaseProtocol; import org.apache.hadoop.yarn.api.records.ApplicationAttemptId; import org.apache.hadoop.yarn.api.records.ApplicationId; import org.apache.hadoop.yarn.api.records.Resource; import org.apache.hadoop.yarn.api.records.YarnApplicationState; +import org.apache.hadoop.yarn.server.resourcemanager.ClientRMService; import org.apache.hadoop.yarn.server.resourcemanager.RMContext; import org.apache.hadoop.yarn.server.resourcemanager.RMContextImpl; import org.apache.hadoop.yarn.server.resourcemanager.ResourceManager; @@ -73,7 +75,8 @@ public void configure(Binder binder) { mockRm(rmContext); binder.bind(ResourceManager.class).toInstance (mockRmWithFairScheduler); - + binder.bind(ApplicationBaseProtocol.class).toInstance( + mockRmWithFairScheduler.getClientRMService()); } catch (IOException e) { throw new IllegalStateException(e); } @@ -112,6 +115,8 @@ public void configure(Binder binder) { mockRmWithApps(rmContext); binder.bind(ResourceManager.class).toInstance (mockRmWithFairScheduler); + binder.bind(ApplicationBaseProtocol.class).toInstance( + mockRmWithFairScheduler.getClientRMService()); } catch (IOException e) { throw new IllegalStateException(e); @@ -168,8 +173,10 @@ private static ResourceManager mockRm(RMContext rmContext) throws IOException { ResourceManager rm = mock(ResourceManager.class); ResourceScheduler rs = mockFairScheduler(); + ClientRMService clientRMService = mockClientRMService(rmContext); when(rm.getResourceScheduler()).thenReturn(rs); when(rm.getRMContext()).thenReturn(rmContext); + when(rm.getClientRMService()).thenReturn(clientRMService); return rm; } @@ -188,8 +195,10 @@ private static ResourceManager mockRmWithApps(RMContext rmContext) throws IOException { ResourceManager rm = mock(ResourceManager.class); ResourceScheduler rs = mockFairSchedulerWithoutApps(rmContext); + ClientRMService clientRMService = mockClientRMService(rmContext); when(rm.getResourceScheduler()).thenReturn(rs); when(rm.getRMContext()).thenReturn(rmContext); + when(rm.getClientRMService()).thenReturn(clientRMService); return rm; } @@ -213,4 +222,7 @@ public FSAppAttempt getApplicationAttempt(ApplicationAttemptId return fs; } + public static ClientRMService mockClientRMService(RMContext rmContext) { + return mock(ClientRMService.class); + } } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java index ba5c73be693d8..eb426795683df 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java @@ -347,7 +347,7 @@ private void verifySubQueue(JSONObject info, String q, int numExpectedElements = 13; boolean isParentQueue = true; if (!info.has("queues")) { - numExpectedElements = 24; + numExpectedElements = 25; isParentQueue = false; } assertEquals("incorrect number of elements", numExpectedElements, info.length()); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ProxyUriUtils.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ProxyUriUtils.java index d678edf9b9d8a..e130225c89297 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ProxyUriUtils.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/ProxyUriUtils.java @@ -181,7 +181,7 @@ public static URI getUriFromTrackingPlugins(ApplicationId id, /** * Returns the scheme if present in the url - * eg. "https://issues.apache.org/jira/browse/YARN" > "https" + * eg. "https://issues.apache.org/jira/browse/YARN" {@literal ->} "https" */ public static String getSchemeFromUrl(String url) { int index = 0; diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmIpFilter.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmIpFilter.java index f1a8be665b490..e7617f0e0fffe 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmIpFilter.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmIpFilter.java @@ -152,8 +152,10 @@ public void doFilter(ServletRequest req, ServletResponse resp, } } if (user == null) { - LOG.warn("Could not find " + WebAppProxyServlet.PROXY_USER_COOKIE_NAME - + " cookie, so user will not be set"); + if (LOG.isDebugEnabled()) { + LOG.debug("Could not find " + WebAppProxyServlet.PROXY_USER_COOKIE_NAME + + " cookie, so user will not be set"); + } chain.doFilter(req, resp); } else { final AmIpPrincipal principal = new AmIpPrincipal(user); diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md index 3c32cddde14b8..1cb963e8737c4 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md @@ -69,6 +69,8 @@ The `CapacityScheduler` supports the following features: * **Resource-based Scheduling** - Support for resource-intensive applications, where-in a application can optionally specify higher resource-requirements than the default, there-by accomodating applications with differing resource requirements. Currently, *memory* is the the resource requirement supported. +* **Queue Mapping based on User or Group** - This feature allows users to map a job to a specific queue based on the user or group. + Configuration ------------- @@ -151,6 +153,30 @@ Configuration **Note:** An *ACL* is of the form *user1*, *user2spacegroup1*, *group2*. The special value of * implies *anyone*. The special value of *space* implies *no one*. The default is * for the root queue if not specified. + * Queue Mapping based on User or Group + + The `CapacityScheduler` supports the following parameters to configure the queue mapping based on user or group: + +| Property | Description | +|:---- |:---- | +| `yarn.scheduler.capacity.queue-mappings` | This configuration specifies the mapping of user or group to aspecific queue. You can map a single user or a list of users to queues. Syntax: `[u or g]:[name]:[queue_name][,next_mapping]*`. Here, *u or g* indicates whether the mapping is for a user or group. The value is *u* for user and *g* for group. *name* indicates the user name or group name. To specify the user who has submitted the application, %user can be used. *queue_name* indicates the queue name for which the application has to be mapped. To specify queue name same as user name, *%user* can be used. To specify queue name same as the name of the primary group for which the user belongs to, *%primary_group* can be used.| +| `yarn.scheduler.capacity.queue-mappings-override.enable` | This function is used to specify whether the user specified queues can be overridden. This is a Boolean value and the default value is *false*. | + +Example: + +``` + + yarn.scheduler.capacity.queue-mappings + u:user1:queue1,g:group1:queue2,u:%user:%user,u:user2:%primary_group + + Here, is mapped to , is mapped to , + maps users to queues with the same name as user, is mapped + to queue name same as respectively. The mappings will be + evaluated from left to right, and the first valid mapping will be used. + + +``` + ###Other Properties * Resource Calculator diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md index e516afbde70b5..d23505d7a20a5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md @@ -141,12 +141,10 @@ Notes ----- ContainerId string format is changed if RM restarts with work-preserving recovery enabled. It used to be such format: - - Container_{clusterTimestamp}_{appId}_{attemptId}_{containerId}, e.g. Container_1410901177871_0001_01_000005. +`Container_{clusterTimestamp}_{appId}_{attemptId}_{containerId}`, e.g. `Container_1410901177871_0001_01_000005`. It is now changed to: - - Container_e{epoch}_{clusterTimestamp}_{appId}_{attemptId}_{containerId}, e.g. Container_e17_1410901177871_0001_01_000005. +`Container_`**e{epoch}**`_{clusterTimestamp}_{appId}_{attemptId}_{containerId}`, e.g. `Container_`**e17**`_1410901177871_0001_01_000005`. Here, the additional epoch number is a monotonically increasing integer which starts from 0 and is increased by 1 each time RM restarts. If epoch number is 0, it is omitted and the containerId string format stays the same as before. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md index 4889936166c3f..cb8a5d33c1610 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md @@ -16,144 +16,122 @@ YARN Timeline Server ==================== * [Overview](#Overview) -* [Current Status](#Current_Status) -* [Basic Configuration](#Basic_Configuration) -* [Advanced Configuration](#Advanced_Configuration) -* [Generic-data related Configuration](#Generic-data_related_Configuration) -* [Per-framework-date related Configuration](#Per-framework-date_related_Configuration) -* [Running Timeline server](#Running_Timeline_server) -* [Accessing generic-data via command-line](#Accessing_generic-data_via_command-line) -* [Publishing of per-framework data by applications](#Publishing_of_per-framework_data_by_applications) + * [Introduction](#Introduction) + * [Current Status](#Current_Status) + * [Timeline Structure](#Timeline_Structure) +* [Deployment](#Deployment) + * [Configurations](#Configurations) + * [Running Timeline server](#Running_Timeline_server) + * [Accessing generic-data via command-line](#Accessing_generic-data_via_command-line) +* [Publishing of application specific data](#Publishing_of_application_specific_data) Overview --------- +--------- -Storage and retrieval of applications' current as well as historic information in a generic fashion is solved in YARN through the Timeline Server (previously also called Generic Application History Server). This serves two responsibilities: +### Introduction -* Generic information about completed applications - - Generic information includes application level data like queue-name, user information etc in the ApplicationSubmissionContext, list of application-attempts that ran for an application, information about each application-attempt, list of containers run under each application-attempt, and information about each container. Generic data is stored by ResourceManager to a history-store (default implementation on a file-system) and used by the web-UI to display information about completed applications. + Storage and retrieval of application's current as well as historic information in a generic fashion is solved in YARN through the Timeline Server. This serves two responsibilities: -* Per-framework information of running and completed applications - - Per-framework information is completely specific to an application or framework. For example, Hadoop MapReduce framework can include pieces of information like number of map tasks, reduce tasks, counters etc. Application developers can publish the specific information to the Timeline server via TimelineClient from within a client, the ApplicationMaster and/or the application's containers. This information is then queryable via REST APIs for rendering by application/framework specific UIs. +#### Application specific information -Current Status --------------- + Supports collection of information completely specific to an application or framework. For example, Hadoop MapReduce framework can include pieces of information like number of map tasks, reduce tasks, counters etc. Application developers can publish the specific information to the Timeline server via TimelineClient, the ApplicationMaster and/or the application's containers. This information is then queryable via REST APIs for rendering by application/framework specific UIs. -Timeline sever is a work in progress. The basic storage and retrieval of information, both generic and framework specific, are in place. Timeline server doesn't work in secure mode yet. The generic information and the per-framework information are today collected and presented separately and thus are not integrated well together. Finally, the per-framework information is only available via RESTful APIs, using JSON type content - ability to install framework specific UIs in YARN isn't supported yet. +#### Generic information about completed applications + + Previously this was done by Application History Server but with timeline server its just one use case of Timeline server functionality. Generic information includes application level data like queue-name, user information etc in the ApplicationSubmissionContext, list of application-attempts that ran for an application, information about each application-attempt, list of containers run under each application-attempt, and information about each container. Generic data is published by ResourceManager to the timeline store and used by the web-UI to display information about completed applications. + -Basic Configuration -------------------- +### Current Status -Users need to configure the Timeline server before starting it. The simplest configuration you should add in `yarn-site.xml` is to set the hostname of the Timeline server. + The essential functionality of the timeline server have been completed and it can work in both secure and non secure modes. The generic history service is also built on timeline store. In subsequent releases we will be rolling out next generation timeline service which is scalable and reliable. Currently, Application specific information is only available via RESTful APIs using JSON type content. The ability to install framework specific UIs in YARN is not supported yet. -```xml - - The hostname of the Timeline service web application. - yarn.timeline-service.hostname - 0.0.0.0 - -``` +### Timeline Structure -Advanced Configuration ----------------------- +![Timeline Structure] (./images/timeline_structure.jpg) -In addition to the hostname, admins can also configure whether the service is enabled or not, the ports of the RPC and the web interfaces, and the number of RPC handler threads. +#### TimelineDomain -```xml - - Address for the Timeline server to start the RPC server. - yarn.timeline-service.address - ${yarn.timeline-service.hostname}:10200 - + Domain is like namespace for Timeline server and users can host multiple entities, isolating them from others. Timeline server Security is defined at this level. Domain majorly stores owner info, read & write ACL information, created and modified time stamp information. Domain is uniquely identified by ID. - - The http address of the Timeline service web application. - yarn.timeline-service.webapp.address - ${yarn.timeline-service.hostname}:8188 - +#### TimelineEntity - - The https address of the Timeline service web application. - yarn.timeline-service.webapp.https.address - ${yarn.timeline-service.hostname}:8190 - + Entity contains the the meta information of some conceptual entity and its related events. The entity can be an application, an application attempt, a container or whatever the user-defined object. It contains Primary filters which will be used to index the entities in TimelineStore, such that users should carefully choose the information they want to store as the primary filters. The remaining data can be stored as other information. Entity is uniquely identified by EntityId and EntityType. - - Handler thread count to serve the client RPC requests. - yarn.timeline-service.handler-thread-count - 10 - +#### TimelineEvent - - Enables cross-origin support (CORS) for web services where - cross-origin web response headers are needed. For example, javascript making - a web services request to the timeline server. - yarn.timeline-service.http-cross-origin.enabled - false - + TimelineEvent contains the information of an event that is related to some conceptual entity of an application. Users are free to define what the event means, such as starting an application, getting allocated a container and etc. - - Comma separated list of origins that are allowed for web - services needing cross-origin (CORS) support. Wildcards (*) and patterns - allowed - yarn.timeline-service.http-cross-origin.allowed-origins - * - +Deployment +---------- - - Comma separated list of methods that are allowed for web - services needing cross-origin (CORS) support. - yarn.timeline-service.http-cross-origin.allowed-methods - GET,POST,HEAD - +###Configurations - - Comma separated list of headers that are allowed for web - services needing cross-origin (CORS) support. - yarn.timeline-service.http-cross-origin.allowed-headers - X-Requested-With,Content-Type,Accept,Origin - +#### Basic Configuration - - The number of seconds a pre-flighted request can be cached - for web services needing cross-origin (CORS) support. - yarn.timeline-service.http-cross-origin.max-age - 1800 - -``` +| Configuration Property | Description | +|:---- |:---- | +| `yarn.timeline-service.enabled` | Indicate to clients whether Timeline service is enabled or not. If enabled, the TimelineClient library used by end-users will post entities and events to the Timeline server. Defaults to false. | +| `yarn.resourcemanager.system-metrics-publisher.enabled` | The setting that controls whether yarn system metrics is published on the timeline server or not by RM. Defaults to false. | +| `yarn.timeline-service.generic-application-history.enabled` | Indicate to clients whether to query generic application data from timeline history-service or not. If not enabled then application data is queried only from Resource Manager. Defaults to false. | -Generic-data related Configuration ----------------------------------- +#### Advanced configuration -Users can specify whether the generic data collection is enabled or not, and also choose the storage-implementation class for the generic data. There are more configurations related to generic data collection, and users can refer to `yarn-default.xml` for all of them. +| Configuration Property | Description | +|:---- |:---- | +| `yarn.timeline-service.ttl-enable` | Enable age off of timeline store data. Defaults to true. | +| `yarn.timeline-service.ttl-ms` | Time to live for timeline store data in milliseconds. Defaults to 604800000 (7 days). | +| `yarn.timeline-service.handler-thread-count` | Handler thread count to serve the client RPC requests. Defaults to 10. | +| `yarn.timeline-service.client.max-retries` | Default maximum number of retires for timeline servive client. Defaults to 30. | +| `yarn.timeline-service.client.retry-interval-ms` | Default retry time interval for timeline servive client. Defaults to 1000. | -```xml - - Indicate to ResourceManager as well as clients whether - history-service is enabled or not. If enabled, ResourceManager starts - recording historical data that Timelien service can consume. Similarly, - clients can redirect to the history service when applications - finish if this is enabled. - yarn.timeline-service.generic-application-history.enabled - false - +#### Timeline store and state store configuration - - Store class name for history store, defaulting to file system - store - yarn.timeline-service.generic-application-history.store-class - org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore - -``` +| Configuration Property | Description | +|:---- |:---- | +| `yarn.timeline-service.store-class` | Store class name for timeline store. Defaults to org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore. | +| `yarn.timeline-service.leveldb-timeline-store.path` | Store file name for leveldb timeline store. Defaults to ${hadoop.tmp.dir}/yarn/timeline. | +| `yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms` | Length of time to wait between deletion cycles of leveldb timeline store in milliseconds. Defaults to 300000. | +| `yarn.timeline-service.leveldb-timeline-store.read-cache-size` | Size of read cache for uncompressed blocks for leveldb timeline store in bytes. Defaults to 104857600. | +| `yarn.timeline-service.leveldb-timeline-store.start-time-read-cache-size` | Size of cache for recently read entity start times for leveldb timeline store in number of entities. Defaults to 10000. | +| `yarn.timeline-service.leveldb-timeline-store.start-time-write-cache-size` | Size of cache for recently written entity start times for leveldb timeline store in number of entities. Defaults to 10000. | +| `yarn.timeline-service.recovery.enabled` | Defaults to false. | +| `yarn.timeline-service.state-store-class` | Store class name for timeline state store. Defaults to org.apache.hadoop.yarn.server.timeline.recovery.LeveldbTimelineStateStore. | +| `yarn.timeline-service.leveldb-state-store.path` | Store file name for leveldb timeline state store. | + +#### Web and RPC Configuration + +| Configuration Property | Description | +|:---- |:---- | +| `yarn.timeline-service.hostname` | The hostname of the Timeline service web application. Defaults to 0.0.0.0. | +| `yarn.timeline-service.address` | Address for the Timeline server to start the RPC server. Defaults to ${yarn.timeline-service.hostname}:10200. | +| `yarn.timeline-service.webapp.address` | The http address of the Timeline service web application. Defaults to ${yarn.timeline-service.hostname}:8188. | +| `yarn.timeline-service.webapp.https.address` | The https address of the Timeline service web application. Defaults to ${yarn.timeline-service.hostname}:8190. | +| `yarn.timeline-service.bind-host` | The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen to all interfaces by setting to 0.0.0.0. | +| `yarn.timeline-service.http-cross-origin.enabled` | Enables cross-origin support (CORS) for web services where cross-origin web response headers are needed. For example, javascript making a web services request to the timeline server. Defaults to false. | +| `yarn.timeline-service.http-cross-origin.allowed-origins` | Comma separated list of origins that are allowed for web services needing cross-origin (CORS) support. Wildcards `(*)` and patterns allowed. Defaults to `*`. | +| yarn.timeline-service.http-cross-origin.allowed-methods | Comma separated list of methods that are allowed for web services needing cross-origin (CORS) support. Defaults to GET,POST,HEAD. | +| `yarn.timeline-service.http-cross-origin.allowed-headers` | Comma separated list of headers that are allowed for web services needing cross-origin (CORS) support. Defaults to X-Requested-With,Content-Type,Accept,Origin. | +| `yarn.timeline-service.http-cross-origin.max-age` | The number of seconds a pre-flighted request can be cached for web services needing cross-origin (CORS) support. Defaults to 1800. | + +#### Security Configuration -Per-framework-date related Configuration ----------------------------------------- + Security can be enabled by setting yarn.timeline-service.http-authentication.type to kerberos and further following configurations can be done. -Users can specify whether per-framework data service is enabled or not, choose the store implementation for the per-framework data, and tune the retention of the per-framework data. There are more configurations related to per-framework data service, and users can refer to `yarn-default.xml` for all of them. +| Configuration Property | Description | +|:---- |:---- | +| `yarn.timeline-service.http-authentication.type` | Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple / kerberos / #AUTHENTICATION_HANDLER_CLASSNAME#. Defaults to simple. | +| `yarn.timeline-service.http-authentication.simple.anonymous.allowed` | Indicates if anonymous requests are allowed by the timeline server when using 'simple' authentication. Defaults to true. | +| `yarn.timeline-service.principal` | The Kerberos principal for the timeline server. | +| yarn.timeline-service.keytab | The Kerberos keytab for the timeline server. Defaults to /etc/krb5.keytab. | +| `yarn.timeline-service.delegation.key.update-interval` | Defaults to 86400000 (1 day). | +| `yarn.timeline-service.delegation.token.renew-interval` | Defaults to 86400000 (1 day). | +| `yarn.timeline-service.delegation.token.max-lifetime` | Defaults to 604800000 (7 day). | -```xml +#### Enabling the timeline service and the generic history service + + Following are the basic configuration to start Timeline server. + +``` Indicate to clients whether Timeline service is enabled or not. If enabled, the TimelineClient library used by end-users will post entities @@ -163,69 +141,97 @@ Users can specify whether per-framework data service is enabled or not, choose t - Store class name for timeline store. - yarn.timeline-service.store-class - org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore - - - - Enable age off of timeline store data. - yarn.timeline-service.ttl-enable + The setting that controls whether yarn system metrics is + published on the timeline server or not by RM. + yarn.resourcemanager.system-metrics-publisher.enabled true - Time to live for timeline store data in milliseconds. - yarn.timeline-service.ttl-ms - 604800000 + Indicate to clients whether to query generic application + data from timeline history-service or not. If not enabled then application + data is queried only from Resource Manager. + yarn.timeline-service.generic-application-history.enabled + true ``` -Running Timeline server ------------------------ +### Running Timeline server -Assuming all the aforementioned configurations are set properly, admins can start the Timeline server/history service with the following command: + Assuming all the aforementioned configurations are set properly, admins can start the Timeline server/history service with the following command: - $ yarn timelineserver +``` + $ yarn timelineserver +``` -Or users can start the Timeline server / history service as a daemon: + Or users can start the Timeline server / history service as a daemon: - $ yarn --daemon start timelineserver +``` + $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start timelineserver +``` -Accessing generic-data via command-line ---------------------------------------- +### Accessing generic-data via command-line -Users can access applications' generic historic data via the command line as below. Note that the same commands are usable to obtain the corresponding information about running applications. + Users can access applications' generic historic data via the command line as below. Note that the same commands are usable to obtain the corresponding information about running applications. ``` - $ yarn application -status - $ yarn applicationattempt -list - $ yarn applicationattempt -status - $ yarn container -list - $ yarn container -status + $ yarn application -status + $ yarn applicationattempt -list + $ yarn applicationattempt -status + $ yarn container -list + $ yarn container -status ``` -Publishing of per-framework data by applications +Publishing of application specific data ------------------------------------------------ -Developers can define what information they want to record for their applications by composing `TimelineEntity` and `TimelineEvent` objects, and put the entities and events to the Timeline server via `TimelineClient`. Following is an example: - -```java -// Create and start the Timeline client -TimelineClient client = TimelineClient.createTimelineClient(); -client.init(conf); -client.start(); - -TimelineEntity entity = null; -// Compose the entity -try { - TimelinePutResponse response = client.putEntities(entity); -} catch (IOException e) { - // Handle the exception -} catch (YarnException e) { - // Handle the exception -} - -// Stop the Timeline client -client.stop(); + Developers can define what information they want to record for their applications by composing `TimelineEntity` and `TimelineEvent` objects, and put the entities and events to the Timeline server via `TimelineClient`. Below is an example: + +``` + // Create and start the Timeline client + TimelineClient client = TimelineClient.createTimelineClient(); + client.init(conf); + client.start(); + + try { + TimelineDomain myDomain = new TimelineDomain(); + myDomain.setID("MyDomain"); + // Compose other Domain info .... + + client.putDomain(myDomain); + + TimelineEntity myEntity = new TimelineEntity(); + myEntity.setDomainId(myDomain.getId()); + myEntity.setEntityType("APPLICATION"); + myEntity.setEntityID("MyApp1") + // Compose other entity info + + TimelinePutResponse response = client.putEntities(entity); + + + TimelineEvent event = new TimelineEvent(); + event.setEventType("APP_FINISHED"); + event.setTimestamp(System.currentTimeMillis()); + event.addEventInfo("Exit Status", "SUCCESS"); + // Compose other Event info .... + + myEntity.addEvent(event); + timelineClient.putEntities(entity); + + } catch (IOException e) { + // Handle the exception + } catch (YarnException e) { + // Handle the exception + } + + // Stop the Timeline client + client.stop(); ``` + + **Note** : Following are the points which needs to be observed during updating a entity. + + * Domain ID should not be modified for already existing entity. + + * Its advisable to have same primary filters for all updates on entity. As on modification of primary filter by subsequent updates will result in not fetching the information before the update when queried with updated primary filter. + + * On modification of Primary filter value, new value will be appended with the old value. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md index 28bb678e498f1..b4bed6453140c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md @@ -67,11 +67,11 @@ Usage: `yarn application [options] ` | COMMAND\_OPTIONS | Description | |:---- |:---- | -| -appStates States | Works with -list to filter applications based on input comma-separated list of application states. The valid application state can be one of the following:  ALL, NEW, NEW\_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED | -| -appTypes Types | Works with -list to filter applications based on input comma-separated list of application types. | +| -appStates \ | Works with -list to filter applications based on input comma-separated list of application states. The valid application state can be one of the following:  ALL, NEW, NEW\_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED | +| -appTypes \ | Works with -list to filter applications based on input comma-separated list of application types. | | -list | Lists applications from the RM. Supports optional use of -appTypes to filter applications based on application type, and -appStates to filter applications based on application state. | -| -kill ApplicationId | Kills the application. | -| -status ApplicationId | Prints the status of the application. | +| -kill \ | Kills the application. | +| -status \ | Prints the status of the application. | Prints application(s) report/kill application @@ -82,8 +82,8 @@ Usage: `yarn applicationattempt [options] ` | COMMAND\_OPTIONS | Description | |:---- |:---- | | -help | Help | -| -list ApplicationId | Lists applications attempts from the RM | -| -status Application Attempt Id | Prints the status of the application attempt. | +| -list \ | Lists applications attempts for the given application. | +| -status \ | Prints the status of the application attempt. | prints applicationattempt(s) report @@ -100,8 +100,8 @@ Usage: `yarn container [options] ` | COMMAND\_OPTIONS | Description | |:---- |:---- | | -help | Help | -| -list ApplicationId | Lists containers for the application attempt. | -| -status ContainerId | Prints the status of the container. | +| -list \ | Lists containers for the application attempt. | +| -status \ | Prints the status of the container. | prints container(s) report @@ -118,10 +118,10 @@ Usage: `yarn logs -applicationId [options] ` | COMMAND\_OPTIONS | Description | |:---- |:---- | | -applicationId \ | Specifies an application id | -| -appOwner AppOwner | AppOwner (assumed to be current user if not specified) | -| -containerId ContainerId | ContainerId (must be specified if node address is specified) | +| -appOwner \ | AppOwner (assumed to be current user if not specified) | +| -containerId \ | ContainerId (must be specified if node address is specified) | | -help | Help | -| -nodeAddress NodeAddress | NodeAddress in the format nodename:port (must be specified if container id is specified) | +| -nodeAddress \ | NodeAddress in the format nodename:port (must be specified if container id is specified) | Dump the container logs @@ -133,8 +133,8 @@ Usage: `yarn node [options] ` |:---- |:---- | | -all | Works with -list to list all nodes. | | -list | Lists all running nodes. Supports optional use of -states to filter nodes based on node state, and -all to list all nodes. | -| -states States | Works with -list to filter nodes based on input comma-separated list of node states. | -| -status NodeId | Prints the status report of the node. | +| -states \ | Works with -list to filter nodes based on input comma-separated list of node states. | +| -status \ | Prints the status report of the node. | Prints node report(s) @@ -145,7 +145,7 @@ Usage: `yarn queue [options] ` | COMMAND\_OPTIONS | Description | |:---- |:---- | | -help | Help | -| -status QueueName | Prints the status of the queue. | +| -status \ | Prints the status of the queue. | Prints queue information diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/timeline_structure.jpg b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/timeline_structure.jpg new file mode 100644 index 0000000000000..dbfce25d7055e Binary files /dev/null and b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/timeline_structure.jpg differ diff --git a/hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn b/hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn.sh similarity index 100% rename from hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn rename to hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn.sh