[pull] master from apache:master #56

pull · 2024-01-04T11:22:56Z

See Commits and Changes for more details.

Can you help keep this open source service alive? 💖 Please sponsor : )

### What changes were proposed in this pull request? a followup of #44527 to fix golden files. ### Why are the changes needed? fix tests ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #44585 from cloud-fan/golden. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Kent Yao <[email protected]>

…edClientLoader ### What changes were proposed in this pull request? Try removing Guava from `sharedClasses` as suggested by JoshRosen in https://github.com/apache/spark/pull/33989#issuecomment-928616327 and https://github.com/apache/spark/pull/42493#issuecomment-1687092403 ### Why are the changes needed? Unblock Guava upgrading. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI passed (embedded HMS) and verified in the internal YARN cluster (remote HMS with kerberos-enabled). ``` # already setup hive-site.xml stuff properly to make sure to use remote HMS bin/spark-shell --conf spark.sql.hive.metastore.jars=maven ... scala> spark.sql("show databases").show warning: 1 deprecation (since 2.13.3); for details, enable `:setting -deprecation` or `:replay -deprecation` https://maven-central.storage-download.googleapis.com/maven2/ added as a remote repository with the name: repo-1 Ivy Default Cache set to: /home/hadoop/.ivy2/cache The jars for the packages stored in: /home/hadoop/.ivy2/jars org.apache.hive#hive-metastore added as a dependency org.apache.hive#hive-exec added as a dependency org.apache.hive#hive-common added as a dependency org.apache.hive#hive-serde added as a dependency org.apache.hadoop#hadoop-client-api added as a dependency org.apache.hadoop#hadoop-client-runtime added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-d0d2962d-ae27-4526-a0c7-040a542e1e54;1.0 confs: [default] found org.apache.hive#hive-metastore;2.3.9 in central found org.apache.hive#hive-serde;2.3.9 in central found org.apache.hive#hive-common;2.3.9 in central found org.apache.hive#hive-shims;2.3.9 in central found org.apache.hive.shims#hive-shims-common;2.3.9 in central found org.apache.logging.log4j#log4j-slf4j-impl;2.6.2 in central found org.slf4j#slf4j-api;1.7.10 in central found com.google.guava#guava;14.0.1 in central found commons-lang#commons-lang;2.6 in central found org.apache.thrift#libthrift;0.9.3 in central found org.apache.httpcomponents#httpclient;4.4 in central found org.apache.httpcomponents#httpcore;4.4 in central found commons-logging#commons-logging;1.2 in central found commons-codec#commons-codec;1.4 in central found org.apache.zookeeper#zookeeper;3.4.6 in central found org.slf4j#slf4j-log4j12;1.6.1 in central found log4j#log4j;1.2.16 in central found jline#jline;2.12 in central found io.netty#netty;3.7.0.Final in central found org.apache.hive.shims#hive-shims-0.23;2.3.9 in central found org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.7.2 in central found org.apache.hadoop#hadoop-annotations;2.7.2 in central found com.google.inject.extensions#guice-servlet;3.0 in central found com.google.inject#guice;3.0 in central found javax.inject#javax.inject;1 in central found aopalliance#aopalliance;1.0 in central found org.sonatype.sisu.inject#cglib;2.2.1-v20090111 in central found asm#asm;3.2 in central found com.google.protobuf#protobuf-java;2.5.0 in central found commons-io#commons-io;2.4 in central found com.sun.jersey#jersey-json;1.14 in central found org.codehaus.jettison#jettison;1.1 in central found com.sun.xml.bind#jaxb-impl;2.2.3-1 in central found javax.xml.bind#jaxb-api;2.2.2 in central found javax.xml.stream#stax-api;1.0-2 in central found javax.activation#activation;1.1 in central found org.codehaus.jackson#jackson-core-asl;1.9.13 in central found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in central found org.codehaus.jackson#jackson-jaxrs;1.9.13 in central found org.codehaus.jackson#jackson-xc;1.9.13 in central found com.sun.jersey#jersey-core;1.14 in central found com.sun.jersey.contribs#jersey-guice;1.9 in central found com.sun.jersey#jersey-server;1.14 in central found org.apache.hadoop#hadoop-yarn-common;2.7.2 in central found org.apache.hadoop#hadoop-yarn-api;2.7.2 in central found org.apache.commons#commons-compress;1.9 in central found org.mortbay.jetty#jetty-util;6.1.26 in central found com.sun.jersey#jersey-client;1.9 in central found commons-cli#commons-cli;1.2 in central found log4j#log4j;1.2.17 in central found org.apache.hadoop#hadoop-yarn-server-common;2.7.2 in central found org.fusesource.leveldbjni#leveldbjni-all;1.8 in central found org.apache.hadoop#hadoop-yarn-server-applicationhistoryservice;2.7.2 in central found commons-collections#commons-collections;3.2.2 in central found org.apache.hadoop#hadoop-yarn-server-web-proxy;2.7.2 in central found org.mortbay.jetty#jetty;6.1.26 in central found org.apache.hive.shims#hive-shims-scheduler;2.3.9 in central found org.apache.hive#hive-storage-api;2.4.0 in central found org.apache.commons#commons-lang3;3.1 in central found org.apache.orc#orc-core;1.3.4 in central found io.airlift#aircompressor;0.8 in central found io.airlift#slice;0.29 in central found org.openjdk.jol#jol-core;0.2 in central found org.eclipse.jetty.aggregate#jetty-all;7.6.0.v20120127 in central found org.apache.geronimo.specs#geronimo-jta_1.1_spec;1.1.1 in central found javax.mail#mail;1.4.1 in central found org.apache.geronimo.specs#geronimo-jaspic_1.0_spec;1.0 in central found org.apache.geronimo.specs#geronimo-annotation_1.0_spec;1.1.1 in central found asm#asm-commons;3.1 in central found asm#asm-tree;3.1 in central found org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 in central found joda-time#joda-time;2.8.1 in central found org.apache.logging.log4j#log4j-1.2-api;2.6.2 in central found org.apache.logging.log4j#log4j-web;2.6.2 in central found org.apache.ant#ant;1.9.1 in central found org.apache.ant#ant-launcher;1.9.1 in central found com.tdunning#json;1.8 in central found io.dropwizard.metrics#metrics-core;3.1.0 in central found io.dropwizard.metrics#metrics-jvm;3.1.0 in central found io.dropwizard.metrics#metrics-json;3.1.0 in central found com.github.joshelser#dropwizard-metrics-hadoop-metrics2-reporter;0.1.2 in central found org.apache.hadoop#hadoop-common;2.7.2 in central found org.apache.commons#commons-math3;3.1.1 in central found xmlenc#xmlenc;0.52 in central found commons-httpclient#commons-httpclient;3.1 in central found commons-net#commons-net;3.1 in central found javax.servlet#servlet-api;2.5 in central found net.java.dev.jets3t#jets3t;0.9.0 in central found com.jamesmurty.utils#java-xmlbuilder;0.4 in central found commons-configuration#commons-configuration;1.6 in central found commons-digester#commons-digester;1.8 in central found commons-beanutils#commons-beanutils;1.7.0 in central found commons-beanutils#commons-beanutils-core;1.8.0 in central found org.apache.avro#avro;1.8.2 in central found com.thoughtworks.paranamer#paranamer;2.7 in central found org.xerial.snappy#snappy-java;1.1.1.3 in central found org.tukaani#xz;1.5 in central found com.google.code.gson#gson;2.2.4 in central found org.apache.hadoop#hadoop-auth;2.7.2 in central found org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 in central found org.apache.directory.server#apacheds-i18n;2.0.0-M15 in central found org.apache.directory.api#api-asn1-api;1.0.0-M20 in central found org.apache.directory.api#api-util;1.0.0-M20 in central found com.jcraft#jsch;0.1.42 in central found com.google.code.findbugs#jsr305;3.0.0 in central found org.apache.htrace#htrace-core;3.1.0-incubating in central found javax.servlet.jsp#jsp-api;2.1 in central found org.slf4j#slf4j-log4j12;1.7.14 in central found org.apache.hive#hive-service-rpc;2.3.9 in central found tomcat#jasper-compiler;5.5.23 in central found javax.servlet#jsp-api;2.0 in central found ant#ant;1.6.5 in central found tomcat#jasper-runtime;5.5.23 in central found commons-el#commons-el;1.0 in central found org.apache.thrift#libfb303;0.9.3 in central found net.sf.opencsv#opencsv;2.3 in central found org.apache.parquet#parquet-hadoop-bundle;1.8.1 in central found javolution#javolution;5.5.1 in central found org.apache.hbase#hbase-client;1.1.1 in central found org.apache.hbase#hbase-annotations;1.1.1 in central found com.github.stephenc.findbugs#findbugs-annotations;1.3.9-1 in central found junit#junit;4.11 in central found org.hamcrest#hamcrest-core;1.3 in central found org.apache.hbase#hbase-protocol;1.1.1 in central found io.netty#netty-all;4.0.52.Final in central found org.jruby.jcodings#jcodings;1.0.8 in central found org.jruby.joni#joni;2.1.2 in central found org.apache.hadoop#hadoop-mapreduce-client-core;2.7.2 in central found com.jolbox#bonecp;0.8.0.RELEASE in central found com.zaxxer#HikariCP;2.5.1 in central found org.apache.derby#derby;10.10.2.0 in central found org.datanucleus#datanucleus-api-jdo;4.2.4 in central found org.datanucleus#datanucleus-core;4.1.17 in central found org.datanucleus#datanucleus-rdbms;4.1.19 in central found commons-pool#commons-pool;1.5.4 in central found commons-dbcp#commons-dbcp;1.4 in central found javax.jdo#jdo-api;3.0.1 in central found javax.transaction#jta;1.1 in central found org.datanucleus#javax.jdo;3.2.0-m3 in central found javax.transaction#transaction-api;1.1 in central found org.antlr#antlr-runtime;3.5.2 in central found co.cask.tephra#tephra-api;0.6.0 in central found co.cask.tephra#tephra-core;0.6.0 in central found com.google.inject.extensions#guice-assistedinject;3.0 in central found it.unimi.dsi#fastutil;6.5.6 in central found org.apache.twill#twill-common;0.6.0-incubating in central found org.apache.twill#twill-core;0.6.0-incubating in central found org.apache.twill#twill-api;0.6.0-incubating in central found org.apache.twill#twill-discovery-api;0.6.0-incubating in central found org.apache.twill#twill-zookeeper;0.6.0-incubating in central found org.apache.twill#twill-discovery-core;0.6.0-incubating in central found co.cask.tephra#tephra-hbase-compat-1.0;0.6.0 in central found org.apache.hive#hive-exec;2.3.9 in central found org.apache.hive#hive-llap-tez;2.3.9 in central found org.apache.hive#hive-llap-client;2.3.9 in central found org.apache.hive#hive-llap-common;2.3.9 in central found org.antlr#ST4;4.0.4 in central found org.apache.ivy#ivy;2.4.0 in central found org.codehaus.groovy#groovy-all;2.4.4 in central found stax#stax-api;1.0.1 in central found net.hydromatic#eigenbase-properties;1.1.5 in central found org.codehaus.janino#commons-compiler;2.7.6 in central found org.codehaus.janino#janino;2.7.6 in central found org.apache.hadoop#hadoop-client-api;3.3.6 in central found org.xerial.snappy#snappy-java;1.1.8.2 in central found org.apache.hadoop#hadoop-client-runtime;3.3.6 in central found org.slf4j#slf4j-api;1.7.36 in central found com.google.code.findbugs#jsr305;3.0.2 in central downloading https://repo1.maven.org/maven2/org/apache/hive/hive-metastore/2.3.9/hive-metastore-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-metastore;2.3.9!hive-metastore.jar (1292ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.9/hive-exec-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-exec;2.3.9!hive-exec.jar (1769ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-common/2.3.9/hive-common-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-common;2.3.9!hive-common.jar (360ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-serde/2.3.9/hive-serde-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-serde;2.3.9!hive-serde.jar (370ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-client-api;3.3.6!hadoop-client-api.jar (936ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-client-runtime;3.3.6!hadoop-client-runtime.jar (1418ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-shims/2.3.9/hive-shims-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-shims;2.3.9!hive-shims.jar (361ms) downloading https://repo1.maven.org/maven2/javolution/javolution/5.5.1/javolution-5.5.1.jar ... [SUCCESSFUL ] javolution#javolution;5.5.1!javolution.jar(bundle) (359ms) downloading https://repo1.maven.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.jar ... [SUCCESSFUL ] com.google.guava#guava;14.0.1!guava.jar(bundle) (385ms) downloading https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar ... [SUCCESSFUL ] com.google.protobuf#protobuf-java;2.5.0!protobuf-java.jar(bundle) (362ms) downloading https://repo1.maven.org/maven2/org/apache/hbase/hbase-client/1.1.1/hbase-client-1.1.1.jar ... [SUCCESSFUL ] org.apache.hbase#hbase-client;1.1.1!hbase-client.jar (378ms) downloading https://repo1.maven.org/maven2/com/jolbox/bonecp/0.8.0.RELEASE/bonecp-0.8.0.RELEASE.jar ... [SUCCESSFUL ] com.jolbox#bonecp;0.8.0.RELEASE!bonecp.jar(bundle) (356ms) downloading https://repo1.maven.org/maven2/com/zaxxer/HikariCP/2.5.1/HikariCP-2.5.1.jar ... [SUCCESSFUL ] com.zaxxer#HikariCP;2.5.1!HikariCP.jar(bundle) (356ms) downloading https://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar ... [SUCCESSFUL ] commons-cli#commons-cli;1.2!commons-cli.jar (353ms) downloading https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar ... [SUCCESSFUL ] commons-lang#commons-lang;2.6!commons-lang.jar (372ms) downloading https://repo1.maven.org/maven2/org/apache/derby/derby/10.10.2.0/derby-10.10.2.0.jar ... [SUCCESSFUL ] org.apache.derby#derby;10.10.2.0!derby.jar (398ms) downloading https://repo1.maven.org/maven2/org/datanucleus/datanucleus-api-jdo/4.2.4/datanucleus-api-jdo-4.2.4.jar ... [SUCCESSFUL ] org.datanucleus#datanucleus-api-jdo;4.2.4!datanucleus-api-jdo.jar (361ms) downloading https://repo1.maven.org/maven2/org/datanucleus/datanucleus-core/4.1.17/datanucleus-core-4.1.17.jar ... [SUCCESSFUL ] org.datanucleus#datanucleus-core;4.1.17!datanucleus-core.jar (391ms) downloading https://repo1.maven.org/maven2/org/datanucleus/datanucleus-rdbms/4.1.19/datanucleus-rdbms-4.1.19.jar ... [SUCCESSFUL ] org.datanucleus#datanucleus-rdbms;4.1.19!datanucleus-rdbms.jar (384ms) downloading https://repo1.maven.org/maven2/commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar ... [SUCCESSFUL ] commons-pool#commons-pool;1.5.4!commons-pool.jar (354ms) downloading https://repo1.maven.org/maven2/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar ... [SUCCESSFUL ] commons-dbcp#commons-dbcp;1.4!commons-dbcp.jar (355ms) downloading https://repo1.maven.org/maven2/javax/jdo/jdo-api/3.0.1/jdo-api-3.0.1.jar ... [SUCCESSFUL ] javax.jdo#jdo-api;3.0.1!jdo-api.jar (357ms) downloading https://repo1.maven.org/maven2/org/datanucleus/javax.jdo/3.2.0-m3/javax.jdo-3.2.0-m3.jar ... [SUCCESSFUL ] org.datanucleus#javax.jdo;3.2.0-m3!javax.jdo.jar (357ms) downloading https://repo1.maven.org/maven2/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.jar ... [SUCCESSFUL ] org.antlr#antlr-runtime;3.5.2!antlr-runtime.jar (355ms) downloading https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar ... [SUCCESSFUL ] org.apache.thrift#libfb303;0.9.3!libfb303.jar (181ms) downloading https://repo1.maven.org/maven2/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar ... [SUCCESSFUL ] org.apache.thrift#libthrift;0.9.3!libthrift.jar (183ms) downloading https://repo1.maven.org/maven2/co/cask/tephra/tephra-api/0.6.0/tephra-api-0.6.0.jar ... [SUCCESSFUL ] co.cask.tephra#tephra-api;0.6.0!tephra-api.jar (353ms) downloading https://repo1.maven.org/maven2/co/cask/tephra/tephra-core/0.6.0/tephra-core-0.6.0.jar ... [SUCCESSFUL ] co.cask.tephra#tephra-core;0.6.0!tephra-core.jar (363ms) downloading https://repo1.maven.org/maven2/co/cask/tephra/tephra-hbase-compat-1.0/0.6.0/tephra-hbase-compat-1.0-0.6.0.jar ... [SUCCESSFUL ] co.cask.tephra#tephra-hbase-compat-1.0;0.6.0!tephra-hbase-compat-1.0.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-service-rpc/2.3.9/hive-service-rpc-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-service-rpc;2.3.9!hive-service-rpc.jar (379ms) downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.4/commons-codec-1.4.jar ... [SUCCESSFUL ] commons-codec#commons-codec;1.4!commons-codec.jar (355ms) downloading https://repo1.maven.org/maven2/org/apache/avro/avro/1.8.2/avro-1.8.2.jar ... [SUCCESSFUL ] org.apache.avro#avro;1.8.2!avro.jar(bundle) (392ms) downloading https://repo1.maven.org/maven2/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar ... [SUCCESSFUL ] net.sf.opencsv#opencsv;2.3!opencsv.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop-bundle/1.8.1/parquet-hadoop-bundle-1.8.1.jar ... [SUCCESSFUL ] org.apache.parquet#parquet-hadoop-bundle;1.8.1!parquet-hadoop-bundle.jar (408ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-storage-api/2.4.0/hive-storage-api-2.4.0.jar ... [SUCCESSFUL ] org.apache.hive#hive-storage-api;2.4.0!hive-storage-api.jar (356ms) downloading https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar ... [SUCCESSFUL ] org.apache.commons#commons-lang3;3.1!commons-lang3.jar (358ms) downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.3.4/orc-core-1.3.4.jar ... [SUCCESSFUL ] org.apache.orc#orc-core;1.3.4!orc-core.jar (391ms) downloading https://repo1.maven.org/maven2/jline/jline/2.12/jline-2.12.jar ... [SUCCESSFUL ] jline#jline;2.12!jline.jar (357ms) downloading https://repo1.maven.org/maven2/org/eclipse/jetty/aggregate/jetty-all/7.6.0.v20120127/jetty-all-7.6.0.v20120127.jar ... [SUCCESSFUL ] org.eclipse.jetty.aggregate#jetty-all;7.6.0.v20120127!jetty-all.jar (379ms) downloading https://repo1.maven.org/maven2/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar ... [SUCCESSFUL ] org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.jar(orbit) (359ms) downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar ... [SUCCESSFUL ] joda-time#joda-time;2.8.1!joda-time.jar (361ms) downloading https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-1.2-api/2.6.2/log4j-1.2-api-2.6.2.jar ... [SUCCESSFUL ] org.apache.logging.log4j#log4j-1.2-api;2.6.2!log4j-1.2-api.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-web/2.6.2/log4j-web-2.6.2.jar ... [SUCCESSFUL ] org.apache.logging.log4j#log4j-web;2.6.2!log4j-web.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar ... [SUCCESSFUL ] org.apache.logging.log4j#log4j-slf4j-impl;2.6.2!log4j-slf4j-impl.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar ... [SUCCESSFUL ] org.apache.commons#commons-compress;1.9!commons-compress.jar (359ms) downloading https://repo1.maven.org/maven2/org/apache/ant/ant/1.9.1/ant-1.9.1.jar ... [SUCCESSFUL ] org.apache.ant#ant;1.9.1!ant.jar (390ms) downloading https://repo1.maven.org/maven2/com/tdunning/json/1.8/json-1.8.jar ... [SUCCESSFUL ] com.tdunning#json;1.8!json.jar (353ms) downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-core/3.1.0/metrics-core-3.1.0.jar ... [SUCCESSFUL ] io.dropwizard.metrics#metrics-core;3.1.0!metrics-core.jar(bundle) (356ms) downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-jvm/3.1.0/metrics-jvm-3.1.0.jar ... [SUCCESSFUL ] io.dropwizard.metrics#metrics-jvm;3.1.0!metrics-jvm.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/io/dropwizard/metrics/metrics-json/3.1.0/metrics-json-3.1.0.jar ... [SUCCESSFUL ] io.dropwizard.metrics#metrics-json;3.1.0!metrics-json.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/com/github/joshelser/dropwizard-metrics-hadoop-metrics2-reporter/0.1.2/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar ... [SUCCESSFUL ] com.github.joshelser#dropwizard-metrics-hadoop-metrics2-reporter;0.1.2!dropwizard-metrics-hadoop-metrics2-reporter.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/hive/shims/hive-shims-common/2.3.9/hive-shims-common-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive.shims#hive-shims-common;2.3.9!hive-shims-common.jar (355ms) downloading https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6-tests.jar ... [SUCCESSFUL ] org.apache.zookeeper#zookeeper;3.4.6!zookeeper.jar(test-jar) (362ms) downloading https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar ... [SUCCESSFUL ] org.apache.zookeeper#zookeeper;3.4.6!zookeeper.jar (188ms) downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.4/httpclient-4.4.jar ... [SUCCESSFUL ] org.apache.httpcomponents#httpclient;4.4!httpclient.jar (368ms) downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4/httpcore-4.4.jar ... [SUCCESSFUL ] org.apache.httpcomponents#httpcore;4.4!httpcore.jar (359ms) downloading https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar ... [SUCCESSFUL ] commons-logging#commons-logging;1.2!commons-logging.jar (353ms) downloading https://repo1.maven.org/maven2/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar ... [SUCCESSFUL ] io.netty#netty;3.7.0.Final!netty.jar(bundle) (373ms) downloading https://repo1.maven.org/maven2/org/apache/hive/shims/hive-shims-0.23/2.3.9/hive-shims-0.23-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive.shims#hive-shims-0.23;2.3.9!hive-shims-0.23.jar (359ms) downloading https://repo1.maven.org/maven2/org/apache/hive/shims/hive-shims-scheduler/2.3.9/hive-shims-scheduler-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive.shims#hive-shims-scheduler;2.3.9!hive-shims-scheduler.jar (355ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-resourcemanager/2.7.2/hadoop-yarn-server-resourcemanager-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.7.2!hadoop-yarn-server-resourcemanager.jar (375ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-annotations/2.7.2/hadoop-annotations-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-annotations;2.7.2!hadoop-annotations.jar (361ms) downloading https://repo1.maven.org/maven2/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar ... [SUCCESSFUL ] com.google.inject.extensions#guice-servlet;3.0!guice-servlet.jar (355ms) downloading https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar ... [SUCCESSFUL ] commons-io#commons-io;2.4!commons-io.jar (357ms) downloading https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0.jar ... [SUCCESSFUL ] com.google.inject#guice;3.0!guice.jar (368ms) downloading https://repo1.maven.org/maven2/com/sun/jersey/jersey-json/1.14/jersey-json-1.14.jar ... [SUCCESSFUL ] com.sun.jersey#jersey-json;1.14!jersey-json.jar (356ms) downloading https://repo1.maven.org/maven2/com/sun/jersey/contribs/jersey-guice/1.9/jersey-guice-1.9.jar ... [SUCCESSFUL ] com.sun.jersey.contribs#jersey-guice;1.9!jersey-guice.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.7.2/hadoop-yarn-common-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-common;2.7.2!hadoop-yarn-common.jar (379ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.7.2/hadoop-yarn-api-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-api;2.7.2!hadoop-yarn-api.jar (386ms) downloading https://repo1.maven.org/maven2/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar ... [SUCCESSFUL ] javax.xml.bind#jaxb-api;2.2.2!jaxb-api.jar (356ms) downloading https://repo1.maven.org/maven2/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar ... [SUCCESSFUL ] org.codehaus.jettison#jettison;1.1!jettison.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar ... [SUCCESSFUL ] com.sun.jersey#jersey-core;1.14!jersey-core.jar (360ms) downloading https://repo1.maven.org/maven2/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar ... [SUCCESSFUL ] com.sun.jersey#jersey-client;1.9!jersey-client.jar(bundle) (355ms) downloading https://repo1.maven.org/maven2/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar ... [SUCCESSFUL ] org.mortbay.jetty#jetty-util;6.1.26!jetty-util.jar (355ms) downloading https://repo1.maven.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar ... [SUCCESSFUL ] log4j#log4j;1.2.17!log4j.jar(bundle) (361ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-common/2.7.2/hadoop-yarn-server-common-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-server-common;2.7.2!hadoop-yarn-server-common.jar (367ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.2/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-server-applicationhistoryservice;2.7.2!hadoop-yarn-server-applicationhistoryservice.jar (363ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.7.2/hadoop-yarn-server-web-proxy-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-yarn-server-web-proxy;2.7.2!hadoop-yarn-server-web-proxy.jar (354ms) downloading https://repo1.maven.org/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar ... [SUCCESSFUL ] org.fusesource.leveldbjni#leveldbjni-all;1.8!leveldbjni-all.jar(bundle) (372ms) downloading https://repo1.maven.org/maven2/javax/inject/javax.inject/1/javax.inject-1.jar ... [SUCCESSFUL ] javax.inject#javax.inject;1!javax.inject.jar (355ms) downloading https://repo1.maven.org/maven2/aopalliance/aopalliance/1.0/aopalliance-1.0.jar ... [SUCCESSFUL ] aopalliance#aopalliance;1.0!aopalliance.jar (368ms) downloading https://repo1.maven.org/maven2/org/sonatype/sisu/inject/cglib/2.2.1-v20090111/cglib-2.2.1-v20090111.jar ... [SUCCESSFUL ] org.sonatype.sisu.inject#cglib;2.2.1-v20090111!cglib.jar (384ms) downloading https://repo1.maven.org/maven2/asm/asm/3.2/asm-3.2.jar ... [SUCCESSFUL ] asm#asm;3.2!asm.jar (355ms) downloading https://repo1.maven.org/maven2/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar ... [SUCCESSFUL ] com.sun.xml.bind#jaxb-impl;2.2.3-1!jaxb-impl.jar (367ms) downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar ... [SUCCESSFUL ] org.codehaus.jackson#jackson-core-asl;1.9.13!jackson-core-asl.jar (357ms) downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar ... [SUCCESSFUL ] org.codehaus.jackson#jackson-mapper-asl;1.9.13!jackson-mapper-asl.jar (365ms) downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar ... [SUCCESSFUL ] org.codehaus.jackson#jackson-jaxrs;1.9.13!jackson-jaxrs.jar (356ms) downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar ... [SUCCESSFUL ] org.codehaus.jackson#jackson-xc;1.9.13!jackson-xc.jar (354ms) downloading https://repo1.maven.org/maven2/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar ... [SUCCESSFUL ] javax.xml.stream#stax-api;1.0-2!stax-api.jar (354ms) downloading https://repo1.maven.org/maven2/javax/activation/activation/1.1/activation-1.1.jar ... [SUCCESSFUL ] javax.activation#activation;1.1!activation.jar (354ms) downloading https://repo1.maven.org/maven2/com/sun/jersey/jersey-server/1.14/jersey-server-1.14.jar ... [SUCCESSFUL ] com.sun.jersey#jersey-server;1.14!jersey-server.jar (362ms) downloading https://repo1.maven.org/maven2/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar ... [SUCCESSFUL ] commons-collections#commons-collections;3.2.2!commons-collections.jar (366ms) downloading https://repo1.maven.org/maven2/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar ... [SUCCESSFUL ] org.mortbay.jetty#jetty;6.1.26!jetty.jar (362ms) downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.8/aircompressor-0.8.jar ... [SUCCESSFUL ] io.airlift#aircompressor;0.8!aircompressor.jar (358ms) downloading https://repo1.maven.org/maven2/io/airlift/slice/0.29/slice-0.29.jar ... [SUCCESSFUL ] io.airlift#slice;0.29!slice.jar (355ms) downloading https://repo1.maven.org/maven2/org/openjdk/jol/jol-core/0.2/jol-core-0.2.jar ... [SUCCESSFUL ] org.openjdk.jol#jol-core;0.2!jol-core.jar (357ms) downloading https://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-jta_1.1_spec/1.1.1/geronimo-jta_1.1_spec-1.1.1.jar ... [SUCCESSFUL ] org.apache.geronimo.specs#geronimo-jta_1.1_spec;1.1.1!geronimo-jta_1.1_spec.jar (354ms) downloading https://repo1.maven.org/maven2/javax/mail/mail/1.4.1/mail-1.4.1.jar ... [SUCCESSFUL ] javax.mail#mail;1.4.1!mail.jar (361ms) downloading https://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-jaspic_1.0_spec/1.0/geronimo-jaspic_1.0_spec-1.0.jar ... [SUCCESSFUL ] org.apache.geronimo.specs#geronimo-jaspic_1.0_spec;1.0!geronimo-jaspic_1.0_spec.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-annotation_1.0_spec/1.1.1/geronimo-annotation_1.0_spec-1.1.1.jar ... [SUCCESSFUL ] org.apache.geronimo.specs#geronimo-annotation_1.0_spec;1.1.1!geronimo-annotation_1.0_spec.jar (354ms) downloading https://repo1.maven.org/maven2/asm/asm-commons/3.1/asm-commons-3.1.jar ... [SUCCESSFUL ] asm#asm-commons;3.1!asm-commons.jar (354ms) downloading https://repo1.maven.org/maven2/asm/asm-tree/3.1/asm-tree-3.1.jar ... [SUCCESSFUL ] asm#asm-tree;3.1!asm-tree.jar (356ms) downloading https://repo1.maven.org/maven2/org/apache/ant/ant-launcher/1.9.1/ant-launcher-1.9.1.jar ... [SUCCESSFUL ] org.apache.ant#ant-launcher;1.9.1!ant-launcher.jar (353ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-common;2.7.2!hadoop-common.jar (424ms) downloading https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.1.1/commons-math3-3.1.1.jar ... [SUCCESSFUL ] org.apache.commons#commons-math3;3.1.1!commons-math3.jar (388ms) downloading https://repo1.maven.org/maven2/xmlenc/xmlenc/0.52/xmlenc-0.52.jar ... [SUCCESSFUL ] xmlenc#xmlenc;0.52!xmlenc.jar (354ms) downloading https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar ... [SUCCESSFUL ] commons-httpclient#commons-httpclient;3.1!commons-httpclient.jar (363ms) downloading https://repo1.maven.org/maven2/commons-net/commons-net/3.1/commons-net-3.1.jar ... [SUCCESSFUL ] commons-net#commons-net;3.1!commons-net.jar (360ms) downloading https://repo1.maven.org/maven2/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar ... [SUCCESSFUL ] javax.servlet#servlet-api;2.5!servlet-api.jar (360ms) downloading https://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar ... [SUCCESSFUL ] net.java.dev.jets3t#jets3t;0.9.0!jets3t.jar (363ms) downloading https://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar ... [SUCCESSFUL ] commons-configuration#commons-configuration;1.6!commons-configuration.jar (366ms) downloading https://repo1.maven.org/maven2/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar ... [SUCCESSFUL ] com.google.code.gson#gson;2.2.4!gson.jar (367ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-auth;2.7.2!hadoop-auth.jar (360ms) downloading https://repo1.maven.org/maven2/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar ... [SUCCESSFUL ] com.jcraft#jsch;0.1.42!jsch.jar (360ms) downloading https://repo1.maven.org/maven2/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar ... [SUCCESSFUL ] org.apache.htrace#htrace-core;3.1.0-incubating!htrace-core.jar (375ms) downloading https://repo1.maven.org/maven2/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar ... [SUCCESSFUL ] com.jamesmurty.utils#java-xmlbuilder;0.4!java-xmlbuilder.jar (354ms) downloading https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8/commons-digester-1.8.jar ... [SUCCESSFUL ] commons-digester#commons-digester;1.8!commons-digester.jar (355ms) downloading https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar ... [SUCCESSFUL ] commons-beanutils#commons-beanutils-core;1.8.0!commons-beanutils-core.jar (357ms) downloading https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar ... [SUCCESSFUL ] commons-beanutils#commons-beanutils;1.7.0!commons-beanutils.jar (356ms) downloading https://repo1.maven.org/maven2/com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar ... [SUCCESSFUL ] com.thoughtworks.paranamer#paranamer;2.7!paranamer.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/org/tukaani/xz/1.5/xz-1.5.jar ... [SUCCESSFUL ] org.tukaani#xz;1.5!xz.jar (369ms) downloading https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar ... [SUCCESSFUL ] org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15!apacheds-kerberos-codec.jar(bundle) (364ms) downloading https://repo1.maven.org/maven2/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar ... [SUCCESSFUL ] org.apache.directory.server#apacheds-i18n;2.0.0-M15!apacheds-i18n.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar ... [SUCCESSFUL ] org.apache.directory.api#api-asn1-api;1.0.0-M20!api-asn1-api.jar(bundle) (354ms) downloading https://repo1.maven.org/maven2/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar ... [SUCCESSFUL ] org.apache.directory.api#api-util;1.0.0-M20!api-util.jar(bundle) (355ms) downloading https://repo1.maven.org/maven2/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar ... [SUCCESSFUL ] javax.servlet.jsp#jsp-api;2.1!jsp-api.jar (356ms) downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.7.14/slf4j-log4j12-1.7.14.jar ... [SUCCESSFUL ] org.slf4j#slf4j-log4j12;1.7.14!slf4j-log4j12.jar (354ms) downloading https://repo1.maven.org/maven2/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar ... [SUCCESSFUL ] tomcat#jasper-compiler;5.5.23!jasper-compiler.jar (360ms) downloading https://repo1.maven.org/maven2/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar ... [SUCCESSFUL ] tomcat#jasper-runtime;5.5.23!jasper-runtime.jar (355ms) downloading https://repo1.maven.org/maven2/javax/servlet/jsp-api/2.0/jsp-api-2.0.jar ... [SUCCESSFUL ] javax.servlet#jsp-api;2.0!jsp-api.jar (355ms) downloading https://repo1.maven.org/maven2/ant/ant/1.6.5/ant-1.6.5.jar ... [SUCCESSFUL ] ant#ant;1.6.5!ant.jar (371ms) downloading https://repo1.maven.org/maven2/commons-el/commons-el/1.0/commons-el-1.0.jar ... [SUCCESSFUL ] commons-el#commons-el;1.0!commons-el.jar (355ms) downloading https://repo1.maven.org/maven2/org/apache/hbase/hbase-annotations/1.1.1/hbase-annotations-1.1.1.jar ... [SUCCESSFUL ] org.apache.hbase#hbase-annotations;1.1.1!hbase-annotations.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/hbase/hbase-protocol/1.1.1/hbase-protocol-1.1.1.jar ... [SUCCESSFUL ] org.apache.hbase#hbase-protocol;1.1.1!hbase-protocol.jar (414ms) downloading https://repo1.maven.org/maven2/io/netty/netty-all/4.0.52.Final/netty-all-4.0.52.Final.jar ... [SUCCESSFUL ] io.netty#netty-all;4.0.52.Final!netty-all.jar (400ms) downloading https://repo1.maven.org/maven2/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.jar ... [SUCCESSFUL ] org.jruby.jcodings#jcodings;1.0.8!jcodings.jar (419ms) downloading https://repo1.maven.org/maven2/org/jruby/joni/joni/2.1.2/joni-2.1.2.jar ... [SUCCESSFUL ] org.jruby.joni#joni;2.1.2!joni.jar (369ms) downloading https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.2/hadoop-mapreduce-client-core-2.7.2.jar ... [SUCCESSFUL ] org.apache.hadoop#hadoop-mapreduce-client-core;2.7.2!hadoop-mapreduce-client-core.jar (393ms) downloading https://repo1.maven.org/maven2/com/github/stephenc/findbugs/findbugs-annotations/1.3.9-1/findbugs-annotations-1.3.9-1.jar ... [SUCCESSFUL ] com.github.stephenc.findbugs#findbugs-annotations;1.3.9-1!findbugs-annotations.jar (354ms) downloading https://repo1.maven.org/maven2/junit/junit/4.11/junit-4.11.jar ... [SUCCESSFUL ] junit#junit;4.11!junit.jar (357ms) downloading https://repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar ... [SUCCESSFUL ] org.hamcrest#hamcrest-core;1.3!hamcrest-core.jar (353ms) downloading https://repo1.maven.org/maven2/javax/transaction/jta/1.1/jta-1.1.jar ... [SUCCESSFUL ] javax.transaction#jta;1.1!jta.jar (354ms) downloading https://repo1.maven.org/maven2/javax/transaction/transaction-api/1.1/transaction-api-1.1.jar ... [SUCCESSFUL ] javax.transaction#transaction-api;1.1!transaction-api.jar (368ms) downloading https://repo1.maven.org/maven2/com/google/inject/extensions/guice-assistedinject/3.0/guice-assistedinject-3.0.jar ... [SUCCESSFUL ] com.google.inject.extensions#guice-assistedinject;3.0!guice-assistedinject.jar (355ms) downloading https://repo1.maven.org/maven2/it/unimi/dsi/fastutil/6.5.6/fastutil-6.5.6.jar ... [SUCCESSFUL ] it.unimi.dsi#fastutil;6.5.6!fastutil.jar (894ms) downloading https://repo1.maven.org/maven2/org/apache/twill/twill-common/0.6.0-incubating/twill-common-0.6.0-incubating.jar ... [SUCCESSFUL ] org.apache.twill#twill-common;0.6.0-incubating!twill-common.jar (355ms) downloading https://repo1.maven.org/maven2/org/apache/twill/twill-core/0.6.0-incubating/twill-core-0.6.0-incubating.jar ... [SUCCESSFUL ] org.apache.twill#twill-core;0.6.0-incubating!twill-core.jar (358ms) downloading https://repo1.maven.org/maven2/org/apache/twill/twill-discovery-api/0.6.0-incubating/twill-discovery-api-0.6.0-incubating.jar ... [SUCCESSFUL ] org.apache.twill#twill-discovery-api;0.6.0-incubating!twill-discovery-api.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/twill/twill-discovery-core/0.6.0-incubating/twill-discovery-core-0.6.0-incubating.jar ... [SUCCESSFUL ] org.apache.twill#twill-discovery-core;0.6.0-incubating!twill-discovery-core.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/twill/twill-zookeeper/0.6.0-incubating/twill-zookeeper-0.6.0-incubating.jar ... [SUCCESSFUL ] org.apache.twill#twill-zookeeper;0.6.0-incubating!twill-zookeeper.jar (357ms) downloading https://repo1.maven.org/maven2/org/apache/twill/twill-api/0.6.0-incubating/twill-api-0.6.0-incubating.jar ... [SUCCESSFUL ] org.apache.twill#twill-api;0.6.0-incubating!twill-api.jar (354ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-llap-tez/2.3.9/hive-llap-tez-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-llap-tez;2.3.9!hive-llap-tez.jar (357ms) downloading https://repo1.maven.org/maven2/org/antlr/ST4/4.0.4/ST4-4.0.4.jar ... [SUCCESSFUL ] org.antlr#ST4;4.0.4!ST4.jar (356ms) downloading https://repo1.maven.org/maven2/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar ... [SUCCESSFUL ] org.apache.ivy#ivy;2.4.0!ivy.jar (372ms) downloading https://repo1.maven.org/maven2/org/codehaus/groovy/groovy-all/2.4.4/groovy-all-2.4.4.jar ... [SUCCESSFUL ] org.codehaus.groovy#groovy-all;2.4.4!groovy-all.jar (549ms) downloading https://repo1.maven.org/maven2/stax/stax-api/1.0.1/stax-api-1.0.1.jar ... [SUCCESSFUL ] stax#stax-api;1.0.1!stax-api.jar (354ms) downloading https://repo1.maven.org/maven2/net/hydromatic/eigenbase-properties/1.1.5/eigenbase-properties-1.1.5.jar ... [SUCCESSFUL ] net.hydromatic#eigenbase-properties;1.1.5!eigenbase-properties.jar(bundle) (356ms) downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/2.7.6/commons-compiler-2.7.6.jar ... [SUCCESSFUL ] org.codehaus.janino#commons-compiler;2.7.6!commons-compiler.jar (354ms) downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/2.7.6/janino-2.7.6.jar ... [SUCCESSFUL ] org.codehaus.janino#janino;2.7.6!janino.jar (363ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-llap-client/2.3.9/hive-llap-client-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-llap-client;2.3.9!hive-llap-client.jar (356ms) downloading https://repo1.maven.org/maven2/org/apache/hive/hive-llap-common/2.3.9/hive-llap-common-2.3.9.jar ... [SUCCESSFUL ] org.apache.hive#hive-llap-common;2.3.9!hive-llap-common.jar (358ms) downloading https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.8.2/snappy-java-1.1.8.2.jar ... [SUCCESSFUL ] org.xerial.snappy#snappy-java;1.1.8.2!snappy-java.jar(bundle) (383ms) downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar ... [SUCCESSFUL ] org.slf4j#slf4j-api;1.7.36!slf4j-api.jar (353ms) downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar ... [SUCCESSFUL ] com.google.code.findbugs#jsr305;3.0.2!jsr305.jar (353ms) :: resolution report :: resolve 193214ms :: artifacts dl 63795ms :: modules in use: ant#ant;1.6.5 from central in [default] aopalliance#aopalliance;1.0 from central in [default] asm#asm;3.2 from central in [default] asm#asm-commons;3.1 from central in [default] asm#asm-tree;3.1 from central in [default] co.cask.tephra#tephra-api;0.6.0 from central in [default] co.cask.tephra#tephra-core;0.6.0 from central in [default] co.cask.tephra#tephra-hbase-compat-1.0;0.6.0 from central in [default] com.github.joshelser#dropwizard-metrics-hadoop-metrics2-reporter;0.1.2 from central in [default] com.github.stephenc.findbugs#findbugs-annotations;1.3.9-1 from central in [default] com.google.code.findbugs#jsr305;3.0.2 from central in [default] com.google.code.gson#gson;2.2.4 from central in [default] com.google.guava#guava;14.0.1 from central in [default] com.google.inject#guice;3.0 from central in [default] com.google.inject.extensions#guice-assistedinject;3.0 from central in [default] com.google.inject.extensions#guice-servlet;3.0 from central in [default] com.google.protobuf#protobuf-java;2.5.0 from central in [default] com.jamesmurty.utils#java-xmlbuilder;0.4 from central in [default] com.jcraft#jsch;0.1.42 from central in [default] com.jolbox#bonecp;0.8.0.RELEASE from central in [default] com.sun.jersey#jersey-client;1.9 from central in [default] com.sun.jersey#jersey-core;1.14 from central in [default] com.sun.jersey#jersey-json;1.14 from central in [default] com.sun.jersey#jersey-server;1.14 from central in [default] com.sun.jersey.contribs#jersey-guice;1.9 from central in [default] com.sun.xml.bind#jaxb-impl;2.2.3-1 from central in [default] com.tdunning#json;1.8 from central in [default] com.thoughtworks.paranamer#paranamer;2.7 from central in [default] com.zaxxer#HikariCP;2.5.1 from central in [default] commons-beanutils#commons-beanutils;1.7.0 from central in [default] commons-beanutils#commons-beanutils-core;1.8.0 from central in [default] commons-cli#commons-cli;1.2 from central in [default] commons-codec#commons-codec;1.4 from central in [default] commons-collections#commons-collections;3.2.2 from central in [default] commons-configuration#commons-configuration;1.6 from central in [default] commons-dbcp#commons-dbcp;1.4 from central in [default] commons-digester#commons-digester;1.8 from central in [default] commons-el#commons-el;1.0 from central in [default] commons-httpclient#commons-httpclient;3.1 from central in [default] commons-io#commons-io;2.4 from central in [default] commons-lang#commons-lang;2.6 from central in [default] commons-logging#commons-logging;1.2 from central in [default] commons-net#commons-net;3.1 from central in [default] commons-pool#commons-pool;1.5.4 from central in [default] io.airlift#aircompressor;0.8 from central in [default] io.airlift#slice;0.29 from central in [default] io.dropwizard.metrics#metrics-core;3.1.0 from central in [default] io.dropwizard.metrics#metrics-json;3.1.0 from central in [default] io.dropwizard.metrics#metrics-jvm;3.1.0 from central in [default] io.netty#netty;3.7.0.Final from central in [default] io.netty#netty-all;4.0.52.Final from central in [default] it.unimi.dsi#fastutil;6.5.6 from central in [default] javax.activation#activation;1.1 from central in [default] javax.inject#javax.inject;1 from central in [default] javax.jdo#jdo-api;3.0.1 from central in [default] javax.mail#mail;1.4.1 from central in [default] javax.servlet#jsp-api;2.0 from central in [default] javax.servlet#servlet-api;2.5 from central in [default] javax.servlet.jsp#jsp-api;2.1 from central in [default] javax.transaction#jta;1.1 from central in [default] javax.transaction#transaction-api;1.1 from central in [default] javax.xml.bind#jaxb-api;2.2.2 from central in [default] javax.xml.stream#stax-api;1.0-2 from central in [default] javolution#javolution;5.5.1 from central in [default] jline#jline;2.12 from central in [default] joda-time#joda-time;2.8.1 from central in [default] junit#junit;4.11 from central in [default] log4j#log4j;1.2.17 from central in [default] net.hydromatic#eigenbase-properties;1.1.5 from central in [default] net.java.dev.jets3t#jets3t;0.9.0 from central in [default] net.sf.opencsv#opencsv;2.3 from central in [default] org.antlr#ST4;4.0.4 from central in [default] org.antlr#antlr-runtime;3.5.2 from central in [default] org.apache.ant#ant;1.9.1 from central in [default] org.apache.ant#ant-launcher;1.9.1 from central in [default] org.apache.avro#avro;1.8.2 from central in [default] org.apache.commons#commons-compress;1.9 from central in [default] org.apache.commons#commons-lang3;3.1 from central in [default] org.apache.commons#commons-math3;3.1.1 from central in [default] org.apache.derby#derby;10.10.2.0 from central in [default] org.apache.directory.api#api-asn1-api;1.0.0-M20 from central in [default] org.apache.directory.api#api-util;1.0.0-M20 from central in [default] org.apache.directory.server#apacheds-i18n;2.0.0-M15 from central in [default] org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 from central in [default] org.apache.geronimo.specs#geronimo-annotation_1.0_spec;1.1.1 from central in [default] org.apache.geronimo.specs#geronimo-jaspic_1.0_spec;1.0 from central in [default] org.apache.geronimo.specs#geronimo-jta_1.1_spec;1.1.1 from central in [default] org.apache.hadoop#hadoop-annotations;2.7.2 from central in [default] org.apache.hadoop#hadoop-auth;2.7.2 from central in [default] org.apache.hadoop#hadoop-client-api;3.3.6 from central in [default] org.apache.hadoop#hadoop-client-runtime;3.3.6 from central in [default] org.apache.hadoop#hadoop-common;2.7.2 from central in [default] org.apache.hadoop#hadoop-mapreduce-client-core;2.7.2 from central in [default] org.apache.hadoop#hadoop-yarn-api;2.7.2 from central in [default] org.apache.hadoop#hadoop-yarn-common;2.7.2 from central in [default] org.apache.hadoop#hadoop-yarn-server-applicationhistoryservice;2.7.2 from central in [default] org.apache.hadoop#hadoop-yarn-server-common;2.7.2 from central in [default] org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.7.2 from central in [default] org.apache.hadoop#hadoop-yarn-server-web-proxy;2.7.2 from central in [default] org.apache.hbase#hbase-annotations;1.1.1 from central in [default] org.apache.hbase#hbase-client;1.1.1 from central in [default] org.apache.hbase#hbase-protocol;1.1.1 from central in [default] org.apache.hive#hive-common;2.3.9 from central in [default] org.apache.hive#hive-exec;2.3.9 from central in [default] org.apache.hive#hive-llap-client;2.3.9 from central in [default] org.apache.hive#hive-llap-common;2.3.9 from central in [default] org.apache.hive#hive-llap-tez;2.3.9 from central in [default] org.apache.hive#hive-metastore;2.3.9 from central in [default] org.apache.hive#hive-serde;2.3.9 from central in [default] org.apache.hive#hive-service-rpc;2.3.9 from central in [default] org.apache.hive#hive-shims;2.3.9 from central in [default] org.apache.hive#hive-storage-api;2.4.0 from central in [default] org.apache.hive.shims#hive-shims-0.23;2.3.9 from central in [default] org.apache.hive.shims#hive-shims-common;2.3.9 from central in [default] org.apache.hive.shims#hive-shims-scheduler;2.3.9 from central in [default] org.apache.htrace#htrace-core;3.1.0-incubating from central in [default] org.apache.httpcomponents#httpclient;4.4 from central in [default] org.apache.httpcomponents#httpcore;4.4 from central in [default] org.apache.ivy#ivy;2.4.0 from central in [default] org.apache.logging.log4j#log4j-1.2-api;2.6.2 from central in [default] org.apache.logging.log4j#log4j-slf4j-impl;2.6.2 from central in [default] org.apache.logging.log4j#log4j-web;2.6.2 from central in [default] org.apache.orc#orc-core;1.3.4 from central in [default] org.apache.parquet#parquet-hadoop-bundle;1.8.1 from central in [default] org.apache.thrift#libfb303;0.9.3 from central in [default] org.apache.thrift#libthrift;0.9.3 from central in [default] org.apache.twill#twill-api;0.6.0-incubating from central in [default] org.apache.twill#twill-common;0.6.0-incubating from central in [default] org.apache.twill#twill-core;0.6.0-incubating from central in [default] org.apache.twill#twill-discovery-api;0.6.0-incubating from central in [default] org.apache.twill#twill-discovery-core;0.6.0-incubating from central in [default] org.apache.twill#twill-zookeeper;0.6.0-incubating from central in [default] org.apache.zookeeper#zookeeper;3.4.6 from central in [default] org.codehaus.groovy#groovy-all;2.4.4 from central in [default] org.codehaus.jackson#jackson-core-asl;1.9.13 from central in [default] org.codehaus.jackson#jackson-jaxrs;1.9.13 from central in [default] org.codehaus.jackson#jackson-mapper-asl;1.9.13 from central in [default] org.codehaus.jackson#jackson-xc;1.9.13 from central in [default] org.codehaus.janino#commons-compiler;2.7.6 from central in [default] org.codehaus.janino#janino;2.7.6 from central in [default] org.codehaus.jettison#jettison;1.1 from central in [default] org.datanucleus#datanucleus-api-jdo;4.2.4 from central in [default] org.datanucleus#datanucleus-core;4.1.17 from central in [default] org.datanucleus#datanucleus-rdbms;4.1.19 from central in [default] org.datanucleus#javax.jdo;3.2.0-m3 from central in [default] org.eclipse.jetty.aggregate#jetty-all;7.6.0.v20120127 from central in [default] org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 from central in [default] org.fusesource.leveldbjni#leveldbjni-all;1.8 from central in [default] org.hamcrest#hamcrest-core;1.3 from central in [default] org.jruby.jcodings#jcodings;1.0.8 from central in [default] org.jruby.joni#joni;2.1.2 from central in [default] org.mortbay.jetty#jetty;6.1.26 from central in [default] org.mortbay.jetty#jetty-util;6.1.26 from central in [default] org.openjdk.jol#jol-core;0.2 from central in [default] org.slf4j#slf4j-api;1.7.36 from central in [default] org.slf4j#slf4j-log4j12;1.7.14 from central in [default] org.sonatype.sisu.inject#cglib;2.2.1-v20090111 from central in [default] org.tukaani#xz;1.5 from central in [default] org.xerial.snappy#snappy-java;1.1.8.2 from central in [default] stax#stax-api;1.0.1 from central in [default] tomcat#jasper-compiler;5.5.23 from central in [default] tomcat#jasper-runtime;5.5.23 from central in [default] xmlenc#xmlenc;0.52 from central in [default] :: evicted modules: org.slf4j#slf4j-api;1.7.10 by [org.slf4j#slf4j-api;1.7.36] in [default] org.slf4j#slf4j-log4j12;1.6.1 by [org.slf4j#slf4j-log4j12;1.7.14] in [default] log4j#log4j;1.2.16 by [log4j#log4j;1.2.17] in [default] commons-logging#commons-logging;1.1.3 by [commons-logging#commons-logging;1.2] in [default] asm#asm;3.1 by [asm#asm;3.2] in [default] io.dropwizard.metrics#metrics-core;3.1.2 by [io.dropwizard.metrics#metrics-core;3.1.0] in [default] org.xerial.snappy#snappy-java;1.1.1.3 by [org.xerial.snappy#snappy-java;1.1.8.2] in [default] com.google.code.findbugs#jsr305;3.0.0 by [com.google.code.findbugs#jsr305;3.0.2] in [default] javax.servlet#servlet-api;2.4 by [javax.servlet#servlet-api;2.5] in [default] commons-logging#commons-logging;1.0.3 by [commons-logging#commons-logging;1.2] in [default] org.apache.hadoop#hadoop-auth;2.5.1 by [org.apache.hadoop#hadoop-auth;2.7.2] in [default] io.netty#netty;3.6.2.Final by [io.netty#netty;3.7.0.Final] in [default] com.google.code.findbugs#jsr305;2.0.1 by [com.google.code.findbugs#jsr305;3.0.0] in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 176 | 168 | 168 | 13 || 164 | 164 | --------------------------------------------------------------------- :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS :: retrieving :: org.apache.spark#spark-submit-parent-d0d2962d-ae27-4526-a0c7-040a542e1e54 confs: [default] 164 artifacts copied, 0 already retrieved (195815kB/314ms) +--------------------+ | namespace| +--------------------+ | a1| | a1_dev| | a2| | access| | access_xzx01| | access_xzx02| | activity| | ...| +--------------------+ only showing top 20 rows scala> ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42599 from pan3793/unshare-guava. Authored-by: Cheng Pan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? Documentation update for java version and onpen JDK for spark 4.0.0 ### Why are the changes needed? spark 4.0.0 release ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? No testing required. ### Was this patch authored or co-authored using generative AI tooling? NO Closes #44589 from engrravijain/SPARK-46412. Authored-by: Ravi Jain <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

… 4.3.2 in AppVeyor" This reverts commit dbdeecb.

…esTests` into small test files ### What changes were proposed in this pull request? Break the remaining part of `IndexesTests` into small test files ### Why are the changes needed? testing parallelism ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #44588 from zhengruifeng/ps_test_idx_base_lastlast. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

… save mode ### What changes were proposed in this pull request? This PR renames the error class `_LEGACY_ERROR_TEMP_1308` to `UNSUPPORTED_DATA_SOURCE_SAVE_MODE` and improves its error messages. ### Why are the changes needed? To make the error more user-friendly. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New unit tests ### Was this patch authored or co-authored using generative AI tooling? No Closes #44576 from allisonwang-db/spark-46576-unsupported-save-mode. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Max Gekk <[email protected]>

…file separator to correctly check PySpark library existence ### What changes were proposed in this pull request? This PR is a followup of #44519 that fixes a mistake of separating the paths. It should use `Files.pathSeparator`. ### Why are the changes needed? It works with testing mode, but it doesn't work with production mode otherwise. ### Does this PR introduce _any_ user-facing change? No, because the main change has not been released. ### How was this patch tested? Manually as described in "How was this patch tested?" at #44504. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44590 from HyukjinKwon/SPARK-46530-followup. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? Fix XSD type conversion for some big integer types in XSDToSchema helper utility. NOTE: This is a deviation from spark-xml. ### Why are the changes needed? To correctly map XSD data types to spark data types ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? Unit test ### Was this patch authored or co-authored using generative AI tooling? No Closes #44587 from sandip-db/xml-xsd-datatype. Authored-by: Sandip Agarwala <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…le.ArraySeq` instead of use `mutable.ArraySeq.toSeq` in `Executor` ### What changes were proposed in this pull request? The following code exists in branch-3.5: https://github.com/apache/spark/blob/fb90ade2c7390077d2755fc43b73e63f5cf44f21/core/src/main/scala/org/apache/spark/executor/Executor.scala#L744-L749 From the code comments, when using Scala 2.12, `metricsPoller.getTaskMetricPeaks(taskId)` is wrapped as `WrappedArray` to avoid collection copying, and the subsequent `metricPeaks.toSeq` is a redundant collection conversion for Scala 2.12. However, for Scala 2.13, if `metricsPoller.getTaskMetricPeaks(taskId)` is still wrapped as `WrappedArray/mutable.ArraySeq`, it is impossible to avoid collection conversion, because the subsequent `metricPeaks.toSeq` will trigger a collection copy when using Scala 2.13. So this pr changes the process to directly wrap `metricPeaks` as `immutable.ArraySeq` to ensure the same effect as when using Scala 2.12, which is also more in line with the original comment description. ### Why are the changes needed? Avoid unnecessary collection copying when using Scala 2.13. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #44586 from LuciferYang/executor-metricPeaks. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: yangjie01 <[email protected]>

…afe since JDK9 ### What changes were proposed in this pull request? This PR propose to remove `ThreadLocal` due to `SecureRandom` is thread safe since JDK9. ### Why are the changes needed? Spark already upgraded the JDK version to 17. Before JDK9, we use `ThreadLocal` to keep the thread safe for `SecureRandom`. There are some javadoc of `SecureRandom` since JDK9 show below. ![屏幕快照 2024-01-03 下午8 11 40](https://github.com/apache/spark/assets/8486025/5699624a-6aa4-4172-a07a-11cf7a3272d8) Now, we can remove `ThreadLocal` due to `SecureRandom` is thread safe since JDK9. ### Does this PR introduce _any_ user-facing change? 'No'. ### How was this patch tested? GA tests. ### Was this patch authored or co-authored using generative AI tooling? 'No'. Closes #44577 from beliefer/SPARK-46569. Authored-by: beliefer <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? This (PR) introduces the utilization of fractions instead of slots, which is similar to the CPU strategy, for determining whether a worker offer can provide the necessary resources to tasks. For instance, when an executor reports to the driver with [gpu, ["1,", "2"]], the driver constructs an executor data map. The keys in the map represent the GPU addresses, and their default values are set to 1.0, indicating one whole GPU. Consequently, the available resource amounts for the executor are as follows: { "1" -> 1.0f, "2" -> 1.0f }. When offering resources to a task that requires 1 CPU and 0.08 GPU, the worker offer examines the available resource amounts. It identifies that the capacity of GPU address "1.0" is greater than the task's GPU requirement (1.0 >= 0.08). Therefore, Spark assigns the GPU address "1" to this task. After the assignment, the available resource amounts for this executor are updated to { "1" -> 0.92, "2" -> 1.0}, ensuring that the remaining resources can be allocated to other tasks. In scenarios where other tasks, using different task resource profiles, request varying GPU amounts when dynamic allocation is disabled, Spark applies the same comparison approach. It compares the task's GPU requirement with the available resource amounts to determine if the resources can be assigned to the task. ### Why are the changes needed? The existing resources offering including gpu, fpga is based on "slots per address", which is defined by the default resource profile. and it's a fixed number for all different resource profiles when dynamic allcation is disabled. Consider the below test case, ``` scala withTempDir { dir => val scriptPath = createTempScriptWithExpectedOutput(dir, "gpuDiscoveryScript", """{"name": "gpu","addresses":["0"]}""") val conf = new SparkConf() .setAppName("test") .setMaster("local-cluster[1, 12, 1024]") .set("spark.executor.cores", "12") conf.set("spark.worker.resource.gpu.amount", "1") conf.set("spark.worker.resource.gpu.discoveryScript", scriptPath) conf.set("spark.executor.resource.gpu.amount", "1") conf.set("spark.task.resource.gpu.amount", "0.08") sc = new SparkContext(conf) val rdd = sc.range(0, 100, 1, 4) var rdd1 = rdd.repartition(3) val treqs = new TaskResourceRequests().cpus(1).resource("gpu", 1.0) val rp = new ResourceProfileBuilder().require(treqs).build rdd1 = rdd1.withResources(rp) assert(rdd1.collect().size === 100) } ``` During the initial stages, Spark generates a default resource profile based on the configurations. The calculation for determining the slots per GPU address is performed as "spark.executor.resource.gpu.amount / spark.task.resource.gpu.amount", resulting in a value of 12 (1/0.08 = 12). This means that Spark can accommodate up to 12 tasks running on each GPU address simultaneously. The job is then divided into two stages. The first stage, which consists of 4 tasks, runs concurrently based on the default resource profile. However, the second stage, comprising 3 tasks, runs sequentially using a new task resource profile. This new profile specifies that each task requires 1 CPU and 1.0 full GPU. In reality, the tasks in the second stage are running in parallel, which is the underlying issue. The problem lies in the line `new TaskResourceRequests().cpus(1).resource("gpu", 1.0)`. The value of 1.0 for the GPU, or any value below 1.0 (specifically, (0, 0.5] which is rounded up to 1.0, spark throws an exception if the value is in (0.5, 1)), is merely requesting the number of slots. In this case, it is requesting only 1 slot. Consequently, each task necessitates 1 CPU core and 1 GPU slot, resulting in all tasks running simultaneously. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? To ensure all tests got passed ### Was this patch authored or co-authored using generative AI tooling? No Closes #43494 from wbo4958/SPARK-45527. Authored-by: Bobby Wang <[email protected]> Signed-off-by: Thomas Graves <[email protected]>

…f ORA-04021 ### What changes were proposed in this pull request? This PR mounts `docker-entrypoint-initdb.d` to OracleDocker to set DDL_LOCK_TIMEOUT and retry `alter user system` to ensure the password of SYSTEM user is correctly set. ### Why are the changes needed? https://github.com/apache/spark/actions/runs/7405337059/job/20148144542 https://github.com/apache/spark/actions/runs/7403889993/job/20144539155 https://github.com/apache/spark/actions/runs/7401841987/job/20138483574 Fix test flakiness caused by ORA-04021 ``` 22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: CONTAINER: Resetting SYS and SYSTEM passwords. 24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: 24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: User altered. 24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: 24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ALTER USER SYSTEM IDENTIFIED BY "Th1s1sThe0racle#Pass" 24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: * 24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ERROR at line 1: 24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ORA-65048: error encountered when processing the current DDL statement in 24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: pluggable database FREEPDB1 24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ORA-04021: timeout occurred while waiting to lock object 24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: Help: https://docs.oracle.com/error-help/db/ora-65048/ 24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: 24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ``` ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? OracleIntegrationSuite ### Was this patch authored or co-authored using generative AI tooling? no Closes #44594 from yaooqinn/SPARK-46592. Authored-by: Kent Yao <[email protected]> Signed-off-by: Max Gekk <[email protected]>

…f XXIntegrationSuite in `docker-integration-tests` module ### What changes were proposed in this pull request? The class descriptions of the test cases in the `docker-integration-tests` module provide suggested commands for executing the test cases, but these commands will search for corresponding test cases in all modules. This pr adds the module name `docker-integration-tests` to these commands, so that the test commands only search for corresponding test cases in the `docker-integration-tests` module. ### Why are the changes needed? Refine docs. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? I have manually checked the new command locally. ### Was this patch authored or co-authored using generative AI tooling? No Closes #44597 from LuciferYang/docker-integration-tests-cmd. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Max Gekk <[email protected]>

### What changes were proposed in this pull request? The package name for `SslTestUtils` is defined as `org.apache.spark`, but it is in the `org/apache/spark/util/` directory. This pr corrects its package name to ensure consistency. ### Why are the changes needed? Correct package name of `SslTestUtils` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #44596 from LuciferYang/SslTestUtils-package-name. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Max Gekk <[email protected]>

### What changes were proposed in this pull request? Refactor `data_type_ops` tests: deduplicate the `OpsTestBase` ### Why are the changes needed? make `OpsTestBase` reusable and reuse it in the parity tests ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #44592 from zhengruifeng/ps_test_data_type_ops. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

…S against golden files with other DBMS, starting with Postgres ### What changes were proposed in this pull request? Create `CrossDbmsQueryTestSuite`, which extends `SQLQueryTestHelper` and `DockerIntegrationSuite`. `CrossDbmsQueryTestSuite` is a trait class that allows testing golden files against other DBMS. `PostgreSQLQueryTestSuite` is an implementation of `CrossDbmsQueryTestSuite`. For starters, sql files in the subquery sql-tests are automatically opted into this test. In this PR, all files except for `exists-having.sql` are ignored, otherwise this PR would have 10K+ line changes (I would like to do that in the next PR, if possible). I had to change the syntax for view creation in `exists-having.sql` slightly, and this is reflected in the `analyzer-results` file, but crucially, the query output (in the `results` file) are not changed. Note that this will not be applicable to many of the current sql tests we have due to: - Incompatible SQL syntax between spark sql and postgres. - Incompatible data types. - Difference in numerical precision with doubles. - Missing functions in either system. - Test files with specific configs, such as ANSI, count bug etc. ### Why are the changes needed? For correctness checking of our SQLQueryTestSuites, we want to run SQLQueryTestSuites with Postgres as a reference DBMS. This can be easily extensible to other DBMS. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This is a test-related PR, does not affect system behaviors. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44084 from andylam-db/crossdbms. Authored-by: Andy Lam <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…ionCls` in `MapObjects` ### What changes were proposed in this pull request? The pr aims to support `s.c.immutable.ArraySeq` as `customCollectionCls` in `MapObjects`. ### Why are the changes needed? Because `s.c.immutable.ArraySeq` is a commonly used type in Scala 2.13, we should support it. ### Does this PR introduce _any_ user-facing change? Yes, We support `s.c.immutable.ArraySeq` in `MapObjects`. ### How was this patch tested? - Add new UT: Added a new test for `ArraySeq` in UDFSuite; Also updated `ObjectExpressionsSuite` for `MapObjects`. - Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44591 from panbingkun/SPARK-46586. Lead-authored-by: panbingkun <[email protected]> Co-authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? The pr aims to refine docstring of `reverse/map_contains_key`. ### Why are the changes needed? To improve PySpark documentation. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44561 from panbingkun/SPARK-46560. Lead-authored-by: panbingkun <[email protected]> Co-authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…from_entries/map_concat` ### What changes were proposed in this pull request? This pr refine docstring of `map_from_arrays/map_from_entries/map_concat` and add some new examples. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass Github Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #44595 from LuciferYang/SPARK-46595. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>

### What changes were proposed in this pull request? revert #44592 which happened to disable the parity tests they should be run on connect mode, but actually run on vanilla mode Closes #44605 from zhengruifeng/revert_46593. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

…ive dictionary ### What changes were proposed in this pull request? This PR updates the `options` field to use a case-insensitive dictionary to keep the behavior consistent with the Scala side (which uses `CaseInsensitiveStringMap`). Currently, `options` are stored in a normal Python dictionary which can be confusing to users. For instance: ```python class MyDataSource(DataSource): def __init__(self, options): self.api_key = options.get("API_KEY") # <- This is None spark.read.format(..).option("API_KEY", my_key).load(...) ``` Here, `options` will not have this "API_KEY" as everything is converted to lowercase on the Scala side. ### Why are the changes needed? To improve usability. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New unit tests ### Was this patch authored or co-authored using generative AI tooling? No Closes #44564 from allisonwang-db/spark-46568-ds-options. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…reparation sql ### What changes were proposed in this pull request? `semicolon` is not able to break SQLs in a file into separated statements ### Why are the changes needed? a followup for SPARK-46592 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Check the Oracle docker log #### Before ``` STDOUT: CONTAINER: running /docker-entrypoint-initdb.d/install.sql ... STDOUT: ALTER SESSION SET DDL_LOCK_TIMEOUT = 30;ALTER USER SYSTEM IDENTIFIED BY "Th1s1sThe0racle#Pass" STDOUT: * STDOUT: ERROR at line 1: STDOUT: ORA-00922: missing or invalid option STDOUT: Help: https://docs.oracle.com/error-help/db/ora-00922/ ``` #### After ``` STDOUT: CONTAINER: DONE: running /docker-entrypoint-initdb.d/install.sql STDOUT: Session altered. STDOUT: User altered. ``` ### Was this patch authored or co-authored using generative AI tooling? no Closes #44609 from yaooqinn/SPARK-46592-F. Authored-by: Kent Yao <[email protected]> Signed-off-by: Max Gekk <[email protected]>

…ead pool ### What changes were proposed in this pull request? This PR aims to use a meaningful class name prefix for REST Submission API thread pool instead of the default value of Jetty QueuedThreadPool, `"qtp"+super.hashCode()`. https://github.com/dekellum/jetty/blob/3dc0120d573816de7d6a83e2d6a97035288bdd4a/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L64 ### Why are the changes needed? This is helpful during JVM investigation. **BEFORE (4.0.0-preview2)** ``` $ SPARK_MASTER_OPTS='-Dspark.master.rest.enabled=true' sbin/start-master.sh $ jstack 28217 | grep qtp "qtp1925630411-52" #52 daemon prio=5 os_prio=31 cpu=0.07ms elapsed=19.06s tid=0x0000000134906c10 nid=0xde03 runnable [0x0000000314592000] "qtp1925630411-53" #53 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=19.06s tid=0x0000000134ac6810 nid=0xc603 runnable [0x000000031479e000] "qtp1925630411-54" #54 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=19.06s tid=0x000000013491ae10 nid=0xdc03 runnable [0x00000003149aa000] "qtp1925630411-55" #55 daemon prio=5 os_prio=31 cpu=0.08ms elapsed=19.06s tid=0x0000000134ac9810 nid=0xc803 runnable [0x0000000314bb6000] "qtp1925630411-56" #56 daemon prio=5 os_prio=31 cpu=0.04ms elapsed=19.06s tid=0x0000000134ac9e10 nid=0xda03 runnable [0x0000000314dc2000] "qtp1925630411-57" #57 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=19.06s tid=0x0000000134aca410 nid=0xca03 runnable [0x0000000314fce000] "qtp1925630411-58" #58 daemon prio=5 os_prio=31 cpu=0.04ms elapsed=19.06s tid=0x0000000134acaa10 nid=0xcb03 runnable [0x00000003151da000] "qtp1925630411-59" #59 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=19.06s tid=0x0000000134acb010 nid=0xcc03 runnable [0x00000003153e6000] "qtp1925630411-60-acceptor-0108e9815-ServerConnector1e497474{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #60 daemon prio=3 os_prio=31 cpu=0.11ms elapsed=19.06s tid=0x00000001317ffa10 nid=0xcd03 runnable [0x00000003155f2000] "qtp1925630411-61-acceptor-11d90f2aa-ServerConnector1e497474{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #61 daemon prio=3 os_prio=31 cpu=0.10ms elapsed=19.06s tid=0x00000001314ed610 nid=0xcf03 waiting on condition [0x00000003157fe000] ``` **AFTER** ``` $ SPARK_MASTER_OPTS='-Dspark.master.rest.enabled=true' sbin/start-master.sh $ jstack 28317 | grep StandaloneRestServer "StandaloneRestServer-52" #52 daemon prio=5 os_prio=31 cpu=0.09ms elapsed=60.06s tid=0x00000001284a8e10 nid=0xdb03 runnable [0x000000032cfce000] "StandaloneRestServer-53" #53 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=60.06s tid=0x00000001284acc10 nid=0xda03 runnable [0x000000032d1da000] "StandaloneRestServer-54" #54 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=60.06s tid=0x00000001284ae610 nid=0xd803 runnable [0x000000032d3e6000] "StandaloneRestServer-55" #55 daemon prio=5 os_prio=31 cpu=0.09ms elapsed=60.06s tid=0x00000001284aec10 nid=0xd703 runnable [0x000000032d5f2000] "StandaloneRestServer-56" #56 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=60.06s tid=0x00000001284af210 nid=0xc803 runnable [0x000000032d7fe000] "StandaloneRestServer-57" #57 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=60.06s tid=0x00000001284af810 nid=0xc903 runnable [0x000000032da0a000] "StandaloneRestServer-58" #58 daemon prio=5 os_prio=31 cpu=0.06ms elapsed=60.06s tid=0x00000001284afe10 nid=0xcb03 runnable [0x000000032dc16000] "StandaloneRestServer-59" #59 daemon prio=5 os_prio=31 cpu=0.05ms elapsed=60.06s tid=0x00000001284b0410 nid=0xcc03 runnable [0x000000032de22000] "StandaloneRestServer-60-acceptor-04aefbaa8-ServerConnector44284d85{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #60 daemon prio=3 os_prio=31 cpu=0.13ms elapsed=60.05s tid=0x000000015cda1a10 nid=0xcd03 runnable [0x000000032e02e000] "StandaloneRestServer-61-acceptor-148976251-ServerConnector44284d85{HTTP/1.1, (http/1.1)}{M3-Max.local:6066}" #61 daemon prio=3 os_prio=31 cpu=0.12ms elapsed=60.05s tid=0x000000015cd1c810 nid=0xce03 waiting on condition [0x000000032e23a000] ``` ### Does this PR introduce _any_ user-facing change? No, the thread names are accessed during the debugging. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48924 from dongjoon-hyun/SPARK-50385. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: panbingkun <[email protected]>

…ingBuilder` ### What changes were proposed in this pull request? This PR aims to improve `toString` by `JEP-280` instead of `ToStringBuilder`. In addition, `Scalastyle` and `Checkstyle` rules are added to prevent a future regression. ### Why are the changes needed? Since Java 9, `String Concatenation` has been handled better by default. | ID | DESCRIPTION | | - | - | | JEP-280 | [Indify String Concatenation](https://openjdk.org/jeps/280) | For example, this PR improves `OpenBlocks` like the following. Both Java source code and byte code are simplified a lot by utilizing JEP-280 properly. **CODE CHANGE** ```java - return new ToStringBuilder(this, ToStringStyle.SHORT_PREFIX_STYLE) - .append("appId", appId) - .append("execId", execId) - .append("blockIds", Arrays.toString(blockIds)) - .toString(); + return "OpenBlocks[appId=" + appId + ",execId=" + execId + ",blockIds=" + + Arrays.toString(blockIds) + "]"; ``` **BEFORE** ``` public java.lang.String toString(); Code: 0: new #39 // class org/apache/commons/lang3/builder/ToStringBuilder 3: dup 4: aload_0 5: getstatic #41 // Field org/apache/commons/lang3/builder/ToStringStyle.SHORT_PREFIX_STYLE:Lorg/apache/commons/lang3/builder/ToStringStyle; 8: invokespecial #47 // Method org/apache/commons/lang3/builder/ToStringBuilder."<init>":(Ljava/lang/Object;Lorg/apache/commons/lang3/builder/ToStringStyle;)V 11: ldc #50 // String appId 13: aload_0 14: getfield #7 // Field appId:Ljava/lang/String; 17: invokevirtual #51 // Method org/apache/commons/lang3/builder/ToStringBuilder.append:(Ljava/lang/String;Ljava/lang/Object;)Lorg/apache/commons/lang3/builder/ToStringBuilder; 20: ldc #55 // String execId 22: aload_0 23: getfield #13 // Field execId:Ljava/lang/String; 26: invokevirtual #51 // Method org/apache/commons/lang3/builder/ToStringBuilder.append:(Ljava/lang/String;Ljava/lang/Object;)Lorg/apache/commons/lang3/builder/ToStringBuilder; 29: ldc #56 // String blockIds 31: aload_0 32: getfield #16 // Field blockIds:[Ljava/lang/String; 35: invokestatic #57 // Method java/util/Arrays.toString:([Ljava/lang/Object;)Ljava/lang/String; 38: invokevirtual #51 // Method org/apache/commons/lang3/builder/ToStringBuilder.append:(Ljava/lang/String;Ljava/lang/Object;)Lorg/apache/commons/lang3/builder/ToStringBuilder; 41: invokevirtual #61 // Method org/apache/commons/lang3/builder/ToStringBuilder.toString:()Ljava/lang/String; 44: areturn ``` **AFTER** ``` public java.lang.String toString(); Code: 0: aload_0 1: getfield #7 // Field appId:Ljava/lang/String; 4: aload_0 5: getfield #13 // Field execId:Ljava/lang/String; 8: aload_0 9: getfield #16 // Field blockIds:[Ljava/lang/String; 12: invokestatic #39 // Method java/util/Arrays.toString:([Ljava/lang/Object;)Ljava/lang/String; 15: invokedynamic #43, 0 // InvokeDynamic #0:makeConcatWithConstants:(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String; 20: areturn ``` ### Does this PR introduce _any_ user-facing change? No. This is an `toString` implementation improvement. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#51572 from dongjoon-hyun/SPARK-52880. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

cloud-fan and others added 9 commits January 4, 2024 11:59

Revert "[SPARK-46582][R][INFRA] Upgrade R Tools version from 4.0.2 to…

1cd3a1b

… 4.3.2 in AppVeyor" This reverts commit dbdeecb.

github-actions bot added BUILD CORE SQL DOCS PYTHON PANDAS API ON SPARK STRUCTURED STREAMING KUBERNETES labels Jan 4, 2024

pull bot added ⤵️ pull and removed BUILD CORE SQL DOCS PYTHON PANDAS API ON SPARK STRUCTURED STREAMING KUBERNETES labels Jan 4, 2024

github-actions bot added BUILD CORE SQL labels Jan 4, 2024

github-actions bot added DOCS PYTHON PANDAS API ON SPARK STRUCTURED STREAMING KUBERNETES labels Jan 4, 2024

wbo4958 and others added 12 commits January 4, 2024 09:19

pull bot merged commit 70b90c8 into huangxiaopingRD:master Jan 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[pull] master from apache:master #56

[pull] master from apache:master #56

Uh oh!

pull bot commented Jan 4, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

[pull] master from apache:master #56

[pull] master from apache:master #56

Uh oh!

Conversation

pull bot commented Jan 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

pull bot commented Jan 4, 2024 •

edited

Loading