Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,6 @@ install:

script:
- mvn test -B

arch:
- ppc64le
10 changes: 10 additions & 0 deletions bigtop-packages/src/common/zeppelin/do-component-build
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,16 @@ BUILD_OPTS="-Dhadoop3.2.version=${HADOOP_VERSION} \
-Phadoop3 \
-Pbuild-distr"

if [ $HOSTTYPE = "powerpc64le" ] ; then
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.5.0 \
-Dclassifier=linux-ppcle_64 -Dpackaging=exe -Dfile=/usr/local/protobuf-3.5.1.1/bin/protoc

mvn install:install-file -DgroupId=io.grpc -DartifactId=protoc-gen-grpc-java -Dversion=1.28.0 -Dclassifier=linux-ppcle_64 -Dpackaging=exe -Dfile=/usr/src/grpc-java-1.28.0/compiler/build/exe/java_plugin/protoc-gen-grpc-java
sed -i "s|<node.version>v12.3.1</node.version>|<node.version>v12.22.1</node.version>|" pom.xml
sed -i "s|<npm.version>6.9.0</npm.version>|<npm.version>6.14.12</npm.version>|" pom.xml

fi

if [ "$ZEPPELIN_RUN_TESTS" = "true" ]; then
BUILD_OPTS="$BUILD_OPTS -DskipTests=false"
else
Expand Down
25 changes: 25 additions & 0 deletions bigtop-packages/src/common/zeppelin/patch2-ppc64le.diff
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
diff --git a/zeppelin-web/package.json b/zeppelin-web/package.json
index de83a629d..619a04d0c 100644
--- a/zeppelin-web/package.json
+++ b/zeppelin-web/package.json
@@ -7,7 +7,7 @@
},
"scripts": {
"clean": "rimraf dist && rimraf .tmp",
- "postinstall": "bower install --silent",
+ "postinstall": "bower install --silent --allow-root",
"prebuild": "npm-run-all clean lint:once",
"build:dist": "npm-run-all prebuild && grunt pre-webpack-dist && webpack && grunt post-webpack-dist",
"build:ci": "npm-run-all prebuild && grunt pre-webpack-ci && webpack && grunt post-webpack-dist",
diff --git a/zeppelin-web/pom.xml b/zeppelin-web/pom.xml
index 54e97053d..2433addd5 100644
--- a/zeppelin-web/pom.xml
+++ b/zeppelin-web/pom.xml
@@ -83,7 +83,7 @@
</goals>
<configuration>
<skip>${web.e2e.enabled}</skip>
- <arguments>install --no-lockfile</arguments>
+ <arguments>install --no-lockfile --unsafe-perm</arguments>
</configuration>
</execution>
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
diff --git a/compiler/build.gradle b/compiler/build.gradle
index 60d3a43..55acee0 100644
--- a/compiler/build.gradle
+++ b/compiler/build.gradle
@@ -105,7 +105,6 @@ model {
// Link other (system) libraries dynamically.
// Clang under OSX doesn't support these options.
linker.args "-Wl,-Bstatic", "-lprotoc", "-lprotobuf", "-static-libgcc",
- "-static-libstdc++",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of removing this, adding libstdc++-static to the packages to be installed seemed to resolve the compiling issue on CentOS 7.
sekikn@b94539c#diff-9337fb34935804ab7b56cd89ae94e14491ad4c64804daa9b1843eeec336772edR67

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure that would work on Ubuntu. But the fix is a backport of an upstream patch grpc/grpc-java#8141

"-Wl,-Bdynamic", "-lpthread", "-s"
}
addEnvArgs("LDFLAGS", linker.args)
47 changes: 47 additions & 0 deletions bigtop_toolchain/manifests/grpc.pp
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

class bigtop_toolchain::grpc {

require bigtop_toolchain::jdk
require bigtop_toolchain::protobuf

$grpc_version = '1.28.0'
$proto_version = '3.5.1.1'
$proto_home = "/usr/local/protobuf-3.5.1.1"

if ($architecture == 'ppc64le') {

exec { "download grpc-java ${grpc_version}":
cwd => "/usr/src",
command => "/usr/bin/wget https://github.com/grpc/grpc-java/archive/refs/tags/v${grpc_version}.tar.gz && mkdir -p grpc-java-${grpc_version} && /bin/tar -xvzf v${grpc_version}.tar.gz -C grpc-java-${grpc_version} --strip-components=1",
creates => "/usr/src/grpc-java-${grpc_version}",
}

file { "/usr/src/grpc-java-${grpc_version}/grpc-java-${grpc_version}-add-support-for-ppc64le.patch":
source => "puppet:///modules/bigtop_toolchain/grpc-java-${grpc_version}-add-support-for-ppc64le.patch",
require => Exec["download grpc-java ${grpc_version}"],
}

exec { "build grpc-java ${grpc_version}":
cwd => "/usr/src/grpc-java-${grpc_version}",
command => "/usr/bin/patch -p1 < grpc-java-${grpc_version}-add-support-for-ppc64le.patch && export LDFLAGS=-L/${proto_home}/lib && export CXXFLAGS=-I/${proto_home}/include && export LD_LIBRARY_PATH=/${proto_home}/lib && cd compiler && ../gradlew java_pluginExecutable -PskipAndroid=true",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think setting environment variables in the environment attribute is better for code simplicity.
sekikn@b94539c#diff-f2963d87c19d71d34863fb0039c7b3d3d0e3699b77bfd7bd57009ca568f51da6R78

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I am not a PP expert. My first time, I used https://github.com/apache/bigtop/blob/master/bigtop_toolchain/manifests/gradle.pp as a guide. Let me know if you need me to change it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh got it:
environment => ['CXXFLAGS=-I/usr/local/protobuf-3.5.1.1/include', 'LDFLAGS=-L/usr/local/protobuf-3.5.1.1/lib'],

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sekikn, were you able to try to run the PR with all distros? If not, let me know. I'll try to do it over the weekend. Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@afsanjar Sorry, I couldn't have enough time to do it. It's really helpful if you could try it on this weekend!

creates => "/usr/local/grpc-java-${grpc_version}",
require => File["/usr/src/grpc-java-${grpc_version}/grpc-java-${grpc_version}-add-support-for-ppc64le.patch"],
timeout => 3000
}
}

}
1 change: 1 addition & 0 deletions bigtop_toolchain/manifests/installer.pp
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
include bigtop_toolchain::env
include bigtop_toolchain::user
include bigtop_toolchain::renv
include bigtop_toolchain::grpc

stage { 'last':
require => Stage['main'],
Expand Down