Skip to content
This repository was archived by the owner on Nov 6, 2020. It is now read-only.

snap: reenable i386, arm64, armhf architecture publishing #10386

Merged
General-Beck merged 16 commits into from
Feb 25, 2019
+57 −37
Merged

snap: reenable i386, arm64, armhf architecture publishing #10386

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
61f5e1e
snap: reenable i386, arm64, armhf architecture publishing
5chdn Feb 20, 2019
02116dd
gitlab: fix indent
5chdn Feb 20, 2019
46eefcb
gitlab: fix yml syntax
5chdn Feb 20, 2019
ba4ca7b
Linker for crosscomile
General-Beck Feb 20, 2019
66d003b
fix target to linker
General-Beck Feb 20, 2019
76a14d7
new docker image
General-Beck Feb 20, 2019
62fd60a
fix lint, add build to this PR
General-Beck Feb 20, 2019
59fb257
calc SHA3 using rhash
General-Beck Feb 20, 2019
655f925
add new images for i386, armhf
General-Beck Feb 20, 2019
da071cb
show snap target & artifacts
General-Beck Feb 20, 2019
6c0fff2
set CARGO_TARGET for publish snap
General-Beck Feb 20, 2019
15a88da
move detect Version to publish snap
General-Beck Feb 21, 2019
eae7ef1
rm libc6 dep from snap-template up pub-snap script
General-Beck Feb 21, 2019
4f0de83
Merge branch 'master' into a5-snap-archs
TriplEight Feb 22, 2019
e3a7364
clean up cargo config before add linker
General-Beck Feb 22, 2019
33e7a12
move linker config to docker images
General-Beck Feb 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 43 additions & 16 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ stages:
- publish
- optional

image: parity/rust:gitlab-ci

image: parity/rust-parity-ethereum-build:xenial

@soc1c soc1c Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the logic behind changing this now?

@TriplEight TriplEight Feb 22, 2019

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this docker image has been tested,
it's the preparation for using cache in docker volumes

@General-Beck General-Beck Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new CI images

@gabreal gabreal Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would prefer a separate pr for that, just in case one has to revert.

variables:
GIT_STRATEGY: fetch
GIT_SUBMODULE_STRATEGY: recursive
Expand Down Expand Up @@ -35,14 +34,6 @@ variables:
paths:
- artifacts/

.determine_version: &determine_version
- VERSION="$(sed -r -n '1,/^version/s/^version = "([^"]+)".*$/\1/p' Cargo.toml)"

@soc1c soc1c Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this removed?

@TriplEight TriplEight Feb 22, 2019

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was used only in publish-snap.sh so moved there.
Is there a reason for it to stay here?

@General-Beck General-Beck Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used only for publish snap

- DATE_STR="$(date +%Y%m%d)"
- ID_SHORT="$(echo ${CI_COMMIT_SHA} | cut -c 1-7)"
- test "${SCHEDULE_TAG:-${CI_COMMIT_REF_NAME}}" = "nightly" && VERSION="${VERSION}-${ID_SHORT}-${DATE_STR}"
- export VERSION
- echo "Version = ${VERSION}"

test-linux:
stage: test
variables:
Expand All @@ -62,7 +53,7 @@ test-audit:
tags:
- linux-docker

build-linux:
build-linux: &build-linux
stage: build
only: *releaseable_branches
script:
Expand All @@ -72,6 +63,24 @@ build-linux:
tags:
- linux-docker

build-linux-i386:
<<: *build-linux
image: parity/rust-parity-ethereum-build:i386
variables:
CARGO_TARGET: i686-unknown-linux-gnu

build-linux-arm64:
<<: *build-linux
image: parity/rust-parity-ethereum-build:arm64
variables:
CARGO_TARGET: aarch64-unknown-linux-gnu

build-linux-armhf:
<<: *build-linux
image: parity/rust-parity-ethereum-build:armhf
variables:
CARGO_TARGET: armv7-unknown-linux-gnueabihf
Comment thread
TriplEight marked this conversation as resolved.

build-darwin:
stage: build
only: *releaseable_branches
Expand Down Expand Up @@ -108,24 +117,43 @@ publish-docker:
script:
- scripts/gitlab/publish-docker.sh parity

publish-snap:
publish-snap: &publish-snap
stage: publish
only: *releaseable_branches
# <<: *no_git
image: snapcore/snapcraft
variables:
BUILD_ARCH: amd64
cache: {}
before_script: *determine_version
dependencies:
- build-linux
tags:
- rust-stable
- linux-docker

@soc1c soc1c Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why linux-docker?

@General-Beck General-Beck Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean up the tags

script:
- scripts/gitlab/publish-snap.sh
allow_failure: true
<<: *collect_artifacts

publish-snap-i386:
<<: *publish-snap
variables:
BUILD_ARCH: i386
dependencies:
- build-linux-i386

publish-snap-arm64:
<<: *publish-snap
variables:
BUILD_ARCH: arm64
dependencies:
- build-linux-arm64

publish-snap-armhf:
<<: *publish-snap
variables:
BUILD_ARCH: armhf
dependencies:
- build-linux-armhf

publish-onchain:
stage: publish
only: *releaseable_branches
Expand Down Expand Up @@ -191,4 +219,3 @@ build-android:
- linux-docker
allow_failure: true
<<: *collect_artifacts

12 changes: 5 additions & 7 deletions scripts/gitlab/build-unix.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ echo "CC: " $CC
echo "CXX: " $CXX
#strip ON
export RUSTFLAGS=" -C link-arg=-s"
# Linker for crosscomile
echo "_____ Linker _____"
cat .cargo/config

echo "_____ Building target: "$CARGO_TARGET" _____"
if [ "${CARGO_TARGET}" = "armv7-linux-androideabi" ]
Expand Down Expand Up @@ -43,11 +46,6 @@ fi
echo "_____ Calculating checksums _____"
for binary in $(ls)
do
rhash --sha256 $binary -o $binary.sha256
if [ "${CARGO_TARGET}" = "armv7-linux-androideabi" ]
then
echo "> ${binary} cannot be hashed with cross-compiled binary"
else
./parity tools hash $binary > $binary.sha3
fi
rhash --sha256 $binary -o $binary.sha256 #do we still need this hash (SHA2)?

@soc1c soc1c Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHA256 is used for Vanity Service and Homebrew (Releases) And One-line Installer

@TriplEight TriplEight Feb 22, 2019

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

rhash --sha3-256 $binary -o $binary.sha3

@gabreal gabreal Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hashes are different, is that okay?

% target/release/parity tools hash target/release/parity
58b49924c583b45871cac4077da8f99820a684df8cca9a81c742080e991f671d
% rhash --sha3-256 target/release/parity
95fcba97a2059bbae16f997c3ea7b6c6c0f9ef5990313f55eb8536a1b3bef9e3  target/release/parity
% rhash --sha256 target/release/parity                
19c7ac31b99f5a45e6a4236367df18467321a3fc9f492c19f80cede098d47abf  target/release/parity

@General-Beck General-Beck Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can't use parity to calculate the hash for other architectures

done
21 changes: 8 additions & 13 deletions scripts/gitlab/publish-snap.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ set -u # treat unset variables as error
# gsub(/ /, "", $2) deletes whitespaces
TRACK=`awk -F '=' '/^track/ {gsub(/"/, "", $2); gsub(/ /, "", $2); print $2}' ./util/version/Cargo.toml`
echo Track is: $TRACK

# prepare variables
VERSION=v"$(sed -r -n '1,/^version/s/^version = "([^"]+)".*$/\1/p' Cargo.toml)"
SNAP_PACKAGE="parity_"$VERSION"_"$BUILD_ARCH".snap"
CARGO_TARGET="$(ls artifacts)"

@soc1c soc1c Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What? ls artifacts gives a reliable CARGO_TARGET?

@General-Beck General-Beck Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, from artifacts dir

# Choose snap release channel based on parity ethereum version track
case ${TRACK} in
nightly) export GRADE="devel" CHANNEL="edge";;
Expand All @@ -21,24 +24,16 @@ esac
case ${CI_COMMIT_REF_NAME} in
beta|stable) export GRADE="stable" CHANNEL="candidate";;
esac

VERSION="v"$VERSION
SNAP_PACKAGE="parity_"$VERSION"_"$BUILD_ARCH".snap"

echo "__________Create snap package__________"
echo "Release channel :" $GRADE " Branch/tag: " $CI_COMMIT_REF_NAME
echo $VERSION:$GRADE:$BUILD_ARCH
# cat scripts/snap/snapcraft.template.yaml | envsubst '$VERSION:$GRADE:$BUILD_ARCH:$CARGO_TARGET' > snapcraft.yaml
# a bit more necromancy (substitutions):
pwd
cd /builds/$CI_PROJECT_PATH/scripts/snap/
echo $VERSION:$GRADE:$BUILD_ARCH:$CARGO_TARGET

sed -e 's/$VERSION/'"$VERSION"'/g' \
-e 's/$GRADE/'"$GRADE"'/g' \
-e 's/$BUILD_ARCH/'"$BUILD_ARCH"'/g' \
-e 's/$CARGO_TARGET/'"$CARGO_TARGET"'/g' \
snapcraft.template.yaml > /builds/$CI_PROJECT_PATH/snapcraft.yaml
cd /builds/$CI_PROJECT_PATH
pwd
scripts/snap/snapcraft.template.yaml > snapcraft.yaml

apt update
apt install -y --no-install-recommends rhash
cat snapcraft.yaml
Expand Down
2 changes: 1 addition & 1 deletion scripts/snap/snapcraft.template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ parts:
cp -v ethkey $SNAPCRAFT_PART_INSTALL/usr/bin/ethkey
cp -v ethstore $SNAPCRAFT_PART_INSTALL/usr/bin/ethstore
cp -v whisper $SNAPCRAFT_PART_INSTALL/usr/bin/whisper
stage-packages: [libc6, libudev1, libstdc++6, cmake, libdb5.3]

@soc1c soc1c Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we added libc6 because it was failling without. It's working now?

@TriplEight TriplEight Feb 22, 2019

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and it was the reason it failed here: https://github.com/paritytech/devops/issues/307

@General-Beck General-Beck Feb 22, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

stage-packages: [libudev1, libstdc++6, cmake, libdb5.3]