Skip to content

GitHub Actions #9514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bcardiff merged 7 commits into from
Jun 22, 2020
Merged

GitHub Actions #9514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
72bead3
GitHub Actions for Linux build
waj Jun 3, 2020
fc9745c
Added macOS workflow
waj Jun 12, 2020
7b84b25
Break spec output lines
waj Jun 19, 2020
15e2e4d
Group output on GitHub Actions
waj Jun 19, 2020
9183425
Alpine, MT and 32bit jobs for Linux CI
waj Jun 19, 2020
78d9c5a
Check format job
waj Jun 19, 2020
062d81d
Fix interpolation for GitHub log grouping
waj Jun 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions .github/workflows/linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
name: Linux CI

on: [push, pull_request]
Copy link
Member

@Blacksmoke16 Blacksmoke16 Jun 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this produce duplicate builds? I.e. one for the push and another for the PR?

Copy link
Member

@jhass jhass Jun 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No.

Copy link
Member

@oprypin oprypin Jun 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well it does produce duplicates in some sense, when core contributors push a branch to the main repo and then create a PR from that. It's possible to tell it to test only when the push is to master, but then you lose the ability to just push a branch just for the purpose of running CI on it.

So this is fine. It's been this way in win.yml since the start, anyway.


env:
TRAVIS_OS_NAME: linux
SPEC_SPLIT_DOTS: 160

jobs:
test_linux:
env:
ARCH: x86_64
ARCH_CMD: linux64
runs-on: ubuntu-latest
steps:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Prepare System
run: bin/ci prepare_system

- name: Prepare Build
run: bin/ci prepare_build

- name: Test
run: bin/ci build

test_linux_32:
env:
ARCH: i386
ARCH_CMD: linux32
runs-on: ubuntu-latest
steps:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Prepare System
run: bin/ci prepare_system

- name: Prepare Build
run: bin/ci prepare_build

- name: Test
run: bin/ci with_build_env 'make std_spec threads=1'
Copy link
Member

@jhass jhass Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of trying to run the compiler specs here too?

Copy link
Member Author

@waj waj Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compiler specs are executed later using the compiled compiler. But it takes a lot of time, so this is just a "fail fast" step to catch more common issues before that.

Copy link
Member

@jhass jhass Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note I'm talking about test_linux_32 specifically here. I don't see any other 32bit tests here?

Copy link
Member Author

@waj waj Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry! Yup, that's probably a good idea. I just transferred as literal as possible what's currently running on CircleCI, but I don't know why it's not running the whole suite with bin/ci build. @bcardiff do you know why? Resource limits maybe?

Copy link
Member

@bcardiff bcardiff Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason was that the resources were not enough to build the compiler in 32 bits environment. I doubt they will be able to succeed.

Ref: #8283

Copy link
Member

@jhass jhass Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean otoh, while things like 32bit ARM is still widely used, 32bit x86 is basically dead anyways. I guess we wouldn't be too terrible people if we just drop it as a platform when most Linux distributions and OSes already did so.

Copy link
Member

@bcardiff bcardiff Jun 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we have some other 32 bits platform, I will accept to drop 32bits x86 builds. If we drop it before I fear there will be more things broken on 32 bits due to 64 bit assumptions.

Maybe what we can do is to drop the 32 bits compiler and use cross-compilation to test 32 bits std-spec. If we do things that way it will be less things to maintain in the release process. And at the same time pave the way for something between tier 1 and tier 2 (or improve what is offered in tier 2).

Copy link
Member

@jhass jhass Jun 22, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pocked a bit at 32bit ARVMv7 support in the last couple of days and the current state is... not pretty.

  • Stdlib specs break on some random OpenSSL session handshake rejects that sound more like memory corruption than anything else, or perhaps another round of C ABI issues.
  • Atomic requires some more from compiler-rt on the platform and just linking with clang -rtlib=compiler-rt had me run into all kinds of linker issues.
  • compiler_spec just runs out of memory no matter what I do, that is not actual memory but what it can allocate via bdwgc, regardless of --enable-large-config.
  • Running an armhf docker container on an aarch64 host was so broken, not even date could do anything but display random values.
  • My attempt to virt-install --arch=armv7l on the aarch64 host would just have the VM hog a CPU and not even show a POST message via virsh console.
  • The icing on the cake is that non of my test setups would then even run into the LFS issues other people are seeing. So just setting up a CI somehow doesn't mean we're actually testing that part apparently...

So, that's some road ahead of us and will likely be in the Windows kind of state for a while where we just can run a whatever works right now CI.

Copy link
Member

@bcardiff bcardiff Jun 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bumpy road ahead 😞 .

Let's decouple the discussion of better 32 bits infra for after this PR then. Do you agree @jhass?

Copy link
Member

@jhass jhass Jun 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good.


test_alpine:
env:
ARCH: x86_64-musl
ARCH_CMD: linux64
runs-on: ubuntu-latest
steps:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Prepare System
run: bin/ci prepare_system

- name: Prepare Build
run: bin/ci prepare_build

- name: Test
run: bin/ci build

test_preview_mt:
env:
ARCH: x86_64
ARCH_CMD: linux64
runs-on: ubuntu-latest
steps:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Prepare System
run: bin/ci prepare_system

- name: Prepare Build
run: bin/ci prepare_build

- name: Make Crystal
run: bin/ci with_build_env 'make crystal'

- name: Test
run: bin/ci with_build_env 'CRYSTAL_WORKERS=4 make std_spec threads=1 FLAGS="-D preview_mt"'

check_format:
env:
ARCH: x86_64
ARCH_CMD: linux64
runs-on: ubuntu-latest
steps:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Prepare System
run: bin/ci prepare_system

- name: Prepare Build
run: bin/ci prepare_build

- name: Check Format
run: bin/ci format
25 changes: 25 additions & 0 deletions .github/workflows/macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: macOS CI

on: [push, pull_request]

env:
TRAVIS_OS_NAME: osx
LLVM_CONFIG: /usr/local/opt/llvm/bin/llvm-config
PKG_CONFIG_PATH: /usr/local/opt/openssl/lib/pkgconfig
SPEC_SPLIT_DOTS: 160

jobs:
test_macos:
runs-on: macos-latest
steps:
- name: Download Crystal source
uses: actions/checkout@v2

- name: Prepare System
run: bin/ci prepare_system

- name: Prepare Build
run: bin/ci prepare_build

- name: Test
run: bin/ci build
15 changes: 14 additions & 1 deletion bin/ci
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,15 @@ on_osx() {
fail_on_error on_os "osx" "${@}"
}

on_github() {
if [ "$GITHUB_ACTIONS" = "true" ]; then
eval "${@}"
return $?
else
return 0
fi
}

prepare_system() {
on_linux 'echo '"'"'{"ipv6":true, "fixed-cidr-v6":"2001:db8:1::/64"}'"'"' | sudo tee /etc/docker/daemon.json'
on_linux sudo service docker restart
Expand Down Expand Up @@ -112,7 +121,8 @@ prepare_build() {
on_osx 'pushd ~;gunzip -c ~/crystal.tar.gz | tar xopf -;mv crystal-0.34.0-1 crystal;popd'

on_osx 'brew unlink python@2 || true'
on_osx brew install z3 llvm@10 gmp libevent pcre openssl pkg-config
on_osx brew install z3 llvm@10 gmp libevent pcre pkg-config
on_osx brew reinstall openssl
on_osx brew link --force llvm@10
# Note: brew link --force might show:
# Warning: Refusing to link macOS-provided software: llvm
Expand All @@ -137,6 +147,7 @@ verify_version() {

with_build_env() {
command="$1"
on_github "echo '::group::$1'"

# Ensure non GMT timezone
export TZ="America/New_York"
Expand Down Expand Up @@ -165,6 +176,7 @@ with_build_env() {
-v /etc/group:/etc/group \
-w /mnt \
-e CRYSTAL_CACHE_DIR="/tmp/crystal" \
-e SPEC_SPLIT_DOTS \
"$DOCKER_TEST_IMAGE" \
"$ARCH_CMD" /bin/sh -c "'$command'"

Expand All @@ -174,6 +186,7 @@ with_build_env() {
CRYSTAL_CACHE_DIR="/tmp/crystal" \
/bin/sh -c "'$command'"

on_github echo "::endgroup::"
}

usage() {
Expand Down
20 changes: 20 additions & 0 deletions src/spec/formatter.cr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,31 @@ module Spec

# :nodoc:
class DotFormatter < Formatter
@count = 0
@split = 0

def initialize(*args)
super

if split = ENV["SPEC_SPLIT_DOTS"]?
@split = split.to_i
end
end

def report(result)
@io << Spec.color(LETTERS[result.kind], result.kind)
split_lines
@io.flush
end

private def split_lines
return unless @split > 0
if (@count += 1) >= @split
@io.puts
@count = 0
end
end

def finish(elapsed_time, aborted)
@io.puts
end
Expand Down