FreeBSD CI #20585

iblislin · 2017-02-12T15:06:18Z

Hi,
I build a freebsd ci: https://julia.iblis.cnmc.tw
I need someone help me to

add a webhook (with Content-Type set to json ) for listening push and pull request events to this endpoint: https://julia.iblis.cnmc.tw/change_hook/github
visit https://julia.iblis.cnmc.tw/wh to authorize the oauth repo:status scope.

and the checking will available like this: iblislin#3

The text was updated successfully, but these errors were encountered:

tkelman · 2017-02-12T20:16:44Z

Does it pass?

JeffBezanson · 2017-02-12T23:09:09Z

@iblis17 Thanks for working on this! It would be great to get this working. What sort of resources is this running on?

iblislin · 2017-02-13T01:04:55Z

@tkelman No... tons of test cases failed. please check out this https://julia.iblis.cnmc.tw/#/builders/1/builds/3

@JeffBezanson It is an old rack server, Dell R710, in my compus lab... I can use it until I finish my master degree (about one year later

ararslan · 2017-02-13T01:17:02Z

I can't connect to the URLs you've posted. Not sure why. Is that happening for anyone else?

@iblis17 What options have you enabled for building? Is this FreeBSD 11.0-RELEASE?

iblislin · 2017-02-13T01:42:48Z

I can't connect to the URLs you've posted. Not sure why. Is that happening for anyone else?

hmm, DNS issue (i use this domain for github webhook and works fine) ? or you can try this https://140.113.31.207/
I use the letsencrypt's SSL certificate, will it cause connection problem?

What options have you enabled for building? Is this FreeBSD 11.0-RELEASE?

Make.user : https://gist.github.com/iblis17/b4dca8221dcb96efcccde24f9cc2fa8d
OS: FreeBSD ionic 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r313193: Sat Feb 4 17:46:46 CST 2017 root@ionic:/usr/obj/usr/src/sys/GENERIC amd64

ararslan · 2017-02-13T02:48:23Z

Are the build logs publicly accessible as they are for Travis and AppVeyor, our other CI services?

iblislin · 2017-02-13T02:55:12Z

Yes, it's public.

The web ui of buildbot 0.9 requires websocket. Any error message on your browser's js console?

tkelman · 2017-02-13T04:07:28Z

freebsd has an ilp64 system openblas? that's surprising and likely to cause problems unless its internal symbols have been renamed

ararslan · 2017-02-13T04:46:40Z

Any error message on your browser's js console?

Nope, just doesn't connect.

freebsd has an ilp64 system openblas?

Where did you see that? OpenBLAS has to be installed from a port, and it doesn't look like the OpenBLAS build flag INTERFACE64 (which enables ILP64) is enabled by default.

iblislin · 2017-02-13T05:17:53Z

On this system, openblas INTERFACE64 flag is enabled.

$ pkg info openblas-0.2.19,1
openblas-0.2.19,1
Name           : openblas
Version        : 0.2.19,1
Installed on   : Mon Feb 13 10:43:43 2017 CST
Origin         : math/openblas
Architecture   : freebsd:12:x86:64
Prefix         : /usr/local
Categories     : math
Licenses       : BSD3CLAUSE
Maintainer     : [email protected]
WWW            : https://github.com/xianyi/OpenBLAS
Comment        : Optimized BLAS library based on GotoBLAS2
Options        :
        AVX            : on
        AVX2           : on
        DYNAMIC_ARCH   : off
        INTERFACE64    : on
        OPENMP         : off
Shared Libs required:
        libquadmath.so.0
        libgfortran.so.3
Shared Libs provided:
        libopenblas.so.0
        libopenblasp.so.0
Annotations    :
Flat size      : 78.5MiB
Description    :
OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version.
OpenBLAS is an open source project supported by
Lab of Parallel Software and Computational Science, ISCAS.

NOTE: If you want to specify your CPU microarchitecture manually,
please use TARGET_CPU_ARCH knob, e.g., "make TARGET_CPU_ARCH=NEHALEM".
This value is set TARGET build flag.

WWW: https://github.com/xianyi/OpenBLAS

iblislin · 2017-02-13T05:23:43Z

Any error message on your browser's js console?

Nope, just doesn't connect.

@ararslan hmm ... may traceroute julia.iblis.cnmc.tw give you an idea?

tkelman · 2017-02-13T06:05:35Z

If you do that without renaming the symbols, numpy and most other programs that use blas aren't going to work properly.

iblislin · 2017-02-13T06:45:49Z

Seems openblas in ports do not rename the symbols?
I will disable INTERFACE64 and build julia again.

tkelman · 2017-02-13T06:54:47Z

Julia can handle it, but a lot of other software can't. Just a warning about that option.

iblislin · 2017-02-13T13:14:36Z

I add the missing arpack-ng and rebuilded: https://julia.iblis.cnmc.tw/#/builders/1/builds/9
One of test failures related to file still exists: #8078

ararslan · 2017-02-14T15:52:42Z

traceroute shows that the connection gets to TWGate, gets forwarded around a bit, then stalls for a long time before quitting.

iblislin · 2017-02-16T05:03:47Z

hmm TWGate is my ISP's ISP...
I built a reverse proxy: https://julia1.iblis.cnmc.tw/
Please give it a try.

ararslan · 2017-02-16T05:12:36Z

The owner of julia1.iblis.cnmc.tw has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

iblislin · 2017-02-16T06:00:46Z

@ararslan (Y) I use my self-signed ssl certificate on julia1.iblis.cnmc.tw. so... seems it does connect? I will migrate nginx setting from julia.iblis.cnmc.tw to julia1.iblis.cnmc.tw and change the DNS record later.

iblislin · 2017-02-17T12:33:17Z

@ararslan the DNS record changed. Please try again: https://julia.iblis.cnmc.tw

ararslan · 2017-02-17T19:13:47Z

Thanks, now it works for me.

tkelman · 2017-03-30T08:15:10Z

Looks like #20798 and one of the recently added libgit2 tests are problematic?

iblislin · 2017-03-30T09:24:43Z

@tkelman seems libgit2 tests are fine after build 510 now.

tkelman · 2017-03-30T09:28:03Z

Ah okay, maybe I was looking at something from right before #21220

iblislin · 2017-03-30T16:01:02Z

Oops! build 514 (6659b59) popup the libgit2 error again.....

The CI worker is using system libgit2 -- libgit2-0.24.0

iblislin · 2017-03-30T17:23:35Z

@tkelman I guess I hit this problem: libgit2/libgit2#4169
My personal ~/.gitconfig is a symbolic link...

I replace it with a normal file, and gmake test-libgit2 passes now.

iblislin · 2017-06-13T22:53:06Z

Now, the build pass, thanks to @ararslan 's fabulous work. :)

Time to consider to turn on a webhook?

There are two worker for the build task, one owns both buildbot master and a worker on it, it's the faster one.
And the other one is an old IBM x3650 machine (10 years-old :p), it takes about 3 hr to finish a build. Is 3 hr acceptable for PR build?

iblislin · 2017-06-20T03:44:11Z

@ararslan Could you confirm that?

ararslan · 2017-06-20T17:24:46Z

Sorry, haven't had time to look at that much. Will take a closer look soon.

iblislin · 2017-07-11T10:16:18Z

Ready to enable webhook?
All recent builds passed.

ararslan · 2017-07-11T17:43:41Z

Which tests do you run? In particular, libgit2-online and pkg should both be failing after the libgit2 upgrade. For reference, Travis and AppVeyor test all, download, libgit2-online, and pkg.

iblislin · 2017-07-11T23:35:40Z

I run gmake testall (in line 3 of
https://julia.iblis.cnmc.tw/#/builders/1/builds/34/steps/5/logs/stdio)
I will add the orther testing... maybe i misunderstand testall

ararslan · 2017-07-11T23:44:48Z

Ah, I see. I recommend taking a look at how Travis and AppVeyor run the tests. In particular, they do the equivalent of

julia test/runtests.jl all download libgit2-online pkg

I don't believe that download et al. are part of the testall Make target.

iblislin · 2017-07-12T02:58:22Z

I reconfigure my buildbot with: ['env', 'JULIA_CPU_CORES=6', 'gmake', 'testall', 'test-download', 'test-pkg', 'test-libgit2-online']
Now the test-pkg failed as expected.

https://julia.iblis.cnmc.tw/#/builders/1/builds/38/steps/5/logs/stdio

iblislin · 2017-07-13T11:49:54Z

good enough now?

tkelman · 2017-07-13T11:55:57Z

We can try it unless anyone has any reasons not to - if it proves unreliable, slow, or burdensome to keep it working then maybe we can re-evaluate after a while, but worth getting more checks I think. How long should the server it's hosted on remain available for this purpose?

iblislin · 2017-07-13T12:04:44Z

How long should the server it's hosted on remain available for this purpose?

You meaning how long we can have this service enabled? maybe 1 year ... (until I finish my master's degree

ararslan · 2017-07-13T16:31:16Z

How long does it take to do an incremental build and test at this point? (You set it up to do incremental rather than fresh builds, right?)

iblislin · 2017-07-13T16:36:01Z

About 40-50 mins on an increamental build.
I have 2 workers, the max len of running queue is 2.

ararslan · 2017-07-13T16:42:52Z

Okay, that sounds great. +1 from me for turning on the webhook now.

iblislin · 2017-07-13T18:57:22Z

hmm, just found that if one do rebase again before previous build finish, the old build will not be canceled. Is this the limitation of webhook? I'm curious about travis can handle this situation or not?

tkelman · 2017-07-13T19:00:11Z

travis has a special feature for it. buildbot might have something you could turn on? otherwise we have a code snippet for fast-failing that queries the travis or appveyor api to see if any newer builds are queued for the same PR

iblislin · 2017-07-13T19:01:44Z

Ok, I will investigate on it

iblislin · 2017-07-13T19:02:42Z

(For reference, the build queue is here: https://julia.iblis.cnmc.tw/#/builders/1

ararslan · 2017-07-14T18:52:52Z

Another feature that would be great is if you could implement build skipping. Currently someone can add [ci skip] to a commit message and that will keep Travis and AppVeyor from running on that commit. Would it be possible to detect that and skip running the FreeBSD CI if [ci skip] is given?

ararslan · 2017-07-14T18:53:59Z

Akin to that, another useful feature would be to be able to skip just the FreeBSD build, just as we can currently skip the Windows build with [av skip] (though unfortunately Travis provides no equivalent for skipping macOS and/or Linux). Maybe something like [bsd skip].

iblislin · 2017-07-15T04:13:23Z

Now ci skipping should work, I just implemented it:

[ci skip]
[skip ci]
[bsd skip]
[skip bsd]

ararslan · 2017-07-15T18:13:59Z

Wonderful, thanks!

tkelman · 2017-07-15T18:15:36Z

I tried it in #22821 but it didn't seem to work. Does buildbot have a built-in feature for that already?

iblislin · 2017-07-15T23:56:05Z

@tkelman I am checking my code again...
No, it's not a built-in feature. PR is here: buildbot/buildbot#3443

tkelman · 2017-08-16T17:08:13Z

I think everyone's pretty happy with this, so closing.

ararslan · 2017-08-16T17:34:04Z

Thanks again for this, Iblis! It's fantastic to have better support for FreeBSD, plus CI that runs way faster than any of the others.

tkelman added the system:freebsd Affects only FreeBSD label Feb 12, 2017

JeffBezanson added the building Build system, or building Julia or its dependencies label Feb 12, 2017

tkelman mentioned this issue Jun 11, 2017

RFC: Improve the platform support section of the README #22319

Merged

This was referenced Jun 26, 2017

debuginfo: tweak the order of get_function_name_and_base and dladdr on FreeBSD #22547

Closed

test/file: repeated running test_13559 triggers SystemError randomly #22566

Merged

iblislin mentioned this issue Jul 15, 2017

Use default system compiler version on Travis #22820

Merged

tkelman closed this as completed Aug 16, 2017

rdeits mentioned this issue Jun 2, 2018

Improve docs for Union{T,Nothing} #27382

Merged

FreeBSD CI #20585

FreeBSD CI #20585

Comments

iblislin commented Feb 12, 2017 • edited Loading

tkelman commented Feb 12, 2017

JeffBezanson commented Feb 12, 2017

iblislin commented Feb 13, 2017

ararslan commented Feb 13, 2017

iblislin commented Feb 13, 2017 • edited Loading

ararslan commented Feb 13, 2017

iblislin commented Feb 13, 2017 • edited Loading

tkelman commented Feb 13, 2017

ararslan commented Feb 13, 2017

iblislin commented Feb 13, 2017

iblislin commented Feb 13, 2017

tkelman commented Feb 13, 2017

iblislin commented Feb 13, 2017

tkelman commented Feb 13, 2017

iblislin commented Feb 13, 2017

ararslan commented Feb 14, 2017

iblislin commented Feb 16, 2017 • edited Loading

ararslan commented Feb 16, 2017

iblislin commented Feb 16, 2017

iblislin commented Feb 17, 2017

ararslan commented Feb 17, 2017

tkelman commented Mar 30, 2017

iblislin commented Mar 30, 2017

tkelman commented Mar 30, 2017

iblislin commented Mar 30, 2017 • edited Loading

iblislin commented Mar 30, 2017

iblislin commented Jun 13, 2017

iblislin commented Jun 20, 2017

ararslan commented Jun 20, 2017

iblislin commented Jul 11, 2017

ararslan commented Jul 11, 2017

iblislin commented Jul 11, 2017

ararslan commented Jul 11, 2017

iblislin commented Jul 12, 2017

iblislin commented Jul 13, 2017

tkelman commented Jul 13, 2017

iblislin commented Jul 13, 2017

ararslan commented Jul 13, 2017

iblislin commented Jul 13, 2017

ararslan commented Jul 13, 2017

iblislin commented Jul 13, 2017 • edited Loading

tkelman commented Jul 13, 2017

iblislin commented Jul 13, 2017

iblislin commented Jul 13, 2017

ararslan commented Jul 14, 2017

ararslan commented Jul 14, 2017 • edited Loading

iblislin commented Jul 15, 2017

ararslan commented Jul 15, 2017

tkelman commented Jul 15, 2017

iblislin commented Jul 15, 2017

tkelman commented Aug 16, 2017

ararslan commented Aug 16, 2017

iblislin commented Feb 12, 2017 •

edited

Loading

iblislin commented Feb 13, 2017 •

edited

Loading

iblislin commented Feb 13, 2017 •

edited

Loading

iblislin commented Feb 16, 2017 •

edited

Loading

iblislin commented Mar 30, 2017 •

edited

Loading

iblislin commented Jul 13, 2017 •

edited

Loading

ararslan commented Jul 14, 2017 •

edited

Loading