Run travis on rustc-master instead of nightly #2941

oli-obk · 2018-07-20T08:26:30Z

No description provided.

Manishearth · 2018-07-20T08:28:55Z

appveyor.yml

@@ -17,6 +17,11 @@ install:
    - rustup-init.exe -y --default-host %TARGET% --default-toolchain nightly
    - set PATH=%PATH%;C:\Users\appveyor\.cargo\bin;C:\Users\appveyor\.rustup\toolchains\nightly-%TARGET%\bin
    - if defined MSYS2_BITS set PATH=%PATH%;C:\msys64\mingw%MSYS2_BITS%\bin
+    - cargo install rustup-toolchain-install-master || echo "rustup-toolchain-install-master already installed"
+    - set RUSTC_HEAD=`git ls-remote https://github.com/rust-lang/rust.git HEAD | tr -d ",." | tr " \t" "\n" | grep -e "HEAD" -v`


let's use rustup-toolchain-install-master -n master -f from kennytm/rustup-toolchain-install-master#6 when it merges

though I'm okay with merging this first and switching over later

oli-obk · 2018-07-20T08:31:29Z

hmm

+++travis_retry rustup-toolchain-install-master bc14d71622378cf942a834e7c2b5358b9901f775
./ci/base-tests.sh: line 4: travis_retry: command not found

mati865 · 2018-07-20T08:37:12Z

Could ~/.cargo/bin not be in PATH on Travis?

Anyway AppVeyor error looks worse:

= note: libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_end referenced in function _ZN4core3ptr13drop_in_place17h3973868bbd333ff7E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_easy_encoder referenced in function _ZN3xz26stream6Stream16new_easy_encoder17h444e67914679cfe5E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_alone_encoder referenced in function _ZN3xz26stream6Stream16new_lzma_encoder17h7a72f1d68ee5b478E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_stream_encoder referenced in function _ZN3xz26stream6Stream18new_stream_encoder17h698b6d2d519d1273E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_stream_decoder referenced in function _ZN3xz26stream6Stream18new_stream_decoder17hf2a8e53459d7da61E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_alone_decoder referenced in function _ZN3xz26stream6Stream16new_lzma_decoder17h2950602432ddcbd3E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_auto_decoder referenced in function _ZN3xz26stream6Stream16new_auto_decoder17hd4a704f64eb88947E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_code referenced in function _ZN3xz26stream6Stream7process17h416f627689d69b52E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_memlimit_get referenced in function _ZN3xz26stream6Stream8memlimit17hd44a326b1824d06eE
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_memlimit_set referenced in function _ZN3xz26stream6Stream12set_memlimit17h7f8ee5e5612bb876E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_lzma_preset referenced in function _ZN3xz26stream11LzmaOptions10new_preset17hb7cf472abb481456E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_check_is_supported referenced in function _ZN3xz26stream5Check12is_supported17h0c95786a9b51894dE
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_mf_is_supported referenced in function _ZN3xz26stream11MatchFinder12is_supported17h1143b95e3df085d9E
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_stream_encoder_mt_memusage referenced in function _ZN3xz26stream15MtStreamBuilder8memusage17h515e8f411dc5c4caE
          libxz2-2575134034ce30a1.rlib(xz2-2575134034ce30a1.xz26-c80f76e38c7a6925a75b74075834931c.rs.rcgu.o) : error LNK2019: unresolved external symbol lzma_stream_encoder_mt referenced in function _ZN3xz26stream15MtStreamBuilder7encoder17h4b074a2a8ad5634dE
          C:\Users\appveyor\AppData\Local\Temp\1\cargo-installqJxNyT\release\deps\rustup_toolchain_install_master-bb30363af798ef93.exe : fatal error LNK1120: 15 unresolved externals
          
error: aborting due to previous error
error:
 failed to compile `rustup-toolchain-install-master v1.1.0`, intermediate artifacts can be found at `C:\Users\appveyor\AppData\Local\Temp\1\cargo-installqJxNyT`

oli-obk · 2018-07-20T09:17:28Z

hmm... the macro imports aren't working.

Manishearth · 2018-07-20T09:20:11Z

Probably means it's not using the toolchain -- maybe it's obeying the rust-toolchain file?

…

On Fri, Jul 20, 2018, 2:17 AM Oliver Schneider ***@***.***> wrote: hmm... the macro imports aren't working. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2941 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSL0_LHiZLSvXpMSp8pStZqWgdMh4ks5uIaApgaJpZM4VXpQR> .

Manishearth · 2018-08-03T18:38:01Z

Rebased and updated to use new rustup-toolchain-install-master

mati865 · 2018-08-03T19:30:58Z

Primary build should succeed if you apply this workaround #3001
Other builds failed due to networking issue

phansch · 2018-08-06T15:40:03Z

I re-triggered the build after #3001 was merged. The master installer still sometimes fails with a Forbidden from the GitHub API. Perhaps we are hitting some rate limit?

fetching master commit hash... thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: ClientError(Forbidden), url: Some("https://api.github.com/repos/rust-lang/rust/commits/master") }

Looks like there are also different AppVeyor errors now:

= note: LINK : fatal error LNK1181: cannot open input file 'lzma.lib.lib'

and https://ci.appveyor.com/project/llogiq/rust-clippy/build/1.0.228#L265

oli-obk · 2018-08-07T14:24:37Z

Appveyor might be failing due to msvc vs mingw. Not sure if we are taking any care in choosing the correct thing

mati865 · 2018-08-07T14:34:39Z

@oli-obk .lib are static msvc libs, mingw does use .a extension for static lib.
Probably it's bug in the build system adding extension twice.

oli-obk · 2018-08-13T16:22:29Z

So.. travis is failing due to

/home/travis/rust/cargo/bin/clippy-driver: error while loading shared libraries: librustc_driver-3571cf80b2c91813.so: cannot open shared object file: No such file or directory

I'm not sure where the master-toolchain's libs are hidden, but we probably need to add it to the LD_LIBRARY_PATH. I would have assumed rustup taking care of this though :/ Not sure what's going on

mati865 · 2018-08-13T17:21:11Z

@oli-obk you are not installing toolchain components, see kennytm/rustup-toolchain-install-master#5

oli-obk · 2018-08-14T07:26:31Z

are you sure? the error looks like it's not finding the rustc libs, which rustc itself definitely needs (and rustc works)

mati865 · 2018-08-14T09:02:24Z

Please disregard my previous comment, I went too far with new component stuff. This repository needs to build Clippy everytime...

To debug this you can add following commands:

which rustc
rustc --print sysroot
ls $(rustc +master --print sysroot)/lib

It could be taking libs from the other toolchains.

…s_retry`

mati865 · 2018-08-20T21:22:45Z

My "debug" build:

$ which rustc
/home/travis/.cargo/bin/rustc
$ rustc --print sysroot
/home/travis/.rustup/toolchains/master
$ ls $(rustc +master --print sysroot)/lib
[..]
librustc_driver-b59ab39da03fc19f.so
[..]

/home/travis/.cargo/bin/clippy-driver: error while loading shared libraries: librustc_driver-b59ab39da03fc19f.so: cannot open shared object file: No such file or directory

Can we dump sysroot path from clippy-driver?

flip1995 · 2018-08-21T11:03:36Z

The master-toolchain libs can be found at $(rustc --print sysroot)/lib after setting rustup default master. See https://github.com/flip1995/rust-clippy/blob/prs/ci/base-tests.sh#L15

flip1995 · 2018-08-21T16:06:13Z

The appveyor linking error seems to be related with MinGW: alexcrichton/xz2-rs@eceb287

I'm not sure why, but without MinGW the linking error disappears. But after that the dogfood test fails appveyor. ~~I'm not sure if this is related to MinGW though..~~ (EDIT: I tested it, it's not)

I got travis to work, except sometimes

fetching master commit hash... thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: ClientError(Forbidden), url: Some("https://api.github.com/repos/rust-lang/rust/commits/master") }

https://github.com/flip1995/rust-clippy/commits/prs

Should I push this, to this branch?

mati865 · 2018-08-21T16:18:49Z

Windows build system seems really borked then.

Error { kind: ClientError(Forbidden), url: Some("https://api.github.com/repos/rust-lang/rust/commits/master") }
Guess it's hitting GitHub request, is there any way to fetch last rust build hash and pass it across all Travis builds?
Anyway maybe @kennytm knows something about this error.

kennytm · 2018-08-21T17:07:57Z

GitHub will reject API requests from the CI since the IP is shared. You need to create a GitHub token. @mati865

See https://github.com/kennytm/rustup-toolchain-install-master/blob/master/.travis.yml for an example (the $GITHUB_TOKEN is a secret env var which you could define in the Travis and AppVeyor settings)

flip1995 · 2018-08-21T22:31:42Z

Thanks @kennytm!

Just tested it on my fork and it works like a charm! We now need someone to create the token for Clippy and add it to travis and appveyor.

flip1995 · 2018-08-22T17:11:19Z

After adding the GITHUB_TOKEN to travis and appveyor, atleast travis will compile and work with the rustc master. appveyor will compile with rustc master, but the dogfood test fails because of a weird linking error (I couldn't find the culprit for this...).

If no GITHUB_TOKEN is set CI will try to use the nightly toolchain.

mati865 · 2018-08-22T22:38:51Z

@matthiaskrgr has found out the reason for this weird issue:

/home/travis/.cargo/bin/clippy-driver: error while loading shared libraries: librustc_driver-b59ab39da03fc19f.so: cannot open shared object file: No such file or directory

In other words:
Removing rust-toolchain file in CI script could make export LD_LIBRARY_PATH=$(rustc --print sysroot)/lib redundant and maybe even fix AppVeyor.

flip1995 · 2018-08-22T22:47:19Z

@mati865 we're already doing this in Travis and in appveyor.

The appveyor error is somehow connected to the master toolchain. Even when you change nothing in the appveyor script, except the toolchain (from nightly to master) , this error appears. Also adding $(rustc --print sysroot)/lib to the path in appveyor didn't fix the problem. I'm all out of ideas on this one...

mati865 · 2018-08-22T23:26:56Z

Oops my bad, guess it's the time to get some sleep.

Looking at AppVeyor error again:

stderr: error: unable to unlink old fallback exe
info: caused by: Access is denied. (os error 5)

It just means the handle to the file is still open. On Windows files cannot be renamed/moved until very last handle is closed.
No idea why this happens with master toolchain.

flip1995 · 2018-08-30T14:48:12Z

Do we want to move forward with this? The next steps would be:

Create a GITHUB_TOKEN and add it to Appveyor and Travis (@oli-obk can you do this? I don't have the permission for Appveyor) GitHub Doc
Fix dogfood test under windows

phansch · 2018-09-02T08:44:48Z

Just realized that I can add new environment variables on travis, too. I will give it a try now. I don't have access to AppVeyor, though.

Manishearth · 2018-09-02T08:56:22Z

Yeah, just fix it for travis and land this, let me know what we need to do for appveyor and I'll do it. I don't know how to grant access to appveyor.

phansch · 2018-09-02T09:12:23Z

Did a merge instead of a rebase because the prs branch was 50+ commits behind master and I'm not sure if any of these commits are used somewhere. Let's see what Travis is saying now.

For AppVeyor, the docs say that there's an 'Environment Variables' tab in the AppVeyor project settings. So essentially, you create a new Token with public_repo access here and copy the token over to AppVeyor.

Manishearth · 2018-09-02T09:25:11Z

Done

Manishearth · 2018-09-02T09:35:40Z

Feel free to merge when ready.

phansch · 2018-09-02T09:57:29Z

Travis looks fine now, but I think AppVeyor is not detecting the GITHUB_TOKEN correctly and doesn't install rust from master:
https://ci.appveyor.com/project/rust-lang-libs/rust-clippy/build/1.0.1536#L29
I guess it should show the output of the master installer?

Manishearth · 2018-09-02T14:37:11Z

Yeah not sure what's going on. Feel free to merge and open a new issue for appveyor, at least we'll have _some_ reliable CI.

…

On Sun, Sep 2, 2018, 3:27 PM Philipp Hansch ***@***.***> wrote: Travis looks fine now, but I think AppVeyor is not detecting the GITHUB_TOKEN correctly and doesn't install rust from master: https://ci.appveyor.com/project/rust-lang-libs/rust-clippy/build/1.0.1536#L29 I guess it should show the output of the master installer? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2941 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSLke9TYzsxKrNBlkBmdlJROq2DIdks5uW6uKgaJpZM4VXpQR> .

flip1995 · 2018-09-02T20:50:04Z

It seems that no env var GITHUB_TOKEN is defined in the Appveyor settings. It worked on my fork setup: https://ci.appveyor.com/project/flip1995/rust-clippy/build/1.0.28

@Manishearth Is there maybe a typo in the env var name? Lower case, instead of upper case? 🤔

Manishearth · 2018-09-03T02:26:47Z

Oh, I added it to Manishearth/rust.

I don't have access to rust-lang-nursery, but I pinged nrc to get it added.

Relatedly: What should we do for PRs then? They don't have access to these env vars, right? Should we use a different method for getting the master hash?

kennytm · 2018-09-03T02:38:30Z

You could clone the rust-lang/rust repo and find the (full 40 digits) hash. I think git clone isn't rate limited by IP (otherwise these CI services simply won't work). Cloning would be slower though.

Manishearth · 2018-09-03T02:44:40Z

Yeah, I really want to avoid that but it might be the only option

Manishearth · 2018-09-03T02:44:54Z

Anyway, merging this since at least master gets good CI

Manishearth · 2018-09-03T02:48:32Z

I'm going to add a CGI script for this on my server so we don't need the token.

Manishearth · 2018-09-03T02:55:22Z

Yeah, shallow cloning rust is the other option but i thought it was terrible amd maybe slow so didn't pick it. But really both suck, i don't know which to pick. I guess cloning is nicer since it's less infra I have to maintain.

Manishearth · 2018-09-03T03:13:40Z

`git ls-remote https://github.com/rust-lang/rust.git master` with some parsing works. I'll incorporate this into this PR (we can also make this part of rustup-toolchain-install-master)

…

On Mon, Sep 3, 2018, 8:08 AM kennytm ***@***.***> wrote: You could clone the rust-lang/rust repo and find the (full 40 digits) hash. I think git clone isn't rate limited by IP (otherwise these CI services simply won't work). Cloning would be slower though. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2941 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABivSOYeaKIajCbFZIOurevBNFzEVFWaks5uXJYogaJpZM4VXpQR> .

Manishearth reviewed Jul 20, 2018

View reviewed changes

Manishearth force-pushed the prs branch from add3bb0 to 2d31ea2 Compare August 3, 2018 18:37

phansch closed this Aug 6, 2018

phansch reopened this Aug 6, 2018

Manishearth mentioned this pull request Aug 7, 2018

Allow print/write with multiple newlines #3014

Merged

oli-obk force-pushed the prs branch from 2d31ea2 to fc9b0ca Compare August 8, 2018 13:20

oli-obk and others added 3 commits August 20, 2018 11:21

Try running appveyor on master instead of nightly

f8a3814

Keep the rustc master install in the travis file so we can use `travi…

d6af688

…s_retry`

Remove rust-toolchain file in CI

f969cf2

Manishearth force-pushed the prs branch from f6ab35c to 43d372c Compare August 20, 2018 18:21

Manishearth force-pushed the prs branch from 43d372c to f969cf2 Compare August 20, 2018 21:59

Add LD_LIBRARY_PATH and GITHUB_TOKEN

205db6f

flip1995 mentioned this pull request Aug 28, 2018

range_plus_one help suggestion should not remove braces #3103

Open

phansch closed this Sep 2, 2018

phansch reopened this Sep 2, 2018

Merge remote-tracking branch 'upstream/master' into prs

8e0de3d

Trigger rebuild for AppVeyor (and fix grammar)

273e11f

Manishearth closed this Sep 3, 2018

Manishearth reopened this Sep 3, 2018

Manishearth merged commit 1d13c1f into master Sep 3, 2018

Manishearth deleted the prs branch September 3, 2018 02:45

Run travis on rustc-master instead of nightly #2941

Run travis on rustc-master instead of nightly #2941

Conversation

oli-obk commented Jul 20, 2018

Manishearth Jul 20, 2018

Choose a reason for hiding this comment

oli-obk commented Jul 20, 2018

mati865 commented Jul 20, 2018

oli-obk commented Jul 20, 2018

Manishearth commented Jul 20, 2018 via email

Manishearth commented Aug 3, 2018

mati865 commented Aug 3, 2018

phansch commented Aug 6, 2018

oli-obk commented Aug 7, 2018

mati865 commented Aug 7, 2018

oli-obk commented Aug 13, 2018

mati865 commented Aug 13, 2018

oli-obk commented Aug 14, 2018

mati865 commented Aug 14, 2018

mati865 commented Aug 20, 2018

flip1995 commented Aug 21, 2018

flip1995 commented Aug 21, 2018 • edited Loading

mati865 commented Aug 21, 2018

kennytm commented Aug 21, 2018

flip1995 commented Aug 21, 2018

flip1995 commented Aug 22, 2018

mati865 commented Aug 22, 2018

flip1995 commented Aug 22, 2018

mati865 commented Aug 22, 2018

flip1995 commented Aug 30, 2018

phansch commented Sep 2, 2018

Manishearth commented Sep 2, 2018

phansch commented Sep 2, 2018

Manishearth commented Sep 2, 2018

Manishearth commented Sep 2, 2018

phansch commented Sep 2, 2018

Manishearth commented Sep 2, 2018 via email

flip1995 commented Sep 2, 2018

Manishearth commented Sep 3, 2018

kennytm commented Sep 3, 2018

Manishearth commented Sep 3, 2018

Manishearth commented Sep 3, 2018

Manishearth commented Sep 3, 2018

Manishearth commented Sep 3, 2018

Manishearth commented Sep 3, 2018 via email

flip1995 commented Aug 21, 2018 •

edited

Loading