Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustc: implement argsfiles for command line #63175

Merged
merged 3 commits into from
Aug 22, 2019
Merged

Conversation

jsgf
Copy link
Contributor

@jsgf jsgf commented Jul 31, 2019

Many tools, such as gcc and gnu-ld, support "args files" - that is, being able to specify @file on the command line. This causes file to be opened and parsed for command line options. They're separated with whitespace; whitespace can be quoted with double or single quotes, and everything can be \-escaped. Args files may recursively include other args files via @file2.

See https://sourceware.org/binutils/docs/ld/Options.html#Options for the documentation of gnu-ld's @file parameters.

This is useful for very large command lines, or when command lines are being generated into files by other tooling.

@rust-highfive
Copy link
Collaborator

r? @petrochenkov

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 31, 2019
@petrochenkov
Copy link
Contributor

I clearly remember I've seen an issue about this, but I can't find it right now.

MSVC supports them as well, with the same syntax.

@petrochenkov petrochenkov added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Jul 31, 2019
@rust-highfive

This comment has been minimized.

@jsgf jsgf force-pushed the argsfile branch 2 times, most recently from cf1bc47 to fbcaacf Compare July 31, 2019 22:38
@Mark-Simulacrum
Copy link
Member

cc @alexcrichton since I believe you added our support for calling linkers with this and are probably generally interested due to Cargo concerns

@Centril Centril added the relnotes Marks issues that should be documented in the release notes of the next release. label Jul 31, 2019
Some(FileArg::Arg(arg)) => return Some(arg),
Some(FileArg::Path(path)) => {
let file =
if self.stack.len() >= MAX_RECURSION { None } else { fs::read(&path).ok() };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels a bit error-prone to simply ignore @files that are "too deep". Maybe we can at least eprintln or something here? There's probably no good way to give a good error message here...

One possible "solution" is to just panic and bump MAX_RECURSION to some very large number like 1000 which would be relatively unreasonable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented error handling is pretty awkward - it also just includes the literal @file string as an arg if it can't open/read the file. The recursion handling is at least consistent with that low standard.

I'm happy to scatter some eprintlns (if that's the best that's available here) around to make noise about things going wrong.

@Mark-Simulacrum
Copy link
Member

Though this matches the ld spec, it is worth noting that this is the first place in rustc/cargo/etc where we're splitting by whitespace and implementing quoting rules around that. I haven't reviewed the implementation here closely but maybe we could avoid doing that somehow?

@petrochenkov
Copy link
Contributor

r? @alexcrichton

@Centril Centril added the needs-fcp This change is insta-stable, so needs a completed FCP to proceed. label Jul 31, 2019
@cramertj
Copy link
Member

Hey, I was just wanting this! Wonderful :)

@jsgf
Copy link
Contributor Author

jsgf commented Jul 31, 2019

@Mark-Simulacrum I was following this as a convention rather than novelty for the sake of it. I chose gnu-ld as a model because it was the first even slightly detailed description of the file syntax which made sense (tho there's still a fair amount of ambiguity).

I could go with something simpler like "each arg is on its own line", though that raises questions like:

  1. What if an arg contains a newline? Doesn't seem completely unreasonable.
  2. How to define newlines on Windows? Insist on full \r\n, or also handle Unix \n line breaks? What does \r on its own mean?

The this implementation:

  1. consumes leading and trailing whitespace in an unsurprising way
  2. allows for empty args by quoting ""
  3. handles space, tab, \r and \n uniformly as whitespace which eliminates line-ending ambiguity
  4. Handles '...' and "..." in a shell-like way, as well as \ quoting.

This has some amount of superfluous complexity, but I think that's offset by making it compatible with typical shell quoting rules (so, for example, Python's shlex module can be used to quote args in this form - with some care).

@Mark-Simulacrum
Copy link
Member

I completely agree that this is a viable and maybe best implementation; just wanted to raise the concern that we've elsewhere always tried to shy away from shlex or the like.

One solution might be to use a null byte as a separator, since iirc they're not valid utf-8 (and rustc generally doesn't work with anything requiring them, anyway). Since these files are presumably mostly programmatically generated that doesn't seem too bad.

@jsgf
Copy link
Contributor Author

jsgf commented Jul 31, 2019

@Mark-Simulacrum \0-separated is an option, but its a bit painful to generate from a shell script and would make the files be treated as binary and unmergable by a source control system (eg, tests in rustc itself).

@jsgf
Copy link
Contributor Author

jsgf commented Aug 1, 2019

BTW, where does $DIR expansion happen? It doesn't seem to be happening for @$DIR/.... Does the $ need to be the first non-whitespace character?

@Mark-Simulacrum
Copy link
Member

$DIR among other expansions happens in normalize_output.

I think my personal preference is \0 as the separator over implementing shlex-alikes but I don't care too much.

@rust-highfive

This comment has been minimized.

@jsgf
Copy link
Contributor Author

jsgf commented Aug 1, 2019

@Mark-Simulacrum Ah thanks, I had thought that it could be used in the .rs file as an expansion. Test fixed (I hope).

@rust-highfive

This comment has been minimized.

@jsgf jsgf force-pushed the argsfile branch 2 times, most recently from 746623c to f3620d5 Compare August 1, 2019 15:45
@jsgf
Copy link
Contributor Author

jsgf commented Aug 20, 2019

OK:

  • moved the named args for usage to be first in the stack
  • folded all previous work into a single commit
  • fixed up error message normalization for windows and addressed other comments
  • added commit to move argfile expansion to run_compiler

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-20T02:07:38.2708746Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-20T02:07:38.2889638Z ##[command]git config gc.auto 0
2019-08-20T02:07:38.2956615Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-20T02:07:38.3007332Z ##[command]git config --get-all http.proxy
2019-08-20T02:07:38.3129869Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63175/merge:refs/remotes/pull/63175/merge
---
2019-08-20T02:08:12.4801322Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-20T02:08:12.4801369Z 
2019-08-20T02:08:12.4801543Z   git checkout -b <new-branch-name>
2019-08-20T02:08:12.4801568Z 
2019-08-20T02:08:12.4801609Z HEAD is now at c3148b5e0 Merge 05e845ee966945870722ac4100baca5c58f01d9d into c1b08dd26036e14f061b99b20cd6f169e29046f3
2019-08-20T02:08:12.4951186Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-20T02:08:12.4953985Z ==============================================================================
2019-08-20T02:08:12.4954034Z Task         : Bash
2019-08-20T02:08:12.4954088Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-20T02:14:27.5664047Z    Compiling serde_json v1.0.40
2019-08-20T02:14:29.3950943Z    Compiling tidy v0.1.0 (/checkout/src/tools/tidy)
2019-08-20T02:14:40.0575161Z     Finished release [optimized] target(s) in 1m 28s
2019-08-20T02:14:40.0660676Z tidy check
2019-08-20T02:14:40.5836707Z tidy error: /checkout/src/librustc_driver/lib.rs:152: line longer than 100 chars
2019-08-20T02:14:40.8681487Z tidy error: /checkout/src/test/ui/commandline-argfile-missing.rs:5: line longer than 100 chars
2019-08-20T02:14:42.0269063Z some tidy checks failed
2019-08-20T02:14:42.0269247Z 
2019-08-20T02:14:42.0269247Z 
2019-08-20T02:14:42.0270047Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor"
2019-08-20T02:14:42.0270205Z 
2019-08-20T02:14:42.0270228Z 
2019-08-20T02:14:42.0270271Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
2019-08-20T02:14:42.0270317Z Build completed unsuccessfully in 0:01:31
2019-08-20T02:14:42.0270317Z Build completed unsuccessfully in 0:01:31
2019-08-20T02:14:42.0313654Z == clock drift check ==
2019-08-20T02:14:42.0331134Z   local time: Tue Aug 20 02:14:42 UTC 2019
2019-08-20T02:14:42.1847520Z   network time: Tue, 20 Aug 2019 02:14:42 GMT
2019-08-20T02:14:42.1848233Z == end clock drift check ==
2019-08-20T02:14:43.6071817Z ##[error]Bash exited with code '1'.
2019-08-20T02:14:43.6103766Z ##[section]Starting: Checkout
2019-08-20T02:14:43.6105879Z ==============================================================================
2019-08-20T02:14:43.6105939Z Task         : Get sources
2019-08-20T02:14:43.6106007Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@jsgf jsgf force-pushed the argsfile branch 2 times, most recently from 9f51b9e to b409b65 Compare August 20, 2019 03:23
@rust-highfive
Copy link
Collaborator

The job mingw-check of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-20T03:23:53.8868432Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-20T03:23:53.9090586Z ##[command]git config gc.auto 0
2019-08-20T03:23:53.9156280Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-20T03:23:53.9210440Z ##[command]git config --get-all http.proxy
2019-08-20T03:23:53.9352662Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63175/merge:refs/remotes/pull/63175/merge
---
2019-08-20T03:24:28.3731432Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-20T03:24:28.3731699Z 
2019-08-20T03:24:28.3732123Z   git checkout -b <new-branch-name>
2019-08-20T03:24:28.3732345Z 
2019-08-20T03:24:28.3732540Z HEAD is now at c6dc7feb0 Merge b409b652026a17efa565c493b491b98b341bff14 into c1b08dd26036e14f061b99b20cd6f169e29046f3
2019-08-20T03:24:28.3906418Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-20T03:24:28.3909265Z ==============================================================================
2019-08-20T03:24:28.3909321Z Task         : Bash
2019-08-20T03:24:28.3909383Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-20T03:33:35.9604151Z     Checking rustc_interface v0.0.0 (/checkout/src/librustc_interface)
2019-08-20T03:33:38.6037137Z error[E0308]: mismatched types
2019-08-20T03:33:38.6040008Z     --> src/librustc_driver/lib.rs:1216:22
2019-08-20T03:33:38.6040723Z      |
2019-08-20T03:33:38.6041306Z 1216 |         run_compiler(args, &mut callbacks, None, None)
2019-08-20T03:33:38.6043053Z      |                      |
2019-08-20T03:33:38.6044424Z      |                      expected reference, found struct `std::vec::Vec`
2019-08-20T03:33:38.6044424Z      |                      expected reference, found struct `std::vec::Vec`
2019-08-20T03:33:38.6046236Z      |                      help: consider borrowing here: `&args`
2019-08-20T03:33:38.6046845Z      |
2019-08-20T03:33:38.6047518Z      = note: expected type `&[std::string::String]`
2019-08-20T03:33:38.6048298Z                 found type `std::vec::Vec<std::string::String>`
2019-08-20T03:33:38.6963703Z error: aborting due to previous error
2019-08-20T03:33:38.6963947Z 
2019-08-20T03:33:38.6965073Z For more information about this error, try `rustc --explain E0308`.
2019-08-20T03:33:38.7173787Z error: Could not compile `rustc_driver`.
---
2019-08-20T03:33:38.7271516Z == clock drift check ==
2019-08-20T03:33:38.7288112Z   local time: Tue Aug 20 03:33:38 UTC 2019
2019-08-20T03:33:38.8142321Z   network time: Tue, 20 Aug 2019 03:33:38 GMT
2019-08-20T03:33:38.8147364Z == end clock drift check ==
2019-08-20T03:33:40.1099488Z ##[error]Bash exited with code '1'.
2019-08-20T03:33:40.1140431Z ##[section]Starting: Checkout
2019-08-20T03:33:40.1141923Z ==============================================================================
2019-08-20T03:33:40.1141972Z Task         : Get sources
2019-08-20T03:33:40.1142015Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-20T05:03:58.1101205Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-20T05:03:58.1326856Z ##[command]git config gc.auto 0
2019-08-20T05:03:58.1412341Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-20T05:03:58.1463165Z ##[command]git config --get-all http.proxy
2019-08-20T05:03:58.1589690Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63175/merge:refs/remotes/pull/63175/merge
---
2019-08-20T05:04:33.1629252Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-20T05:04:33.1629285Z 
2019-08-20T05:04:33.1629523Z   git checkout -b <new-branch-name>
2019-08-20T05:04:33.1629571Z 
2019-08-20T05:04:33.1629625Z HEAD is now at b0dc87b26 Merge 3f58528665cbd8b31944e91b7fbe1aa8213f95a2 into d8d99baf565bb389b82553dfe1f6e56ddc650184
2019-08-20T05:04:33.1787668Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-20T05:04:33.1790461Z ==============================================================================
2019-08-20T05:04:33.1790520Z Task         : Bash
2019-08-20T05:04:33.1790568Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-20T06:06:33.2035333Z .................................................................................................... 1500/8944
2019-08-20T06:06:38.7235932Z .................................................................................................... 1600/8944
2019-08-20T06:06:51.6170635Z ........................................i...............i........................................... 1700/8944
2019-08-20T06:06:59.6227267Z .................................................................................................... 1800/8944
2019-08-20T06:07:14.0275866Z ................................iiiii............................................................... 1900/8944
2019-08-20T06:07:24.6660293Z .................................................................................................... 2100/8944
2019-08-20T06:07:27.2455276Z .................................................................................................... 2200/8944
2019-08-20T06:07:31.8949093Z .................................................................................................... 2300/8944
2019-08-20T06:07:38.9176490Z .................................................................................................... 2400/8944
---
2019-08-20T06:10:40.8690916Z ....................i...............i............................................................... 4700/8944
2019-08-20T06:10:52.8843027Z .................................................................................................... 4800/8944
2019-08-20T06:10:59.1910571Z .................................................................................................... 4900/8944
2019-08-20T06:11:10.4603880Z .................................................................................................... 5000/8944
2019-08-20T06:11:15.8631320Z .ii.ii.............................................................................................. 5100/8944
2019-08-20T06:11:30.9268436Z .................................................................................................... 5300/8944
2019-08-20T06:11:38.0554698Z .........................................................i.......................................... 5400/8944
2019-08-20T06:11:45.2773319Z .................................................................................................... 5500/8944
2019-08-20T06:11:54.2301630Z .................................................................................................... 5600/8944
2019-08-20T06:11:54.2301630Z .................................................................................................... 5600/8944
2019-08-20T06:12:04.5947612Z ..................................................ii...i..ii...........i............................ 5700/8944
2019-08-20T06:12:27.0140698Z .................................................................................................... 5900/8944
2019-08-20T06:12:32.1281603Z .................................................................................................... 6000/8944
2019-08-20T06:12:32.1281603Z .................................................................................................... 6000/8944
2019-08-20T06:12:42.8104644Z ...................................................i..ii............................................ 6100/8944
2019-08-20T06:13:07.1331620Z .................................................................................................i.. 6300/8944
2019-08-20T06:13:09.4004620Z .................................................................................................... 6400/8944
2019-08-20T06:13:11.6840223Z .....................................................................i.............................. 6500/8944
2019-08-20T06:13:14.7015826Z .................................................................................................... 6600/8944
---
2019-08-20T06:17:19.0152086Z 
2019-08-20T06:17:19.0152854Z ---- [ui] ui/commandline-argfile-missing.rs stdout ----
2019-08-20T06:17:19.0153181Z diff of stderr:
2019-08-20T06:17:19.0153440Z 
2019-08-20T06:17:19.0154012Z - error: Failed to load argument file: IO Error: $DIR/commandline-argfile-missing.args: $FILE_MISSING (os error $ERR)
2019-08-20T06:17:19.0154665Z + error: Failed to load argument file: IO Error: $DIR/commandline-argfile-missing.args: No such file or directory (os error $ERR)
2019-08-20T06:17:19.0155451Z 3 
2019-08-20T06:17:19.0155672Z 
2019-08-20T06:17:19.0155876Z 
2019-08-20T06:17:19.0156109Z The actual stderr differed from the expected stderr.
2019-08-20T06:17:19.0156109Z The actual stderr differed from the expected stderr.
2019-08-20T06:17:19.0156712Z Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/commandline-argfile-missing/commandline-argfile-missing.stderr
2019-08-20T06:17:19.0157295Z To update references, rerun the tests and pass the `--bless` flag
2019-08-20T06:17:19.0157880Z To only update this specific test, also pass `--test-args commandline-argfile-missing.rs`
2019-08-20T06:17:19.0158391Z error: 1 errors occurred comparing output.
2019-08-20T06:17:19.0158615Z status: exit code: 1
2019-08-20T06:17:19.0158615Z status: exit code: 1
2019-08-20T06:17:19.0159979Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/commandline-argfile-missing.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/commandline-argfile-missing" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--cfg" "cmdline_set" "@/checkout/src/test/ui/commandline-argfile-missing.args" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/commandline-argfile-missing/auxiliary" "-A" "unused"
2019-08-20T06:17:19.0160958Z ------------------------------------------
2019-08-20T06:17:19.0161528Z 
2019-08-20T06:17:19.0162069Z ------------------------------------------
2019-08-20T06:17:19.0162391Z stderr:
2019-08-20T06:17:19.0162391Z stderr:
2019-08-20T06:17:19.0162854Z ------------------------------------------
2019-08-20T06:17:19.0163480Z error: Failed to load argument file: IO Error: /checkout/src/test/ui/commandline-argfile-missing.args: No such file or directory (os error 2)
2019-08-20T06:17:19.0163962Z 
2019-08-20T06:17:19.0164447Z ------------------------------------------
2019-08-20T06:17:19.0164708Z 
2019-08-20T06:17:19.0164922Z 
---
2019-08-20T06:17:19.0187710Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:536:22
2019-08-20T06:17:19.0188087Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-08-20T06:17:19.0201861Z 
2019-08-20T06:17:19.0201930Z 
2019-08-20T06:17:19.0213266Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-08-20T06:17:19.0213687Z 
2019-08-20T06:17:19.0214478Z 
2019-08-20T06:17:19.0217002Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-08-20T06:17:19.0217222Z Build completed unsuccessfully in 1:06:13
2019-08-20T06:17:19.0217222Z Build completed unsuccessfully in 1:06:13
2019-08-20T06:17:19.0271045Z == clock drift check ==
2019-08-20T06:17:19.0288319Z   local time: Tue Aug 20 06:17:19 UTC 2019
2019-08-20T06:17:19.1656338Z   network time: Tue, 20 Aug 2019 06:17:19 GMT
2019-08-20T06:17:19.1659223Z == end clock drift check ==
2019-08-20T06:17:20.0203858Z ##[error]Bash exited with code '1'.
2019-08-20T06:17:20.0246333Z ##[section]Starting: Checkout
2019-08-20T06:17:20.0248203Z ==============================================================================
2019-08-20T06:17:20.0248279Z Task         : Get sources
2019-08-20T06:17:20.0248331Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@alexcrichton
Copy link
Member

@bors: delegate+

Implementation looks good to me, feel free to r+ when tests are good

@bors
Copy link
Contributor

bors commented Aug 20, 2019

✌️ @jsgf can now approve this pull request

@jsgf jsgf force-pushed the argsfile branch 2 times, most recently from 07fc983 to 44753eb Compare August 20, 2019 17:47
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-08-20T17:47:51.5190858Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-08-20T17:47:51.5426086Z ##[command]git config gc.auto 0
2019-08-20T17:47:51.5492033Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-08-20T17:47:51.5556467Z ##[command]git config --get-all http.proxy
2019-08-20T17:47:51.5700960Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/63175/merge:refs/remotes/pull/63175/merge
---
2019-08-20T17:48:27.2987858Z do so (now or later) by using -b with the checkout command again. Example:
2019-08-20T17:48:27.2988658Z 
2019-08-20T17:48:27.2989214Z   git checkout -b <new-branch-name>
2019-08-20T17:48:27.2989468Z 
2019-08-20T17:48:27.2989775Z HEAD is now at 6c18e4db6 Merge 44753eb57fa79ea6551631d3213f416edec96a64 into 51879c3abaedb926739095d19a2af638ee6a07d8
2019-08-20T17:48:27.3142618Z ##[section]Starting: Collect CPU-usage statistics in the background
2019-08-20T17:48:27.3145441Z ==============================================================================
2019-08-20T17:48:27.3145539Z Task         : Bash
2019-08-20T17:48:27.3145586Z Description  : Run a Bash script on macOS, Linux, or Windows
---
2019-08-20T18:52:22.6167931Z .................................................................................................... 1500/8946
2019-08-20T18:52:28.4312602Z .................................................................................................... 1600/8946
2019-08-20T18:52:41.6173423Z ...........................................i..............i......................................... 1700/8946
2019-08-20T18:52:50.1213304Z .................................................................................................... 1800/8946
2019-08-20T18:53:04.8962476Z ..................................iiiii............................................................. 1900/8946
2019-08-20T18:53:15.9803740Z .................................................................................................... 2100/8946
2019-08-20T18:53:18.6876034Z .................................................................................................... 2200/8946
2019-08-20T18:53:23.4005744Z .................................................................................................... 2300/8946
2019-08-20T18:53:30.7817108Z .................................................................................................... 2400/8946
---
2019-08-20T18:56:36.7410689Z ......................i...............i............................................................. 4700/8946
2019-08-20T18:56:48.8107888Z .................................................................................................... 4800/8946
2019-08-20T18:56:55.1567007Z .................................................................................................... 4900/8946
2019-08-20T18:57:06.5939656Z .................................................................................................... 5000/8946
2019-08-20T18:57:12.1346348Z ...ii.ii............................................................................................ 5100/8946
2019-08-20T18:57:27.4528885Z .................................................................................................... 5300/8946
2019-08-20T18:57:34.4970373Z ...........................................................i........................................ 5400/8946
2019-08-20T18:57:41.8624854Z .................................................................................................... 5500/8946
2019-08-20T18:57:50.5032955Z .................................................................................................... 5600/8946
2019-08-20T18:57:50.5032955Z .................................................................................................... 5600/8946
2019-08-20T18:58:01.3347447Z ....................................................ii...i..ii...........i.......................... 5700/8946
2019-08-20T18:58:25.4024974Z .................................................................................................... 5900/8946
2019-08-20T18:58:30.5880612Z .................................................................................................... 6000/8946
2019-08-20T18:58:30.5880612Z .................................................................................................... 6000/8946
2019-08-20T18:58:38.5929304Z .....................................................i..ii.......................................... 6100/8946
2019-08-20T18:59:06.4461003Z ...................................................................................................i 6300/8946
2019-08-20T18:59:08.7848895Z .................................................................................................... 6400/8946
2019-08-20T18:59:11.1439353Z .......................................................................i............................ 6500/8946
2019-08-20T18:59:14.3166663Z .................................................................................................... 6600/8946
---
2019-08-20T19:04:05.0920848Z  finished in 21.788
2019-08-20T19:04:05.1127947Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:04:05.2790280Z 
2019-08-20T19:04:05.2791005Z running 148 tests
2019-08-20T19:04:08.6898988Z i....iii......iii..iiii....i............................i..i..................i....i.........ii.i.i. 100/148
2019-08-20T19:04:10.6581216Z .iiii..............i.........iii.i......ii......
2019-08-20T19:04:10.6582255Z 
2019-08-20T19:04:10.6583910Z  finished in 5.545
2019-08-20T19:04:10.6793512Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:04:10.8480289Z 
---
2019-08-20T19:04:12.9914658Z  finished in 2.312
2019-08-20T19:04:13.0143714Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:04:13.1870858Z 
2019-08-20T19:04:13.1871089Z running 9 tests
2019-08-20T19:04:13.1871891Z iiiiiiiii
2019-08-20T19:04:13.1872301Z 
2019-08-20T19:04:13.1872348Z  finished in 0.173
2019-08-20T19:04:13.2067241Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:04:13.3735533Z 
---
2019-08-20T19:04:32.0949888Z  finished in 18.887
2019-08-20T19:04:32.1159280Z Check compiletest suite=debuginfo mode=debuginfo-gdb+lldb (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:04:32.2894190Z 
2019-08-20T19:04:32.2894481Z running 122 tests
2019-08-20T19:04:57.6363169Z .iiiii...i.....i..i...i..i.i.i..i.ii..i.i.....i..i....i..........iiii..........i...ii...i.......ii.i 100/122
2019-08-20T19:05:02.5658207Z .i.i......iii.i.....ii
2019-08-20T19:05:02.5658679Z 
2019-08-20T19:05:02.5662232Z  finished in 30.450
2019-08-20T19:05:02.5669754Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:05:02.5670115Z Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
2019-08-20T19:19:36.1053146Z running 970 tests
2019-08-20T19:19:36.1217393Z .................................................................................................... 100/970
2019-08-20T19:19:36.1322550Z .................................................................................................... 200/970
2019-08-20T19:19:36.1418081Z .................................................................................................... 300/970
2019-08-20T19:19:36.2328098Z ................................................................................................ii.. 400/970
2019-08-20T19:19:36.3509409Z .................................................................................................... 600/970
2019-08-20T19:19:36.3757809Z .................................................................................................... 700/970
2019-08-20T19:19:36.3870675Z .................................................................................................... 800/970
2019-08-20T19:19:36.9312814Z .................................................................................................... 900/970
2019-08-20T19:19:36.9312814Z .................................................................................................... 900/970
2019-08-20T19:19:38.1225310Z ......................................................................
2019-08-20T19:19:38.1231941Z test result: ok. 968 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out
2019-08-20T19:19:38.1232349Z 
2019-08-20T19:19:38.1241024Z    Doc-tests core
2019-08-20T19:19:43.5558545Z 
2019-08-20T19:19:43.5566893Z running 2380 tests
2019-08-20T19:19:57.2493150Z ......iiiii......................................................................................... 100/2380
2019-08-20T19:20:11.1194321Z .........................................................................ii......................... 200/2380
2019-08-20T19:20:43.8391928Z .................................................................................................... 400/2380
2019-08-20T19:20:43.8391928Z .................................................................................................... 400/2380
2019-08-20T19:20:56.1957455Z ..............................i..i.................iiii............................................. 500/2380
2019-08-20T19:21:22.7430922Z .................................................................................................... 700/2380
2019-08-20T19:21:35.9339874Z .................................................................................................... 800/2380
2019-08-20T19:21:49.1624488Z .................................................................................................... 900/2380
2019-08-20T19:22:02.4776207Z .................................................................................................... 1000/2380
---
2019-08-20T19:27:53.0446596Z 
2019-08-20T19:27:53.0447725Z running 991 tests
2019-08-20T19:28:17.9362255Z i................................................................................................... 100/991
2019-08-20T19:28:31.7987500Z .................................................................................................... 200/991
2019-08-20T19:28:41.1487525Z .................iii......i......i...i......i....................................................... 300/991
2019-08-20T19:28:46.5907169Z .................................................................................................... 400/991
2019-08-20T19:28:55.2114088Z ..................................i..i.................................ii........................... 500/991
2019-08-20T19:29:11.3850391Z .................................................................................................... 700/991
2019-08-20T19:29:11.3850391Z .................................................................................................... 700/991
2019-08-20T19:29:20.1938327Z .................iiii............................................................................... 800/991
2019-08-20T19:29:35.8802339Z .................................................................................................... 900/991
2019-08-20T19:29:44.0625022Z .......................................iiii................................................
2019-08-20T19:29:44.0625864Z 
2019-08-20T19:29:44.0835838Z  finished in 264.478
2019-08-20T19:29:44.0853338Z Testing unwind stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2019-08-20T19:29:44.2512366Z     Finished release [optimized] target(s) in 0.16s
---
2019-08-20T19:31:37.3936862Z    Compiling rustc_driver v0.0.0 (/checkout/src/librustc_driver)
2019-08-20T19:31:37.3937673Z error[E0583]: file not found for module `tests`
2019-08-20T19:31:37.3938117Z  --> src/librustc_driver/args.rs:9:5
2019-08-20T19:31:37.3938489Z   |
2019-08-20T19:31:37.3938889Z 9 | mod tests;
2019-08-20T19:31:37.3940120Z   |
2019-08-20T19:31:37.3940120Z   |
2019-08-20T19:31:37.3940640Z   = help: name the file either args/tests.rs or args/tests/mod.rs inside the directory "src/librustc_driver"
2019-08-20T19:31:37.3941199Z error: aborting due to previous error
2019-08-20T19:31:37.3941380Z 
2019-08-20T19:31:37.3942704Z For more information about this error, try `rustc --explain E0583`.
2019-08-20T19:31:37.3943245Z error: Could not compile `rustc_driver`.
2019-08-20T19:31:37.3943245Z error: Could not compile `rustc_driver`.
2019-08-20T19:31:37.3943438Z 
2019-08-20T19:31:37.3943821Z To learn more, run the command again with --verbose.
2019-08-20T19:31:37.3943979Z 
2019-08-20T19:31:37.3944104Z 
2019-08-20T19:31:37.3945206Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "2" "--release" "--locked" "--color" "always" "--features" "" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "-p" "rustc_driver" "--" "--quiet"
2019-08-20T19:31:37.3946120Z 
2019-08-20T19:31:37.3946255Z 
2019-08-20T19:31:37.3946400Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-08-20T19:31:37.3946567Z Build completed unsuccessfully in 1:36:09
2019-08-20T19:31:37.3946567Z Build completed unsuccessfully in 1:36:09
2019-08-20T19:31:37.3946713Z == clock drift check ==
2019-08-20T19:31:37.3946868Z   local time: Tue Aug 20 19:31:37 UTC 2019
2019-08-20T19:31:37.3947038Z   network time: Tue, 20 Aug 2019 19:31:37 GMT
2019-08-20T19:31:37.3947180Z == end clock drift check ==
2019-08-20T19:31:37.8512557Z ##[error]Bash exited with code '1'.
2019-08-20T19:31:37.8556060Z ##[section]Starting: Checkout
2019-08-20T19:31:37.8557814Z ==============================================================================
2019-08-20T19:31:37.8557892Z Task         : Get sources
2019-08-20T19:31:37.8557941Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

This will make @path work with miri and other non-standard entrypoints.

Also since this simplifies librustc_driver::args, move it into a simple source file. Also
remove the tests since they're doing nothing more than checking `str::lines` has the right
behaviour.
@jsgf
Copy link
Contributor Author

jsgf commented Aug 22, 2019

@bors r+

@bors
Copy link
Contributor

bors commented Aug 22, 2019

📌 Commit d949774 has been approved by jsgf

@bors
Copy link
Contributor

bors commented Aug 22, 2019

⌛ Testing commit d949774 with merge 54ebbd0...

bors added a commit that referenced this pull request Aug 22, 2019
rustc: implement argsfiles for command line

Many tools, such as gcc and gnu-ld, support "args files" - that is, being able to specify @file on the command line.  This causes `file` to be opened and parsed for command line options. They're separated with whitespace; whitespace can be quoted with double or single quotes, and everything can be \\-escaped. Args files may recursively include other args files via `@file2`.

See https://sourceware.org/binutils/docs/ld/Options.html#Options for the documentation of gnu-ld's @file parameters.

This is useful for very large command lines, or when command lines are being generated into files by other tooling.
@bors
Copy link
Contributor

bors commented Aug 22, 2019

💥 Test timed out

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 22, 2019
@jsgf
Copy link
Contributor Author

jsgf commented Aug 22, 2019

@bors retry

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 22, 2019
@bors
Copy link
Contributor

bors commented Aug 22, 2019

⌛ Testing commit d949774 with merge 201e52e...

bors added a commit that referenced this pull request Aug 22, 2019
rustc: implement argsfiles for command line

Many tools, such as gcc and gnu-ld, support "args files" - that is, being able to specify @file on the command line.  This causes `file` to be opened and parsed for command line options. They're separated with whitespace; whitespace can be quoted with double or single quotes, and everything can be \\-escaped. Args files may recursively include other args files via `@file2`.

See https://sourceware.org/binutils/docs/ld/Options.html#Options for the documentation of gnu-ld's @file parameters.

This is useful for very large command lines, or when command lines are being generated into files by other tooling.
@bors
Copy link
Contributor

bors commented Aug 22, 2019

☀️ Test successful - checks-azure
Approved by: jsgf
Pushing 201e52e to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Aug 22, 2019
@bors bors merged commit d949774 into rust-lang:master Aug 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants