src/test/bench: restructure

[emberian]

This should make benchmarks easier to understand. But, it doesn't work.
BENCH_RS in mk/tests.mk has everything, from what I can tell in remake, but
only those that are direct children of src/test/bench get build and run.
@graydon, can you lend your expertise? I can't make heads or tails of this
makefile.

[brson]

The code that configures the compiletest driver to run the benchmarks is https://github.com/cmr/rust/blob/2272fa28b29c588068a05580afd800fb77a027ff/mk/tests.mk#L505

These variables are plugged into this macro: https://github.com/cmr/rust/blob/2272fa28b29c588068a05580afd800fb77a027ff/mk/tests.mk#L505

This says to run all the .rs files in src/test/bench, but now the benchmarks are no longer in that directory. The compiletest driver accepts a single --src-base argument and runs files in that directory (if it descended into subdirectories then it would end up trying to run some non-test files under test/run-pass/*/).

There are probably three ways to make this work: 1) teach compiletest to accept multilpe src-basees, then adjust tests.mk to support that; 2) split the benchmark test runs into bench, bench-rt, bench-shootout, etc., add the appropriate new configs to tests.mk; 3) teach compiletest to descend into subdirectories to find tests, xfail all the non-tests that would be discovered under this scheme.

My preference is 1, but it's also the hardest. I don't like 3 that much because it will create even more 'xfail-test not a test'-type comments.

[emberian]

@brson I'll take a shot at compiletest in the morning. thanks!

[emberian]

@brson So the compiletest bits are done, but I can't figure out what to do to the makefile.

I pushed my attempt, but it outputs:

x86_64-unknown-linux-gnu/stage1/bin/compiletest --compile-lib-path x86_64-unknown-linux-gnu/stage1/lib --run-lib-path x86_64-unknown-linux-gnu/stage1/lib/rustc/x86_64-unknown-linux-gnu/lib --rustc-path x86_64-unknown-linux-gnu/stage1/bin/rustc --clang-path /usr/bin/clang++ --llvm-bin-path /home/cmr/hacking/rust/build/llvm/x86_64-unknown-linux-gnu/Release+Asserts/bin --aux-base /home/cmr/hacking/rust/src/test/auxiliary/ --stage-id stage1-x86_64-unknown-linux-gnu --target x86_64-unknown-linux-gnu --adb-path=/usr/bin/adb --adb-test-dir= --rustcflags " -O --target=x86_64-unknown-linux-gnu" --verbose --src-base /home/cmr/hacking/rust/src/test/bench bench/rt bench/shootout bench/std/ --build-base x86_64-unknown-linux-gnu/test/bench/ --ratchet-metrics tmp/check-stage1-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-bench-metrics.json --mode run-pass --logfile tmp/check-stage1-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-bench.log && touch tmp/check-stage1-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-bench.ok

Which, as you can see, has the wrong src-base except for the first one. I don't really know make at all.

[brson]

@cmr: here's a diff against your patch:

-               $(foreach base,$$(CTEST_SRC_BASE_$(4)), \
-               --src-base $$(S)src/test/$(base))/ \
+        $$(foreach base,$$(CTEST_SRC_BASE_$(4)), \
+          --src-base $$(S)src/test/$$(base))/ \

[catamorphism]

@cmr: Needs a rebase.

[catamorphism]

One of the pretty printer tests segfaulted -- could this be a stack size issue?

[alexcrichton]

That's odd, it's the second case of segfaulting during pretty printing:

http://buildbot.rust-lang.org/builders/auto-mac-64-nopt-t/builds/256/steps/test/logs/stdio

[emberian]

Seems there might be some latent OS X issues?

[huonw]

(I've seen more than 2 segfaults there in the day, fwiw.)