-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extreme stack traces #157
Comments
Please use our cukes group for support. There are some libraries to help deal with stack traces on NPM. We could reduce the traces a bit by filtering Cucumber's source from it. I'd be curious to see your complete testing stack though as 160 lines seem a bit much. Are you using webdriverjs or something similar? |
I wrote a little stack trace filter. It will only display stack trace "sites" related to your own code (based on the file path). It could be smarter but this is a good starting point. Add the following to var path = require('path');
var filteredPathPrefix = path.resolve(__dirname, '..', '..');
var originalPrepareStackTrace;
if (originalPrepareStackTrace = Error.prepareStackTrace) {
Error.prepareStackTrace = function (error, stack) {
var originalString = originalPrepareStackTrace(error, stack);
var string = 'Error: ' + error.message + "\n";
var lines = originalString.replace('Error: ' + error.message + "\n", '').replace(/\s*$/, '').split("\n");
var i;
for (i = 0; i < lines.length; i++) {
var line = lines[i];
var callSite = stack[i];
if (callSite.getFileName().indexOf(filteredPathPrefix) == 0)
string = string + line + "\n";
};
return string;
};
} I'm reopening the issue as it could make sense to get something similar (better) built in Cucumber.js. |
Thanks for reopening. Here is a simple failing test that gives me 160 lines of stack trace:
I don't have much experience with cucumber but I do have with testing in general and I really think this is a hindered (even if there are workarounds), By default IMHO only the error message needs to be printed. |
Well I do not completely agree. When doing TDD/BDD, the stack trace helps you figure out what should be done next. An error message alone is often not enough to work efficiently. That being said, I agree everything related to the testing framework should be hidden from the stack trace. |
I actually agree with that and did so before as well. It was a weird choice of words on my behalf in my last post. |
My stack trace is
"at anonymous" goes on for pages and pages for one single assertion error. FWIW, I'm running with chai, chai-as-promised, protractor and selenium. Please, please, please add a flag to turn off stack traces? In a webdriver context, it seems the stack traces are not going to be very helpful -- all I really need to know is that my webapp isn't doing the right thing! I have to scroll so far up to find the actual errors... I'm going to try hacking my console but I wish the stack trace religion could soften for those of us who really get no value from them. |
Doh! Somehow I managed to overlook that there is a fine solution sitting right in this issue. I should get more sleep. Thank you @jbpros, your snippet was exactly what I needed to get me started. Here is my webdriver-compatible patch: // monkey-patch stack trace more-or-less compatible with webdriver
var isMyStack = new RegExp(
path.join(
__dirname,
'../..', // go up two dirs to get to repo root -- ymmv, adjust accordingly
'[^node_modules/]' // exclude node_modules
)
.replace(/\//g, '\\/')
);
Error.prepareStackTrace = function (err, stack) {
var i;
var frame;
var results = [];
for (i = 0; i < stack.length; i++) {
frame = stack[i].toString();
if (isMyStack.test(frame)) {
results.push(' at ' + frame);
}
}
// believe it or not you need to let webdriver throw away the first line
// so prepend a newline if intending to return anything
return results.length ? '\n' + results.join('\n') : '';
}; It is based on a regular expression that keeps away any code that is coming from outside of my repo or from node_modules in my repo. This might be a fairly decent pattern for others, subject to modifying the relative directory resolution for the location of the code that does the monkeying (I happen to do it in my World module). I certainly hope it helps out anyone who is using webdriver and cucumber-js together. For me, it gets rid of the protractor, webdriver and cucumber internals, which together are a very long list of uninteresting stack frames -- with this patch, I'm left with the two stack relevant trace lines and that makes me happy. The gory parts of the story... I suspect anyone who's using webdriver -- probably a healthy chunk of cucumber-js users -- might be in this boat. Both cucumber and webdriver are messing around with error objects. This seems maybe to be causing an issue in that:
/^ at(?: (?:(?:((?:new )?(?:\[object Object\]|[a-zA-Z_$][\w$]*(?:\.[a-zA-Z_$][\w$]*)*))\.|(new )))?((?:[a-zA-Z_$][\w$]*|<anonymous>))(?: \[as ([a-zA-Z_$][\w$]*)\])?)? (?:\((.*)\)|(.*))$/ So it drops anything that doesn't look like that and replaces is with " at <anonymous>":
The apparent lost rendering of an AssertionError message is only one minor symptom (in fact, I still see the AssertionError messages even after witnessing some of their demises, so they must be tough little buggers). I do believe that for the actual stack traces, anything that looks at all out of the ordinary is being "murdered" by webdriver -- and hence we "webdriver victims" sometimes get a long list of "at <anonymous>". It's also one of the reasons that, while your particular monkeypatch put me onto the solution, it didn't work for me. First, it depends on someone already having provided a prepareStackTrace override... I'm going to guess that it works if you run cucumber-js from the command line because it requires coffee-script, which does such a thing. So you're basing your patch on the coffee-script-tweaking process being available, I guess. If you don't run cucumber-js from the command line and don't otherwise include coffeescript, then it just flat out errors. If you do load coffeescript, it goes back to the bigger problem of webdriver-the-intolerant being very finicky about what a valid stack trace line looks like. After I figured out that I wasn't going to have a preexisting prepareStackTrace function to leverage, I was playing fast and loose with my formatting, and thus webdriver was clobbering my trace. One I got that worked out I was home free. |
Thanks Stu, this is very useful feedback. I think I understand the whole thing and I didn't realise To be honest, I'm not sure what to do now. It looks like we'll be fighting against stealth ennemies (i.e. other libraries) that are messing with the stack traces in unpredictable ways (and already kinda fighting between themselves). I guess we could detect the presence of webdriver, chai, test-my-thing and whatnot but I can foresee how things will go out of control pretty quickly. What if we didn't care about this at all in Cucumber.js but published separate packages to handle the different well-known libs? There could be a cucumber-js-webdriver package that would insert your filter, another one called cucumber-js-phantom that would address Phantom.js-specific issues. Those could even provide helper methods. Just like cucumber-rails. Thoughts? |
I think making stack traces optional altogether in the reporter would be great. I think factoring out reporters so that they can be developed independently of Cucumber-JS would be great (since I'm also needing the XUnit reporter that is in limbo lacking its own tests in a different issue here). As to what to do about stack traces in this reporter, it would great to provide a hook in a convenient place in Cucumber-JS so that we could customize what's reported. That all said, I think first and foremost if Cucumber didn't rewrite the stack traces, or rewrote them to V8's expected format (which seems to be compatible elsewhere as well), then webdriver in particular wouldn't choke on them, and in fact would be able to suppress them itself.... but to be honest I can't remember all of the code I was looking at, and the path through it that the stack trace content takes, to know for sure whether that would be enough for webdriver to not cause the clutter. At the least it wouldn't clobber everything with |
Sorry -- forgot to mention -- I think webdriver at the very least should be accommodated somehow. Too may Cucumber users are testing webapps in end-to-end scenarios for the current situation, where Selenium is the main way to drive them -- the patch works for me so I'm ok but it wasn't a pretty experience for me! :) |
For what it's worth I do the patch as a side of effect of "Node-require"'ing the module that exposes my World constructor via the protractor harness, all before the testing actually kicks in, then I make that constructor a global object so that I don't have to require it all over the place. |
@jbpros |
@jlin412 could you share some code to reproduce this? |
@jbpros Here is my code: Feature File: Step Definition:
Commandline:
Scenario Outline: To login and click on each menu tabs # test/e2e/features/smoke.feature:4 Done, without errors. |
@jlin412 and if you remove the filter, you got the whole suite running? If so, I'm afraid something else than us is trying to mess around with the stack trace and fail to do so because of the filtering. |
@jlin412 can you confirm everything runs as expected without the filter? |
@jbpros I confirm that everything still runs as expected without the filter. The stack_trace.js seems to filter out chai as promise exception or error. What should I do to enable that? Is there a way that I can modify the existing code that you provided. |
Following on from the above monkey patch, here's a shorter version: var path = require('path');
var filteredPathPrefix = path.resolve(__dirname, '..');
var originalPrepareStackTrace;
var stackTraceDepth = 1;
if (originalPrepareStackTrace = Error.prepareStackTrace) {
Error.prepareStackTrace = function (error, stack) {
var message = [error.name + ': ' + error.message];
for (var i = 0; i < stack.length; i++ ) {
if (stack[i].getFileName().indexOf(filteredPathPrefix) == 0) message.push(' at ' + stack[i]);
}
return message.slice(0, stackTraceDepth + 1).join('\n');
};
} |
@pumbers not sure what magic you did in your code. I didn't get the same issue anymore and error message filter works great. |
@pumbers Nice patch. Just what I needed. However, +1 for the command line option to suppress the stack trace. |
is this somehow usable? I just need the first line of an assert and the rest optional. It really takes the fun out of a gulp-watch stlyle development. |
I agree that it is particularly painful when it completely "buries the lead" in a watch console. I and others have overcome it. Have you tried any of our hacks? |
No. I have not yet. Lets get through these hacks: The first one by jbpros does not tell me where to actually call these. Do I just put a file there and magically it gets filtered? I tried following the instructions, but putting a file into the support folder does not cut it. The second one by stu-salsbury is a webdriver compatible change. I do not use webdriver. I use zombie.js pumbers proposes something that looks quite useful, bit there is no way to see where I have to apply the change to in order to make it work. All in all, I did not find any "patch" or "fix" in this thread that was useful to me. I am sorry, maybe I am just a bit too much of a "present me the fix on a plate" guy. But I really wonder why not ever someone has had the idea to fix this once and for all since cucumber is kind of a tool where you work from exception to exception in order to make all acceptance tests pass. So everyone ever has scrolled all the lines up for every condition that failed? regards |
In case you're using assertions, I've just released the library "cucumber-assert" which allows you to assert things and get a nice error message w/o 5 pages of stacktrace: https://github.com/djungowski/cucumber-assert |
Mocha encounter the same problem, and there's a nice solution using pipe to shorten stack trace. |
I think I just found a gem. clarify is a module that removes Node.js-related stack frames from the stack trace. Under the hood, it uses stack-chain module, which allows stack frames to be extended, filtered, and replaced. Therefore, I created a package require('hide-stack-frames-from')('cucumber') |
Thai, How about submitting a PR using this code to give an option to hide On Thursday, January 8, 2015, Thai Pangsakulyanont [email protected]
|
@jlin412 I am already happy putting that line of code in some I also use it to hide stack frames from Bluebird, and etc. That's why I prefer being able to configure all of them from a single supporting script. |
Works like a charms and fixes my problem. thanks |
thanks @pumbers ! |
Just a heads up, not sure if there is a better solution today on March 2016. I tried using Instead I used @laurelnaiad's snippet with the addition of the error message. Indeed the algorithm was removing any custom message sent by
So at the end I ended up keeping only the first line of any error message:
|
FWIW, as a developer using Cucumber JS for BDD, I like to have a full stack trace. As a developer creating BDD reports for the rest of the team, I like to use various reporter modules that hide the trace in an HTML report. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
I am just trying out cucmber-js and I notice that when a test fails, an enormous stack trace is produced (160 lines). This makes it really hard to understand which test failed and why. Is there some kind of workaround or solution to this problem?
The text was updated successfully, but these errors were encountered: