Enable screenshots functionality

[inikulin]

No description provided.

[AlexanderMoskovkin]

I'm working on this

[AlexanderMoskovkin]

@inikulin @VasilyStrelyaev please check it and write your opinion regarding the issue:

• we have the following API for screenshots:

runner.screenshots (path, takeOnFails = false)

• we don't make any screenshot if path is not set

When screenshotPath is set:

• act.screenshot() in the step:
  we save the screenshot to the {screenshotPath}/{testRunId}/{userAgent}/{stepName}.png file, for example: screenshots/1445437598847/Chrome 46.0.2490 _ Windows 7 0.0.0/1.Make screenshot step.png.
  NOTE: now our user arent looks like Chrome 46.0.2490 / Windows 7 0.0.0. We should replace the / symbol or use only the first part (Chrome 46.0.2490) for the folder name
• act.screenshot(filePath):
  we save the screenshot to the filePath relative to the test dir regardless the screenshotPath. Example:

act.screenshot('my-screenshots/1.png');

Saves to {testDir}/my-screenshots/1.png

• an error in the test: if the takeOnFails is true we save the screenshot like in the act.screenshot() case. I have two questions here:

1. should we add the Error prefix to the filename? (./Error - 1.Failed step.png)

2. The error may be appeared on page load when test steps are not started. How we will name the screenshot file, Error - Page load or somehow else?

And one more thing. Now we always create a thumbnail for each screenshot and put it in the thumbnails dir:

./
---screenshot.png
---thumbnails/
   ---screenshot.png

Maybe should we make an option for thumbnails creating?

[VasilyStrelyaev]

1. A postfix probably - to preserve the sort order in case steps are numbered? 3.My third step - Fail.png ? And, yeah, Fail, not to confuse with JS errors that can be skipped and all... Agree?
2. I like it. Although, if the test fails on page load, we will have no more screenshots... But, for clarity, we probably should add - Page Load
3. Can it somehow affect the gallery in Control Panel / Studio (even if this option will be available in code only - still - I could save the report to the same folder as I usually do in Studio - right ) ? If no, than OK.

[VasilyStrelyaev]

We should replace the / symbol

With @, hm?

[VasilyStrelyaev]

Or enclose the OS name in brackets

[VasilyStrelyaev]

Why do I use act.screenshot(filePath) anyway?
I have the screeshot overwritten each time... Weird...

[AlexanderMoskovkin]

Why do I use act.screenshot(filePath) anyway?

It's actual for CI. The typical scenario is: save screenshots during test run on the testing machine (it's maybe a virtual machine), compare them with original images then clear them or save to some storage

[AlexanderMoskovkin]

3. Can it somehow affect the gallery in Control Panel / Studio

The studio will works with screenshots as before

[inikulin]

I have the screeshot overwritten each time... Weird..

Wrong. You will just specifiy the root dir for your screenshots. Then for each test run dir with the unique ID will be created. So, nothing will be overwritten

[AlexanderMoskovkin]

Wrong. You will just specifiy the root dir for your screenshots.

I suppose, @VasilyStrelyaev writes not about runner.screenshot(path) that set the root dir, but about the action with argument act.screenshot('./some-nane.png'). In this case you set fixed path for the screenshot and it will be overwritten each test run. But as I wrote above it's for CI

[inikulin]

@VasilyStrelyaev @AlexanderMoskovkin Well, if you explicitly specify the path to the screenshot, then it's uniqueness is your burden now. I don't see any problem here.

[AlexanderMoskovkin]

@VasilyStrelyaev @inikulin
Guys, one more question about reporting with screenshots. We should add screenshot path to errors and to a test that contains act.screenshot actions.
So, what do you think about the following reports?

err fixture1Test1
    1) Chrome 15.0.874 / Mac OS X 10.8.1
       Assertion failed at step "Check for arrays equality":


           eq(["12345678901"], ["00000000000"])

       Arrays differ at index 0:

       Expected: [0]: "12345678901"
       Actual:   [0]: "00000000000"
                   ^

       Screenshot: D:\screenshots\1445437598847\Chrome 46.0.2490 @
     Windows 7 0.0.0\1.Failed step - Fail.png


    2) Chrome 15.0.874 / Mac OS X 10.8.1
       IFrame loading timed out.

       Screenshot: D:\screenshots\1445437598847\Chrome 46.0.2490 @
     Windows 7 0.0.0\Page Load - Fail.png

ok fixture2Test2
    Screenshots: 
        D:\screenshots\1445437598847\Chrome 46.0.2490 @ Windows
    7 0.0.0\1.The first step.png
        D:\screenshots\1445437598847\Chrome 46.0.2490 @ Windows
    7 0.0.0\2.The second step.png

ok fixture2Test3

Should we highlight the Screenshot: label and screenshot path with a specific color?

[inikulin]

1)Let's move all error screenshots to the errors dir in the test screenshot path, so you will not need to add suffix for them
2)In report just give the path to the test screenshots dir

E.g.:

Running tests in: Chrome, Firefox

  fixture1 (./fixture1.js)
    ✓ fixture1test1 (unstable) (screenshots: D:\screenshots\1445437598847)
    ✖ fixture1test2 (screenshots: D:\screenshots\144543754328847)

        1) Chrome
           Assertion failed at step "Step":

               eq(["12345678901"], ["00000000000"])

           Arrays differ at index 0:

           Expected: [0]: "12345678901"
           Actual:   [0]: "00000000000"
                           ^


        2) Chrome
           Assertion failed at step "Step":

               notEq("test", "test")

           Expected: not "test"
           Actual:   "test"


        3) Firefox
           Assertion failed at step "Step":

               ok(false)

           Expected: not null, not undefined, not false, not NaN and not
           ''
           Actual:   false


    ✓ fixture1test3

  fixture2 (./fixture2.js)
    ✓ fixture2test1 (screenshots: D:\screenshots\4345437598847)
    ✓ fixture2test2 (screenshots: D:\screenshots\6545437598847)


  2/6 failed (15m 25s)

[AlexanderMoskovkin]

1. Which option:
   screenshots-path/errors/{testId}/{userAgent}/file.png
screenshots-path/{testId}/errors/{userAgent}/file.png}
screenshots-path/{testId}/{userAgent}/errors/file.png ?

2. I like it

[inikulin]

1. Which option:
   screenshots-path/errors/{testId}/{userAgent}/file.png
   screenshots-path/{testId}/errors/{userAgent}/file.png}
   screenshots-path/{testId}/{userAgent}/errors/file.png ?

I like the third one

[VasilyStrelyaev]

Cool!

[saeidzebardast]

@AlexanderMoskovkin is there any option to disable creating thumbnails?

[AlexanderMoskovkin]

@saeidzebardast Not yet, but we're going to do it in the contex of the #1550 issue.

[graeme-plangrid]

I can't seem to find anywhere so I'l ask here. Has there been any talk of supporting screenshot name pattern? I'd love if the created screenshots generated on failure could be named based on the test name...but currently it only seems to be named via the index, as I get: 1.png

[AlexanderMoskovkin]

Hi @graeme-plangrid,

Take a look at the #2152 issue. Does this proposal feet your need. Feel free to add you comments there. We already have a PR for this and we'll review and merge it once we're done with our current tasks

[peterpfberlin]

@AlexanderMoskovkin

i would appreciate a clarification:

If im running old testcafe tests, which were recorded manually from the browser recorder, is this currently the only was in which screenshots are still supported when Testcafe is integrated in a command line/CI environment

[AlexanderMoskovkin]

Hi @peterpfberlin,

TestCafe supports screenshot functionality for both tests captured with the browser recorder and written using JavaScript API.

[graemeRharvey]

@AlexanderMoskovkin Yes, this looks good! Thanks for pointing me in the right direction.

[lock]

This thread has been automatically locked since it is closed and there has not been any recent activity. Please open a new issue for related bugs or feature requests. We recommend you ask TestCafe API, usage and configuration inquiries on StackOverflow.