Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugin "stepByStepReport" doesn't synchronize screenshots with steps #4405

Closed
mirao opened this issue Jun 21, 2024 · 13 comments · Fixed by #4413
Closed

Plugin "stepByStepReport" doesn't synchronize screenshots with steps #4405

mirao opened this issue Jun 21, 2024 · 13 comments · Fixed by #4413

Comments

@mirao
Copy link
Contributor

mirao commented Jun 21, 2024

What are you trying to achieve?

A screenshot should be done after each step in the scenario - there are 4 steps in the scenario, so the screenshot should display a result after each of those 4 steps.

What do you get instead?

All screenshots are done at the very beginning of the test.
E.g. you can't see there filling of the text in the search field or a result of the search query:
record.zip

image

Provide console output if related. Use --verbose mode for more details.

$ codeceptjs run --verbose
***************************************
nodeInfo:  18.20.3
osInfo:  Linux 6.5 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
cpuInfo:  (16) x64 AMD Ryzen 9 7940HS w/ Radeon 780M Graphics
chromeInfo:  126.0.6478.114
edgeInfo:  "N/A"
firefoxInfo:  undefined
safariInfo:  N/A
If you need more detailed info, just run this: npx codeceptjs info
***************************************
CodeceptJS v3.6.4 #StandWithUkraine
Using test root "/home/mirao/workspace/tmp"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport, tryTo, retryFailedStep, retryTo, eachElement

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0000.png
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0001.png
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0002.png
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0003.png
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 1688ms


  OK  | 1 passed   // 2s
◉ Step-by-step preview: file:///home/mirao/workspace/tmp/output/records.html

Provide test source code if related

Feature('StepByStep');

Scenario('test something',  ({ I }) => {
    I.amOnPage("https://codecept.io")
    I.fillField("#algolia-search-input", "Step by Step")
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

Details

  • CodeceptJS version: 3.6.4
  • NodeJS Version: 18.20.3
  • Operating System: Ubuntu 22.04
  • Playwright 1.44.1
  • Configuration file:
import { setHeadlessWhen, setCommonPlugins } from '@codeceptjs/configure';
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

export const config: CodeceptJS.MainConfig = {
  tests: './*_test.ts',
  output: './output',
  helpers: {
    Playwright: {
      browser: 'chromium',
      url: 'http://localhost',
      show: true
    }
  },
  plugins: {
    stepByStepReport: {
      enabled: true,
        deleteSuccessful: false,
      },
  },
  include: {
    I: './steps_file'
  },
  name: 'tmp'
}
@mirao mirao changed the title Plugin "stepByStep" doesn't synchronize screenshots with steps Plugin "stepByStepReport" doesn't synchronize screenshots with steps Jun 21, 2024
@mirao
Copy link
Contributor Author

mirao commented Jun 24, 2024

It seems that CodeceptJS creates a screenshot when adding the scenario steps into a "recorder" and not when executing the steps.

@mirao
Copy link
Contributor Author

mirao commented Jun 24, 2024

Looks like broken by #4385 .
It works for me with 3.6.2 and doesn't work with 3.6.3 and 3.6.4

3.6.2:

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
    I am on page "https://codecept.io"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0000.png
    I fill field "#algolia-search-input", "Step by Step"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0001.png
    I press key "Enter"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0002.png
    I see element "#stepbystepreport"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0003.png
  ✔ OK in 2971ms


  OK  | 1 passed   // 4s

CC: @kobenguyent

@kobenguyent
Copy link
Collaborator

hey @mirao would you mind trying this test version [email protected]? Hope this resolves the issue.

  Github login
    I am on page "https://github.com"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0000.png
    I click "Sign in"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0001.png
    I fill field "Username", "davert"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0002.png
    I fill field "Password", "123345"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0003.png
    I click "Login"
    [1]  Error (Non-Terminated) | Error: Clickable element "Login" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1] Error | Error: Clickable element "Login" was not found by text|CSS|XPath undefined...
    [1] <teardown>  Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/Github_login.failed.png
 › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0004.png
  ✖ FAILED in 1940ms

@mirao
Copy link
Contributor Author

mirao commented Jun 26, 2024

@kobenguyent It doesn't work for me with [email protected]

CodeceptJS v3.6.5-beta.1 #StandWithUkraine
Using test root "/home/mirao/workspace/codeceptjs/tests/my"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport

My --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 4011ms

And I'm getting only record_hash/index.html without screenshots and without records.html.

This is with 3.6.4:

CodeceptJS v3.6.4 #StandWithUkraine
Using test root "/home/mirao/workspace/codeceptjs/tests/my"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport

My --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0000.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0001.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0002.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0003.png
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 3081ms


  OK  | 1 passed   // 4s
◉ Step-by-step preview: file:///home/mirao/workspace/codeceptjs/tests/my/output/records.html

@kobenguyent
Copy link
Collaborator

@mirao sorry, wrong push, please try with this test version [email protected]

@mirao
Copy link
Contributor Author

mirao commented Jun 27, 2024

@kobenguyent It works.
But I noticed that if you have BeforeSuite() in a test, it fails 🐛 (it didn't fail with 3.6.4)

Test

Feature("StepByStep");

BeforeSuite(({ I }) => {
    I.say("Before Suite");
});

Scenario("test something", ({ I }) => {
    I.amOnPage("https://codecept.io");
    I.fillField("#algolia-search-input", "Step by Step");
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

output

CodeceptJS v3.6.5-beta.2 #StandWithUkraine
Using test root "/home/mirao/workspace/codeceptjs/tests/my"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport, tryTo, retryFailedStep, retryTo, eachElement

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
    --- STARTED "before all" hook: BeforeSuite for "test something" ---
    I say "Before Suite"
    [1]  Error (Non-Terminated) | TypeError: Cannot read properties of undefined (reading 'toString') | (err) => { step.status = 'failed'; step.endTime = ...
    [1] Error | TypeError: Cannot read properties of undefined (reading 'toString') (e) => { const err = recorder.getAsyncErr() === nu...
 › <screenshotOnFail> BeforeSuite/AfterSuite do not have any access to the browser, hence it could not take screenshot.
 › <stepByStepReport> BeforeSuite/AfterSuite do not have any access to the browser, hence it could not take screenshot.
    [1] <teardown>  Stopping recording promises
  ✖ FAILED in 8ms

    [2]  Starting recording promises

-- FAILURES:

  1) StepByStep
       "before all" hook: BeforeSuite for "test something":
     Cannot read properties of undefined (reading 'toString')
      at persistStep (node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@jest_pddntehogpft3mtwd3fjgeop4e/node_modules/codeceptjs/lib/plugin/stepByStepReport.js:158:63)
      at /home/mirao/workspace/codeceptjs/tests/my/node_modules/.pnpm/[email protected]_@[email protected]_@[email protected]_@[email protected]__@jest_pddntehogpft3mtwd3fjgeop4e/node_modules/codeceptjs/lib/plugin/stepByStepReport.js:110:60
  
  Scenario Steps:
  - I.say("Before Suite") at Context.<anonymous> (./StepByStep_test.ts:4:7)
  


  FAIL  | 0 passed, 1 failed   // 211ms

@kobenguyent
Copy link
Collaborator

kobenguyent commented Jun 27, 2024

@mirao one more try with [email protected], please.

login --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
    --- STARTED "before all" hook: BeforeSuite for "test something" ---
    I say "Before Suite"
   Before Suite
    --- ENDED "before all" hook: BeforeSuite for "test something" ---
  test something
    I am on page "https://codecept.io"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0000.png
    I fill field "#algolia-search-input", "Step by Step"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0001.png
    I press key "Enter"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0002.png
    I see element "#stepbystepreport"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0003.png
  ✔ OK in 3318ms


  OK  | 1 passed   // 5s
◉ Step-by-step preview: file:///Users/t/Desktop/projects/codeceptjs-playwright-fun/output/records.html

@mirao
Copy link
Contributor Author

mirao commented Jun 27, 2024

It works well with [email protected]. Only issue is that AfterSuite() isn't omitted from the plugin. It doesn't cause a failure on exit, but it looks nasty. The issue is even in 3.6.4
Could you fix it, please?

test:

Feature("StepByStep");

BeforeSuite(({ I }) => {
  I.say("Before Suite");
});

Scenario("test something", ({ I }) => {
  I.amOnPage("https://codecept.io");
  I.fillField("#algolia-search-input", "Step by Step");
  I.pressKey("Enter");
  I.seeElement("#stepbystepreport");
});

AfterSuite(({ I }) => {
  I.say("After Suite");
});

output:

CodeceptJS v3.6.5-beta.3 #StandWithUkraine
Using test root "/home/mirao/workspace/tmp"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport, tryTo, retryFailedStep, retryTo, eachElement

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
    --- STARTED "before all" hook: BeforeSuite for "test something" ---
    I say "Before Suite"
   Before Suite
    --- ENDED "before all" hook: BeforeSuite for "test something" ---
  test something
    I am on page "https://codecept.io"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0000.png
    I fill field "#algolia-search-input", "Step by Step"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0001.png
    I press key "Enter"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0002.png
    I see element "#stepbystepreport"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0003.png
  ✔ OK in 4392ms

    --- STARTED "after all" hook: AfterSuite for "test something" ---
    I say "After Suite"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0004.png
    › <Can't save step screenshot: Error: page.screenshot: Target page, context or browser has been closed> 
   After Suite
    --- ENDED "after all" hook: AfterSuite for "test something" ---

  OK  | 1 passed   // 5s
◉ Step-by-step preview: file:///home/mirao/workspace/tmp/output/records.html

@kobenguyent
Copy link
Collaborator

@mirao let's try with [email protected].

@mirao
Copy link
Contributor Author

mirao commented Jun 28, 2024

@kobenguyent Works well now, thanks.

One more issue: When you have two scenarios and the 1st one fails, then the plugin stops creating screenshots for the 2nd scenario and finally records.html isn't created 🐛
Could you look at it? Or may I create a separate issue for it? The bug is even in 3.6.4

test:

Feature("StepByStep");

BeforeSuite(({ I }) => {
    I.say("Before Suite");
});

Scenario("test something 1", ({ I }) => {
    I.amOnPage("https://codecept.io");
    I.fillField("#blabla", "Step by Step"); // A typo in element
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

Scenario("test something 2", ({ I }) => {
    I.amOnPage("https://codecept.io");
    I.fillField("#algolia-search-input", "Step by Step");
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

AfterSuite(({ I }) => {
    I.say("After Suite");
});

output:

StepByStep --
    [1]  Starting recording promises
    Timeouts: 600
 › [Session] Starting singleton browser session
 › Open login screen
    --- STARTED "before all" hook: BeforeSuite for "test something 1" ---
    I say "Before Suite"
   Before Suite
    --- ENDED "before all" hook: BeforeSuite for "test something 1" ---
  test something 1
 › Test Timeout: 600s
    I am on page "https://codecept.io"
    › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/s4h-web/output/record_0fc08ef3432d0381979450d4c2fa03d13b0844e2d6d8a7b6ec83da5cc693519e/0000.png
    I fill field "#blabla", "Step by Step"
    [1]  Error (Non-Terminated) | Error: Field "#blabla" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1]  Error (Non-Terminated) | Error: Field "#blabla" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1]  Error (Non-Terminated) | Error: Field "#blabla" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1] Error | Error: Field "#blabla" was not found by text|CSS|XPath undefined...
    [1] <teardown>  Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/s4h-web/output/test_somet_1719563268.failed.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/s4h-web/output/record_0fc08ef3432d0381979450d4c2fa03d13b0844e2d6d8a7b6ec83da5cc693519e/0001.png
  ✖ FAILED in 2045ms

    [2]  Starting recording promises
  test something 2
 › Test Timeout: 600s
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 2133ms

    --- STARTED "after all" hook: AfterSuite for "test something 2" ---
    I say "After Suite"
   After Suite
    --- ENDED "after all" hook: AfterSuite for "test something 2" ---

-- FAILURES:

  1) StepByStep
       test something 1:
     Field "#blabla" was not found by text|CSS|XPath
      at new ElementNotFound (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/[email protected]_patch_hash=eotzuczidxj7tycvn36zpciqru_@[email protected]_@babel+prese_tysehjua4vp7stc2brmcbmndoe/node_modules/codeceptjs/lib/helper/errors/ElementNotFound.js:15:11)
      at assertElementExists (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/[email protected]_patch_hash=eotzuczidxj7tycvn36zpciqru_@[email protected]_@babel+prese_tysehjua4vp7stc2brmcbmndoe/node_modules/codeceptjs/lib/helper/Playwright.js:3714:11)
      at Playwright.fillField (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/[email protected]_patch_hash=eotzuczidxj7tycvn36zpciqru_@[email protected]_@babel+prese_tysehjua4vp7stc2brmcbmndoe/node_modules/codeceptjs/lib/helper/Playwright.js:1741:5)
  
  Scenario Steps:
  - I.fillField("#blabla", "Step by Step") at Test.<anonymous> (./StepByStep_test.ts:9:7)
  - I.amOnPage("https://codecept.io") at Test.<anonymous> (./StepByStep_test.ts:8:7)
  
  Artifacts:
  - screenshots: /home/mirao/workspace/codeceptjs/tests/s4h-web/output/record_0fc08ef3432d0381979450d4c2fa03d13b0844e2d6d8a7b6ec83da5cc693519e/0000.png
  - screenshot: /home/mirao/workspace/codeceptjs/tests/s4h-web/output/test_somet_1719563268.failed.png


  FAIL  | 1 passed, 1 failed   // 6s

@mirao
Copy link
Contributor Author

mirao commented Jun 28, 2024

One more issue: When you have two scenarios and the 1st one fails, then the plugin stops creating screenshots for the 2nd scenario and finally records.html isn't created 🐛

Well, it seems to be by design. disableScreenshotOnFail is true by default. If I set it to false then it works as expected.
https://codecept.io/plugins/#stepbystepreport

@mirao
Copy link
Contributor Author

mirao commented Jun 28, 2024

@kobenguyent So you can close the task, everything seems to be solved. Thanks.

@kobenguyent
Copy link
Collaborator

@mirao thanks for your testing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants