Skip to content

update Cucumber.HtmlFormatter dependency#749

Merged
gasparnagy merged 6 commits intomainfrom
update_html_formatter_dependency
Aug 11, 2025
Merged

update Cucumber.HtmlFormatter dependency#749
gasparnagy merged 6 commits intomainfrom
update_html_formatter_dependency

Conversation

@gasparnagy
Copy link
Contributor

🤔 What's changed?

⚡️ What's your motivation?

🏷️ What kind of change is this?

  • 📖 Documentation (improvements without changing code)
  • 🏦 Refactoring/debt/DX (improvement to code design, tooling, etc. without changing behaviour)
  • 🐛 Bug fix (non-breaking change which fixes a defect)
  • ⚡ New feature (non-breaking change which adds new behaviour)
  • 💥 Breaking change (incompatible changes to the API)

♻️ Anything particular you want feedback on?

📋 Checklist:

  • I've changed the behaviour of the code
    • I have added/updated tests to cover my changes.
  • My change requires a change to the documentation.
    • I have updated the documentation accordingly.
  • Users should know about my change
    • I have added an entry to the "[vNext]" section of the CHANGELOG, linking to this pull request & included my GitHub handle to the release contributors list.

This text was originally taken from the template of the Cucumber project, then edited by hand. You can modify the template here.

gasparnagy added a commit to reqnroll/Reqnroll.ExploratoryTestProjects that referenced this pull request Aug 11, 2025
@clrudolphi
Copy link
Contributor

clrudolphi commented Aug 11, 2025

The test failures so far are due to the mechanisms by which we're validating the HTML formatter output.
This will likely require some rethinking and reimplementation of the testing approach.
Problem 1: Comparing Envelopes embedded in the HTML against the ndjson.
Problem 2: HTML validation only checks the Envelopes; there is no validation of any other customized HTML (like Title).

Problem 1: Comparing Envelopes within the HTML
The validation suite is primarily uses acceptance test style testing. It presumes that there is a 'golden' copy of the expected output against which the test output can be compared. The validation suite uses the CCK as the source of the 'golden copy' of the ndjson. The CCK has no HTML output to use as 'golden copy' for HTML validation.
Therefore the approach taken was to first validate that the ndjson was generated correctly by comparing envelopes against the CCK golden set of ndjson envelopes. Then, for HTML validation, we pull the serialized Envelopes out of the test output HTML and compare that (as a string) against the test output ndjson (which was just validated in the prior step). This new release of HtmlFormatter performs a character substitution of the less-than symbol (with a multi-character encoding) which then will not successfully compare, on a char-by-char basis, against the serialized ndjson envelopes. This affects those scenarios that use the less-than character, such as Scenario Outlines (which uses '<column_name>' in the feature source) and those scenarios that throw exceptions as stack traces often reference the names of generic types (typeA<TTypeB>).

Problem 2: HTML Validation is limited to validating the embedded envelopes payload only
The test suite does not validate that the generated HTML reflects any desired properties beyond the content of the Envelopes. This latest release of the HtmlFormatter allows for setting a custom title and icon. This PR attempts to set a title specific to Reqnroll and to use the Reqnroll icon. This test does not validate that those are correct.

Proposed Approach to address these issues:
Separate HTML Validation from NDJSON validation - I suggest we create a folder for HTML validation and create a separate set of tests.

  • One such test/s would validate the custom HTML rendering (title & icon).
  • Second set of test/s would validate the envelope content, focusing on encoding issues. We would not need to test all CCK scenarios, but instead could use one of them as a basis (or create our own).

This more focused approach should result in a test suite that runs faster (fewer html comparisons as we're no longer attempting this for each CCK scenario). Since it is a set of dedicated tests, there will be less confusion regarding whether problems are related to ndjson or HTML.

@gasparnagy
Copy link
Contributor Author

I think I get the problem. But we should postpone this to after v3. So for now, let's fix the tests somehow and let's schedule the big test refactoring for a bit later.

…he 'expected' envelope stream that the HtmlFormatter has done. The serialized strings should then match.
@clrudolphi
Copy link
Contributor

I have modified the HTML validation suite such that the expected serialized Envelopes have the same encoding substitution as that performed by the HtmlFormatter. The string comparisons now pass.

With the deferral of restructuring the tests and deferral of validation of the Html Customizations, this PR is ready for merge.

@clrudolphi clrudolphi marked this pull request as ready for review August 11, 2025 16:13
@gasparnagy gasparnagy merged commit de12f86 into main Aug 11, 2025
6 checks passed
@gasparnagy gasparnagy deleted the update_html_formatter_dependency branch August 11, 2025 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants