Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky random publication tests #2039

Merged
merged 2 commits into from
Mar 10, 2021
Merged

Fix flaky random publication tests #2039

merged 2 commits into from
Mar 10, 2021

Conversation

kevindew
Copy link
Member

@kevindew kevindew commented Mar 8, 2021

I was able to find three causes to a problem where unique schemas weren't being generated for the random publication test.

  1. Generating for an unexpected document type (fix in first commit)
  2. Assumptions of attachment presence in code that don't match schema rules (fix in 2nd commit)
  3. Invalid schemas being generated due to an array expecting unique items (fix in Generate unique items for arrays with the "uniqueItems" property govuk_schemas#63)

More info in the commits.

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

kevindew added 2 commits March 8, 2021 11:30
By running `merge` rather than `merge!` this wasn't actually mutating
the underlying hash and thus not having an effect.

This helps to resolve a problem where a new publication document type
has been added and causes some tests to fail. For example:

```
  test_random_but_valid_items_do_not_error                       ERROR (0.39s)
Minitest::UnexpectedError:         ActionView::Template::Error: translation missing: zh.content_item.schema_name.standard
           app/views/content_items/publication.html.erb:10
           app/controllers/content_items_controller.rb:149:in `block in render_template'
           app/controllers/content_items_controller.rb:183:in `block in with_locale'
           app/controllers/content_items_controller.rb:183:in `with_locale'
           app/controllers/content_items_controller.rb:148:in `render_template'
           app/controllers/content_items_controller.rb:25:in `show'
```
This resolves a couple of scenarios where assumptions are made that
documents will have attachments and attachments with particular ids
exist. This is data that developers assume is the case (and most likely
is) but is beyond the guarantees that schemas make.

This mismatch can cause tests to fail when valid schemas are generated
which cannot be rendered.
@bevanloon bevanloon temporarily deployed to government-f-flaky-test-pyvtsf March 8, 2021 14:40 Inactive
Copy link
Contributor

@gclssvglx gclssvglx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great - thanks @kevindew. Nice spot with the merge! 👏

@kevindew kevindew merged commit 7090299 into master Mar 10, 2021
@kevindew kevindew deleted the flaky-test branch March 10, 2021 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants