Skip to content

SO Tagging: fix flaky test and re-enable it #82930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 18, 2020
Merged

SO Tagging: fix flaky test and re-enable it #82930

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d216b9e
fix flaky test and re-enable it
pgayvallet Nov 9, 2020
c2d1238
wait for table to load before to perform operations
pgayvallet Nov 12, 2020
374814f
Merge remote-tracking branch 'upstream/master' into kbn-82556-fix-sot…
pgayvallet Nov 12, 2020
db86c3b
move everything out of ciGroup2 for flaky test runner
pgayvallet Nov 12, 2020
46da2af
add debug block for flaky runner
pgayvallet Nov 13, 2020
2909b19
use correct vis name
pgayvallet Nov 13, 2020
d37a5b7
Merge remote-tracking branch 'upstream/master' into kbn-82556-fix-sot…
pgayvallet Nov 16, 2020
c77f036
remove test sync
pgayvallet Nov 16, 2020
5e9a93c
Revert "move everything out of ciGroup2 for flaky test runner"
pgayvallet Nov 16, 2020
5861e50
Merge remote-tracking branch 'upstream/master' into kbn-82556-fix-sot…
pgayvallet Nov 18, 2020
61980e9
Merge remote-tracking branch 'upstream/master' into kbn-82556-fix-sot…
pgayvallet Nov 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions x-pack/test/saved_object_tagging/functional/tests/visualize_integration.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
const listingTable = getService('listingTable');
const testSubjects = getService('testSubjects');
const find = getService('find');
const PageObjects = getPageObjects(['visualize', 'tagManagement', 'visEditor']);
const PageObjects = getPageObjects(['visualize', 'tagManagement', 'visEditor', 'header']);

/**
* Select tags in the searchbar's tag filter.
Expand All @@ -31,6 +31,8 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
// click elsewhere to close the filter dropdown
const searchFilter = await find.byCssSelector('main .euiFieldSearch');
await searchFilter.click();

await PageObjects.header.waitUntilLoadingHasFinished();
};

const selectSavedObjectTags = async (...tagNames: string[]) => {
Expand All @@ -56,6 +58,7 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
describe('listing', () => {
beforeEach(async () => {
await PageObjects.visualize.gotoVisualizationLandingPage();
await PageObjects.header.waitUntilLoadingHasFinished();
});

it('allows to manually type tag filter query', async () => {
Expand Down Expand Up @@ -83,7 +86,7 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
});

describe('creating', () => {
it.skip('allows to assign tags to the new visualization', async () => {
it('allows to assign tags to the new visualization', async () => {
await PageObjects.visualize.navigateToNewVisualization();

await PageObjects.visualize.clickMarkdownWidget();
Expand All @@ -95,7 +98,10 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
await selectSavedObjectTags('tag-1');

await testSubjects.click('confirmSaveSavedObjectButton');
await testSubjects.existOrFail('saveVisualizationSuccess');

await PageObjects.visualize.gotoVisualizationLandingPage();
await PageObjects.header.waitUntilLoadingHasFinished();
Copy link
Contributor

@mshustov mshustov Nov 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: There could be a lot of reasons for waiting for a DOM element to be rendered. What if we increase timeout in selectFilterTags method instead?

Copy link
Contributor Author

@pgayvallet pgayvallet Nov 9, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH after 100 runs against my local I couldn't reproduce the flaky failure even once. The failure message indicates that for some reason, 1. the tag filter was not applied (Visualization 2 (tag-2) is present in the table) and 2. the new vis was not created (My new markdown viz is not), and I don't really how we could come as far as the last assertion in that state without failing before in the test. So I kinda did the best I could with additional synchronizations / waiting steps to see if that would resolves it on CI (renamed the PR accordingly).

I can do what you suggest though

Copy link
Contributor

@mshustov mshustov Nov 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pgayvallet Have you tried to leverage https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/build
to run a flaky test on CI?

Copy link
Contributor Author

@pgayvallet pgayvallet Nov 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not. Can this be ran against a PR branch?

Copy link
Contributor

@mshustov mshustov Nov 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can configure it
2020-11-10_12-30-14

Copy link
Contributor Author

@pgayvallet pgayvallet Nov 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First run had a failure on another test - https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/978/testReport/junit/X-Pack%20Saved%20Object%20Tagging%20Functional%20Tests/x-pack_test_saved_object_tagging_functional_tests_dashboard_integration%C2%B7ts/Kibana_Pipeline___agent_2___saved_objects_tagging___functional_tests_dashboard_integration_creating_allows_to_create_a_tag_from_the_tag_selector/

but the error message was a little more explicit on that one. trying something else by waiting for the table to load


await selectFilterTags('tag-1');
const itemNames = await listingTable.getAllItemsNames();
Expand Down Expand Up @@ -133,7 +139,9 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
expect(await tagModal.isOpened()).to.be(false);

await testSubjects.click('confirmSaveSavedObjectButton');

await PageObjects.visualize.gotoVisualizationLandingPage();
await PageObjects.header.waitUntilLoadingHasFinished();

await selectFilterTags('my-new-tag');
const itemNames = await listingTable.getAllItemsNames();
Expand All @@ -154,6 +162,7 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {

await testSubjects.click('confirmSaveSavedObjectButton');
await PageObjects.visualize.gotoVisualizationLandingPage();
await PageObjects.header.waitUntilLoadingHasFinished();

await selectFilterTags('tag-2');
const itemNames = await listingTable.getAllItemsNames();
Expand Down