Skip to content

Conversation

@dnfield
Copy link
Contributor

@dnfield dnfield commented Feb 22, 2019

This PR wraps all test output so that errors are printed last. It also sends all test result data to a bigquery table we can look at to get better info about what tests failed and when without needing to visit specific builder pages.

It's only implemented for Cirrus at the moment. It should be possible to implement for LUCI as well, but I'd rather tackle that separately.

@dnfield dnfield added a: tests "flutter test", flutter_test, or one of our tests c: contributor-productivity Team-specific productivity, code health, technical debt. work in progress; do not review labels Feb 22, 2019
@dnfield
Copy link
Contributor Author

dnfield commented Feb 27, 2019

This is working on Cirrus, need to check into how I can make it work for LUCI.

@dnfield dnfield changed the title Test reporter 🚧 Test reporter Mar 6, 2019
@dnfield
Copy link
Contributor Author

dnfield commented Mar 6, 2019

This is still TODO on LUCI but would be an improvement for Cirrus. Marking ready for review.

/cc @goderbauer

Copy link
Contributor

@gspencergoog gspencergoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32384589-a60f0e74-c078-11e7-9bc1-e5b5287aea9d

Very cool. I can't wait to see the data analysis we can do with this...

@dnfield dnfield merged commit 20e0f13 into flutter:master Mar 6, 2019
@dnfield dnfield deleted the test_reporter branch March 6, 2019 21:13
@jonahwilliams
Copy link
Contributor

This doesn't appear to be working properly: https://cirrus-ci.com/task/4970475997364224

@dnfield
Copy link
Contributor Author

dnfield commented Mar 7, 2019

Something's messed up on Windows. Checking.

@dnfield
Copy link
Contributor Author

dnfield commented Mar 7, 2019

That will be fixed by #28970

}

Future<bq.BigqueryApi> _getBigqueryApi() async {
// TODO(dnfield): How will we do this on LUCI?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (privateKey == null || privateKey.isEmpty) {
return null;
}
final auth.ServiceAccountCredentials accountCredentials = auth.ServiceAccountCredentials( //.fromJson(credentials);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: stray comment on this line?

@Hixie
Copy link
Contributor

Hixie commented Mar 22, 2019

What's the background here?

@dnfield
Copy link
Contributor Author

dnfield commented Mar 22, 2019

The compact formatter makes it really hard to find what tests actually failed - particularly since Cirrus only shows us the tail of the log (which almost never contains the test that failed).

I also would like to see all of our test results in a database so we could find failing tests rather than checking failing bots.

This is currently turned off though, as it had a couple problems I haven't been able to fully sort through yet.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

a: tests "flutter test", flutter_test, or one of our tests c: contributor-productivity Team-specific productivity, code health, technical debt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants