diff --git a/src/Microsoft.DotNet.Helix/Sdk/Readme.md b/src/Microsoft.DotNet.Helix/Sdk/Readme.md index ac3d16a0406..5b8774440e4 100644 --- a/src/Microsoft.DotNet.Helix/Sdk/Readme.md +++ b/src/Microsoft.DotNet.Helix/Sdk/Readme.md @@ -286,3 +286,80 @@ You may assume that all the following variables are set on any given Helix clien - **HELIX_CURRENT_LOG** : Path to the current work item's console log (note: will typically have file handles open) +## Test Retry +Helix supports retrying and reporting on partial successes for tests based on repository specific configuration. +When the configuration matches a test failure, the test assembly is reexecuted, and the results compared. +Tests that failed partially (only in some executions) will be reported to Azure DevOps as "Passed on Rerun", +which will include each iteration as a sub-result of the failing test. +If a test passes or fails in all attempts, only a single report is made representing the first execution. + +To opt-in and configure test retries when using helix, create file in the reporitory at "eng/test-configuration.json" + +### test-configuraion.json format +```json +{ + "version" : 1, + "defaultOnFailure": "fail", + "localRerunCount" : 2, + "retryOnRules": [ + {"testName": {"regex": "^System\\.Networking\\..*"}}, + {"testAssembly": {"wildcard": "System.SomethingElse.*" }}, + {"failureMessage": "network disconnected" }, + ], + "failOnRules": [ + ], + "quarantineRules": [ + ] +} +``` + +## Description +### version +Schema version for compatibility (curent version is 1) + +### defaultOnFailure +- default: "fail" + +One of "fail" or "rerun" +
+
fail
If a test fails, the default behavior if no rules match is to fail the test immediate
+
rerun
If a test fails, the default behavior is no rules match is to rerun the test according to the localRerun/remoteRerun counts
+
+ +## localRerunCount +- default: 1 + +This number indicates the number of times a test that needs to be "rerun" should be rerun on the local computer immediately. +This is the fastest rerun option, because the payloads don't need to be redownloaded, so it always the first attempted re-execution method. + +In the example, with a value of "2", that means that the test will need to fail 3 times before being marks as failed (1 intial failure, and 2 rerun failures). + +## rules +The three "rules" entries are lists of rules that will be used to match test to determine desired behavior. In the case of multiple rule matches: +- if a quarantine rule matches, the test is quarantined +- if the default behavior is "rerun" and a "fail" rule matches, the test is failed +- if the default behavior is "fail" and a "rerun" rule matches, the test is rerun +- default behavior is used + +A "rule" consists of a property, and then a rule object + +### Properties +
+
testName
The name of the test, including namespace, class name, and method name
+
testAssembly
The name of the assembly containing the test
+
failureMessage
The failure message logged by the test
+
callstack (multiline)
The callstack reported by the test execution
+
+ +### Rule object +For all rules, if a property is designated "multiline", then the rule must match a line, otherwise the entire value is used. + +All comparisons are case-insensitive +#### Raw string (e.g. "rule string") +True if the property value exactly matches the string +#### {"contains": "value"} +True if the property contains (case-insensitive) the value string +#### {"wildcard": "value with * wildcard"} +The same as a raw string, but "*" can match any number of characters, and "?" can match one character +#### {"regex": "value with .* regex"} +true if the property matches the regular expression