Add test-configuration.json documentation

src/Microsoft.DotNet.Helix/Sdk/Readme.md

@@ -286,3 +286,80 @@ You may assume that all the following variables are set on any given Helix clien
 - **HELIX_CURRENT_LOG** : Path to the current work item's console log (note: will typically have file handles open)
 
 
+## Test Retry
+Helix supports retrying and reporting on partial successes for tests based on repository specific configuration.
+When the configuration matches a test failure, the test assembly is reexecuted, and the results compared.
+Tests that failed partially (only in some executions) will be reported to Azure DevOps as "Passed on Rerun",
+which will include each iteration as a sub-result of the failing test.
+If a test passes or fails in all attempts, only a single report is made representing the first execution.
+
+To opt-in and configure test retries when using helix, create file in the reporitory at "eng/test-configuration.json"
+
+### test-configuraion.json format
+```json
+{
+  "version" : 1,
+  "defaultOnFailure": "fail",
+  "localRerunCount" : 2,
+  "retryOnRules": [
+    {"testName": {"regex": "^System\\.Networking\\..*"}},
+    {"testAssembly": {"wildcard": "System.SomethingElse.*" }},
+    {"failureMessage": "network disconnected" },
+  ],
+  "failOnRules": [
+  ],
+  "quarantineRules": [
+  ]
+}
+```
+
+## Description
+### version
+Schema version for compatibility (curent version is 1)
+
+### defaultOnFailure
+- default: "fail"
+
+One of "fail" or "rerun"
+<dl>
+<dt>fail</dt><dd>If a test fails, the default behavior if no rules match is to fail the test immediate</dd>
+<dt>rerun</dt><dd>If a test fails, the default behavior is no rules match is to rerun the test according to the localRerun/remoteRerun counts</dd>
+</dl>
+
+## localRerunCount
+- default: 1
+
+This number indicates the number of times a test that needs to be "rerun" should be rerun on the local computer immediately.
+This is the fastest rerun option, because the payloads don't need to be redownloaded, so it always the first attempted re-execution method.
+
+In the example, with a value of "2", that means that the test will need to fail 3 times before being marks as failed (1 intial failure, and 2 rerun failures).
+
+## rules
+The three "rules" entries are lists of rules that will be used to match test to determin desired behavior. In the case of multiple rule matches:
+- if a quarantine rule matches, the test is quarantined
+- if the default behavior is "rerun" and a "fail" rule matches, the test is failed
+- if the default bheavior is "fail" and a "rerun" rule matches, the test is rerun
+- default behavior is used
+
+A "rule" consists of a property, and then a rule object
+
+### Properties
+<dl>
+<dt>testName</dt><dd>The name of the test, including namespace, class name, and method name</dd>
+<dt>testAssembly</dt><dd>The name of the assembly containing the test</dd>
+<dt>failureMessage</dt><dd>The failure message logged by the test</dd>
+<dt>callstack (multiline)</dt><dd>The callstack reported by the test execution</dd>
+</dl>
+
+### Rule object
+For all rules, if a property is designated "multiline", then the rule must match a line, otherwise the entire value is used.
+
+All comparisons are case-insensitive
+#### Raw string (e.g. "rule string")
+True if the property value exactly matches the string
+#### {"contains": "value"}
+True if the property contains (case-insensitive) the value string
+#### {"wildcard": "value with * wildcard"}
+The same as a raw string, but "*" can match any number of characters, and "?" can match one character
+#### {"regex": "value with .* regex"}
+true if the propety matches the regular expression