From 0f42b03a2f86ade307aba633ea6e7b130aba0e46 Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 12 Aug 2016 09:57:07 +0200 Subject: [PATCH 1/7] Added documentation and example for EiffelTestSuiteFinishedEvent. --- README.md | 2 +- eiffel-vocabulary/EiffelTestSuiteFinished.md | 53 +++++++++++++++++++ .../EiffelTestSuiteFinishedEvent/simple.json | 34 ++++++++++++ 3 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 eiffel-vocabulary/EiffelTestSuiteFinished.md create mode 100644 examples/events/EiffelTestSuiteFinishedEvent/simple.json diff --git a/README.md b/README.md index 84ad53d0..8002ebfb 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ __IMPORTANT NOTICE:__ The contents of this repository currectly reflect a __DRAF 1. EiffelTestCaseStartedEvent 1. EiffelTestCaseFinishedEvent 1. [EiffelTestSuiteStartedEvent](./eiffel-vocabulary/EiffelTestSuiteStartedEvent.md) - 1. EiffelTestSuiteFinishedEvent + 1. [EiffelTestSuiteFinishedEvent](./eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md) 1. EiffelTestExecutionRecipeCollectionCreated 1. EiffelAnnouncementEvent 1. EiffelConfigurationChangedEvent diff --git a/eiffel-vocabulary/EiffelTestSuiteFinished.md b/eiffel-vocabulary/EiffelTestSuiteFinished.md new file mode 100644 index 00000000..c3772897 --- /dev/null +++ b/eiffel-vocabulary/EiffelTestSuiteFinished.md @@ -0,0 +1,53 @@ +# EiffelTestSuiteFinishedEvent +The EiffelTestSuiteFinishedEvent declares that a previously started test suite (declared by [EiffelTestSuiteStartedEvent](./EiffelTestSuiteStartedEvent.md) has finished and reports the outcome. + +Note that while similar, the __data.outcome__ object is different from that of [EiffelActivityFinishedEvent](./EiffelActivityFinishedEvent.md). The outcome of the test suite reports not only the conclusion of the test suite execution - whether the tests were successfully executed - but also passes a verdict on the item(s) under test. To highlight this conceptual difference, both __data.outcome.verdict__ and __data.outcome.conclusion__ are included. + +## Data Members +### data.outcome +__Type:__ Object +__Required:__ Yes +__Description:__ The outcome of the test suite. + +#### data.outcome.verdict +__Type:__ String +__Required:__ No +__Legal values:__ PASSED, FAILED, INCONCLUSIVE +__Description:__ A terse standardized verdict on the item under test. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method to summarize the verdict of the test suite as a whole, but may be skipped. +PASSED signifies that the item(s) under test successfully passed the test suite. +FAILED signifies that the item(s) under test failed to pass the test suite. +INCONCLUSIVE signifies that the verdict of the test suite was inconclusive. This SHOULD be the case if __data.outcome.conclusion__ is not __SUCCESSFUL__, but may in combination with a __SUCCESSFUL__ conclusion be used to represent unreliability or flakiness. + +#### data.outcome.conclusion +__Type:__ String +__Required:__ No +__Legal values:__ SUCCESSFUL, FAILED, ABORTED, TIMED_OUT, INCONCLUSIVE +__Description:__ A terse standardized conclusion of the test suite, designed to be machine readable. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method to summarize the conclusion of the test suite as a whole, but may be skipped. +SUCCESSFUL signifies that the test suite was successfully concluded. Note that this does not imply that the item under test passed the tests. +FAILED signifies that the test suite could not be successfully executed. To exemplify, one or more tests failued to run due to required environments being unavailable. +ABORTED signifies that the test suite was aborted before it could be concluded. +TIMED_OUT signifies that the test suite did not conclude within the allowed time frame. +INCONCLUSIVE signifies that the outcome of the test suite could not be determined. + +#### data.outcome.description +__Type:__ String +__Required:__ No +__Description:__ A verbose description of the test suite outcome, designed to provide human readers with further information. + +### data.persistentLogs +__Type:__ Object[] +__Required:__ No +__Description:__ An array of persistent log files generated during execution. + +#### data.persistentLogs.name +__Type:__ String +__Required:__ Yes +__Description:__ The name of the log file. + +#### data.persistentLogs.uri +__Type:__ String +__Required:__ Yes +__Description:__ The URI at which the log can be retrieved. + +## Examples +* [Simple example](../examples/events/EiffelTestSuiteFinishedEvent/simple.json) \ No newline at end of file diff --git a/examples/events/EiffelTestSuiteFinishedEvent/simple.json b/examples/events/EiffelTestSuiteFinishedEvent/simple.json new file mode 100644 index 00000000..d61b69a5 --- /dev/null +++ b/examples/events/EiffelTestSuiteFinishedEvent/simple.json @@ -0,0 +1,34 @@ +{ + "meta": { + "type": "EiffelTestSuiteFinishedEvent", + "version": "1.0", + "time": 1234567890, + "id": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee0", + "source": { + "domainId": "example.domain" + } + }, + "data": { + "outcome": { + "verdict": "FAILED", + "outcome": "SUCCESSFUL", + "description": "16 test failures." + }, + "persistentLogs": [ + { + "name": "firstLog", + "uri": "http://myHost.com/firstLog" + }, + { + "name": "otherLog", + "uri": "isbn:0-486-27557-4" + } + ] + }, + "links": [ + { + "type": "TEST_SUITE_EXECUTION", + "target": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeee1" + } + ] +} From 1905a009e190576b8fc09addc587d775e34281ff Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 12 Aug 2016 10:00:29 +0200 Subject: [PATCH 2/7] Fixed name of EiffelTestSuiteFinishedEvent documentation. --- ...EiffelTestSuiteFinished.md => EiffelTestSuiteFinishedEvent.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename eiffel-vocabulary/{EiffelTestSuiteFinished.md => EiffelTestSuiteFinishedEvent.md} (100%) diff --git a/eiffel-vocabulary/EiffelTestSuiteFinished.md b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md similarity index 100% rename from eiffel-vocabulary/EiffelTestSuiteFinished.md rename to eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md From a64a91e4c63c0df1c34c091f3fcb2c82f1b294af Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 12 Aug 2016 10:01:25 +0200 Subject: [PATCH 3/7] Added missing bracket. --- eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md index c3772897..63ab6f43 100644 --- a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md +++ b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md @@ -1,5 +1,5 @@ # EiffelTestSuiteFinishedEvent -The EiffelTestSuiteFinishedEvent declares that a previously started test suite (declared by [EiffelTestSuiteStartedEvent](./EiffelTestSuiteStartedEvent.md) has finished and reports the outcome. +The EiffelTestSuiteFinishedEvent declares that a previously started test suite (declared by [EiffelTestSuiteStartedEvent](./EiffelTestSuiteStartedEvent.md)) has finished and reports the outcome. Note that while similar, the __data.outcome__ object is different from that of [EiffelActivityFinishedEvent](./EiffelActivityFinishedEvent.md). The outcome of the test suite reports not only the conclusion of the test suite execution - whether the tests were successfully executed - but also passes a verdict on the item(s) under test. To highlight this conceptual difference, both __data.outcome.verdict__ and __data.outcome.conclusion__ are included. From 51c5fb05c711e3383272993d80e1a3902632f54d Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 12 Aug 2016 10:02:50 +0200 Subject: [PATCH 4/7] Rephrased item(s) as item or items. --- eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md index 63ab6f43..319800a2 100644 --- a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md +++ b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md @@ -1,7 +1,7 @@ # EiffelTestSuiteFinishedEvent The EiffelTestSuiteFinishedEvent declares that a previously started test suite (declared by [EiffelTestSuiteStartedEvent](./EiffelTestSuiteStartedEvent.md)) has finished and reports the outcome. -Note that while similar, the __data.outcome__ object is different from that of [EiffelActivityFinishedEvent](./EiffelActivityFinishedEvent.md). The outcome of the test suite reports not only the conclusion of the test suite execution - whether the tests were successfully executed - but also passes a verdict on the item(s) under test. To highlight this conceptual difference, both __data.outcome.verdict__ and __data.outcome.conclusion__ are included. +Note that while similar, the __data.outcome__ object is different from that of [EiffelActivityFinishedEvent](./EiffelActivityFinishedEvent.md). The outcome of the test suite reports not only the conclusion of the test suite execution - whether the tests were successfully executed - but also passes a verdict on the item or items under test. To highlight this conceptual difference, both __data.outcome.verdict__ and __data.outcome.conclusion__ are included. ## Data Members ### data.outcome @@ -13,9 +13,9 @@ __Description:__ The outcome of the test suite. __Type:__ String __Required:__ No __Legal values:__ PASSED, FAILED, INCONCLUSIVE -__Description:__ A terse standardized verdict on the item under test. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method to summarize the verdict of the test suite as a whole, but may be skipped. -PASSED signifies that the item(s) under test successfully passed the test suite. -FAILED signifies that the item(s) under test failed to pass the test suite. +__Description:__ A terse standardized verdict on the item or items under test. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method to summarize the verdict of the test suite as a whole, but may be skipped. +PASSED signifies that the item or items under test successfully passed the test suite. +FAILED signifies that the item or items under test failed to pass the test suite. INCONCLUSIVE signifies that the verdict of the test suite was inconclusive. This SHOULD be the case if __data.outcome.conclusion__ is not __SUCCESSFUL__, but may in combination with a __SUCCESSFUL__ conclusion be used to represent unreliability or flakiness. #### data.outcome.conclusion From 12eb6d5bc0700a98fc2dc75ec4200f44134ec818 Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 12 Aug 2016 10:04:58 +0200 Subject: [PATCH 5/7] Fixed various typos. --- eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md index 319800a2..917ce4a3 100644 --- a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md +++ b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md @@ -13,7 +13,7 @@ __Description:__ The outcome of the test suite. __Type:__ String __Required:__ No __Legal values:__ PASSED, FAILED, INCONCLUSIVE -__Description:__ A terse standardized verdict on the item or items under test. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method to summarize the verdict of the test suite as a whole, but may be skipped. +__Description:__ A terse standardized verdict on the item or items under test. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method of summarizing the verdict of the test suite as a whole, but may be skipped. PASSED signifies that the item or items under test successfully passed the test suite. FAILED signifies that the item or items under test failed to pass the test suite. INCONCLUSIVE signifies that the verdict of the test suite was inconclusive. This SHOULD be the case if __data.outcome.conclusion__ is not __SUCCESSFUL__, but may in combination with a __SUCCESSFUL__ conclusion be used to represent unreliability or flakiness. @@ -22,7 +22,7 @@ INCONCLUSIVE signifies that the verdict of the test suite was inconclusive. This __Type:__ String __Required:__ No __Legal values:__ SUCCESSFUL, FAILED, ABORTED, TIMED_OUT, INCONCLUSIVE -__Description:__ A terse standardized conclusion of the test suite, designed to be machine readable. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method to summarize the conclusion of the test suite as a whole, but may be skipped. +__Description:__ A terse standardized conclusion of the test suite, designed to be machine readable. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method of summarizing the conclusion of the test suite as a whole, but may be skipped. SUCCESSFUL signifies that the test suite was successfully concluded. Note that this does not imply that the item under test passed the tests. FAILED signifies that the test suite could not be successfully executed. To exemplify, one or more tests failued to run due to required environments being unavailable. ABORTED signifies that the test suite was aborted before it could be concluded. From 1e2001a6883ec52ca58c309c281aca8098e5b190 Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 12 Aug 2016 10:14:16 +0200 Subject: [PATCH 6/7] Fixed type. Made data.outcome optional. --- eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md index 917ce4a3..9a11d869 100644 --- a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md +++ b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md @@ -6,7 +6,7 @@ Note that while similar, the __data.outcome__ object is different from that of [ ## Data Members ### data.outcome __Type:__ Object -__Required:__ Yes +__Required:__ No __Description:__ The outcome of the test suite. #### data.outcome.verdict @@ -22,7 +22,7 @@ INCONCLUSIVE signifies that the verdict of the test suite was inconclusive. This __Type:__ String __Required:__ No __Legal values:__ SUCCESSFUL, FAILED, ABORTED, TIMED_OUT, INCONCLUSIVE -__Description:__ A terse standardized conclusion of the test suite, designed to be machine readable. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method of summarizing the conclusion of the test suite as a whole, but may be skipped. +__Description:__ A terse standardized conclusion of the test suite, designed to be machine readable. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method of summarizing the conclusion of the test suite as a whole, but may be skipped. SUCCESSFUL signifies that the test suite was successfully concluded. Note that this does not imply that the item under test passed the tests. FAILED signifies that the test suite could not be successfully executed. To exemplify, one or more tests failued to run due to required environments being unavailable. ABORTED signifies that the test suite was aborted before it could be concluded. From a8dccd9f5cd8aa2fd730f0cf032b70c8bf44913d Mon Sep 17 00:00:00 2001 From: Daniel Stahl Date: Fri, 26 Aug 2016 09:08:43 +0200 Subject: [PATCH 7/7] Fixed typo. --- eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md index 9a11d869..79b7d704 100644 --- a/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md +++ b/eiffel-vocabulary/EiffelTestSuiteFinishedEvent.md @@ -24,7 +24,7 @@ __Required:__ No __Legal values:__ SUCCESSFUL, FAILED, ABORTED, TIMED_OUT, INCONCLUSIVE __Description:__ A terse standardized conclusion of the test suite, designed to be machine readable. Unlike in [EiffelTestCaseFinishedEvent](./EiffelTestCaseFinishedEvent.md), this property is optional. It offers a method of summarizing the conclusion of the test suite as a whole, but may be skipped. SUCCESSFUL signifies that the test suite was successfully concluded. Note that this does not imply that the item under test passed the tests. -FAILED signifies that the test suite could not be successfully executed. To exemplify, one or more tests failued to run due to required environments being unavailable. +FAILED signifies that the test suite could not be successfully executed. To exemplify, one or more tests failed to run due to required environments being unavailable. ABORTED signifies that the test suite was aborted before it could be concluded. TIMED_OUT signifies that the test suite did not conclude within the allowed time frame. INCONCLUSIVE signifies that the outcome of the test suite could not be determined.