Schema change / purl-test: Add validation tests

[mjherzog]

This issue is intended to raise the visibility and documentation of a proposed addition to purl-test-schema.json from its introduction in PR #614 which proposes a new test type for validation. The PR covers the proposed change to the schema and corresponding updates to the 32 registered PURL types. The context for adding this issue is that context any normative change to a PURL schema should be reviewed approved by TC54/TG2. The purl-test-schema.json is not included in the proposed PURL 1.0 Ecma standard, but it is a normative element of the PURL specification.

The gist of the proposal is to add a validation test type :

"purl_validation_message": {
      "title": "PURL validation message",
      "description": "Validation severity and message to use as a test expected output when validating a PURL.",
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "severity": {
          "title": "Validation severity",
          "description": "Validation severity.",
          "type": "string",
          "enum": [
            "info",
            "warning",
            "error"
          ],
          "meta:enum": {
            "info": "Informational validation message",
            "warning": "Warning validation message",
            "error": "Error validation message"
          }
        },
        "message": {
          "title": "Validation message",
          "description": "Validation message.",
          "type": "string"
        }
      }
    },

PR #614 adds the new purl_validation_message property and includes other changes to implement the change in the schema. See the PR for details.

[sjn]

This needs also a "success" severity.

[pombredanne]

@sjn re:

This needs also a "success" severity.

IMHO success is the default state, do you think we need this explicitly?

[sjn]

@sjn re:

This needs also a "success" severity.

IMHO success is the default state, do you think we need this explicitly?

Yes, because sometimes error conditions don't produce an output.
If you expect an output and don't get it, you can derive useful information from this.
(Though this only works if you are outputting anything useful beyond what is expected from the other severity levels)

For example: how do you detect if a test unexpectedly exits with a 0 return code?

You can detect this failure mode if you expect a final success output.