Adding templates to the DSL

[BrentBlanckaert]

Templates are handled in the same file as the translations. So using to following will now also proces templates:

python -m tested.nat_translation ./exercise/simple-example/templates/suite.yaml en

[BrentBlanckaert]

Some examples of what is currently possible:

Usage of object template

translations:
  animal:
    en: "animals"
    nl: "dieren"
tabs:
  - tab: '{{translations.animal}}'
    templates:
      default_template:
        expression:
          javascript: "{{translations.animal}}_javascript({{ templates.input }})"
          typescript: "{{translations.animal}}_typescript({{ templates.input }})"
          java: "Submission.{{translations.animal}}_java({{ templates.input }})"
          python: !natural_language
            en: "{{translations.animal}}_python_en({{ templates.input }})"
            nl: "{{translations.animal}}_python_nl({{ templates.input }})"
        return: "{{ templates.output }}"
    testcases:
      - placeholder:
          name: "default_template"
          data:
            input: !natural_language
              en: "Sardines"
              nl: "Sardienen"
            output: !natural_language
              en: "fish"
              nl: "vis"
      - placeholder:
          name: "default_template"
          data:
            input: !natural_language
              en: "Tiger"
              nl: "Tijger"
            output: !natural_language
              en: "mammal"
              nl: "zoogdier"

Usage of list template

"units":
- "unit": "echo"
  "translations":
    "monkeys":
      en: "monkeys"
      nl: "apen"
  "templates":
    "drinks":
      - "expression": "drinks({{ templates.input1 }})"
        "return": "{{ templates.ok }}"
      - "expression": "drinks({{ templates.input2 }})"
        "return": "{{ templates.ok }}"
      - "expression": "drinks({{ templates.input3 }})"
        "return": "{{ templates.ok }}"
  "scripts":
    "placeholder":
      "name": "drinks"
      "data":
        "ok": "OK!"
        "input1": "Beer"
        "input2": "Soda"
        "input3": "Water"

Usage of scalar template

"units":
- "unit": "IO"
  "translations":
    "monkeys":
      en: "monkeys"
      nl: "apen"
  "templates":
    "heartbeats": !natural_language
      "en": "{{templates.animals}} have {{templates.billion_heartbeats_per_lifetime}} billion heartbeats"
      "nl": "{{templates.animals}} hebben {{templates.billion_heartbeats_per_lifetime}} miljard hartslagen"
  "scripts":
  - "stdin": |-
      horses
      44
      40
    "stdout": "horses have 0.93 billion heartbeats"
  - "stdin": |-
      {{translations.monkeys}}
      190
      40
      15
    "stdout":
      "placeholder":
        "name": "heartbeats"
        "data":
          "animals": "{{translations.monkeys}}"
          "billion_heartbeats_per_lifetime": 1.50

[BrentBlanckaert]

New suggestion for templates

Usage of object template

Basic usage

tabs:
  - tab: 'animals'
    templates:
      default_animal:
        expression:
          java: "Submission.animals({{ input }})"
          python: "animals({{ input }})"
        return: "{{ output }}"
    testcases:
      - template:
          name: "default_animal"
          parameters:
            - input: "Sardines"
              output: "fish"
            - input: "Tiger"
              output: "mammal"
      - template: 
          name: "default_animal"
          parameters: # Parameters kan bijvoorbeeld ook een object zijn.
            input: "Whale"
            output: "mammal"

A templates map can be defined in the exact same places as a translations map (globally, in a tab and in a context). The templates map will be a map with keys that will be the names for the templates. Under that name you define the actual template.

In the test-suite, you can then define a template. A template will always need a name. This will be used to match the defined keys in the templates maps. In a template, you can also specify parameters (not manditory). In this case input and output are specified. In the found templates the placesholders with the matching placeholders will be replaced with the defined parameters.

With repeats

tabs:
  - tab: 'random'
    templates:
      default_random:
        expression: "random()"
        return: "{{ output }}"
    testcases:
      - template: 
          name: "default_random"
          parameters:
            - output: 0
              repeat: 100
            - output: 9
              repeat: 100

Default template

For a default template, you should define it globally.

default_template:
  expression: "calculator({{ expr }})"
  return: "{{ result }}"
templates:
  print_calc:
    expression: "print_steps({{ expr }})"
    return: "{{ result }}"
tabs:
  - tab: 'calculator'
    testcases:
      - parameters:
          - expr: "2 + 10"
            result: 12
          - expr: "9 + 5"
            result: 14
          - expr: "99 + 3"
            result: 102
          - expr: "4 * 13"
            result: 52
      - template: 
          name: "print_calc"
          parameters:
            - expr: "2 + 10"
              result: "2 + 10 = 12"
            - expr: "9 + 5"
              result: "9 + 5 = 14"
            - expr: "99 + 3"
              result: "99 + 3 = 102"
            - expr: "4 * 13"
              result: "4 * 13 = 4 * 10 + 4 * 3 = 40 + 12 = 52"

To access the default template, you just have to specify the parameters.

Using translations

translations:
  animals:
    en: "animals"
    nl: "dieren"
tabs:
  - tab: '{{ animals }}'
    templates:
      default_animal:
        expression:
          java: "Submission.{{ translations.animals }}({{ templates.animals }})"
          python: "{{translations.animals}}({{ templates.animals }})"
        return: "{{ output }}"
    testcases:
      - template:
          name: "default_animal"
          parameters:
            - input: "Sardines"
              output: "fish"
            - input: "Tiger"
              output: "mammal"

This is a very rare case in which the template uses a key that is also defined in a translations map. In this case, namespaces should be used (throwing an error?).

Usage of list template

"units":
- "unit": "drinks"
  "templates":
    "default_drinks":
      - "expression": "drinks({{ input1 }})"
        "return": "{{ ok }}"
      - "expression": "drinks({{ input2 }})"
        "return": "{{ ok }}"
      - "expression": "drinks({{ input3 }})"
        "return": "{{ ok }}"
  "scripts":
    "placeholder":
      "name": "default_drinks"
      "data":
        "ok": "OK!"
        "input1": "Beer"
        "input2": "Soda"
        "input3": "Water"

Usage of scalar template

Not sure if usage in stdin, stdout, stderr, return, expressions, etc., should be supported or not.

"units":
- "unit": "IO"
  "translations":
    "monkeys":
      en: "monkeys"
      nl: "apen"
  "templates":
    "default_heartbeats": !natural_language
      "en": "{{animals}} have {{billion_heartbeats_per_lifetime}} billion heartbeats"
      "nl": "{{animals}} hebben {{billion_heartbeats_per_lifetime}} miljard hartslagen"
  "scripts":
  - "stdin": |-
      horses
      44
      40
    "stdout": "horses have 0.93 billion heartbeats"
  - "stdin": |-
      {{monkeys}}
      190
      40
      15
    "stdout":
      "placeholder":
        "name": "default_heartbeats"
        "data":
          "animals": "{{monkeys}}"
          "billion_heartbeats_per_lifetime": 1.50

[BrentBlanckaert]

Suggestion 3 for templates

Basic usage

tabs:
  - tab: 'animals'
    templates:
      animal_IO:
        expression:
          java: "Submission.animals({{ input }})"
          python: "animals({{ input }})"
        return: "{{ output }}"
    testcases:
      - template:  "animal_IO"
        parameters:
           input: "Sardines"
           output: "fish"
      - template: "animal_IO"
        parameters:
           input: "Whale"
           output: "mammal"
        return: "mammal (not a fish)"

A templates map can be defined in the exact same places as a translations map (globally, in a tab and in a context). The templates map will be a map with keys that will be the names for the templates. Under that name you define the actual template.

After that you can define a template inside a testcase. A string is passed with it. This will be used to match the defined keys in the templates maps. Once a match is found template will be replaced with the found match.

With a template, you can also specify parameters (not manditory). In this case input and output are specified. In the found templates the placesholders with the matching placeholders will be replaced with the defined parameters.

In the second testcase a return is specified. This will overwrite the return in the template.

A template inside a template is not allowed.

With repeats

templates:
  one_name:
    stdin: |-
      1
      {{ name }}
    stdout: "{{ username }}"
units:
- unit: IO
  scripts:
    - template: one_name
      parameters:
        name: Graham Chapman
        username: gchap
    - repeat:
        template: one_name
        parameters:
          - name: John Cleese
            username: jclee
          - name: Terry Gilliam
            username: tgill
          - name: Eric Idle
            username: eidle
          - name: Terry Jones
            username: tjone
          - name: Michael Palin
            username: mpali

A repeat can also be defined, serving as a loop to generate multiple test cases. In a repeat, specifying parameters is always mandatory. The key difference from the previous definition of parameters is that it is now a list. Each item in this list will be iterated over to generate the test cases. In the example above, this results in six test cases based on the provided parameters.

Using translations

translations:
  animals:
    en: "animals"
    nl: "dieren"
tabs:
  - tab: '{{ animals }}'
    templates:
      animal_IO:
        expression:
          java: "Submission.{{ animals }}({{ animals }})"
          python: "{{animals}}({{ animals }})"
        return: "{{ output }}"
    testcases:
      - template: "animal_IO"
        parameters:
          animals: "Sardines"
          output: "fish"

This is a very rare case in which the template uses a key that is also defined in a translations map. In this case, an error should be thrown.

Non-string parameters

Up until now, all examples have used string parameters, but what if you'd like to specify an integer or a list?
If Jinja is used to replace "{{key}}", then key can be mapped to anything the user wants—but Jinja will always return a string.
That’s why Jinja can’t be used in such cases. To fight this the following is proposed:

templates:
  calculator_IO:
    expression: "calculator('{{ expr }}')"
    return: !parameter "result"
  temp2:
    expression: "calculator('{{ expr }}')"
    return: !oracle
      value: !parameter "result"
      oracle: "builtin"
  temp3:
    arguments: !parameter "list"
    stdout: !parameter "res"
    exit_code: !parameter "code"
  temp4:
    expression: "calculator('{{ expr }}')"
    return: !natural_language
      en: !parameter "test"
      nl: !parameter "test"
  temp5:
    expression: "calculator('{{ expr }}')"
    return: !oracle
      value: !parameter "result"
      oracle: "custom_check"
      file: "file.txt"
  temp6:
    arguments:
      - !parameter "list1"
      - "second"
      - "third"
    stdout: !parameter "res"
    exit_code: !parameter "code"
tabs:
  - tab: 'calculator'
    testcases:
      - template: "calculator_IO"
        parameters:
          expr: "2 + 10"
          result: 12
      - template: "calculator_IO"
        parameters:
          expr: "4 * 13"
          result: 52

In this case, Jinja is not used, and a parameter is specified using the !parameter tag. This will be replaced with the exact content provided in the testcase, so the result will be the integer 12 or the integer 15.