Tool: DATm: Diderot's Automated testing

Details

• Branch: Diderot-Dev
• Use: Test operators on and between tensors/image data based on correctness
• Tool: DATm:Diderot’s Automated Testing tool
• Text: ICSE-AST paper and Testing chapter in Dissertation

  Testing environment variables in Pg 102 Adding a new operator in Pg 113

User Guide

Getting Started

Quick instructions

1. Checkout github directory for DATm

   git clone https://github.com/cchiw/DATm.git

2. Change cpath in Frame to your absolute path to diderot branches. See Set Up about other variables you might want to change. 3. Starting Testing with command line arguments. See Section on Running DATm.

   python3 datm.py

Set Up: variables and testing frame settings

Settings in the testing process can be changed by commenting in/out variables in the Frame (input.py).

• Change branch(s_branch) : Tell DATm the name of branch being tested. Some branch names are built in(vis15,Diderot-Dev, Chiw17). Comment in the right s_branch variable in Frame

   #s_branch  = branch_vis15
   s_branch = branch_dev 
   #s_branch = branch_chiw17
   #s_branch  = branch_other
  

or comment in branch_other and set the variable to a string in shared/base_constants.py

• Complexity (s_layer): The core computation in a Diderot test program can be simple or more complicated. s_layer indicates the number of operators to apply in a core computation. That number can be 1, 2, or 3.

• type of field(c_pde_test) : DATm test tensors and fields. The fields can either be made by nrrd files or by Firedrake (outside tool to solve PDE solutions). For an original Diderot Field types created with nrrd, set variable c_pde_test to False in Frame. For PDE solutions set the variable to true and change the path in fem/makedefs.gmk.

type of search (s_random_range) For an exhaustive testing approach, set variable s_random_range to 0 in Frame. For randomized testing set the variable to x where the probability of a single test case being generated is 1 in x+1.

• order of coefficients(s_coeff_style) : The order of coefficients for the polynomial creating synthetic data. The data can either be linear, quadratic or cubic.

   #s_coeff_style = coeff_linear
   s_coeff_style = coeff_quadratic
   #s_coeff_style = coeff_cubic
  

• testing environment: You can comment in and out variables in Frame. This includes variables to change the number of samples, type of arguments,..

  More details in Pg 102 in Dissertation,

Labels

• Each test case has a testing label

  Of the form "p_o1...l2"

Running DATm: command-line commands and targetting testing

The testing environment is indicated by the Frame. The scope helps target a specific operator, test, or family of programs.

• Run everything:

  python3 datm.py

• Test a single operator

  python3 datm.py id # where id is a number Note Each operator has a unique id. List printed to screen and copies to rst/stash/results_ops.txt

• Family of computations
  Rerun (group of) tests by using 1-4 integers from the testing label. For instance, the label

  “p_o27_o0_t0_tN_tN_l2”

  can rerun with command

  python datm.py 27 0 0

Results passes/fails

Great, everything is running now, but how do I look at the results? In the directory rst/stash are several text files that record the test cases (with labels)

• results_final.txt:The results of each test case
• results_terrible.txt: Reports test cases with errors
• results_ty.txt:Test Types used
• results_ops.txt:Operators with ids

Development

• Adding a new operator to DATm:
  1. Add to operator constant: shared/obj_operator.py
  2. Add case to type-checker: shared/obj_typechecker.py
  3. Add way to evaluate that operator applied to polynomials: nc/nc_eval.py

  More details in Pg 113 in Dissertation

Organization and flow

Command line arguments

graph LR
L(input.py) -- python3 datm.py ... --> Cr{core}
Cr --> C{Cte}
Cr --> f{fem}
C --> R{rst}
f --> R{rst}

Loading

Iterate over test cases

graph LR
C(datm.py) -- Handler --> ICC(core_iter_cmd.py) 
ICC-- Iterates  -->IC(core_iter.py)
IC -- Typechecks --> OTY(obj_typechecker.py)
IC-- Runs single test  -->CC(cte_core.py)

Loading

Single Test case

graph LR
CC(cte_core.py) -- Creates arguments -->NF(nc_createField.py)
CC-- Writes Diderot program --> CW(cte_writeDiderot.py)
CC -- Observed data --> BO(base_observed.py)
CC-- Correct  output --> CE(cte_eval.py)
CC-- Compare  output --> NC(nc_compare.py)

Loading