Skip to content

Latest commit

 

History

History
162 lines (145 loc) · 8.29 KB

benchdnn_general_info.md

File metadata and controls

162 lines (145 loc) · 8.29 KB
Raw

General Notes and Details

benchdnn follows the concept of state machine. Command line options either common or driver-specific together with default option values form a set of states. Execution iterates over all states when a problem descriptor (or dimensions) is parsed. This means that options that are specified after descriptor won't be applied for the previous descriptor, only for the next one if/when met. By default, each driver has the set of option values that results in a single state for a given problem descriptor.

Return status

Returns 1 if any submitted tests returned status FAILED or UNIMPLEMENTED, 0 otherwise.

Running Tests

oneDNN comes with its own testing infrastructure enabled through CMake. If the project is built with DNNL_BUILD_TESTS set to TRUE, then Cmake will add test targets automatically. Targets can be triggered by the following command:

    ctest [-R ".*benchdnn.*"]

, where the content in brackets is optional and limits testing to benchdnn only. The pattern can be extended further for finer granularity.

The available tests can be found in the oneDNN directory: tests/benchdnn/inputs/

Glossary

Abbreviation Description
src Source/input image
wei Weights (or filter)
bia Bias
dst Destination/output image
acc Accumulation (typically in terms of data type)

Modes

benchdnn supports several execution flows, or "modes". The driver takes the following steps to execute any flow:

  1. Parse user input.
  2. Iterate over multiple selected options for each problem descriptor and create a driver problem object (benchdnn internal abstraction) for each unique set of all options available for the driver. Each problem object continues performing the next steps.
  3. Call backend API to create backend objects to execute.
  4. Create memory objects for backend and reference paths and fill them with reasonable data.
  5. Execute backend path.
  6. Correctness validation:
    • Check that padded area, if present, is properly zeroed for each memory.
    • For GPU: check that the backend didn't write out-of-boundary.
    • Execute reference path.
    • Setup compare object.
    • Compare outputs of backend and reference and save the status.
  7. Bitwise validation:
    • Save results from the first run.
    • Refresh data in tensors if needed.
    • Execute backend path for a second time.
    • Compare outputs to be bitwise exact.
  8. Performance validation:
    • Execute backend path in a loop until one of selected criterion to stop is triggered. Refer to performance options for details.
  9. Report a test case status and repro line.
    • If performance validation was requested, print a performance report output based on selected options and collected statistics.
  10. Repeat steps 2-7 until all setups are validated.
  11. Report the summary and return the status.

Each mode is standalone since most of them include one another, unless specified otherwise. The following modes (--mode) are supported:

  • Listing (L). This flow executes steps 1-2. It allows to verify input files by parsing syntax and check, if all problem repro lines are valid.
  • Initialization (I). This flow executes steps 1-3. It allows to verify successful backend objects creation (especially large problems that take excessive memory and/or time to execute).
  • Execution (R). This flow executes steps 1-5. It saves time from running correctness when it is not needed.
  • Correctness (C). This is the default driver flow. It executes all steps, skipping bitwise and performance validation steps.
  • Bitwise (B). This flow executes all steps, skipping correctness and performance steps. The mode purpose is to check that two different library runs of the same problem return the exactly the same output. For some applications it may be a hard requirement. To increase the chances of detecting a potential issue, completely random float values are used for the accumulation order to impact the output results. It is important to note that if the case is passing, it doesn't necessarily mean the problem is free of issues; it may happen that it was not triggered.
  • Performance (P): This flow executes all steps, skipping steps 6 and 7.
  • Fast Performance (F): This flow executes Performance mode with P and M modifiers (see below) enabled and updated maximum measuring time per case.
  • Correctness & performance (CP). This flow executes all steps, skipping the bitwise step. It's not recommended for the usage due to filling data conflicts between correctness and performance modes.

Mode modifiers

Modes may have extensions to their default behavior. Those extensions may be enabled by special mode modifiers (--mode-modifier). They have limited scope and applicability. See details next to each modifier to know their limits. The following modifiers are supported:

  • Parallel test object creation (P). This is an extension of step 4, when several backend objects, up to the number of threads identified on the system, are created in parallel and then executed in order. This allows to overlap creation overhead. Applicable for both CPU and GPU and for all modes but listing. Note: this modifier changes the default scratchpad mode from library to mode because of thread-safety issue. The library scratchpad mode can't be used unless "-DDNNL_ENABLE_CONCURRENT_EXEC=ON" is enabled at the build time. Otherwise scratchpad pointers are invalidated due to threads used for creation are no longer alive at the point when execution time comes.
  • Disabling usage of host memory (M). This is an extension of performance mode when all work with host memory is disabled. It includes mapping/unmapping memory objects and also skipping filling functions with their reorders. Every value of a device memory object is assigned with a special value directly. This is applicable for GPU only.

Problem Statuses

Each problem in benchdnn receives a status reflecting the outcome of the problem execution. Following statuses are supported (in order of processing the problem):

  • LISTED. It means that a driver problem object was created, and the reproducer line might be reported. The execution was stopped before creating any library objects.
  • SKIPPED. Same as LISTED but the execution was stopped intentionally for the reason given in the short description, e.g. "Case not supported" or "Skip-impl option hit". Note: Nvidia backend is treated specially. See a note below.
  • INVALID_ARGUMENTS. It means that the library API returned an error due to incorrect argument values. It is treated as a failure.
  • UNIMPLEMENTED. It means that the library does not have an implementation for a requested problem. It is treated as a failure. Note: All Nvidia backend unimplemented status errors are always treated as SKIPPED "(Case not supported)" to simplify validation.
  • INITIALIZED. It means that a problem was initialized, and the primitive creation was successful, but there was no execution call or validation.
  • EXECUTED. It means that a problem was run, and the library execution call was successful, but the correctness was not validated.
  • PASSED. It means that a problem passed the correctness validation, and the library output matches the driver's reference output.
  • MISTRUSTED. It means that the quality of correctness validation is under question. This often happens when the ratio of the number of zero values to the total number of elements in the output exceeds a certain threshold. One possible reason is improper filling of input data for a given driver, specific algorithm, or a specific problem. Though the validation may not fulfil the purpose, as long as values are same for driver reference and the library outputs, it is not treated as a failure.
  • FAILED. It means that a problem did not pass the correctness validation, and the library output differs from the driver's reference output.
  • UNTESTED. It means that none of above statuses were assigned, and the execution was aborted at unexpected place. It is treated as a failure.

Input Files Naming Convention

Benchdnn follows certain guidelines regarding input files naming convention.