Compiler cache (cl_cache) is a mechanism that can be used to improve the performance of just-in-time (JIT) compilation. When a kernel is compiled, the binary is stored in the cache. If a compilation of the same kernel is requested, the binary from the cache is used instead of being recompiled. Utilizing the cache can substitute the need for compiling kernels in the primary source format - OpenCL C as well as in an intermediate representation, such as SPIRV. Compute Runtime implements a persistent cache, which means storing compiled kernels on a long-term basis. Cached files are preserved across multiple runs of OpenCL/Level Zero applications and only deleted once eviction is needed.
The cache mechanism in Compute Runtime is supported by two APIs: OpenCL and Level Zero.
The following file extensions are used:
- OpenCL: .cl_cache
- Level Zero: .l0_cache
Each cache file has a <hash>.<extension> format.
Hash is a unique identifier for a file, created based on the following attributes:
Kernel sources
- Translation Input
Provided options
- Build API Options
- Build Internal Options
User defined constants
- Specialization Constant Identifiers
- Specialization Constant Values
Identifying the compiler's unique version
- IGC Revision
- IGC Library Size
- IGC Library Modification Time
Additional information
- Hardware Info
The resulting hash is identical for a particular set of variables, which means that the same variables always generate the same hash. This ensures that we always read/write the right binary file under the given conditions.
| Environment Variable | Value | Description |
|---|---|---|
| NEO_CACHE_PERSISTENT | 0: disabled 1: enabled Default: 1 |
Enable or disable on-disk binary cache. When enabled Compute Runtime will try to cache and reuse compiled binaries. |
| NEO_CACHE_DIR | <Absolute path> Default: %LocalAppData%\NEO\neo_compiler_cache |
Path to persistent cache directory. If NEO_CACHE_DIR is not set and %LocalAppData% could not be accessed, on-disk cache is disabled. |
| NEO_CACHE_MAX_SIZE | <Size in bytes> Default: 1 GB |
Maximum size of compiler cache in bytes. Total size of files stored in the cache will never exceed this value. If adding a new binary would cause the cache to exceed its limit, the eviction mechanism is triggered. Set to 0 to disable size-based cache eviction. |
| Environment Variable | Value | Description |
|---|---|---|
| NEO_CACHE_PERSISTENT | 0: disabled 1: enabled Default: 1 |
Enable or disable on-disk binary cache. When enabled Compute Runtime will try to cache and reuse compiled binaries. |
| NEO_CACHE_DIR | <Absolute path> Default: $XDG_CACHE_HOME/neo_compiler_cache |
Path to persistent cache directory. Default value is $XDG_CACHE_HOME/neo_compiler_cache if $XDG_CACHE_HOME is set, $HOME/.cache/neo_compiler_cache otherwise. If neither NEO_CACHE_DIR, $XDG_CACHE_HOME nor $HOME is defined, on-disk cache is disabled. |
| NEO_CACHE_MAX_SIZE | <Size in bytes> Default: 1GB |
Maximum size of compiler cache in bytes. Total size of files stored in the cache will never exceed this value. If adding a new binary would cause the cache to exceed its limit, the eviction mechanism is triggered. Set to 0 to disable size-based cache eviction. |
When persistent cache is enabled, at first occurance driver creates config.file which contains the directory size and is also entry point to caching mechanism. As cache files are created/evicted, the contents of config.file are automatically updated. Such mechanism prevents the total cache size from being calculated at each compilation and prevents iterating through all files each time.
Each write to disk has following steps:
- lock config.file (advisor lock)
- create temporary file
- write contents to file
- rename temporary file to proper hash name
- store updated directory size in the config.file
- unlock config.file
Since Compute Runtime cl_cache is persistent type of cache, it is not cleared when the application has finished or the system is rebooted.
It stays until eviction is needed, which is when the cache limit set by the NEO_CACHE_MAX_SIZE is about to be reached.
At such point, Compute Runtime activates the eviction mechanism, which works as follows:
- lock config.file (advisor lock)
- gather all cache files
- sort files by last access time value (read/write)
- remove least recently accessed files with a total size of 1/3
NEO_CACHE_MAX_SIZE - store updated directory size in the config.file
- unlock config.file
The eviction mechanism first removes the least recently accessed files, which are least likely to be reused. This keeps the cache as up-to-date as possible.
- By using mutex and file locking mechanism, cl_cache provides thread and process safety
- Skipping multiple compilation results in performance gains
- Automatic eviction of old cache
The cl_cache mechanism consists of many stages and an error can occur in any of them.
For this reason, we have placed error checking at each critical point.
If a problem occurred, the information about it can be read with a debug key:
PrintDebugMessages=0/1 : when enabled, some debug messages will be propagated to console
Additionally, the BinaryCacheTrace flag can be used to provide deeper insights into the hash creation process:
BinaryCacheTrace=0/1 : when enabled, cl_cache will generate a trace file containing values of all attributes used to calculate the unique hash for a compiled binary. This information can be invaluable in understanding why a specific kernel is or isn't being retrieved from the cache as expected.
Note: PrintDebugMessages propagates all debug messages from the driver - it is not exclusive to cl_cache.
-
Since the cl_cache mechanism automatically keeps track of the size of all created and deleted cache files, the user should not manually tamper with the generated files. If this were to happen, config.file would not have the correct directory size which could result in failure to meet the limit set by
NEO_CACHE_MAX_SIZE. However, it is safe to manually delete the entire cache directory or all files at once. -
If we encounter issues opening or reading data from the config.file (e.g., due to corruption), the cache mechanism will not be utilized, and we will fall back to the standard compilation path. In this situation, information about the error will be printed in debug messages.
-
cl_cache relies on the last access time of files to manage its eviction process effectively. If last access time updates are not enabled in the filesystem, cl_cache will be unable to accurately prioritize files for eviction based on their usage frequency. Consequently, files may not be evicted in the optimal order, potentially affecting cache performance. To address this limitation, future plans include implementing a backup sorting method for eviction, which may utilize file size or creation time as criteria to ensure efficient cache management even when last access time is unavailable.
-
When generating a unique hash, Compiler Cache does not take into account the environment variables of external components. Changes in these variables may not trigger cache invalidation, which can lead to unexpected behavior and difficult to debug errors.