Skip to content

Commit

Permalink
Address Sanitizer support in CoreCLR (#74623)
Browse files Browse the repository at this point in the history
* Enable AddressSanitizer in CoreCLR, Libs, and Host and update runtime-sanitized to run tests with ASAN on Linux and Mac

* Unify on HAS_ADDRESS_SANITIZER define name

Centralize setting up sanitizer flags.

Remove last usage of CLR_CMAKE_ENABLE_ASAN outside of eng/native

Remove unnecessary diffs

Use the cpuid intrinsic and enable ASAN on InitJitHelpers1.

Add comments for the places where we have ASAN disabled

Undo changes to src/coreclr/jit/CMakeLists.txt

Add docs and fix the docs in the build script.

Add docs for the SkipVCEnvInit hook

* Remove CRT runtime changes. We won't need this when we finally onboard to Windows with some changes they have coming down the pipeline, so remove them for now to reduce the diff of this PR.

* Remove suppression now that we've fixed the underlying issue.

* Remove some Windows-only sanitizer CMake that we won't need when ASAN is ready for us to consume on Windows.

* Set schedule for runtime-sanitized pipeline

* Remove workaround in JIT memory set/copy helpers now that the JIT bug has been fixed.

* Add missing helix queues setup template.

* Fix missing command to build nativeaot runtime tests as nativeaot.

* Add a scenario name to trigger the extended timeout in the libraries test helix configuration.

* Remove extraneous whitespace.

* Fix using the cross-targetting ILC when sanitizers are enabled. Also pass through our TargetOS and TargetArchitecture variables to the publish command for our native sanitizers targets to correctly do their extra logic.

* Disable LSAN on the CustomMain test.

* Disable some tests on sanitized runtimes.

* Copy the sanitizer runtime for OSX NativeAOT runtime tests.

* A little cleanup to try to get the build right now that we are always doing cross-builds on Linux.

* Fix crossgen-corelib.proj syntax.

* Fix cross-os dac builds to not include the host architecture in the output path.

* Split the debugger components into a separate component and make the "unsanitized cross components" build into an "unsanitized debugger components" build as it can't be part of the regular cross-components build

* Always write out the host-arch path on Windows and update the cross-dac build script to expect that.

* Change to use the dynamic runtime on Windows, as ASAN in VS is moving to a dynamic-only model.

* Update docker images to include the sanitizer runtimes in the crossrootfs images.

* Fix explicit image tags

* The unsanitized build should be of the target architecture, not the host architecture. As a result, we still need a cross-arch build for the cross-arch use cases when sanitized, as well as an unsanitized target arch build for the debugger tools.

* Turn off using the sigaltstack for NativeAOT tests that don't use the asansupport.cpp default options

* Make sure the shared ASAN runtime is present for the nativeaot/SmokeTests/SharedLibrary test.

* Disable crossgen2 tests with sanitizers as they don't get us interesting coverage.

* Fix custom default options and disable some more crossgen2-based tests.

* Simplify lookup of asan runtime on mac and fix copying the shared runtime for the SharedLibrary NativeAOT test.

* Disable test that's failing for weird reasons.

* Fix one more alloc-dealloc mismatch that only started to show up after test merging increased allocations in the runtime

* Disable the System.Text.Json test suite on sanitized builds as it causes SO failures on Mac

* Fix test exclusion

* PR feedback.

* Fix mac build

* Do review changes

* Add libbootstrapper object files to the platform manifest now that NativeAOT has

* Disable tests that check size on sanitized builds

* Use the built-in `include_guard` option

* Disable use-after-return checking in ASAN. CoreCLR doesn't do well with parallel stacks.

* Only pass the no UAR flag on C and CXX with Clang (not AppleClang).
  • Loading branch information
jkoritzinsky authored Jul 18, 2023
1 parent a654a77 commit 2811e7c
Show file tree
Hide file tree
Showing 108 changed files with 938 additions and 249 deletions.
16 changes: 16 additions & 0 deletions docs/workflow/building/coreclr/linux-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,19 @@ Just like you can use specialized Docker images, you can also do any of the supp
## Create the Core_Root

The Core_Root provides one of the main ways to test your build. Full instructions on how to build it in the [CoreCLR testing doc](/docs/workflow/testing/coreclr/testing.md), and we also have a detailed guide on how to use it for your own testing in [its own dedicated doc](/docs/workflow/testing/using-corerun-and-coreroot.md).

## Native Sanitizers

CoreCLR can be built with native sanitizers like AddressSanitizer to help catch memory safety issues. To build the project with native sanitizers, add the `-fsanitize address` argument to the build script like the following:

```bash
build.sh -s clr -fsanitize address
```

When building the repo with any native sanitizers, you should build all native components in the repo with the same set of sanitizers.

The following sanitizers are supported for CoreCLR on Linux:

| Sanitizer Name | `-fsanitize` argument | Support Status |
|-----------------|-----------------------|----------------|
| AddressSanitize | `address` | regularly tested on x64 |
16 changes: 16 additions & 0 deletions docs/workflow/building/coreclr/macos-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,19 @@ It is possible to get a macOS ARM64 build using an Intel x64 Mac and vice versa,
## Create the Core_Root

The Core_Root provides one of the main ways to test your build. Full instructions on how to build it in the [CoreCLR testing doc](/docs/workflow/testing/coreclr/testing.md), and we also have a detailed guide on how to use it for your own testing in [its own dedicated doc](/docs/workflow/testing/using-corerun-and-coreroot.md).

## Native Sanitizers

CoreCLR can be built with native sanitizers like AddressSanitizer to help catch memory safety issues. To build the project with native sanitizers, add the `-fsanitize address` argument to the build script like the following:

```bash
build.sh -s clr -fsanitize address
```

When building the repo with any native sanitizers, you should build all native components in the repo with the same set of sanitizers.

The following sanitizers are supported for CoreCLR on macOS:

| Sanitizer Name | `-fsanitize` argument | Support Status |
|-----------------|-----------------------|----------------|
| AddressSanitize | `address` | regularly tested on x64 |
4 changes: 4 additions & 0 deletions docs/workflow/building/coreclr/nativeaot.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,10 @@ Build library tests by passing the `libs.tests` subset together with the `/p:Tes
* [ILC Compiler Architecture](/docs/design/coreclr/botr/ilc-architecture.md)
* [Managed Type System](/docs/design/coreclr/botr/managed-type-system.md)

## Native Sanitizers

Using native sanitizers with NativeAOT requires additional care compared to using them with CoreCLR. In addition to passing the `-fsanitize` flag to the command that builds NativeAOT, you must also pass the `EnableNativeSanitizers` MSBuild property to any commands that build projects with a sanitized NativeAOT build to ensure that any sanitizer runtimes are correctly linked with the project.

## Further Reading

If you want to know more about working with _NativeAOT_ in general, you can check out their [more in-depth docs](/src/coreclr/nativeaot/docs/README.md) in the `src/coreclr/nativeaot` subtree.
20 changes: 20 additions & 0 deletions docs/workflow/building/coreclr/windows-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,23 @@ build.cmd -s clr -c Release -arch arm64 -msbuild
```

Since this is still in an experimental phase, the recommended way for building ARM64 is cross-compiling from an x64 machine. Instructions on how to do this can be found at the [cross-building doc](/docs/workflow/building/coreclr/cross-building.md#cross-compiling-for-arm32-and-arm64).

## Native Sanitizers

CoreCLR can be built with native sanitizers like AddressSanitizer to help catch memory safety issues. To build the project with native sanitizers, add the `-fsanitize address` argument to the build script like the following:

```cmd
build.cmd -s clr -fsanitize address
```

When building the repo with any native sanitizers, you should build all native components in the repo with the same set of sanitizers.

The following sanitizers are supported for CoreCLR on Windows:

| Sanitizer Name | Minimum VS Version | `-fsanitize` argument | Support Status |
|----------------|--------------------|-----------------------|----------------|
| AddressSanitizer | not yet released | `address` | experimental |

## Using a custom compiler environment

If you ever need to use a custom compiler environment for the native builds on Windows, you can set the `SkipVCEnvInit` environment variable to `1`. The build system will skip discovering Visual Studio and initializing its development environment when this flag is used. This is only required for very advanced scenarios and should be used rarely.
10 changes: 10 additions & 0 deletions docs/workflow/building/libraries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,16 @@ By default the `build` script only builds the product libraries and none of the

For Windows, replace `./build.sh` with `build.cmd`.

### Building the native components with native sanitizers

The libraries native components can be built with native sanitizers like AddressSanitizer to help catch memory safety issues. To build the project with native sanitizers, add the `-fsanitize` argument to the build script like the following:

```bash
build.sh -s libs -fsanitize address
```

When building the repo with any native sanitizers, you should build all native components in the repo with the same set of sanitizers.

### How to build native components only

The libraries build contains some native code. This includes shims over libc, openssl, gssapi, and zlib. The build system uses CMake to generate Makefiles using clang. The build also uses git for generating some version information.
Expand Down
29 changes: 25 additions & 4 deletions eng/Subsets.props
Original file line number Diff line number Diff line change
Expand Up @@ -253,8 +253,11 @@
Category="clr" />
</ItemGroup>

<!-- Build the CoreCLR cross tools when we're doing a cross build and either we're building any CoreCLR native tools for platforms CoreCLR fully supports or when someone explicitly requests them -->
<ItemGroup Condition="(('$(ClrRuntimeBuildSubsets)' != '' and '$(PrimaryRuntimeFlavor)' == 'CoreCLR') or $(_subset.Contains('+clr.crossarchtools+'))) and ('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)')">
<!--
Build the CoreCLR cross tools when we're doing a cross build and either we're building any CoreCLR native tools for platforms CoreCLR fully supports or when someone explicitly requests them.
The cross tools are used as part of the build process with the downloaded build tools, so we need to build them for the host architecture and build them as unsanitized binaries.
-->
<ItemGroup Condition="(('$(ClrRuntimeBuildSubsets)' != '' and '$(PrimaryRuntimeFlavor)' == 'CoreCLR') or $(_subset.Contains('+clr.crossarchtools+'))) and ('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or '$(EnableNativeSanitizers)' != '')">
<ProjectToBuild
Include="$(CoreClrProjectRoot)runtime.proj"
AdditionalProperties="%(AdditionalProperties);
Expand All @@ -265,6 +268,24 @@
NoPgoOptimize=true;
CrossBuild=false;
CMakeArgs=$(CMakeArgs) -DCLR_CROSS_COMPONENTS_BUILD=1"
UndefineProperties="EnableNativeSanitizers"
Category="clr" />
</ItemGroup>

<!--
Build the debugging components of CoreCLR for the same target architecture as an unsanitized build whenever we build a sanitized coreclr build.
These components are loaded into a debugger process, which generally is not a sanitized executable.
-->
<ItemGroup Condition="'$(ClrRuntimeBuildSubsets)' != '' and '$(EnableNativeSanitizers)' != ''">
<ProjectToBuild
Include="$(CoreClrProjectRoot)runtime.proj"
AdditionalProperties="%(AdditionalProperties);
ClrDebugSubset=true;
PgoInstrument=false;
NoPgoOptimize=true;
CrossBuild=$(CrossBuild);
BuildSubdirectory=unsanitized"
UndefineProperties="EnableNativeSanitizers"
Category="clr" />
</ItemGroup>

Expand Down Expand Up @@ -317,8 +338,8 @@
<ProjectToBuild Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler.csproj" Category="clr" Condition="'$(NativeAotSupported)' == 'true'" />
<ProjectToBuild Include="$(CoreClrProjectRoot)nativeaot\BuildIntegration\BuildIntegration.proj" Category="clr" Condition="'$(NativeAotSupported)' == 'true'" />

<ProjectToBuild Condition="'$(NativeAotSupported)' == 'true' and ('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or '$(HostOS)' != '$(TargetOS)')" Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler_crossarch.csproj" Category="clr" />
<ProjectToBuild Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(HostOS)' != '$(TargetOS)'" Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_crossarch.csproj" Category="clr" />
<ProjectToBuild Condition="'$(NativeAotSupported)' == 'true' and ('$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or '$(HostOS)' != '$(TargetOS)' or '$(EnableNativeSanitizers)' != '')" Include="$(CoreClrProjectRoot)tools\aot\ILCompiler\ILCompiler_crossarch.csproj" Category="clr" />
<ProjectToBuild Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(HostOS)' != '$(TargetOS)' or '$(EnableNativeSanitizers)' != ''" Include="$(CoreClrProjectRoot)tools\aot\crossgen2\crossgen2_crossarch.csproj" Category="clr" />

<ProjectToBuild Condition="'$(TargetOS)' == 'windows' or ('$(TargetOS)' == 'linux' and ('$(TargetArchitecture)' == 'x64' or '$(TargetArchitecture)' == 'arm64')) or '$(TargetOS)' == 'osx'" Include="$(CoreClrProjectRoot)tools\SuperFileCheck\SuperFileCheck.csproj" Category="clr" />
</ItemGroup>
Expand Down
15 changes: 10 additions & 5 deletions eng/build.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Param(
[switch]$msbuild,
[string]$cmakeargs,
[switch]$pgoinstrument,
[string[]]$fsanitize,
[Parameter(ValueFromRemainingArguments=$true)][String[]]$properties
)

Expand Down Expand Up @@ -84,10 +85,13 @@ function Get-Help() {
Write-Host ""

Write-Host "Native build settings:"
Write-Host " -cmakeargs User-settable additional arguments passed to CMake."
Write-Host " -ninja Use Ninja to drive the native build. (default)"
Write-Host " -msbuild Use MSBuild to drive the native build. This is a no-op for Mono."
Write-Host " -pgoinstrument Build the CLR with PGO instrumentation."
Write-Host " -cmakeargs User-settable additional arguments passed to CMake."
Write-Host " -ninja Use Ninja to drive the native build. (default)"
Write-Host " -msbuild Use MSBuild to drive the native build. This is a no-op for Mono."
Write-Host " -pgoinstrument Build the CLR with PGO instrumentation."
Write-Host " -fsanitize (address) Build the native components with the specified sanitizers."
Write-Host " Sanitizers can be specified with a comma-separated list."
Write-Host ""

Write-Host "Command-line arguments not listed above are passed through to MSBuild."
Write-Host "The above arguments can be shortened as much as to be unambiguous."
Expand Down Expand Up @@ -220,7 +224,7 @@ if ($vs) {

# Put our local dotnet.exe on PATH first so Visual Studio knows which one to use
$env:PATH=($env:DOTNET_ROOT + ";" + $env:PATH);

# Disable .NET runtime signature validation errors which errors for local builds
$env:VSDebugger_ValidateDotnetDebugLibSignatures=0;

Expand Down Expand Up @@ -269,6 +273,7 @@ foreach ($argument in $PSBoundParameters.Keys)
# configuration and arch can be specified multiple times, so they should be no-ops here
"configuration" {}
"arch" {}
"fsanitize" { $arguments += " /p:EnableNativeSanitizers=$($PSBoundParameters[$argument])"}
default { $arguments += " /p:$argument=$($PSBoundParameters[$argument])" }
}
}
Expand Down
16 changes: 16 additions & 0 deletions eng/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ usage()
echo " --keepnativesymbols Optional argument: set to true to keep native symbols/debuginfo in generated binaries."
echo " --ninja Optional argument: set to true to use Ninja instead of Make to run the native build."
echo " --pgoinstrument Optional argument: build PGO-instrumented runtime"
echo " --fsanitize Optional argument: Specify native sanitizers to instrument the native build with. Supported values are: 'address'."
echo ""

echo "Command line arguments starting with '/p:' are passed through to MSBuild."
Expand Down Expand Up @@ -509,6 +510,21 @@ while [[ $# > 0 ]]; do
shift 1
;;

-fsanitize)
if [ -z ${2+x} ]; then
echo "No value for -fsanitize is supplied. See help (--help) for supported values." 1>&2
exit 1
fi
arguments="$arguments /p:EnableNativeSanitizers=$2"
shift 2
;;

-fsanitize=*)
sanitizers="${opt/#-fsanitize=/}" # -fsanitize=address => address
arguments="$arguments /p:EnableNativeSanitizers=$sanitizers"
shift 2
;;

*)
extraargs="$extraargs $1"
shift 1
Expand Down
2 changes: 1 addition & 1 deletion eng/liveBuilds.targets
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
<CoreCLRSharedFrameworkDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'sharedFramework'))</CoreCLRSharedFrameworkDir>
<CoreCLRCrossgen2Dir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'crossgen2'))</CoreCLRCrossgen2Dir>
<CoreCLRILCompilerDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'ilc-published'))</CoreCLRILCompilerDir>
<CoreCLRCrossILCompilerDir Condition="'$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or '$(HostOS)' != '$(TargetOS)'">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', '$(BuildArchitecture)', 'ilc'))</CoreCLRCrossILCompilerDir>
<CoreCLRCrossILCompilerDir Condition="'$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)' or '$(HostOS)' != '$(TargetOS)' or '$(EnableNativeSanitizers)' != ''">$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', '$(BuildArchitecture)', 'ilc'))</CoreCLRCrossILCompilerDir>
<CoreCLRAotSdkDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'aotsdk'))</CoreCLRAotSdkDir>
<CoreCLRBuildIntegrationDir>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)', 'build'))</CoreCLRBuildIntegrationDir>

Expand Down
12 changes: 12 additions & 0 deletions eng/native/build-commons.sh
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,7 @@ usage()
echo "-portablebuild: pass -portablebuild=false to force a non-portable build."
echo "-skipconfigure: skip build configuration."
echo "-keepnativesymbols: keep native/unmanaged debug symbols."
echo "-fsanitize: Enable native sanitizers"
echo "-verbose: optional argument to enable verbose build output."
echo ""
echo "Additional Options:"
Expand Down Expand Up @@ -392,6 +393,17 @@ while :; do
__CMakeArgs="$__CMakeArgs -DCLR_CMAKE_KEEP_NATIVE_SYMBOLS=true"
;;

-fsanitize)
__CMakeArgs="$__CMakeArgs -DCLR_CMAKE_ENABLE_SANITIZERS=$2"
EnableNativeSanitizers=$2
shift
;;
-fsanitize=*)
sanitizers="${lowerI/#-fsanitize=/}" # -fsanitize=address => address
__CMakeArgs="$__CMakeArgs -DCLR_CMAKE_ENABLE_SANITIZERS=$sanitizers"
EnableNativeSanitizers=$sanitizers
;;

ninja|-ninja)
__UseNinja=1
;;
Expand Down
Loading

0 comments on commit 2811e7c

Please sign in to comment.