From 60b8f018042a8055ce49fe656d8f9c8349bcaf10 Mon Sep 17 00:00:00 2001 From: Doug Ilijev Date: Wed, 13 Jul 2016 11:26:37 -0700 Subject: [PATCH] MicroBuild v2: Fix binary and package version metadata; refactor and improve scripts. Binary and Package Versions =========================== Binary File Version Metadata ---------------------------- Enables correct embedding of version string in file info. All binaries will now embed File Version metadata as follows: ...0 (:...) Example: 1.2.1.0 (release/1.2:9786fcdbcaa6963a5ccc6648b7aa9bd9dd51d9ec.00008.12122.161007-1756) `MAJOR`, `MINOR`, and `PATCH` correspond to SemVer fields and will match the version of ChakraCore the package corresponds to. The fourth field would have been used to indicate security fixes (a.k.a. `QFE`) but because a fourth field is not allowed in NuGet package version numbers, but is standard for versions of binaries produced from VS projects, we are keeping the fourth field in the binary data and locking it to 0, and omitting it from NuGet package version numbers and release numbers from now on. Release NuGet Package Versions ------------------------------ The format of version strings for released NuGet packages will continue to be: Microsoft.ChakraCore....nupkg Example: Microsoft.ChakraCore.1.2.1.nupkg Preview NuGet Package Versions ------------------------------ The format of version strings for preview NuGet packages will now be: Microsoft.ChakraCore...-.nupkg Where `` is: preview-- This solves the problem of getting enough information into the package preview version string (``) to order builds correctly while still guaranteeing that we fit the requirements for the `` field: 1. It must start with a letter 2. It can only contain the characters [a-zA-Z0-9-] 3. It must be 20 characters or less Example: Microsoft.ChakraCore.1.2.1-preview-00008-12122.nupkg Miscellaneous Version-Related Changes ------------------------------------- Add `PATCH` version to `CommonDefines.h` so that it doesn't need to be manually coded into builds. (By reducing user intervention for package versions we make it more feasible to automatically release preview packages from multiple branches.) Extract `MAJOR`, `MINOR`, `PATCH`, and `QFE` version numbers from the source code by grepping for values of the `#defines` that correspond to those values. Refactorings and Script Parameters ================================== Merge `pogo_build.ps1` into `run_build.ps1`. Passing the required pogo-related flags to `run_build.ps1` produces a POGO build. No need to have separate scripts. Compute `ObjectDirectory` and use that for `Chakra.Generated.BuildInfo.props` unless `TF_BUILD_BUILDDIRECTORY` is specified in the PATH. Infer `$binpath`. For many parameters that were previously specified as environment variables from the environment of the build infrastructure, it is now possible to specify them as script parameters directly, where script parameters will take precedence over the values of the corresponding environment variables. This can be helpful when running scripts manually where you want to quickly experiment with the value of a variable without having to write it to the environment first. Since it is sometime valuable to get that variable from the environment, if the script parameter is unspecified, the value of the environment variable will be used instead. Additionally, if any parameter is unspecified by either script parameter or environment variable, a reasonable default is encoded in the script logic. In order of precedence, all values will be determined by checking for values in the following order, if applicable: 1. Script parameter 2. Environment variable 3. Script logic to retrieve value 4. Default value Add parameters with corresponding environment variables to specify skipping certain paths through the build scripts, which can save time when testing certain changes by ignoring expensive but unimportant parts of the builds. Specifying an environment value at the time the build is queued is easier than modifying the build definition, so adding the environment variable option adds convenience when testing with the production build definitions. * `run_build.ps1 -skipPogo` or set `SKIP_POGO` to a non-empty string will skip the POGO phase of the build, if it would have been a POGO build. Useful if the build binaries produced at the correct locations is enough and you're most interested in testing the non-POGO-related aspects of the build scripts. * `post_build.ps1 -skipTests` or set `SKIP_TESTS` to a non-empty string will skip running tests, which can be lengthy. This is useful if the quality of the binaries produced is not important and you're interested in testing other aspects of the build scripts. Miscellaneous ------------- Reduce dependencies on data from build server environments so that these scripts can be used to perform local builds. Update `UseValueOrDefault` to take any number of arguments. Reduce dependency on `locate_msbuild.ps1`. Add `$BuildSubtype` to 'not enough info' error. Check for the existence of `nuget.exe` before trying to invoke it. This helps when running in environments where NuGet is not installed or is not available. Can avoid errors related to NuGet package restore when running on a build server with a custom NuGet location and a specific NuGet Task separate from our scripts. Keeping the NuGet invocation in our scripts allows us to still do local builds without MicroBuild infrastructure configured on the dev machine (we don't take a hard dependency on the aforementioned NuGet Task. --- .gitignore | 2 + Build/Common.Build.Default.props | 7 +- Build/scripts/finalize_build.ps1 | 2 +- Build/scripts/init_build.ps1 | 160 +++++++++++++++++++++---------- Build/scripts/pogo_build.ps1 | 143 --------------------------- Build/scripts/post_build.ps1 | 12 ++- Build/scripts/pre_build.ps1 | 12 +-- Build/scripts/pre_post_util.ps1 | 14 ++- Build/scripts/run_build.ps1 | 107 +++++++++++++++++---- Build/scripts/util.ps1 | 18 ++-- bin/CoreCommon.ver | 9 +- lib/Common/CommonDefines.h | 3 +- 12 files changed, 254 insertions(+), 235 deletions(-) delete mode 100644 Build/scripts/pogo_build.ps1 diff --git a/.gitignore b/.gitignore index 535f61bf7c2..1880a582f6e 100644 --- a/.gitignore +++ b/.gitignore @@ -32,3 +32,5 @@ Build/Chakra.Core.VC.opendb test/benchmarks/*.txt test/benchmarks/*.dpl *.nupkg +_DROP +packages.config diff --git a/Build/Common.Build.Default.props b/Build/Common.Build.Default.props index f5b03dfac81..bcf47e801dd 100644 --- a/Build/Common.Build.Default.props +++ b/Build/Common.Build.Default.props @@ -58,7 +58,6 @@ amd64 - $(OutBaseDir)\$(SolutionName) @@ -68,7 +67,11 @@ - + + $(TF_BUILD_BUILDDIRECTORY) + $(IntBaseDir)\obj\$(PlatformPathName.ToLower())_$(Configuration.ToLower()) + + diff --git a/Build/scripts/finalize_build.ps1 b/Build/scripts/finalize_build.ps1 index 4d158f51c7e..94c4ede1dc6 100644 --- a/Build/scripts/finalize_build.ps1 +++ b/Build/scripts/finalize_build.ps1 @@ -101,7 +101,7 @@ $buildFlavorJson | Add-Member -type NoteProperty -name flavor -value $Env:BuildC $buildFlavorJson | Add-Member -type NoteProperty -name subtype -value $Env:BuildSubtype $buildFlavorJson | ConvertTo-Json | Write-Output -$buildFlavorJson | ConvertTo-Json | Out-File $buildFlavorJsonFile -Encoding ascii +$buildFlavorJson | ConvertTo-Json | Out-File $buildFlavorJsonFile -Encoding utf8 # # Copy outputs to metadata directory diff --git a/Build/scripts/init_build.ps1 b/Build/scripts/init_build.ps1 index 18430024624..e84f107da74 100644 --- a/Build/scripts/init_build.ps1 +++ b/Build/scripts/init_build.ps1 @@ -12,23 +12,47 @@ # before running the Pre-Build script. param ( + [ValidateSet("x86", "x64", "arm", "")] + [string]$arch = "", + [ValidateSet("debug", "release", "test", "codecoverage", "")] + [string]$flavor = "", + [ValidateSet("default", "codecoverage", "pogo")] + [string]$subtype = "default", + [string]$buildtype, + [string]$envConfigScript = "ComputedEnvironment.cmd", [string[]]$supportedPogoBuildTypes = @("x64_release", "x86_release"), - [Parameter(Mandatory=$True)] + [string]$verMajor = "", + [string]$verMinor = "", + [string]$verPatch = "", + [string]$verSecurity = "", + + [string]$dropRoot, + + [switch]$cleanBinDir, + [string]$oauth ) -# If $Env:BuildType is specified, extract BuildPlatform and BuildConfiguration -# Otherwise, if $Env:BuildPlatform and $Env:BuildConfiguration are specified, construct $BuildType -$BuildPlatform = $Env:BuildPlatform -$BuildConfiguration = $Env:BuildConfiguration -$BuildType = $Env:BuildType -$BuildSubtype = "default" # will remain as "default" or become e.g. "pogo", "codecoverage" +# +# Define values for variables based on parameters and environment variables +# with default values in case the environment variables are not defined. +# + +. $PSScriptRoot\util.ps1 +$gitExe = GetGitPath + +$BuildType = UseValueOrDefault $buildtype $Env:BuildType +$BuildPlatform = UseValueOrDefault $arch $Env:BuildPlatform +$BuildConfiguration = UseValueOrDefault $flavor $Env:BuildConfiguration +$BuildSubtype = UseValueOrDefault $subtype $Env:BuildSubtype -if (Test-Path Env:\BuildType) { - $BuildType = $Env:BuildType +# If $BuildType is specified, extract BuildPlatform and BuildConfiguration +# Otherwise, if $BuildPlatform and $BuildConfiguration are specified, construct $BuildType +# $BuildSubtype will remain as "default" if not already specified, or become e.g. "pogo", "codecoverage" +if ($BuildType) { $buildTypeSegments = $BuildType.split("_") $BuildPlatform = $buildTypeSegments[0] $BuildConfiguration = $buildTypeSegments[1] @@ -42,12 +66,10 @@ if (Test-Path Env:\BuildType) { $BuildSubtype = "codecoverage" # keep information about codecoverage in the subtype } - if (-not ($BuildSubtype -in @("default","pogo","codecoverage"))) { + if (-not ($BuildSubtype -in @("default", "pogo", "codecoverage"))) { Write-Error "Unsupported BuildSubtype: $BuildSubtype" } -} elseif ((Test-Path Env:\BuildPlatform) -and (Test-Path Env:\BuildConfiguration)) { - $BuildPlatform = $Env:BuildPlatform - $BuildConfiguration = $Env:BuildConfiguration +} elseif ($BuildPlatform -and $BuildConfiguration) { $BuildType = "${BuildPlatform}_${BuildConfiguration}" } else { Write-Error (@" @@ -56,67 +78,92 @@ if (Test-Path Env:\BuildType) { BuildType={0} BuildPlatform={1} BuildConfiguration={2} + BuildSubtype={3} -"@ -f $Env:BuildType, $Env:BuildPlatform, $Env:BuildConfiguration) +"@ -f $BuildType, $BuildPlatform, $BuildConfiguration, $BuildSubtype) exit 1 } +$CommitHash = UseValueOrDefault $Env:BUILD_SOURCEVERSION $(iex "${gitExe} rev-parse HEAD") + +$branchFullName = UseValueOrDefault $Env:BUILD_SOURCEBRANCH $(iex "${gitExe} rev-parse --symbolic-full-name HEAD") + +$SourcesDirectory = UseValueOrDefault $Env:BUILD_SOURCESDIRECTORY $(GetRepoRoot) +$BinariesDirectory = UseValueOrDefault (Join-Path $SourcesDirectory "Build\VcBuild") +$ObjectDirectory = Join-Path $BinariesDirectory "obj\${BuildPlatform}_${BuildConfiguration}" + +$DropRoot = UseValueOrDefault $dropRoot $Env:DROP_ROOT (Join-Path $(GetRepoRoot) "_DROP") + # set up required variables and import pre_post_util.ps1 $arch = $BuildPlatform $flavor = $BuildConfiguration $OuterScriptRoot = $PSScriptRoot # Used in pre_post_util.ps1 . "$PSScriptRoot\pre_post_util.ps1" -$gitExe = GetGitPath - $BuildName = ConstructBuildName -arch $BuildPlatform -flavor $BuildConfiguration -subtype $BuildSubtype -$branch = $Env:BUILD_SOURCEBRANCH -if (-not $branch) { - $branch = iex "$gitExe rev-parse --symbolic-full-name HEAD" -} - -$BranchName = $branch.split('/',3)[2] +$BranchName = $branchFullName.split('/',3)[2] $BranchPath = $BranchName.replace('/','\') -$CommitHash = $Env:BUILD_SOURCEVERSION + if (-not $CommitHash) { - $CommitHash = iex "$gitExe rev-parse HEAD" + $CommitHash = iex "${gitExe} rev-parse HEAD" } +$CommitShortHash = $(iex "${gitExe} rev-parse --short $CommitHash") -$Username = (iex "$gitExe log $CommitHash -1 --pretty=%ae").split('@')[0] -$CommitDateTime = [DateTime]$(iex "$gitExe log $CommitHash -1 --pretty=%aD") +$Username = (iex "${gitExe} log $CommitHash -1 --pretty=%ae").split('@')[0] +$CommitDateTime = [DateTime]$(iex "${gitExe} log $CommitHash -1 --pretty=%aD") $CommitTime = Get-Date $CommitDateTime -Format yyMMdd.HHmm # # Get Build Info # -$info = GetBuildInfo $oauth $CommitHash +$buildPushDate = $null +$buildPushIdString = $null -$BuildPushDate = [datetime]$info.push.date -$PushDate = Get-Date $BuildPushDate -Format yyMMdd.HHmm +if (-not $oauth) +{ + $buildPushIdPart1 = 65535 + $buildPushIdPart2 = 65535 + $buildPushIdString = "65535.65535" + $buildPushDate = [DateTime]$CommitDateTime +} +else +{ + $info = GetBuildInfo $oauth $CommitHash + $_, $buildPushIdPart1, $buildPushIdPart2, $buildPushIdString = GetBuildPushId $info + $buildPushDate = [DateTime]$info.push.date +} + +$PushDate = Get-Date $buildPushDate -Format yyMMdd.HHmm + +$VersionMajor = UseValueOrDefault $verMajor $Env:VERSION_MAJOR (GetVersionField "CHAKRA_CORE_MAJOR_VERSION") "0" +$VersionMinor = UseValueOrDefault $verMinor $Env:VERSION_MINOR (GetVersionField "CHAKRA_CORE_MINOR_VERSION") "0" +$VersionPatch = UseValueOrDefault $verPatch $Env:VERSION_PATCH (GetVersionField "CHAKRA_CORE_PATCH_VERSION") "0" +$VersionSecurity = UseValueOrDefault $verSecurity $Env:VERSION_QFE (GetVersionField "CHAKRA_CORE_VERSION_RELEASE_QFE") "0" -$buildPushId, $buildPushIdPart1, $buildPushIdPart2, $buildPushIdString = GetBuildPushId $info +$VersionString = "${VersionMajor}.${VersionMinor}.${VersionPatch}" # Only use MAJOR.MINOR.PATCH to align with SemVer -$VersionMajor = UseValueOrDefault "$Env:VERSION_MAJOR" "1" -$VersionMinor = UseValueOrDefault "$Env:VERSION_MINOR" "2" -$VersionPatch = UseValueOrDefault "$Env:VERSION_PATCH" "0" -$VersionQFE = UseValueOrDefault "$Env:VERSION_QFE" "0" +$buildVersionString = "{0}-{1}" -f $buildPushIdPart1.ToString("00000"), $buildPushIdPart2.ToString("00000") +$PreviewVersionString = "${VersionString}-preview-${buildVersionString}" -$VersionString = "${VersionMajor}.${VersionMinor}.${VersionPatch}.${VersionQFE}" -$PreviewVersionString = "${VersionString}-preview" +$ShortBranch = "commit" +if ($BranchName -eq "master") { + $ShortBranch = "master" +} elseif ($BranchName.StartsWith("release")) { + $ShortBranch = $BranchName.replace("release/","") +} # unless it is a build branch, subdivide the output directory by month if ($BranchPath.StartsWith("build")) { $YearAndMonth = "" } else { - $YearAndMonth = (Get-Date $BuildPushDate -Format yyMM) + "\" + $YearAndMonth = (Get-Date $buildPushDate -Format yyMM) + "\" } $BuildIdentifier = "${buildPushIdString}_${PushDate}_${Username}_${CommitHash}" $ComputedDropPathSegment = "${BranchPath}\${YearAndMonth}${BuildIdentifier}" -$BinariesDirectory = "${Env:BUILD_SOURCESDIRECTORY}\Build\VcBuild" $ObjectDirectory = "${BinariesDirectory}\obj\${BuildPlatform}_${BuildConfiguration}" # Create a sentinel file for each build flavor to track whether the build is complete. @@ -128,9 +175,9 @@ This could mean that the build is in progress, or that it was unable to run to c The contents of this directory should not be relied on until the build completes. "@ -$DropPath = Join-Path $Env:DROP_ROOT $ComputedDropPathSegment +$DropPath = Join-Path $DropRoot $ComputedDropPathSegment New-Item -ItemType Directory -Force -Path $DropPath -New-Item -ItemType Directory -Force -Path (Join-Path $Env:BUILD_SOURCESDIRECTORY "test\logs") +New-Item -ItemType Directory -Force -Path (Join-Path $SourcesDirectory "test\logs") New-Item -ItemType Directory -Force -Path (Join-Path $BinariesDirectory "buildlogs") New-Item -ItemType Directory -Force -Path (Join-Path $BinariesDirectory "logs") @@ -138,26 +185,33 @@ $FlavorBuildIncompleteFile = Join-Path $DropPath "${BuildType}.incomplete" if (-not (Test-Path $FlavorBuildIncompleteFile)) { ($buildIncompleteFileContentsString -f "Build of ${BuildType}") ` - | Out-File $FlavorBuildIncompleteFile -Encoding Ascii + | Out-File $FlavorBuildIncompleteFile -Encoding utf8 } -$PogoConfig = $supportedPogoBuildTypes -contains "${Env:BuildPlatform}_${Env:BuildConfiguration}" +$PogoConfig = $supportedPogoBuildTypes -contains "${BuildPlatform}_${BuildConfiguration}" # Write the $envConfigScript @" set BranchName=${BranchName} +set ShortBranch=${ShortBranch} set BranchPath=${BranchPath} set YearAndMonth=${YearAndMonth} set BuildIdentifier=${BuildIdentifier} -set buildPushIdString=${buildPushIdString} +set VersionMajor=${VersionMajor} +set VersionMinor=${VersionMinor} +set VersionPatch=${VersionPatch} +set VersionSecurity=${VersionSecurity} + +set BuildPushIdString=${buildPushIdString} set VersionString=${VersionString} set PreviewVersionString=${PreviewVersionString} set PushDate=${PushDate} set CommitTime=${CommitTime} set Username=${Username} set CommitHash=${CommitHash} +set CommitShortHash=${CommitShortHash} set ComputedDropPathSegment=${ComputedDropPathSegment} set BinariesDirectory=${BinariesDirectory} @@ -174,24 +228,30 @@ set FlavorBuildIncompleteFile=${FlavorBuildIncompleteFile} set PogoConfig=${PogoConfig} "@ ` - | Out-File $envConfigScript -Encoding Ascii + | Out-File $envConfigScript -Encoding ASCII # Use the VSTS environment vars to construct a backwards-compatible VSO build environment # for the sake of reusing the pre-build and post-build scripts as they are. @" -set TF_BUILD_SOURCEGETVERSION=LG:${branch}:${CommitHash} +set TF_BUILD_SOURCEGETVERSION=LG:${branchFullName}:${CommitHash} set TF_BUILD_DROPLOCATION=${BinariesDirectory} -set TF_BUILD_SOURCESDIRECTORY=${Env:BUILD_SOURCESDIRECTORY} +set TF_BUILD_SOURCESDIRECTORY=${SourcesDirectory} set TF_BUILD_BUILDDIRECTORY=${ObjectDirectory} set TF_BUILD_BINARIESDIRECTORY=${BinariesDirectory} +REM The following variables are only used for logging build metadata. set TF_BUILD_BUILDDEFINITIONNAME=${Env:BUILD_DEFINITIONNAME} set TF_BUILD_BUILDNUMBER=${Env:BUILD_BUILDNUMBER} set TF_BUILD_BUILDURI=${Env:BUILD_BUILDURI} "@ ` - | Out-File $envConfigScript -Encoding Ascii -Append + | Out-File $envConfigScript -Encoding ASCII -Append + +# Print contents of $envConfigScript as a sanity check +Write-Output "" +Get-Content $envConfigScript | Write-Output +Write-Output "" # Export VSO variables that can be consumed by other VSO tasks where the task # definition in VSO itself needs to know the value of the variable. @@ -216,10 +276,10 @@ Write-Output "Setting VSO variable VSO_VersionString = ${VersionString}" Write-Output "##vso[task.setvariable variable=VSO_VersionString;]${VersionString}" # -# Clean up files that might have been left behind from a previous build. +# Optionally ($cleanBinDir): clean up files that might have been left behind from a previous build. # -if ((Test-Path Env:\BUILD_BINARIESDIRECTORY) -and (Test-Path "$Env:BUILD_BINARIESDIRECTORY")) +if ($BinariesDirectory -and (Test-Path "$BinariesDirectory") -and $cleanBinDir) { - Remove-Item -Verbose "${Env:BUILD_BINARIESDIRECTORY}\*" -Recurse + Remove-Item -Verbose "${BinariesDirectory}\*" -Recurse } diff --git a/Build/scripts/pogo_build.ps1 b/Build/scripts/pogo_build.ps1 deleted file mode 100644 index f02fdf8c9d0..00000000000 --- a/Build/scripts/pogo_build.ps1 +++ /dev/null @@ -1,143 +0,0 @@ -#------------------------------------------------------------------------------------------------------- -# Copyright (C) Microsoft. All rights reserved. -# Licensed under the MIT license. See LICENSE.txt file in the project root for full license information. -#------------------------------------------------------------------------------------------------------- - -# Use this script to run a POGO build for the given BuildType (arch, flavor, subtype) - -param ( - [ValidateSet("x86", "x64", "arm")] - [Parameter(Mandatory=$True)] - [string]$arch, - - # We do not use ValidateSet here because this $flavor param is used to name the BuildConfuration - # from the solution file. MsBuild will determine whether it is valid. - [Parameter(Mandatory=$True)] - [string]$flavor, - - [ValidateSet("default", "codecoverage", "pogo")] - [string]$subtype = "pogo", - - [Parameter(Mandatory=$True)] - [string]$solutionFile, - - [switch]$clean, - - # $binDir will be inferred if not provided. - [string]$binDir = "", - [string]$buildlogsSubdir = "buildlogs", - - # Assume NuGet is on the path, otherwise the caller must specify an explicit path. - [string]$nugetExe = "NuGet.exe", - - [string]$logFile = "", - - # - # POGO training parameters - # - - [string[]]$scenarios = @(), - - [Parameter(Mandatory=$True)] - [string]$binpath, - - [string]$binaryName = "ch.exe" -) - -# -# Configure logging -# - -$OuterScriptRoot = $PSScriptRoot -. $PSScriptRoot\pre_post_util.ps1 - -$buildName = ConstructBuildName -arch $arch -flavor $flavor -subtype $subtype - -if (($logFile -eq "") -and (Test-Path Env:\TF_BUILD_BINARIESDIRECTORY)) { - $logFile = "${Env:TF_BUILD_BINARIESDIRECTORY}\logs\pogo_build.${buildName}.log" -} - -if (($logFile -ne "") -and (Test-Path $logFile)) { - Remove-Item $logFile -Force -} - -# -# Only continue with this build if it is a valid pogo build configuration -# - -if ($subtype -ne "pogo") { - WriteMessage "This build's subtype is not pogo (subtype: $subtype). Skipping build." - exit 0 -} - -if ($scenarios.Length -eq 0) { - WriteMessage "No training scenarios selected. Please specify training scenarios using the -scenarios parameter." - exit 0 -} - -# -# NuGet restore -# - -ExecuteCommand "& $nugetExe restore $solutionFile -NonInteractive" - -# -# Setup -# - -$msbuildExe = Locate-MSBuild -if (-not $msbuildExe) { - WriteErrorMessage "Error: Could not find msbuild.exe -- exiting (1)..." - exit 1 -} - -$binDir = UseValueOrDefault "$binDir" "${Env:BinariesDirectory}" "${Env:BUILD_SOURCESDIRECTORY}\Build\VcBuild" -$buildlogsPath = Join-Path $binDir $buildlogsSubdir - -$defaultParams = "$solutionFile /nologo /m /nr:false /p:platform=`"${arch}`" /p:configuration=`"${flavor}`"" -$loggingParams = @( - "/fl1 `"/flp1:logfile=${buildlogsPath}\build.${buildName}.log;verbosity=normal`"", - "/fl2 `"/flp2:logfile=${buildlogsPath}\build.${buildName}.err;errorsonly`"", - "/fl3 `"/flp3:logfile=${buildlogsPath}\build.${buildName}.wrn;warningsonly`"", - "/verbosity:normal" - ) -join " " - -$targets = "" -if ($clean) { - $targets += "`"/t:Clean,Rebuild`"" -} - -$binary = Join-Path $binpath $binaryName - -# -# Build -# - -function Build($targets="", $pogoParams) { - $buildCommand = "& `"$msbuildExe`" $targets $defaultParams $loggingParams $pogoParams" - ExecuteCommand "$buildCommand" - if ($global:LastExitCode -ne 0) { - WriteErrorMessage "Failed msbuild command:`n$buildCommand`n" - WriteErrorMessage "Build failed. Exiting..." - exit 1 - } -} - -Build -pogoParams "`"/p:POGO_TYPE=PGI`"" -targets "$targets" - -$scenariosParamValue = $scenarios -join ',' -$pogoTrainingCommand = "& `"${PSScriptRoot}\pgo\pogo_training.ps1`" -arch $arch -flavor $flavor -subtype $subtype -binary $binary -scenarios $scenariosParamValue" -ExecuteCommand "$pogoTrainingCommand" - -Build -pogoParams "`"/p:POGO_TYPE=PGO`"" - -# -# Clean up -# - -if (("$binpath" -ne "") -and (Test-Path $binpath)) { - # remove *.pgc, *.pgd, and pgort* - Get-ChildItem -Recurse -Path $binpath "*" ` - | ? { $_.Name -match "(.*\.pg[cd]|pgort.*)" } ` - | % { Remove-Item -Force $_.FullName } -} diff --git a/Build/scripts/post_build.ps1 b/Build/scripts/post_build.ps1 index caec69cb86a..82652e514fe 100644 --- a/Build/scripts/post_build.ps1 +++ b/Build/scripts/post_build.ps1 @@ -37,9 +37,15 @@ param ( [string[]]$pogo = @(), [string]$pogoscript = "", - [switch]$noaction + # + # Skip flags + # + + [switch]$skipTests # or set $Env:SKIP_TESTS before invoking build ) +$skipTests = $skipTests -or (Test-Path Env:\SKIP_TESTS) + $global:exitcode = 0 if ($arch -eq "*") { @@ -105,7 +111,9 @@ if ($arch -eq "*") { } # run tests - ExecuteCommand("$bvtcmdpath -$arch$flavor") + if (-not $skipTests) { + ExecuteCommand("$bvtcmdpath -$arch$flavor") + } } # check prefast diff --git a/Build/scripts/pre_build.ps1 b/Build/scripts/pre_build.ps1 index 043ce75e3f1..c2d23d8217a 100644 --- a/Build/scripts/pre_build.ps1 +++ b/Build/scripts/pre_build.ps1 @@ -20,7 +20,7 @@ # $Env:TF_BUILD_BUILDDIRECTORY (a.k.a. $objpath) # $Env:TF_BUILD_BINARIESDIRECTORY (a.k.a. $binpath) # -# Optional information: +# Optional information (metadata only): # $Env:TF_BUILD_BUILDDEFINITIONNAME # $Env:TF_BUILD_BUILDNUMBER # $Env:TF_BUILD_BUILDURI @@ -32,7 +32,6 @@ param ( [Parameter(Mandatory=$True)] [ValidateSet("debug", "release", "test", "codecoverage")] [string]$flavor, - [ValidateSet("default", "codecoverage", "pogo")] [string]$subtype = "default", @@ -43,7 +42,6 @@ param ( [string]$corePath = "core", - [Parameter(Mandatory=$True)] [string]$oauth ) @@ -108,12 +106,12 @@ if (Test-Path Env:\TF_BUILD_SOURCEGETVERSION) $info = GetBuildInfo $oauth $commitHash - $BuildDate = ([datetime]$info.push.date).toString("yyMMdd-HHmm") + $BuildDate = ([DateTime]$info.push.date).toString("yyMMdd-HHmm") $buildPushId, $buildPushIdPart1, $buildPushIdPart2, $buildPushIdString = GetBuildPushId $info # commit message - $command = "$gitExe log -1 --name-status -p $commitHash" + $command = "$gitExe log -1 --name-only -p $commitHash" $CommitMessageLines = iex $command $CommitMessage = $CommitMessageLines -join "`r`n" @@ -157,12 +155,12 @@ $CommitMessage $changeJson | Add-Member -type NoteProperty -name PushIdPart2 -value $BuildPushIdPart2 $changeJson | Add-Member -type NoteProperty -name PushIdString -value $BuildPushIdString $changeJson | Add-Member -type NoteProperty -name Username -value $Env:Username - $changeJson | Add-Member -type NoteProperty -name CommitMessage -value $CommitMessage + $changeJson | Add-Member -type NoteProperty -name CommitMessage -value $CommitMessageLines Write-Output "-----" Write-Output $outputJsonFile $changeJson | ConvertTo-Json | Write-Output - $changeJson | ConvertTo-Json | Out-File $outputJsonFile -Encoding Ascii + $changeJson | ConvertTo-Json | Out-File $outputJsonFile -Encoding utf8 $buildInfoOutputDir = $objpath if (-not(Test-Path -Path $buildInfoOutputDir)) { diff --git a/Build/scripts/pre_post_util.ps1 b/Build/scripts/pre_post_util.ps1 index ac7881be2d1..5198f95e991 100644 --- a/Build/scripts/pre_post_util.ps1 +++ b/Build/scripts/pre_post_util.ps1 @@ -4,7 +4,6 @@ #------------------------------------------------------------------------------------------------------- . "$PSScriptRoot\util.ps1" -. "$PSScriptRoot\locate_msbuild.ps1" function WriteCommonArguments() { WriteMessage " Source Path: $srcpath" @@ -12,6 +11,17 @@ function WriteCommonArguments() { WriteMessage "Binaries Path: $binpath" } +function GetVersionField($fieldname) { + $gitExe = GetGitPath + $query = "#define ${fieldname} (\d+)" + $line = (iex "${gitExe} grep -P ""${query}"" :/") + $matches = $line | Select-String $query + if ($matches) { + return $matches[0].Matches.Groups[1].Value + } + return "" +} + function GetBuildInfo($oauth, $commitHash) { # Get the git remote path and construct the REST API URI $gitExe = GetGitPath @@ -21,7 +31,7 @@ function GetBuildInfo($oauth, $commitHash) { # Get the pushId and push date time to use that for build number and build date time $uri = ("{0}/commits/{1}?api-version=1.0" -f $remote, $commitHash) $oauthToken = Get-Content $oauth - $header = @{Authorization=("Basic {0}" -f $oauthToken) } + $header = @{ Authorization=("Basic {0}" -f $oauthToken) } $info = Invoke-RestMethod -Headers $header -Uri $uri -Method GET return $info diff --git a/Build/scripts/run_build.ps1 b/Build/scripts/run_build.ps1 index fcd4599b6c7..afa27074aeb 100644 --- a/Build/scripts/run_build.ps1 +++ b/Build/scripts/run_build.ps1 @@ -3,54 +3,75 @@ # Licensed under the MIT license. See LICENSE.txt file in the project root for full license information. #------------------------------------------------------------------------------------------------------- -# Use this script to run a build command for the given BuildType (arch, flavor, subtype) +# Use this script to run a build for the given BuildType (arch, flavor, subtype) param ( [ValidateSet("x86", "x64", "arm")] [Parameter(Mandatory=$True)] [string]$arch, - # We do not use ValidateSet here because this $flavor param is used to name the BuildConfuration # from the solution file. MsBuild will determine whether it is valid. [Parameter(Mandatory=$True)] [string]$flavor, - [ValidateSet("default", "codecoverage", "pogo")] [string]$subtype = "default", [Parameter(Mandatory=$True)] - [string]$solutionFile = "", + [string]$solutionFile, [switch]$clean, - # $binDir will be inferred if not provided. - [string]$binDir = "", + [string]$binDir = "", # will be inferred if not provided. [string]$buildlogsSubdir = "buildlogs", # assume NuGet is on the path, otherwise the caller must specify an explicit path [string]$nugetExe = "NuGet.exe", - [string]$logFile = "" + [string]$logFile = "", + + # + # Skip flags + # + + [switch]$skipPogo, # or set $Env:SKIP_POGO before invoking build + + # + # POGO training parameters + # + + [string[]]$scenarios = @(), + [string]$binpath, # will be inferred if not provided + [string]$binaryName = "ch.exe" ) +# +# Configure logging +# + $OuterScriptRoot = $PSScriptRoot . $PSScriptRoot\pre_post_util.ps1 +. $PSScriptRoot\locate_msbuild.ps1 + +$buildName = ConstructBuildName -arch $arch -flavor $flavor -subtype $subtype if (($logFile -eq "") -and (Test-Path Env:\TF_BUILD_BINARIESDIRECTORY)) { $logFile = "${Env:TF_BUILD_BINARIESDIRECTORY}\logs\run_build.${Env:BuildName}.log" - if (Test-Path -Path $logFile) { - Remove-Item $logFile -Force - } +} + +if (($logFile -ne "") -and (Test-Path $logFile)) { + Remove-Item $logFile -Force } # # NuGet restore # -ExecuteCommand "& $nugetExe restore $solutionFile -NonInteractive" +if (-not (Get-Command $nugetExe -ErrorAction SilentlyContinue)) { + ExecuteCommand "& $nugetExe restore $solutionFile -NonInteractive" +} # -# Setup and build +# Setup # $msbuildExe = Locate-MSBuild @@ -62,6 +83,12 @@ if (-not $msbuildExe) { $binDir = UseValueOrDefault "$binDir" "${Env:BinariesDirectory}" "${Env:BUILD_SOURCESDIRECTORY}\Build\VcBuild" $buildlogsPath = Join-Path $binDir $buildlogsSubdir +$skipPogo = $skipPogo -or (Test-Path Env:\SKIP_POGO) + +if (("$binpath" -ne "") -or (-not (Test-Path $binpath))) { + $binpath = Join-Path $binDir "bin\${buildName}" +} + $defaultParams = "$solutionFile /nologo /m /nr:false /p:platform=`"${arch}`" /p:configuration=`"${flavor}`"" $loggingParams = @( "/fl1 `"/flp1:logfile=${buildlogsPath}\build.${Env:BuildName}.log;verbosity=normal`"", @@ -75,11 +102,59 @@ if ($clean) { $targets += "`"/t:Clean,Rebuild`"" } -if ($subtype -eq "codecoverage") { - $subtypeParams = "/p:ENABLE_CODECOVERAGE=true" +# +# Build +# + +function Build($targets="", $extraParams) { + $buildCommand = "& `"$msbuildExe`" $targets $defaultParams $loggingParams $extraParams" + ExecuteCommand "$buildCommand" + if ($global:LastExitCode -ne 0) { + WriteErrorMessage "Failed msbuild command:`n$buildCommand`n" + WriteErrorMessage "Build failed. Exiting..." + exit 1 + } } -$buildCommand = "& `"$msbuildExe`" $targets $defaultParams $loggingParams $subtypeParams" -ExecuteCommand "$buildCommand" +if ($subtype -eq "pogo") { + if ($scenarios.Length -eq 0) { + WriteMessage "No training scenarios selected for pogo build. Please specify training scenarios using the -scenarios parameter." + exit 1 + } + + Build -extraParams "`"/p:POGO_TYPE=PGI`"" -targets "$targets" + + if (-not $skipPogo) { + $scenariosParamValue = $scenarios -join ',' + $binary = Join-Path $binpath $binaryName + if (($binary -ne "") -and (Test-Path $binary)) { + $pogoTrainingCommand = "& `"${PSScriptRoot}\pgo\pogo_training.ps1`" -arch $arch -flavor $flavor -subtype $subtype -binary $binary -scenarios $scenariosParamValue" + ExecuteCommand "$pogoTrainingCommand" + } else { + WriteMessage "Binary not found at `"$binary`". Exiting..." + exit 1 + } + + Build -extraParams "`"/p:POGO_TYPE=PGO`"" + } +} else { + $subtypeParams = "" + if ($subtype -eq "codecoverage") { + $subtypeParams = "/p:ENABLE_CODECOVERAGE=true" + } + + Build -extraParams $subTypeParams -targets $targets +} + +# +# Clean up +# + +if (("$binpath" -ne "") -and (Test-Path $binpath)) { + # remove *.pgc, *.pgd, and pgort* + Get-ChildItem -Recurse -Path $binpath "*" ` + | ? { $_.Name -match "(.*\.pg[cd]|pgort.*)" } ` + | % { Remove-Item -Force $_.FullName } +} exit $global:lastexitcode diff --git a/Build/scripts/util.ps1 b/Build/scripts/util.ps1 index 504e84b9d4a..115fe3b3e13 100644 --- a/Build/scripts/util.ps1 +++ b/Build/scripts/util.ps1 @@ -3,14 +3,13 @@ # Licensed under the MIT license. See LICENSE.txt file in the project root for full license information. #------------------------------------------------------------------------------------------------------- -function UseValueOrDefault($value, $defaultvalue, $defaultvalue2) { - if ($value -ne "") { - return $value - } elseif ($defaultvalue -ne "") { - return $defaultvalue - } else { - return $defaultvalue2 +function UseValueOrDefault() { + foreach ($value in $args) { + if ($value) { + return $value + } } + return "" } function GetGitPath() { @@ -26,6 +25,11 @@ function GetGitPath() { return $gitExe } +function GetRepoRoot() { + $gitExe = GetGitPath + return iex "$gitExe rev-parse --show-toplevel" +} + function WriteMessage($str) { Write-Output $str if ($logFile -ne "") { diff --git a/bin/CoreCommon.ver b/bin/CoreCommon.ver index cd2c54e4aef..5f5c91c0962 100644 --- a/bin/CoreCommon.ver +++ b/bin/CoreCommon.ver @@ -18,8 +18,8 @@ /// CHAKRA_VERSION_BUILD_QFE /// /// Binary version string is composed from the following values: -/// CHAKRA_CORE_MAJOR_VERSION . CHAKRA_CORE_MINOR_VERSION . VER_PRODUCTBUILD . CHAKRA_CORE_VERSION_RELEASE_QFE -/// e.g. 1.2.0.0 +/// CHAKRA_CORE_MAJOR_VERSION . CHAKRA_CORE_MINOR_VERSION . CHAKRA_CORE_PATCH_VERSION . CHAKRA_CORE_VERSION_RELEASE_QFE +/// e.g. 1.2.1.0 /// These fields must be explicitly set in the source. /// Note: VER_PRODUCTBUILD is always set to 0. @@ -30,6 +30,7 @@ #define VER_PRODUCTMAJORVERSION CHAKRA_CORE_MAJOR_VERSION #define VER_PRODUCTMINORVERSION CHAKRA_CORE_MINOR_VERSION +#define VER_PRODUCTPATCHVERSION CHAKRA_CORE_PATCH_VERSION // File Flags #if DBG @@ -126,7 +127,7 @@ #endif // defined(CHAKRA_VERSION_BUILD_NUMBER) && defined(CHAKRA_VERSION_BUILD_QFE) -#define VER_PRODUCTVERSION VER_PRODUCTMAJORVERSION,VER_PRODUCTMINORVERSION,VER_PRODUCTBUILD,VER_PRODUCTBUILD_QFE +#define VER_PRODUCTVERSION VER_PRODUCTMAJORVERSION,VER_PRODUCTMINORVERSION,VER_PRODUCTPATCHVERSION,VER_PRODUCTBUILD_QFE #define VER_PRODUCTVERSION_MAJORMINOR2(x,y) #x "." #y #define VER_PRODUCTVERSION_MAJORMINOR1(x,y) VER_PRODUCTVERSION_MAJORMINOR2(x, y) @@ -136,7 +137,7 @@ #define VER_PRODUCTVERSION_STR3(x) VER_PRODUCTVERSION_STR4(x) #define VER_PRODUCTVERSION_STR2(x,y) VER_PRODUCTVERSION_STRING "." #x "." #y #define VER_PRODUCTVERSION_STR1(x,y) VER_PRODUCTVERSION_STR2(x, y) -#define VER_PRODUCTVERSION_STR VER_PRODUCTVERSION_STR1(VER_PRODUCTBUILD, VER_PRODUCTBUILD_QFE) +#define VER_PRODUCTVERSION_STR VER_PRODUCTVERSION_STR1(VER_PRODUCTPATCHVERSION, VER_PRODUCTBUILD_QFE) #ifdef VER_FILEDESCRIPTION_SUFFIX1_STR #ifdef VER_FILEDESCRIPTION_SUFFIX2_STR diff --git a/lib/Common/CommonDefines.h b/lib/Common/CommonDefines.h index c507200ec39..2fe86f6e50a 100644 --- a/lib/Common/CommonDefines.h +++ b/lib/Common/CommonDefines.h @@ -12,9 +12,10 @@ //---------------------------------------------------------------------------------------------------- #define CHAKRA_CORE_MAJOR_VERSION 1 #define CHAKRA_CORE_MINOR_VERSION 2 +#define CHAKRA_CORE_PATCH_VERSION 1 #define CHAKRA_CORE_VERSION_RELEASE 1 #define CHAKRA_CORE_VERSION_PRERELEASE 0 -#define CHAKRA_CORE_VERSION_RELEASE_QFE 0 +#define CHAKRA_CORE_VERSION_RELEASE_QFE 0 // Redundant with PATCH_VERSION. Keep this value set to 0. #define CHAKRA_VERSION_RELEASE 1