Move symbol publishing into a dedicated stage and refactor build artifacts

.github/instructions/onebranch-pipeline-design.instructions.md

@@ -36,6 +36,8 @@ Defined in `stages/build-stages.yml`. Four build stages plus validation, ordered
 - **`build_addons`** (Stage 4) — AKV Provider; `dependsOn: build_dependent`; downloads SqlClient + Abstractions + Logging artifacts
 - **`mds_package_validation`** — Validates signed SqlClient package; `dependsOn: build_dependent`; runs in parallel with Stage 4
 
+Each build job copies PDB files into `$(JOB_OUTPUT)/symbols/` so they are included in the auto-published pipeline artifact alongside the NuGet packages in `$(JOB_OUTPUT)/packages/`.
+
 Stage conditional rules:
 - Wrap stages/jobs in `${{ if }}` compile-time conditionals based on build parameters
 - `buildSqlClient` controls Stages 2, 3, validation, and Logging (when AKV is disabled)
@@ -45,17 +47,30 @@ Stage conditional rules:
 
 ## Job Templates
 
-- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing
-- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing
+- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing → Copy PDBs to artifact
+- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing → Copy PDBs to artifact
 - **`validate-signed-package-job.yml`** — Validates signed MDS package (signature, strong names, folder structure, target frameworks)
 - **`publish-nuget-package-job.yml`** — Reusable release job using OneBranch `templateContext.type: releaseJob` with `inputs` for artifact download; pushes via `NuGetCommand@2`
+- **`publish-symbols-job.yml`** — Reusable symbols job: downloads a build artifact, locates PDBs under `symbols/`, and invokes `publish-symbols-step.yml`
 
 When adding a new csproj-based package:
 - Use `build-signed-csproj-package-job.yml` with appropriate `packageName`, `packageFullName`, `versionProperties`, and `downloadArtifacts`
 - Add build and pack targets to `build.proj`
 - Add version variables to `variables/common-variables.yml`
 - Add artifact name variable to `variables/onebranch-variables.yml`
 
+## Symbols Publishing Stage
+
+- Defined in `stages/publish-symbols-stage.yml`; produces stage `publish_symbols`
+- Entire stage excluded at compile time when `publishSymbols` is false
+- `dependsOn` is conditional based on which `build*` parameters are set, mirroring the build stage dependency graph
+- One job per package (`publish-symbols-job.yml`), each downloading its build artifact and publishing PDBs from `symbols/`
+- Each package's PDBs are published separately with unique artifact names and version information
+- Build jobs copy PDBs into `$(JOB_OUTPUT)/symbols/` so they are included in the auto-published artifact
+- The `publish-symbols-step.yml` accepts a `symbolsFolder` parameter to point at the downloaded PDB location
+- The publish step calls an extracted `Publish-Symbols.ps1` script with structured error handling and diagnostic logging
+- Symbols publishing credentials come from the `Symbols Publishing` variable group
+
 ## Release Stage
 
 - Defined in `stages/release-stages.yml`; produces stage `release_production` (official) or `release_test` (non-official) via `stageNameSuffix` parameter
@@ -98,8 +113,7 @@ When `isPreview` is true, pipeline resolves `effective*Version` variables to pre
 - When adding a new package, add GA version, preview version, and assembly file version entries
 
 Variable groups:
-- `Release Variables` — release configuration (in `common-variables.yml`)
-- `Symbols publishing` — symbol publishing credentials (in `common-variables.yml`)
+- `Symbols Publishing` — symbol publishing credentials (in `onebranch-variables.yml`)
 - `ESRP Federated Creds (AME)` — ESRP signing credentials (in `common-variables.yml`)
 
 ## Code Signing (ESRP)
@@ -115,15 +129,19 @@ Variable groups:
 
 - TSA: enabled only in official pipeline; disabled in non-official to avoid spurious alerts
 - ApiScan: enabled in both; currently `break: false` pending package registration
-- Each build job sets `ob_sdl_apiscan_*` variables pointing to `$(Build.SourcesDirectory)/apiScan/<PackageName>/`
+- Each build job sets `ob_sdl_apiscan_softwareFolder` to `$(JOB_OUTPUT)/assemblies` and `ob_sdl_apiscan_symbolsFolder` to `$(JOB_OUTPUT)/symbols`
 - CodeQL, SBOM, Policheck (`break: true`): enabled in both pipelines
 - asyncSdl `enabled: false` in both; individual sub-tools (CredScan, BinSkim, Armory, Roslyn) configured underneath
 - Policheck exclusions: `$(REPO_ROOT)\.config\PolicheckExclusions.xml`
 - CredScan suppressions: `$(REPO_ROOT)/.config/CredScanSuppressions.json`
 
 ## Artifact Conventions
 
-- `ob_outputDirectory` set to `$(PACK_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory
+- `ob_outputDirectory` set to `$(JOB_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory
+- Each published artifact uses subdirectories to separate file types:
+  - `assemblies/` — DLL assemblies for APIScan (preserving TFM folder structure)
+  - `packages/` — NuGet packages (`.nupkg`, `.snupkg`)
+  - `symbols/` — PDB symbol files (preserving TFM folder structure, shared by APIScan and symbol publishing)
 - Artifact names follow `drop_<stageName>_<jobName>` — defined in `variables/onebranch-variables.yml`
 - Downstream jobs download artifacts via `DownloadPipelineArtifact@2` into `$(Build.SourcesDirectory)/packages`
 - Downloaded packages serve as a local NuGet source for `dotnet restore`

eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml

@@ -36,26 +36,6 @@ parameters:
   - name: signingEsrpConnectedServiceName
     type: string
 
-  # Symbols Publishing Parameters ------------------------------------------
-
-  - name: symbolsAzureSubscription
-    type: string
-    default: 'Symbols publishing Workload Identity federation service-ADO.Net'
-
-  - name: symbolsPublishProjectName
-    type: string
-    default: 'Microsoft.Data.SqlClient.SNI'
-
-  - name: symbolsPublishServer
-    type: string
-
-  - name: symbolsPublishTokenUri
-    type: string
-
-  - name: symbolsUploadAccount
-    type: string
-    default: 'SqlClientDrivers'
-
   # OTHERS +=====================================
 
   # Short package name used in the job name, display strings, filesystem paths, and as a suffix for
@@ -74,11 +54,6 @@ parameters:
   - name: packageFullName
     type: string
 
-  # The version of the package.  This is used for symbol publishing.  It is not used for the DLL or
-  # NuGet package versions since those are supplied via the versionProperties parameter.
-  - name: packageVersion
-    type: string
-
   # The MSBuild build target in build.proj (e.g. BuildLogging).  If not specified, defaults to
   # Build<packageName>.
   - name: buildTarget
@@ -106,10 +81,6 @@ parameters:
   - name: assemblyFileVersion
     type: string
 
-  # True to publish symbols to private and public servers.
-  - name: publishSymbols
-    type: boolean
-
   # Optional list of pipeline artifacts to download before building.  Each entry is an object
   # with 'artifactName' (the pipeline artifact name) and 'displayName' (used in the task label).
   # This replaces hard-coded packageName conditionals so callers declare their own dependencies.
@@ -125,12 +96,12 @@ jobs:
 
     variables:
       # Inform OneBranch that files put in this directory should be uploaded as artifacts.
-      ob_outputDirectory: $(PACK_OUTPUT)
+      ob_outputDirectory: $(JOB_OUTPUT)
 
       # APIScan configuration for this Extension package
       ob_sdl_apiscan_enabled: true
-      ob_sdl_apiscan_softwareFolder: $(REPO_ROOT)/apiScan/${{ parameters.packageName }}/dlls
-      ob_sdl_apiscan_symbolsFolder: $(REPO_ROOT)/apiScan/${{ parameters.packageName }}/pdbs
+      ob_sdl_apiscan_softwareFolder: $(JOB_OUTPUT)/assemblies
+      ob_sdl_apiscan_symbolsFolder: $(JOB_OUTPUT)/symbols
       ob_sdl_apiscan_softwarename: ${{ parameters.packageFullName }}
       ob_sdl_apiscan_versionNumber: ${{ parameters.assemblyFileVersion }}
 
@@ -149,7 +120,7 @@ jobs:
             displayName: Download ${{ artifact.displayName }}
             inputs:
               artifactName: ${{ artifact.artifactName }}
-              targetPath: $(REPO_ROOT)/packages
+              targetPath: $(JOB_INPUT)
 
       # Install the .NET SDK.
       - template: /eng/pipelines/steps/install-dotnet.yml@self
@@ -181,24 +152,27 @@ jobs:
               esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
               pattern: '${{ parameters.packageFullName }}.dll'
 
-      # Copy signed/unsigned DLLs and PDBs to APIScan folders.
+      # Copy DLLs to the assemblies/ subdirectory for APIScan.
       - task: CopyFiles@2
         displayName: Copy DLLs for APIScan
         inputs:
           SourceFolder: $(BUILD_OUTPUT)/Package/bin
           Contents: "**/${{ parameters.packageFullName }}.dll"
-          TargetFolder: ${{ variables.ob_sdl_apiscan_softwareFolder }}
+          TargetFolder: $(JOB_OUTPUT)/assemblies
           # We must preserve the folder structure since our C# projects may produce multiple
           # identically named DLLs for different target frameworks (e.g. netstandard2.0, net5.0,
           # etc.), and we need to keep those separate for APIScan to work correctly.
           flattenFolders: false
 
+      # Copy PDBs into the output directory so they are included in the published pipeline
+      # artifact. The symbols publishing stage will download this artifact and publish PDBs
+      # for this package using the files under symbols/.
       - task: CopyFiles@2
-        displayName: Copy PDBs for APIScan
+        displayName: Copy PDBs for APIScan and Symbols Publishing
         inputs:
           SourceFolder: $(BUILD_OUTPUT)/Package/bin
-          Contents: "**/${{ parameters.packageFullName }}.pdb"
-          TargetFolder: ${{ variables.ob_sdl_apiscan_symbolsFolder }}
+          Contents: '**/${{ parameters.packageFullName }}.pdb'
+          TargetFolder: $(JOB_OUTPUT)/symbols
           flattenFolders: false
 
       # Pack the signed DLLs into NuGet package (NoBuild=true).
@@ -217,22 +191,5 @@ jobs:
               authSignCertName: '${{ parameters.signingAuthSignCertName }}'
               esrpClientId: '${{ parameters.signingEsrpClientId }}'
               esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
-              searchPath: $(PACK_OUTPUT)
+              searchPath: $(JOB_OUTPUT)/packages
               searchPattern: '${{ parameters.packageFullName}}.*nupkg'
-
-      # Publish symbols to servers
-      # @TODO: Get these parameters from variables/libraries
-      - ${{ if eq(parameters.publishSymbols, true) }}:
-          - template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self
-            parameters:
-              artifactName: '${{ parameters.packageFullName }}_symbols_$(System.TeamProject)_$(Build.Repository.Name)_$(Build.SourceBranchName)_${{ parameters.packageVersion }}_$(System.TimelineId)'
-              azureSubscription: '${{ parameters.symbolsAzureSubscription }}'
-              packageName: '${{ parameters.packageFullName }}'
-              publishProjectName: '${{ parameters.symbolsPublishProjectName }}'
-              publishServer: '${{ parameters.symbolsPublishServer }}'
-              publishToInternal: 'true'
-              publishToPublic: 'true'
-              publishTokenUri: '${{ parameters.symbolsPublishTokenUri }}'
-              searchPattern: '**/${{ parameters.packageFullName }}*.pdb'
-              uploadAccount: '${{ parameters.symbolsUploadAccount }}'
-              version: '${{ parameters.packageVersion }}'

eng/pipelines/onebranch/jobs/build-signed-sqlclient-package-job.yml

@@ -7,20 +7,11 @@
 # Job that performs a build of Microsoft.Data.SqlClient using the build2.proj file.
 
 parameters:
-  - name: apiScanDllPath
-    type: string
-
-  - name: apiScanPdbPath
-    type: string
-
   # Whether this build is for an "official" OneBranch pipeline. This is used to enable ESRP signing
   # on the artifacts of this job.
   - name: isOfficial
     type: boolean
 
-  - name: publishSymbols
-    type: boolean
-
   - name: signingAppRegistrationClientId
     type: string
 
@@ -39,21 +30,6 @@ parameters:
   - name: signingEsrpConnectedServiceName
     type: string
 
-  - name: symbolsAzureSubscription
-    type: string
-
-  - name: symbolsPublishProjectName
-    type: string
-
-  - name: symbolsPublishServer
-    type: string
-
-  - name: symbolsPublishTokenUri
-    type: string
-
-  - name: symbolsUploadAccount
-    type: string
-
   # Package Parameters
   - name: abstractionsArtifactName
     type: string
@@ -80,9 +56,9 @@ jobs:
       type: windows
 
     variables:
-      ob_outputDirectory: '$(PACK_OUTPUT)'
-      ob_sdl_apiscan_softwareFolder: ${{ parameters.apiScanDllPath }}
-      ob_sdl_apiscan_symbolsFolder: ${{ parameters.apiScanPdbPath }}
+      ob_outputDirectory: '$(JOB_OUTPUT)'
+      ob_sdl_apiscan_softwareFolder: $(JOB_OUTPUT)/assemblies
+      ob_sdl_apiscan_symbolsFolder: $(JOB_OUTPUT)/symbols
       ob_sdl_apiscan_softwarename: 'Microsoft.Data.SqlClient'
       ob_sdl_apiscan_versionNumber: ${{ parameters.sqlClientAssemblyFileVersion }}
 
@@ -102,13 +78,13 @@ jobs:
         displayName: Download Microsoft.Data.SqlClient.Extensions.Abstractions Artifact
         inputs:
           artifactName: '${{ parameters.abstractionsArtifactName }}'
-          targetPath: '$(REPO_ROOT)/packages'
+          targetPath: '$(JOB_INPUT)'
 
       - task: DownloadPipelineArtifact@2
         displayName: Download Microsoft.Data.SqlClient.Extensions.Logging Artifact
         inputs:
           artifactName: '${{ parameters.loggingArtifactName }}'
-          targetPath: '$(REPO_ROOT)/packages'
+          targetPath: '$(JOB_INPUT)'
 
       # Install the .NET SDK
       - template: /eng/pipelines/steps/install-dotnet.yml@self
@@ -127,13 +103,6 @@ jobs:
           loggingPackageVersion: '${{ parameters.loggingPackageVersion }}'
           sqlClientPackageVersion: '${{ parameters.sqlClientPackageVersion }}'
 
-      # Copy the built DLLs and PDBs to the APIScan output folder for APIScanning post-build
-      - template: /eng/pipelines/onebranch/steps/copy-apiscan-files-sqlclient-step.yml@self
-        parameters:
-          dllPath: '${{ parameters.apiScanDllPath }}'
-          pdbPath: '${{ parameters.apiScanPdbPath }}'
-          referenceType: 'Package'
-
       # Sign the DLLs
       - ${{ if parameters.isOfficial }}:
           - template: /eng/pipelines/onebranch/steps/esrp-dll-signing-step.yml@self
@@ -146,6 +115,26 @@ jobs:
               esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
               pattern: 'Microsoft.Data.SqlClient*.dll'
 
+      # Copy DLLs to the assemblies/ subdirectory for APIScan.
+      - task: CopyFiles@2
+        displayName: Copy DLLs for APIScan
+        inputs:
+          SourceFolder: '$(BUILD_OUTPUT)/Microsoft.Data.SqlClient/Package-Release'
+          Contents: '**/Microsoft.Data.SqlClient.dll'
+          TargetFolder: '$(JOB_OUTPUT)/assemblies'
+          flattenFolders: false
+
+      # Copy PDBs into the output directory so they are included in the published pipeline
+      # artifact. The symbols publishing stage will download this artifact and publish PDBs
+      # for this package using the files under symbols/.
+      - task: CopyFiles@2
+        displayName: Copy PDBs for APIScan and Symbols Publishing
+        inputs:
+          SourceFolder: '$(BUILD_OUTPUT)/Microsoft.Data.SqlClient/Package-Release'
+          Contents: '**/Microsoft.Data.SqlClient*.pdb'
+          TargetFolder: '$(JOB_OUTPUT)/symbols'
+          flattenFolders: false
+
       # Package the build output into a NuGet package
       - template: /eng/pipelines/onebranch/steps/pack-sqlclient-step.yml
         parameters:
@@ -163,20 +152,5 @@ jobs:
               authSignCertName: '${{ parameters.signingAuthSignCertName }}'
               esrpClientId: '${{ parameters.signingEsrpClientId }}'
               esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
-              searchPath: '$(PACK_OUTPUT)'
+              searchPath: '$(JOB_OUTPUT)/packages'
               searchPattern: 'Microsoft.Data.SqlClient.*nupkg'
-
-      - ${{ if parameters.publishSymbols }}:
-          - template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self
-            parameters:
-              artifactName: 'Microsoft.Data.SqlClient_symbols_$(System.TeamProject)_$(Build.Repository.Name)_$(Build.SourceBranchName)_${{ parameters.sqlClientPackageVersion }}_$(System.TimelineId)'
-              azureSubscription: '${{ parameters.symbolsAzureSubscription }}'
-              packageName: 'Microsoft.Data.SqlClient'
-              publishProjectName: '${{ parameters.symbolsPublishProjectName }}'
-              publishServer: '${{ parameters.symbolsPublishServer }}'
-              publishToInternal: true
-              publishToPublic: true
-              publishTokenUri: '${{ parameters.symbolsPublishTokenUri }}'
-              searchPattern: '**/Microsoft.Data.SqlClient*.pdb' # @TODO: This seems very heavy
-              uploadAccount: '${{ parameters.symbolsUploadAccount }}'
-              version: '${{ parameters.sqlClientPackageVersion }}'

eng/pipelines/onebranch/jobs/publish-nuget-package-job.yml

@@ -65,13 +65,13 @@ jobs:
 
     variables:
       - name: ob_outputDirectory
-        value: $(PACK_OUTPUT)
+        value: $(JOB_OUTPUT)
 
       - name: artifactPath
         value: $(Pipeline.Workspace)/${{ parameters.artifactName }}
 
       - name: packageToPush
-        value: $(artifactPath)/${{ coalesce(parameters.packagePath, format('{0}.*.nupkg', parameters.packageName)) }}
+        value: $(artifactPath)/${{ coalesce(parameters.packagePath, format('packages/{0}.*.nupkg', parameters.packageName)) }}
 
     # Template context inputs are used to pass parameters to the deployment job since it doesn't
     # automatically download pipeline artifacts.