Skip to content

Sparse checkout docs repository in release step #1585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
3 commits merged into from
May 18, 2021
Merged

Sparse checkout docs repository in release step #1585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6efb380
Sparse checkout docs repository in release step
benbp May 4, 2021
fb706b4
Detect default branch for custom sparse checkout repository
benbp May 5, 2021
fb05204
Use sparse checkout paths parameter in docs metadata release
benbp May 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 77 additions & 25 deletions eng/common/pipelines/templates/steps/docs-metadata-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,65 @@
# intended to be used as part of a release process
parameters:
ArtifactLocation: 'not-specified'
PackageRepository: 'not-specified'
ReleaseSha: 'not-specified'
RepoId: $(Build.Repository.Name)
WorkingDirectory: ''
ScriptDirectory: eng/common/scripts
TargetDocRepoName: ''
TargetDocRepoOwner: ''
PRBranchName: 'master-rdme'
PRLabels: 'auto-merge'
ArtifactName: ''
Language: ''
DocRepoDestinationPath: '' #usually docs-ref-services/
CIConfigs: '[]'
GHReviewersVariable: ''
GHTeamReviewersVariable: '' # externally set, as eng-common does not have the identity-resolver. Run as pre-step
OnboardingBranch: ''
CloseAfterOpenForTesting: false
SkipPackageJson: false
- name: ArtifactLocation
type: string
default: 'not-specified'
- name: PackageRepository
type: string
default: 'not-specified'
- name: ReleaseSha
type: string
default: 'not-specified'
- name: RepoId
type: string
default: $(Build.Repository.Name)
- name: WorkingDirectory
type: string
default: ''
- name: ScriptDirectory
type: string
default: eng/common/scripts
- name: TargetDocRepoName
type: string
default: ''
- name: TargetDocRepoOwner
type: string
default: ''
- name: PRBranchName
type: string
default: 'master-rdme'
- name: PRLabels
type: string
default: 'auto-merge'
- name: ArtifactName
type: string
default: ''
- name: Language
type: string
default: ''
- name: DocRepoDestinationPath
type: string
default: '' #usually docs-ref-services/
- name: CIConfigs
type: string
default: '[]'
- name: GHReviewersVariable
type: string
default: ''
- name: GHTeamReviewersVariable
type: string
default: '' # externally set, as eng-common does not have the identity-resolver. Run as pre-step
- name: OnboardingBranch
type: string
default: ''
- name: CloseAfterOpenForTesting
type: boolean
default: false
- name: SkipPackageJson
type: object
default: false
- name: SparseCheckoutPaths
type: object
default: null

steps:
- pwsh: |
Expand All @@ -30,14 +71,25 @@ steps:
Write-Host "This script is not executing on Windows, skipping registry modification."
}
displayName: Enable Long Paths if Necessary

- pwsh: |
git clone https://github.com/${{ parameters.TargetDocRepoOwner }}/${{ parameters.TargetDocRepoName }} ${{ parameters.WorkingDirectory }}/repo
displayName: Clone Documentation Repository
ignoreLASTEXITCODE: false

- ${{ if not(parameters.SparseCheckoutPaths) }}:
Copy link
Member

@weshaggard weshaggard May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely like this approach better as it should allow both scenarios to work. It might actually be interesting make this an option in the sparse-checkout template itself if we find more patterns like this.

Copy link
Member Author

@benbp benbp May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you mean perhaps replacing all our checkouts with calls to some checkout template, and we can have a parameter like sparse or full? I initially tried doing a sparse checkout with /* but that actually doubled the checkout time, hence why I went this route instead.

Copy link
Member

@weshaggard weshaggard May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes having this split of full vs sparse clone might be a future improvement to consider.

- pwsh: |
git clone https://github.com/${{ parameters.TargetDocRepoOwner }}/${{ parameters.TargetDocRepoName }} ${{ parameters.WorkingDirectory }}/repo
displayName: Clone Documentation Repository
ignoreLASTEXITCODE: false

- ${{ if parameters.SparseCheckoutPaths }}:
- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
parameters:
SkipDefaultCheckout: true
Repositories:
- Name: ${{ parameters.TargetDocRepoOwner }}/${{ parameters.TargetDocRepoName }}
WorkingDirectory: ${{ parameters.WorkingDirectory }}/repo
Paths: ${{ parameters.SparseCheckoutPaths }}

- template: /eng/common/pipelines/templates/steps/set-default-branch.yml
parameters:
WorkingDirectory: ${{ parameters.WorkingDirectory }}/repo
WorkingDirectory: ${{ parameters.WorkingDirectory }}/repo
- task: PowerShell@2
displayName: 'Apply Documentation Updates From Artifact'
inputs:
Expand Down
5 changes: 3 additions & 2 deletions eng/common/pipelines/templates/steps/sparse-checkout.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ steps:
displayName: Init sparse checkout ${{ repo.Name }}
workingDirectory: ${{ coalesce(repo.WorkingDirectory, format('{0}/{1}', '$(System.DefaultWorkingDirectory)', repo.Name)) }}

- pwsh: git checkout ${{ repo.Commitish }}
displayName: Sparse checkout at ${{ repo.Commitish }}
- pwsh: |
git checkout ${{ repo.Commitish }} # this will use the default branch if repo.Commitish is empty
Copy link
Member

@weshaggard weshaggard May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it pick-up the default branch? Are you sure git checkout without parameters checks out the default branch? As best I can tell it will be a noop.

Copy link
Member Author

@benbp benbp May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my testing (local and in-pipeline), it does. According to the docs:

You could omit , in which case the command degenerates to "check out the current branch", which is a glorified no-op with rather expensive side-effects to show only the tracking information, if exists, for the current branch.

Since we're in a state where sparse checkout paths are set but no blobs are downloaded, this seems to have the effect of grabbing all the relevant blobs.

I could switch it back to using the regex that we're using elsewhere, but as this seems to work I thought it was much cleaner.

Copy link
Member

@weshaggard weshaggard May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this works than great. If not I was think we could use the default branch template and the $(DefaultBranch) variable in this place.

Copy link
Member Author

@benbp benbp May 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually only discovered this because I copied that DefaultBranch code over and forgot to change the remote from Source to origin, so it failed to set the variable and just ran git checkout, which worked!

displayName: Sparse checkout at ${{ coalesce(repo.Commitish, 'default branch') }}
workingDirectory: ${{ coalesce(repo.WorkingDirectory, format('{0}/{1}', '$(System.DefaultWorkingDirectory)', repo.Name)) }}
6 changes: 3 additions & 3 deletions eng/pipelines/templates/steps/sync-repo-branch.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ parameters:
# Example:
# Azure/azure-rest-api-specs:
# Branch: master
# TargetRepos:
# TargetRepos:
# Azure/azure-rest-api-specs-pr:
# azure-sdk/azure-rest-api-specs:
# Branch: dev
# Rebase: true
# Azure/azure-sdk-for-go:
# Branch: master
# TargetRepos:
# TargetRepos:
# Azure/azure-sdk-for-go-pr:
- name: Repos
type: object
Expand Down Expand Up @@ -40,7 +40,7 @@ steps:
Write-Host "No source branch. Fetch default branch $defaultBranch."
$SourceBranch = $defaultBranch
}

git fetch --no-tags Source $SourceBranch
if ($LASTEXITCODE -ne 0) {
Write-Host "#`#vso[task.logissue type=error]Failed to fetch ${SourceRepo}:${SourceBranch}"
Expand Down