Skip to content

Use sparse checkout instead of remote tracking #25847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sima-zhu merged 3 commits into from
Dec 8, 2021
Merged

Use sparse checkout instead of remote tracking #25847

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
37423d5
Add code owner file in sparse checkout
sima-zhu Nov 23, 2021
e583d7f
Merge branch 'main' of https://github.com/Azure/azure-sdk-for-java
sima-zhu Dec 7, 2021
bbf2f7d
Git sparse checkout on azure-sdk repo
sima-zhu Dec 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 14 additions & 28 deletions eng/pipelines/docindex.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ jobs:
vmImage: ubuntu-20.04
variables:
DocRepoLocation: $(Pipeline.Workspace)/docs
DailyDocRepoLocation: $(Pipeline.Workspace)/daily
DocRepoOwner: Azure
DocRepoName: azure-docs-sdk-java
DailyDocsRepoOwner: azure-sdk
steps:
# Sync docs repo onboarding files/folders
- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
Expand Down Expand Up @@ -50,29 +50,16 @@ jobs:
- template: /eng/common/pipelines/templates/steps/set-daily-docs-branch-name.yml
parameters:
DailyBranchVariableName: DailyDocsBranchName
# Create a branch to track docs fork repo onboarding files/folders
- pwsh: |
$ErrorActionPreference = "Continue"
# git remote on fork repo
$GitUrl = "https://$(azuresdk-github-pat)@github.com/$(DailyDocsRepoOwner)/$(DocRepoName).git"
$remoteName = "$(DailyDocsRepoOwner)"
Write-Host "git remote add $remoteName $GitUrl"
git remote add $remoteName $GitUrl
if ($LASTEXITCODE -ne 0)
{
Write-Error "Unable to add remote LASTEXITCODE=$($LASTEXITCODE), see command output above."
exit $LASTEXITCODE
}

# git fetch on fork repo
Write-Host "git fetch $remoteName $(DailyDocsBranchName)"
git fetch $remoteName "$(DailyDocsBranchName)"
Copy link
Copy Markdown
Member

@weshaggard weshaggard Dec 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benbp it would be good to understand why adding another remote and then fetching a branch to checkout doesn't work as expected when using sparse checkout. https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1233498&view=logs&j=dc056dfc-c0cf-5958-c8c4-8da4f91a3739&t=d08ed1f2-0ec2-5498-48a9-682643d126ca

Copy link
Copy Markdown
Contributor Author

@sima-zhu sima-zhu Dec 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on git man documentation, git fetch do downloading the deltas besides of sparse checkout files.
There is no sparse checkout like functionality for git fetch
https://git-scm.com/docs/git-remote#Documentation/git-remote.txt-emaddem

Copy link
Copy Markdown
Contributor Author

@sima-zhu sima-zhu Dec 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://stackoverflow.com/questions/4114887/is-it-possible-to-do-a-sparse-checkout-without-checking-out-the-whole-repository

Copy link
Copy Markdown
Member

@weshaggard weshaggard Dec 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that is the same situation that we have here. We already do the no-checkout on clone but we should be able to add a remote and then fetch a new set of commits down and checkout just those changes.


# create a branch tracking fork repo daily branch
Write-Host "git checkout -b $(DailyDocsBranchName) -t $remoteName/$(DailyDocsBranchName)"
git checkout -b $(DailyDocsBranchName) -t "$remoteName/$(DailyDocsBranchName)"
displayName: Checkout fork repo daily branch
workingDirectory: $(DocRepoLocation)
- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
parameters:
SkipDefaultCheckout: true
Paths:
- package.json
- metadata/
Repositories:
- Name: azure-sdk/$(DocRepoName)
WorkingDirectory: $(DailyDocRepoLocation)

# Docs daily updates is supposed to download packages from public feed repository, so we have to specify additional repositories in a POM or the profile.
# Here is maven documentation: https://maven.apache.org/guides/mini/guide-multiple-repositories.html
- powershell: |
Expand All @@ -87,12 +74,11 @@ jobs:
Copy-Item "./eng/repo-docs/docms/daily.update.setting.xml" -Destination "~/.m2/settings.xml"
displayName: 'Configure mvn'
workingDirectory: $(Build.SourcesDirectory)

- task: Powershell@2
inputs:
pwsh: true
filePath: eng/common/scripts/Update-DocsMsPackages.ps1
arguments: -DocRepoLocation "$(DocRepoLocation)"
arguments: -DocRepoLocation $(DailyDocRepoLocation)
displayName: Update Docs Onboarding for Daily branch
- template: /eng/common/pipelines/templates/steps/git-push-changes.yml
parameters:
Expand All @@ -101,7 +87,7 @@ jobs:
CommitMsg: "Update targeting packages based on release metadata. (Daily docs)"
TargetRepoName: $(DocRepoName)
TargetRepoOwner: $(DocRepoOwner)
WorkingDirectory: $(DocRepoLocation)
WorkingDirectory: $(DailyDocRepoLocation)
ScriptDirectory: $(Build.SourcesDirectory)/eng/common/scripts

- task: PowerShell@2
Expand All @@ -114,4 +100,4 @@ jobs:
-Project "Content%20CI"
-DefinitionId 3188
-Base64EncodedAuthToken "$(azuresdk-apidrop-devops-queue-build-pat)"
-BuildParametersJson (@{ params = (Get-Content ./eng/dailydocsconfig.json -Raw) -replace '%%DailyDocsBranchName%%', "$(DailyDocsBranchName)" } | ConvertTo-Json)
-BuildParametersJson (@{ params = (Get-Content ./eng/dailydocsconfig.json -Raw) -replace '%%DailyDocsBranchName%%', "$(DailyDocsBranchName)" } | ConvertTo-Json)