-
Notifications
You must be signed in to change notification settings - Fork 325
Finetuning Extension code #6459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 8 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
41030f1
Moving code Azure fork (#6429)
achauhan-scc bb7f9a1
Refactoring jobs list command to align with the LLD | Add retry, envi…
saanikaguptamicrosoft 4e01756
Refactoring jobs show command to align with the LLD (#6434)
saanikaguptamicrosoft 5db7d48
Create command implementation for FineTuning CLI (#6437)
ZubaeyrMSFT 62a90c5
Build and Release Pipelines for Finetuning CLI (#6449)
ZubaeyrMSFT 53d2524
adding init command (#6442)
achauhan-scc 57a9e97
network isloation changes. (#6458)
ZubaeyrMSFT b562fd0
Cleaning code to raise a PR to main (#6457)
achauhan-scc eacfee5
updating version to 0.0.7-preview (#6462)
ZubaeyrMSFT 481edde
Rename changelog.md to CHANGELOG.md
ZubaeyrMSFT 5155d1c
adding copyright header text
ZubaeyrMSFT a738c5f
applying pr review suggestion: remove providers section from extensio…
ZubaeyrMSFT 16eab72
updating version.txt to 0.0.8-preview
ZubaeyrMSFT 4a60e3d
simplying utils for file check
ZubaeyrMSFT 3bcb9a6
reanming _uploadFile to uploadFile
ZubaeyrMSFT 49ce76e
removing duplicate import
ZubaeyrMSFT f54c0fe
removing unnecessary todo
ZubaeyrMSFT d8664b4
Adding cspell dict
ZubaeyrMSFT d11851d
updating cspell.yaml
ZubaeyrMSFT 9c39857
updating cspell.yaml
ZubaeyrMSFT 59530fa
Updating finetune word in global cspell.yaml
ZubaeyrMSFT 439c5b8
PR Review Changes: Updating ParseGithubUrl func (#6476)
ZubaeyrMSFT 6d1becb
Updating extension name in read me file. (#6483)
achauhan-scc 6d288aa
resolving review comments
ZubaeyrMSFT 408d746
Merge branch 'Model-Customization-Dev' of https://github.com/Azure/az…
ZubaeyrMSFT 9a7c06d
resolving review comments
ZubaeyrMSFT 92d7b8c
using arm resourceId parser
ZubaeyrMSFT f73698b
Merge branch 'main' of https://github.com/Azure/azure-dev into Model-…
ZubaeyrMSFT da93fca
Finetuning CLU - Using OSS package go-retry instead of custom logic (…
ZubaeyrMSFT e8962f5
Allowing user to override the fine tuning route (#6484)
achauhan-scc 6cd886a
spell check fix (#6486)
achauhan-scc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # `azd` Demo Extension | ||
|
|
||
| An AZD Demo extension | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Ensure script fails on any error | ||
| $ErrorActionPreference = 'Stop' | ||
|
|
||
| # Get the directory of the script | ||
| $EXTENSION_DIR = Split-Path -Parent $MyInvocation.MyCommand.Path | ||
|
|
||
| # Change to the script directory | ||
| Set-Location -Path $EXTENSION_DIR | ||
|
|
||
| # Create a safe version of EXTENSION_ID replacing dots with dashes | ||
| $EXTENSION_ID_SAFE = $env:EXTENSION_ID -replace '\.', '-' | ||
|
|
||
| # Define output directory | ||
| $OUTPUT_DIR = if ($env:OUTPUT_DIR) { $env:OUTPUT_DIR } else { Join-Path $EXTENSION_DIR "bin" } | ||
|
|
||
| # Create output directory if it doesn't exist | ||
| if (-not (Test-Path -Path $OUTPUT_DIR)) { | ||
| New-Item -ItemType Directory -Path $OUTPUT_DIR | Out-Null | ||
| } | ||
|
|
||
| # Get Git commit hash and build date | ||
| $COMMIT = git rev-parse HEAD | ||
| if ($LASTEXITCODE -ne 0) { | ||
| Write-Host "Error: Failed to get git commit hash" | ||
| exit 1 | ||
| } | ||
| $BUILD_DATE = (Get-Date -Format "yyyy-MM-ddTHH:mm:ssZ") | ||
|
|
||
| # List of OS and architecture combinations | ||
| if ($env:EXTENSION_PLATFORM) { | ||
| $PLATFORMS = @($env:EXTENSION_PLATFORM) | ||
| } | ||
| else { | ||
| $PLATFORMS = @( | ||
| "windows/amd64", | ||
| "windows/arm64", | ||
| "darwin/amd64", | ||
| "darwin/arm64", | ||
| "linux/amd64", | ||
| "linux/arm64" | ||
| ) | ||
| } | ||
|
|
||
| $APP_PATH = "$env:EXTENSION_ID/internal/cmd" | ||
|
|
||
| # Loop through platforms and build | ||
| foreach ($PLATFORM in $PLATFORMS) { | ||
| $OS, $ARCH = $PLATFORM -split '/' | ||
|
|
||
| $OUTPUT_NAME = Join-Path $OUTPUT_DIR "$EXTENSION_ID_SAFE-$OS-$ARCH" | ||
|
|
||
| if ($OS -eq "windows") { | ||
| $OUTPUT_NAME += ".exe" | ||
| } | ||
|
|
||
| Write-Host "Building for $OS/$ARCH..." | ||
|
|
||
| # Delete the output file if it already exists | ||
| if (Test-Path -Path $OUTPUT_NAME) { | ||
| Remove-Item -Path $OUTPUT_NAME -Force | ||
| } | ||
|
|
||
| # Set environment variables for Go build | ||
| $env:GOOS = $OS | ||
| $env:GOARCH = $ARCH | ||
|
|
||
| go build ` | ||
| -ldflags="-X '$APP_PATH.Version=$env:EXTENSION_VERSION' -X '$APP_PATH.Commit=$COMMIT' -X '$APP_PATH.BuildDate=$BUILD_DATE'" ` | ||
| -o $OUTPUT_NAME | ||
|
|
||
| if ($LASTEXITCODE -ne 0) { | ||
| Write-Host "An error occurred while building for $OS/$ARCH" | ||
| exit 1 | ||
| } | ||
| } | ||
|
|
||
| Write-Host "Build completed successfully!" | ||
| Write-Host "Binaries are located in the $OUTPUT_DIR directory." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Get the directory of the script | ||
| EXTENSION_DIR="$(cd "$(dirname "$0")" && pwd)" | ||
|
|
||
| # Change to the script directory | ||
| cd "$EXTENSION_DIR" || exit | ||
|
|
||
| # Create a safe version of EXTENSION_ID replacing dots with dashes | ||
| EXTENSION_ID_SAFE="${EXTENSION_ID//./-}" | ||
|
|
||
| # Define output directory | ||
| OUTPUT_DIR="${OUTPUT_DIR:-$EXTENSION_DIR/bin}" | ||
|
|
||
| # Create output and target directories if they don't exist | ||
| mkdir -p "$OUTPUT_DIR" | ||
|
|
||
| # Get Git commit hash and build date | ||
| COMMIT=$(git rev-parse HEAD) | ||
| BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) | ||
|
|
||
| # List of OS and architecture combinations | ||
| if [ -n "$EXTENSION_PLATFORM" ]; then | ||
| PLATFORMS=("$EXTENSION_PLATFORM") | ||
| else | ||
| PLATFORMS=( | ||
| "windows/amd64" | ||
| "windows/arm64" | ||
| "darwin/amd64" | ||
| "darwin/arm64" | ||
| "linux/amd64" | ||
| "linux/arm64" | ||
| ) | ||
| fi | ||
|
|
||
| APP_PATH="$EXTENSION_ID/internal/cmd" | ||
|
|
||
| # Loop through platforms and build | ||
| for PLATFORM in "${PLATFORMS[@]}"; do | ||
| OS=$(echo "$PLATFORM" | cut -d'/' -f1) | ||
| ARCH=$(echo "$PLATFORM" | cut -d'/' -f2) | ||
|
|
||
| OUTPUT_NAME="$OUTPUT_DIR/$EXTENSION_ID_SAFE-$OS-$ARCH" | ||
|
|
||
| if [ "$OS" = "windows" ]; then | ||
| OUTPUT_NAME+='.exe' | ||
| fi | ||
|
|
||
| echo "Building for $OS/$ARCH..." | ||
|
|
||
| # Delete the output file if it already exists | ||
| [ -f "$OUTPUT_NAME" ] && rm -f "$OUTPUT_NAME" | ||
|
|
||
| # Set environment variables for Go build | ||
| GOOS=$OS GOARCH=$ARCH go build \ | ||
| -ldflags="-X '$APP_PATH.Version=$EXTENSION_VERSION' -X '$APP_PATH.Commit=$COMMIT' -X '$APP_PATH.BuildDate=$BUILD_DATE'" \ | ||
| -o "$OUTPUT_NAME" | ||
|
|
||
| if [ $? -ne 0 ]; then | ||
| echo "An error occurred while building for $OS/$ARCH" | ||
| exit 1 | ||
| fi | ||
| done | ||
|
|
||
| echo "Build completed successfully!" | ||
| echo "Binaries are located in the $OUTPUT_DIR directory." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # Release History | ||
|
|
||
| ## 0.0.1 - Initial Version |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| param( | ||
| [string] $Version = (Get-Content "$PSScriptRoot/version.txt"), | ||
| [string] $SourceVersion = (git rev-parse HEAD), | ||
| [switch] $CodeCoverageEnabled, | ||
| [switch] $BuildRecordMode, | ||
| [string] $MSYS2Shell, # path to msys2_shell.cmd | ||
| [string] $OutputFileName | ||
| ) | ||
|
|
||
| # Remove any previously built binaries | ||
| go clean | ||
|
|
||
| if ($LASTEXITCODE) { | ||
| Write-Host "Error running go clean" | ||
| exit $LASTEXITCODE | ||
| } | ||
|
|
||
| # Run `go help build` to obtain detailed information about `go build` flags. | ||
| $buildFlags = @( | ||
| # remove all file system paths from the resulting executable. | ||
| # Instead of absolute file system paths, the recorded file names | ||
| # will begin either a module path@version (when using modules), | ||
| # or a plain import path (when using the standard library, or GOPATH). | ||
| "-trimpath", | ||
|
|
||
| # Use buildmode=pie (Position Independent Executable) for enhanced security across platforms | ||
| # against memory corruption exploits across all major platforms. | ||
| # | ||
| # On Windows, the -buildmode=pie flag enables Address Space Layout | ||
| # Randomization (ASLR) and automatically sets DYNAMICBASE and HIGH-ENTROPY-VA flags in the PE header. | ||
| "-buildmode=pie" | ||
| ) | ||
|
|
||
| if ($CodeCoverageEnabled) { | ||
| $buildFlags += "-cover" | ||
| } | ||
|
|
||
| # Build constraint tags | ||
| # cfi: Enable Control Flow Integrity (CFI), | ||
| # cfg: Enable Control Flow Guard (CFG), | ||
| # osusergo: Optimize for OS user accounts | ||
| $tagsFlag = "-tags=cfi,cfg,osusergo" | ||
|
|
||
| # ld linker flags | ||
| # -s: Omit symbol table and debug information | ||
| # -w: Omit DWARF symbol table | ||
| # -X: Set variable at link time. Used to set the version in source. | ||
|
|
||
| $ldFlag = "-ldflags=-s -w -X 'azure.ai.finetune/internal/cmd.Version=$Version' -X 'azure.ai.finetune/internal/cmd.Commit=$SourceVersion' -X 'azure.ai.finetune/internal/cmd.BuildDate=$(Get-Date -Format o)' " | ||
|
|
||
| if ($IsWindows) { | ||
| $msg = "Building for Windows" | ||
| Write-Host $msg | ||
| } | ||
| elseif ($IsLinux) { | ||
| Write-Host "Building for linux" | ||
| } | ||
| elseif ($IsMacOS) { | ||
| Write-Host "Building for macOS" | ||
| } | ||
|
|
||
| # Add output file flag based on specified output file name | ||
| $outputFlag = "-o=$OutputFileName" | ||
|
|
||
| # collect flags | ||
| $buildFlags += @( | ||
| $tagsFlag, | ||
| $ldFlag, | ||
| $outputFlag | ||
| ) | ||
|
|
||
| function PrintFlags() { | ||
| param( | ||
| [string] $flags | ||
| ) | ||
|
|
||
| # Attempt to format flags so that they are easily copy-pastable to be ran inside pwsh | ||
| $i = 0 | ||
| foreach ($buildFlag in $buildFlags) { | ||
| # If the flag has a value, wrap it in quotes. This is not required when invoking directly below, | ||
| # but when repasted into a shell for execution, the quotes can help escape special characters such as ','. | ||
| $argWithValue = $buildFlag.Split('=', 2) | ||
| if ($argWithValue.Length -eq 2 -and !$argWithValue[1].StartsWith("`"")) { | ||
| $buildFlag = "$($argWithValue[0])=`"$($argWithValue[1])`"" | ||
| } | ||
|
|
||
| # Write each flag on a newline with '`' acting as the multiline separator | ||
| if ($i -eq $buildFlags.Length - 1) { | ||
| Write-Host " $buildFlag" | ||
| } | ||
| else { | ||
| Write-Host " $buildFlag ``" | ||
| } | ||
| $i++ | ||
| } | ||
| } | ||
|
|
||
| $oldGOEXPERIMENT = $env:GOEXPERIMENT | ||
| # Enable the loopvar experiment, which makes the loop variaible for go loops like `range` behave as most folks would expect. | ||
| # the go team is exploring making this default in the future, and we'd like to opt into the behavior now. | ||
| $env:GOEXPERIMENT = "loopvar" | ||
|
|
||
| try { | ||
| Write-Host "Running: go build ``" | ||
| PrintFlags -flags $buildFlags | ||
| go build @buildFlags | ||
| if ($LASTEXITCODE) { | ||
| Write-Host "Error running go build" | ||
| exit $LASTEXITCODE | ||
| } | ||
|
|
||
| if ($BuildRecordMode) { | ||
| # Modify build tags to include record | ||
| $recordTagPatched = $false | ||
| for ($i = 0; $i -lt $buildFlags.Length; $i++) { | ||
| if ($buildFlags[$i].StartsWith("-tags=")) { | ||
| $buildFlags[$i] += ",record" | ||
| $recordTagPatched = $true | ||
| } | ||
| } | ||
| if (-not $recordTagPatched) { | ||
| $buildFlags += "-tags=record" | ||
| } | ||
| # Add output file flag for record mode | ||
| $recordOutput = "-o=$OutputFileName-record" | ||
| if ($IsWindows) { $recordOutput += ".exe" } | ||
| $buildFlags += $recordOutput | ||
|
|
||
| Write-Host "Running: go build (record) ``" | ||
| PrintFlags -flags $buildFlags | ||
| go build @buildFlags | ||
| if ($LASTEXITCODE) { | ||
| Write-Host "Error running go build (record)" | ||
| exit $LASTEXITCODE | ||
| } | ||
| } | ||
|
|
||
| Write-Host "go build succeeded" | ||
| } | ||
| finally { | ||
| $env:GOEXPERIMENT = $oldGOEXPERIMENT | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.