Skip to content
This repository was archived by the owner on Aug 26, 2025. It is now read-only.

Commit 31d2f2a

Browse files
danieljurekwanlwanlAlanceremsyycraych1
authored
Migrate typespec validation (Azure#31033)
* test breaking change mlc in js (#3451) * ` * ` * Revert "`" This reverts commit c26fc6a. * Update specificationRepositoryConfiguration.json (#3457) * Update specificationRepositoryConfiguration.json (#3459) * Update specificationRepositoryConfiguration.json (#3462) Revert python repo branch change as it was deleted. * Update specificationRepositoryConfiguration.json (#3463) * Migrate typespec-validation.yml to GH Actions * path, fetch depth * Introduce error * Test a feature that AI says will collapse logs (doubtful it'll work) * Revert "Test a feature that AI says will collapse logs (doubtful it'll work)" This reverts commit 04aed83. * Log $LASTEXITCODE * Invalid syntax * Use error logging, remove continue-on-error because that is different behavior in GitHub Actions compared to ignoreLASTEXITCODE * Errors also get annotations in GitHub Actions * String replacement * Revert Logging-Functions.ps1 * Test composite action * File location * Invocation * Add typespec-validation-all.yml * Migrate typespec-validation-all.yml * Revert main.tsp invalid spec * fetch-depth: 2 * Long paths * checkout@v4 * Remove pipelines * *.yml -> *.yaml * Do not change specificationRepositoryConfiguration.json * Job names (for checks) * Add Skip/Take logic and matrix * Syntax * Rename * Remove "shell" * Try defaults.run.shell (unlikely to work) * bash * Name for actions/setup-node@v4 does not appear in the logs. Remove. * Use sharding semantics * Shard * Review feedback * Review feedback * +1 * Revert "Shard"... Display is confusing because it starts at 0 and doesn't reach total-shards. Math is not allowed in expressions. This reverts commit 035bec73044d60c07307fd6efc19b7295258a507. * Shard only if TotalShards > 0 * Remove description (schema validaiton does not pass, let's see what GH Actions says) * The `description` property is not absolutely required by GH Actions at this time but adding back in because schema requires it. * Add array functions and tests * Move Copy-ApiVersion.Tests.ps1 out of folder (adjust paths so they are accurate) * Add trailing newline * Add trailing newline * Update .github/workflows/typespec-validation.yaml Co-authored-by: Mike Harder <[email protected]> * Review feedback * Update eng/scripts/TypeSpec-Validation.ps1 Co-authored-by: Mike Harder <[email protected]> --------- Co-authored-by: Wanpeng Li <[email protected]> Co-authored-by: Peng Jiahui <[email protected]> Co-authored-by: Yuchao Yan <[email protected]> Co-authored-by: Ray Chen <[email protected]> Co-authored-by: Mike Harder <[email protected]>
1 parent e6094e6 commit 31d2f2a

File tree

9 files changed

+237
-71
lines changed

9 files changed

+237
-71
lines changed

‎.github/actions/setup-node-npm-ci/action.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: Setup Node 20 and run `npm ci`
2+
description: Uses specified Node version and runs npm commands to set up the environment for REST API CI
3+
4+
inputs:
5+
node-version:
6+
description: 'Node version to use'
7+
default: 20.x
8+
9+
runs:
10+
using: "composite"
11+
12+
steps:
13+
- uses: actions/setup-node@v4
14+
with:
15+
node-version: ${{ inputs.node-version }}
16+
17+
- run: npm ci
18+
shell: bash
19+
20+
- run: npm ls -a
21+
shell: bash
22+
continue-on-error: true

‎.github/workflows/typespec-validation-all.yaml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: TypeSpec Validation All
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- RPSaaSMaster
8+
- typespec-next
9+
10+
pull_request:
11+
branches:
12+
- main
13+
- RPSaaSMaster
14+
- typespec-next
15+
paths:
16+
- .gitattributes
17+
- .prettierrc.json
18+
- package-lock.json
19+
- package.json
20+
- tsconfig.json
21+
- eng/**
22+
- specification/suppressions.yaml
23+
- specification/common-types/**
24+
25+
# Workflow and workflow dependencies
26+
- .github/workflows/typespec-validation-all.yaml
27+
- .github/actions/setup-node-npm-ci/**
28+
29+
schedule:
30+
# Run 4x/day
31+
- cron: '0 0,6,12,18 * * * '
32+
33+
jobs:
34+
typespec-validation-all:
35+
name: TypeSpec Validation All
36+
strategy:
37+
matrix:
38+
os: [ubuntu-latest, windows-latest]
39+
# shards must start at 0 and increment by 1
40+
shard: [0, 1, 2]
41+
# total-shards must be an accurate count of the number of shards
42+
total-shards: [3]
43+
44+
runs-on: ${{ matrix.os }}
45+
46+
steps:
47+
- name: Enable git long paths
48+
if: runner.os == 'Windows'
49+
run: git config --global core.longpaths true
50+
51+
- uses: actions/checkout@v4
52+
with:
53+
fetch-depth: 2
54+
55+
- name: Setup Node 20 and run `npm ci`
56+
uses: ./.github/actions/setup-node-npm-ci
57+
58+
- name: Validate All Specs
59+
run: |
60+
./eng/scripts/TypeSpec-Validation.ps1 `
61+
-Shard ${{ matrix.shard }} `
62+
-TotalShards ${{ matrix.total-shards }} `
63+
-CheckAll `
64+
-GitClean `
65+
-Verbose
66+
67+
# Effectively the same as ignoreLASTEXITCODE: true in Azure DevOps
68+
exit 0
69+
shell: pwsh

‎.github/workflows/typespec-validation.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: Typespec Validation
2+
3+
on: pull_request
4+
5+
jobs:
6+
typespec-validation:
7+
name: Typespec Validation
8+
runs-on: ubuntu-latest
9+
10+
steps:
11+
- uses: actions/checkout@v4
12+
with:
13+
fetch-depth: 2
14+
15+
- name: Setup Node 20 and run `npm ci`
16+
uses: ./.github/actions/setup-node-npm-ci
17+
18+
- name: Validate Impacted Specs
19+
run: |
20+
./eng/scripts/TypeSpec-Validation.ps1 -GitClean -Verbose
21+
22+
# Effectively the same as ignoreLASTEXITCODE: true in Azure DevOps
23+
exit 0
24+
shell: pwsh

‎eng/pipelines/typespec-validation-all.yml‎

Lines changed: 0 additions & 47 deletions
This file was deleted.

‎eng/pipelines/typespec-validation.yml‎

Lines changed: 0 additions & 16 deletions
This file was deleted.

‎eng/scripts/Array-Functions.ps1‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
function ShardArray($array, $shard, $totalShards) {
2+
if ($totalShards -lt 2) {
3+
return $array
4+
}
5+
6+
if ($shard -ge $totalShards) {
7+
throw "Shard index ($shard) must be less than total shards ($totalShards)"
8+
}
9+
10+
if ($totalShards -gt $array.Length) {
11+
throw "Cannot shard array into more pieces than there are elements"
12+
}
13+
14+
$shardSize = [math]::Ceiling($array.Length / $totalShards)
15+
$start = $shard * $shardSize
16+
$end = [math]::Min($start + $shardSize, $array.Length)
17+
return $array[$start..($end - 1)]
18+
}

‎eng/scripts/Tests/Array-Functions.Tests.ps1‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
BeforeAll {
2+
. "$PSScriptRoot\..\Array-Functions.ps1"
3+
}
4+
5+
# 0..78 makes an array of 79 (prime number) items
6+
7+
Describe "ShardArray" {
8+
Context "Input Validation" {
9+
It "throws when sharding into more pieces than there are elements" {
10+
$array = 0..78
11+
$shard = 0
12+
$totalShards = 150
13+
14+
{ ShardArray $array $shard $totalShards } | Should -Throw
15+
}
16+
17+
It "returns the full array when given <totalShards> shards" -ForEach @(
18+
@{ totalShards = 1 },
19+
@{ totalShards = 0 },
20+
@{ totalShards = -1 },
21+
@{ totalShards = -2 }
22+
) {
23+
$array = 0..78
24+
$shard = 0
25+
26+
$result = ShardArray $array $shard $totalShards
27+
28+
$result | Should -Be $array
29+
}
30+
31+
It "throws when the shard index is greater than the total shards" {
32+
$array = 0..78
33+
$shard = 10
34+
$totalShards = 3
35+
36+
{ ShardArray $array $shard $totalShards } | Should -Throw
37+
}
38+
39+
It "does not throw when totalShards equals the array length" {
40+
$array = 0..78
41+
$shard = 0
42+
$totalShards = 79
43+
44+
{ ShardArray $array $shard $totalShards } | Should -Not -Throw
45+
}
46+
47+
}
48+
49+
Context "Shards arrays" -ForEach @(
50+
@{ array = 0..78; totalShards = 1 },
51+
@{ array = 0..78; totalShards = 2 },
52+
@{ array = 0..11; totalShards = 3 },
53+
@{ array = 0..10; totalShards = 4 },
54+
@{ array = 0..78; totalShards = 79 }
55+
) {
56+
It "returns all of the values in order when sharding (Total Shards: <totalShards>) " {
57+
$shards = New-Object object[] $totalShards
58+
for ($i = 0; $i -lt $totalShards; $i++) {
59+
# Assigning directly avoids flattening the array.
60+
$shards[$i] = ShardArray $array $i $totalShards
61+
}
62+
63+
# Flatten the array for comparison
64+
$actual = $shards | ForEach-Object { $_ }
65+
66+
$actual | Should -Be $array
67+
}
68+
69+
It "returns arrays of expected valid size (Total Shards: <totalShards>)" {
70+
$maxLength = [math]::Ceiling($array.Length / $totalShards)
71+
for ($i = 0; $i -lt $totalShards; $i++) {
72+
$shard = ShardArray $array $i $totalShards
73+
$shard.Length | Should -BeLessOrEqual $maxLength
74+
}
75+
}
76+
}
77+
}

‎eng/scripts/Tests/Copy-ApiVersion/Copy-ApiVersion.Tests.ps1‎ renamed to ‎eng/scripts/Tests/Copy-ApiVersion.Tests.ps1‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
Import-Module Pester
22

33
BeforeAll {
4-
. "$PSScriptRoot\..\..\Copy-ApiVersion-Functions.ps1"
4+
. "$PSScriptRoot\..\Copy-ApiVersion-Functions.ps1"
55
}
66

77
Describe "Copy-ApiVersion regex tests" {
@@ -11,25 +11,25 @@ Describe "Copy-ApiVersion regex tests" {
1111
version = "2024-01-01-preview"
1212
provider = "Microsoft.AgFoodPlatform"
1313
versionStatus = "preview"
14-
specsDir = '..\..\..\..\..\specification\agrifood\resource-manager\Microsoft.AgFoodPlatform\preview\2023-06-01-preview'
14+
specsDir = '..\..\..\..\specification\agrifood\resource-manager\Microsoft.AgFoodPlatform\preview\2023-06-01-preview'
1515
},
1616
@{
1717
version = "2024-01-01"
1818
provider = "Microsoft.Compute\ComputeRP"
1919
versionStatus = "stable"
20-
specsDir = '..\..\..\..\..\specification\compute\resource-manager\Microsoft.Compute\ComputeRP\stable\2023-09-01'
20+
specsDir = '..\..\..\..\specification\compute\resource-manager\Microsoft.Compute\ComputeRP\stable\2023-09-01'
2121
},
2222
@{
2323
version = "7.6-preview.1"
2424
provider = "Microsoft.KeyVault"
2525
versionStatus = "preview"
26-
specsDir = '..\..\..\..\..\specification\keyvault\data-plane\Microsoft.KeyVault\preview\7.6-preview.1'
26+
specsDir = '..\..\..\..\specification\keyvault\data-plane\Microsoft.KeyVault\preview\7.6-preview.1'
2727
},
2828
@{
2929
version = "7.5"
3030
provider = "Microsoft.Compute\ComputeRP"
3131
versionStatus = "stable"
32-
specsDir = '..\..\..\..\..\specification\keyvault\data-plane\Microsoft.KeyVault\stable\7.5'
32+
specsDir = '..\..\..\..\specification\keyvault\data-plane\Microsoft.KeyVault\stable\7.5'
3333
}
3434
) {
3535
param($version, $provider, $versionStatus, $specsDir)
@@ -48,12 +48,12 @@ Describe "Copy-ApiVersion regex tests" {
4848
# TODO: This is fragile. The tests stop working when a service team updates their readme.md. We should instead take fixed copies or something.
4949
It "Default version gets updated" -TestCases @(
5050
@{
51-
inputReadme = '..\..\..\..\..\specification\compute\resource-manager\readme.md'
51+
inputReadme = '..\..\..\..\specification\compute\resource-manager\readme.md'
5252
apiVersion = "2024-01-01"
5353
versionStatus = "stable"
5454
},
5555
@{
56-
inputReadme = '..\..\..\..\..\specification\keyvault\data-plane\readme.md'
56+
inputReadme = '..\..\..\..\specification\keyvault\data-plane\readme.md'
5757
apiVersion = "7.6-preview.1"
5858
versionStatus = "preview"
5959
}

‎eng/scripts/TypeSpec-Validation.ps1‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,35 @@
11
[CmdletBinding()]
22
param (
33
[switch]$CheckAll = $false,
4+
[int]$Shard = 0,
5+
[int]$TotalShards = 1,
46
[switch]$GitClean = $false,
57
[switch]$DryRun = $false,
68
[string]$BaseCommitish = "HEAD^",
79
[string]$TargetCommitish = "HEAD"
810
)
911

12+
if ($TotalShards -gt 0 -and $Shard -ge $TotalShards) {
13+
throw "Shard ($Shard) must be less than TotalShards ($TotalShards)"
14+
}
15+
1016
. $PSScriptRoot/Logging-Functions.ps1
1117
. $PSScriptRoot/Suppressions-Functions.ps1
18+
. $PSScriptRoot/Array-Functions.ps1
19+
20+
$typespecFolders, $checkedAll = &"$PSScriptRoot/Get-TypeSpec-Folders.ps1" `
21+
-BaseCommitish:$BaseCommitish `
22+
-TargetCommitish:$TargetCommitish `
23+
-CheckAll:$CheckAll
1224

13-
$typespecFolders, $checkedAll = &"$PSScriptRoot/Get-TypeSpec-Folders.ps1" -BaseCommitish:$BaseCommitish -TargetCommitish:$TargetCommitish -CheckAll:$CheckAll
25+
if ($TotalShards -gt 1 -and $TotalShards -le $typespecFolders.Count) {
26+
$typespecFolders = shardArray $typespecFolders $Shard $TotalShards
27+
}
28+
29+
Write-Host "Checking $($typespecFolders.Count) TypeSpec folders:"
30+
foreach ($typespecFolder in $typespecFolders) {
31+
Write-Host " $typespecFolder"
32+
}
1433

1534
$typespecFoldersWithFailures = @()
1635
if ($typespecFolders) {

0 commit comments

Comments
 (0)