diff --git a/tools/sdk-generation-pipeline/Dockerfile b/tools/sdk-generation-pipeline/Dockerfile index 320fa42ac81..37461ab3fbc 100644 --- a/tools/sdk-generation-pipeline/Dockerfile +++ b/tools/sdk-generation-pipeline/Dockerfile @@ -46,7 +46,6 @@ RUN apt-get install -y dotnet-sdk-6.0 # install git RUN add-apt-repository ppa:git-core/ppa -y && apt update && apt upgrade -y && apt install git -y RUN git config --global credential.helper store && git config --global core.fileMode false - # install depended packages RUN pip3 install --upgrade wheel PyYAML requests RUN npm install -g typescript diff --git a/tools/sdk-generation-pipeline/documents/docker/README.md b/tools/sdk-generation-pipeline/documents/docker/README.md index 40d6b664131..c309c9411fa 100644 --- a/tools/sdk-generation-pipeline/documents/docker/README.md +++ b/tools/sdk-generation-pipeline/documents/docker/README.md @@ -21,7 +21,7 @@ The docker image will be used in different scenarios: Command ```shell -docker run -it --privileged -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md docker.image:latest --readme={relative_readme} --sdk={sdk_to_generate} +docker run -it --privileged -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md sdkgeneration.azurecr.io/sdk-generation:latest --readme={relative_readme} --sdk={sdk_to_generate} ``` Parameter description: @@ -36,13 +36,13 @@ Parameter description: Example Command: ```shell -docker run -it --privileged -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir docker.image:latest --readme="specification/agrifood/resource-manager/readme.md" --sdk=js,java +docker run -it --privileged -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir sdkgeneration.azurecr.io/sdk-generation:latest --readme="specification/agrifood/resource-manager/readme.md" --sdk=js,java ``` -After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can open your browser and request `http://127.0.0.1:8080/?folder=/work-dir` for further grow up development. +After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can [open your local vscode and connect to docker container](./vscode-connect-docker-container.md) for further grow up development. If you want to re-generate codes after grow up development or changing swagger, please run command in docker container: ```shell -rerun-tasks -readme={relative_readme} --sdk={sdk_to_generate} +rerun-tasks --readme={relative_readme} --sdk={sdk_to_generate} ``` rerun-tasks is a script, which invokes task engine to re-run tasks. @@ -53,41 +53,36 @@ There are two scenarios here: 1. Service team has generated codes locally by using docker image and has exited the docker container. But they want to do grow up development now. 2. Service team has generated codes by using sdk generation pipeline, and sdk generation pipeline creates a work branch. Service team hope to do grow up based on the work branch. -Compared to scenario 1, scenario 2 needs user to clone and checkout the work branch by themselves. It’s very simple with git: +Run docker commands to do grow up development: ```shell -cd {local_work_folder} -git clone -b {work-branch} {repo-url} +docker run -it --privileged -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md sdkgeneration.azurecr.io/sdk-generation:latest --readme={relative_readme} --spec-link={spec-link} --sdk-work-branch={sdk-work-branch-link} ``` Parameter description: -| Parameter | Description | Example | -|-----------------------|-----------------------------------------------------------------------------------------|-----------------------------------------------| -| { local_work_folder } | Required. It's used to point to the work folder, which stores all sdk repositories. | /home/test/work-dir | -| {work_branch } | Required. It's used to point the work branch name generated by SDK generation pipeline. | sdkAuto/workbranch | -| {repo_url} | Required. It's used to point the repository url that work branch is in. | https://github.com/Azure/azure-sdk-for-js.git | +| Parameter | Description | Example | +|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| +| { local_spec_repo_path } | Optional. If you want to change the swagger and re-generate codes, you need to mount the swagger repo. If you only want to do grow up development, no need to mount it. If you input a the link by parameter {spec-link}, docker container helps clone it. | /home/test/azure-rest-api-specs | +| { local_work_folder } | Required. It's used to point to the work folder, which stores all sdk repositories. | /home/test/work-dir | +| { local_autorest_config } | Optional. When you generate data-plane sdk, and there is no autorest configuration in sdk repository or you want to change the autorest configuration, you can set new autorest config in a file and mount it to the docker container. About the content of file, please refer to [document](https://github.com/Azure/azure-rest-api-specs/blob/dpg-doc/documentation/onboard-dpg-in-sdkautomation/add-autorest-configuration-in-spec-comment.md) | /home/test/autorest.md ([Example file](./autorest-config-file-sample.md)) | +| { relative_readme } | Optional. It's used to specify the readme.md file and docker image uses it to start mock server. it's the relative path from {path_to_local_spec_repo}. If not specified, mock server will not start. | specification/agrifood/resource-manager/readme.md | +| { sdk-work-branch-link } | **Only Required in Scenario 2**. It's used to specify the link to sdk work branch generated by sdk generation pipeline, and docker container will use it to clone sdk repo and checkout the work branch. **In scenario 2, please make sure there is no corresponding sdk repo under {local_work_folder}.** | specification/agrifood/resource-manager/readme.md | +| { spec-link } | **Only Required in Scenario 2 and no local repo mounted**. It's used to specify the link to spec repo, which can be PR link, repo Link or branch link. Then docker container will use it to clone spec repo and checkout the properly branch. **In scenario 2, please make sure there is no corresponding spec repo with path '/spec-repo' in docker container.** | specification/agrifood/resource-manager/readme.md | -Then run docker commands to do grow up development: +Example Command: +Scenario 1: ```shell -docker run -it --privileged -v {local_spec_repo_path}:/spec-repo -v {local_work_folder}:/work-dir -v {local_autorest_config}:/autorest.md docker.image:latest --readme={relative_readme} +docker run -it --privileged -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir sdkgeneration.azurecr.io/sdk-generation:latest --readme="specification/agrifood/resource-manager/readme.md" ``` -Parameter description: -| Parameter | Description | Example | -|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------| -| { local_spec_repo_path } | Optional. If you want to change the swagger and re-generate codes, you need to mount the swagger repo. If you only want to do grow up development, no need to mount it. | /home/test/azure-rest-api-specs | -| { local_work_folder } | Required. It's used to point to the work folder, which stores all sdk repositories. | /home/test/work-dir | -| { local_autorest_config } | Optional. When you generate data-plane sdk, and there is no autorest configuration in sdk repository or you want to change the autorest configuration, you can set new autorest config in a file and mount it to the docker container. About the content of file, please refer to [document](https://github.com/Azure/azure-rest-api-specs/blob/dpg-doc/documentation/onboard-dpg-in-sdkautomation/add-autorest-configuration-in-spec-comment.md) | /home/test/autorest.md ([Example file](./autorest-config-file-sample.md)) | -| { relative_readme } | Optional. It's used to specify the readme.md file and docker image uses it to start mock server. it's the relative path from {path_to_local_spec_repo}. If not specified, mock server will not start. | specification/agrifood/resource-manager/readme.md | - -Example Command: +Scenario 2: ```shell -docker run -it --privileged -v /home/test/azure-rest-api-specs:/spec-repo -v /home/test/work-dir:/work-dir docker.image:latest +docker run -it --privileged -v /home/test/work-dir:/work-dir sdkgeneration.azurecr.io/sdk-generation:latest --readme="specification/agrifood/resource-manager/readme.md" --spec-link="https://github.com/Azure/azure-rest-api-specs/pull/19850" --sdk-work-branch="https://github.com/Azure/azure-sdk-for-js/tree/agrifood/dev/branch" ``` -After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can open your browser and request `http://127.0.0.1:8080/?folder=/work-dir` for further grow up development. +After running command, docker container generates SDKs. When SDKs are generated, the docker container doesn't exit, and you can [open your local vscode and connect to docker container](./vscode-connect-docker-container.md) for further grow up development. If you want to re-generate codes after grow up development or changing swagger, please run command in docker container: ```shell -rerun-tasks -readme={relative_readme} --sdk={sdk_to_generate} +rerun-tasks --readme={relative_readme} --sdk={sdk_to_generate} ``` rerun-tasks is a script, which invokes task engine to re-run tasks. @@ -101,7 +96,7 @@ Before running docker command, pipeline must prepare the spec repo and sdk repo. Command: ```shell -docker run --privileged -v {spec_repo_path}:/spec-repo -v {sdk_repo_path}:/sdk-repo -v {local_autorest_config}:/autorest.md -v {output_folder_path}:/tmp/output docker.image:latest --readme={relative_readme} +docker run --privileged -v {spec_repo_path}:/spec-repo -v {sdk_repo_path}:/sdk-repo -v {local_autorest_config}:/autorest.md -v {output_folder_path}:/tmp/output sdkgeneration.azurecr.io/sdk-generation:latest --readme={relative_readme} ``` Parameter description: @@ -115,7 +110,7 @@ Parameter description: Example Command: ```shell -docker run --privileged -v /home/vsts/work/azure-rest-api-specs:/spec-repo -v /home/vsts/work/azure-sdk-for-js:/sdk-repo -v /home/vsts/work/output:/tmp/output docker.image:latest --readme=specification/agrifood/resource-manager/readme.md +docker run --privileged -v /home/vsts/work/azure-rest-api-specs:/spec-repo -v /home/vsts/work/azure-sdk-for-js:/sdk-repo -v /home/vsts/work/output:/tmp/output sdkgeneration.azurecr.io/sdk-generation:latest --readme=specification/agrifood/resource-manager/readme.md ``` After running the command in pipeline, docker will execute tasks automatically. Also, there will be output files generated, which will be used by pipeline's other job, such as upload codes, parsing logs. diff --git a/tools/sdk-generation-pipeline/documents/task-engine/README.md b/tools/sdk-generation-pipeline/documents/task-engine/README.md index 14fc7706366..da6a5499213 100644 --- a/tools/sdk-generation-pipeline/documents/task-engine/README.md +++ b/tools/sdk-generation-pipeline/documents/task-engine/README.md @@ -72,7 +72,8 @@ Input file for generate and build script. "headRef": "refs/pull/1234/merge", "repoHttpsUrl": "https://github.com/Azure/azure-rest-api-specs.git", "relatedReadmeMdFile": "compute/resource-manager/readme.md" - "serviceType": "resource-manager" + "serviceType": "resource-manager", + "skipGeneration": "false" } ``` diff --git a/tools/sdk-generation-pipeline/documents/task-engine/schema/GenerateAndBuildInputSchema.json b/tools/sdk-generation-pipeline/documents/task-engine/schema/GenerateAndBuildInputSchema.json index cbb35569de4..8e93c22c73e 100644 --- a/tools/sdk-generation-pipeline/documents/task-engine/schema/GenerateAndBuildInputSchema.json +++ b/tools/sdk-generation-pipeline/documents/task-engine/schema/GenerateAndBuildInputSchema.json @@ -27,6 +27,14 @@ // The type of related swagger: resource-manager or data-plane "type": "string", "enum": ["resource-manager", "data-plane"] + }, + "autorestConfig": { + // The autorest config got from /autorest.md, which is mounted by user + "type": "string" + }, + "skipGeneration": { + // Whether skip generation + "type": "boolean" } }, "required": ["specFolder", "headSha", "headRef", "repoHttpsUrl", "relatedReadmeMdFile"] diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerContext.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerContext.ts index d1da6df8ab7..ad4ed2293c3 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerContext.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerContext.ts @@ -17,6 +17,10 @@ export class DockerContext { sdkRepo?: string; resultOutputFolder?: string; autorestConfigFilePath?: string; + specLink?: string; + sdkWorkBranchLink?: string; + skipGeneration: boolean; + isPublicRepo: boolean; logger: Logger; /* @@ -34,22 +38,40 @@ export class DockerContext { this.sdkRepo = inputParams.sdkRepo; this.resultOutputFolder = inputParams.resultOutputFolder; this.autorestConfigFilePath = inputParams.autorestConfigFilePath; + this.specLink = inputParams.specLink; + this.sdkWorkBranchLink = inputParams.sdkWorkBranchLink; + this.skipGeneration = inputParams.skipGeneration; this.logger = initializeLogger(path.join(inputParams.resultOutputFolder, inputParams.dockerLogger), 'docker'); if (this.sdkList?.length === 0 && fs.existsSync(this.workDir)) { this.logger.info('Preparing environment to do grow up development'); this.mode = DockerRunningModel.GrowUp; - this.validateSpecRepo(); + this.isPublicRepo = false; + if (!this.specLink) { + try { + this.validateSpecRepo(); + } catch (e) { + throw new Error(`Cannot get spec repo link by parameter --spec-link, or get mounted swagger repo.`); + } + } else { + this.validateSpecLink(); + } + this.validateWorkDir(); + if (!!this.sdkWorkBranchLink) { + this.validateWorkBranchLink(); + } } else if (fs.existsSync(this.workDir)) { this.logger.info('Preparing environment to generate codes and do grow up development in local'); this.mode = DockerRunningModel.CodeGenAndGrowUp; + this.isPublicRepo = false; this.validateSpecRepo(); this.validateReadmeMdPath(); this.validateSdk(); } else { this.logger.info('Preparing environment to generate codes in pipeline'); this.mode = DockerRunningModel.Pipeline; + this.isPublicRepo = inputParams.isPublicRepo; this.validateSdkRepo(); this.validateSpecRepo(); this.validateReadmeMdPath(); @@ -102,4 +124,18 @@ export class DockerContext { throw new Error(`Cannot find ${this.resultOutputFolder}, please mount it to docker container`); } } + + private validateWorkBranchLink() { + const match = this.sdkWorkBranchLink.match(/(https.*\/([^\/]*))\/tree\/(.*)/); + if (match?.length !== 4) { + throw new Error(`Get invalid sdk work branch link: ${this.sdkWorkBranchLink}`); + } + } + + private validateSpecLink() { + const match = this.specLink.match(/http.*/); + if (!match) { + throw new Error(`Get invalid sdk work branch link: ${this.specLink}`); + } + } } diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerTaskEngineContext.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerTaskEngineContext.ts index 603663f3591..d47c1551e9e 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerTaskEngineContext.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/DockerTaskEngineContext.ts @@ -45,6 +45,7 @@ export class DockerTaskEngineContext { changeOwner: boolean; mode: DockerRunningModel; autorestConfig: string | undefined; + skipGeneration: boolean; public async initialize(dockerContext: DockerContext) { // before execute task engine, safe spec repos and sdk repos because they may be owned by others @@ -64,10 +65,11 @@ export class DockerTaskEngineContext { this.readmeMdPath = dockerContext.readmeMdPath; this.specRepo = { repoPath: dockerContext.specRepo, - headSha: dockerTaskEngineConfigProperties.headSha ?? dockerContext.mode === DockerRunningModel.Pipeline? - await new GitOperationWrapper(dockerContext.specRepo).getHeadSha() : '{commit_id}', + headSha: dockerTaskEngineConfigProperties.headSha ?? dockerContext.isPublicRepo? + await new GitOperationWrapper(dockerContext.specRepo).getHeadSha() : '', headRef: dockerTaskEngineConfigProperties.headRef ?? await new GitOperationWrapper(dockerContext.specRepo).getHeadRef(), - repoHttpsUrl: dockerTaskEngineConfigProperties.repoHttpsUrl + repoHttpsUrl: dockerTaskEngineConfigProperties.repoHttpsUrl?? (await new GitOperationWrapper(dockerContext.specRepo).getRemote())?? + `https://github.com/Azure/azure-rest-api-specs` }; this.serviceType = dockerContext.readmeMdPath.includes('data-plane') && dockerTaskEngineConfigProperties.serviceType ? 'data-plane': 'resource-manager'; this.tag = dockerContext.tag; @@ -78,12 +80,16 @@ export class DockerTaskEngineContext { this.changeOwner = dockerTaskEngineConfigProperties.changeOwner; this.mode = dockerContext.mode; this.autorestConfig = extractAutorestConfigs(dockerContext.autorestConfigFilePath, dockerContext.sdkRepo, dockerContext.logger); + this.skipGeneration = dockerContext.skipGeneration; } public async beforeRunTaskEngine() { if (!!this.resultOutputFolder && !fs.existsSync(this.resultOutputFolder)) { fs.mkdirSync(this.resultOutputFolder, { recursive: true }); } + if (!!this.sdkRepo && fs.existsSync(this.sdkRepo)) { + await new GitOperationWrapper(this.sdkRepo).disableFileMode(); + } this.logger.info(`Start to run task engine in ${path.basename(this.sdkRepo)}`); } diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/jobs/GrowUpJob.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/jobs/GrowUpJob.ts index b747b1c49e0..69c69c55e9a 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/jobs/GrowUpJob.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/jobs/GrowUpJob.ts @@ -1,3 +1,7 @@ +import * as fs from 'fs'; +import * as path from 'path'; + +import { GitOperationWrapper } from '../../../../utils/GitOperationWrapper'; import { DockerContext } from '../DockerContext'; import { BaseJob } from './BaseJob'; @@ -9,7 +13,59 @@ export class GrowUpJob extends BaseJob { this.context = context; } + private async cloneSdkWorkBranchIfNotExist() { + if (!this.context.sdkWorkBranchLink) return; + const match = this.context.sdkWorkBranchLink.match(/(https.*\/([^\/]*))\/tree\/(.*)/); + const sdkRepoUrl = match[1]; + const sdkRepo = match[2]; + const branch = match[3]; + if (fs.existsSync(path.join(this.context.workDir, sdkRepo))) { + this.context.logger.info(`${path.join(this.context.workDir, sdkRepo)} has already existed, and no need to clone it.`); + return; + } + const gitOperationWrapper = new GitOperationWrapper(this.context.workDir); + await gitOperationWrapper.cloneBranch(sdkRepoUrl, branch, this.context.logger); + gitOperationWrapper.baseDir = path.join(this.context.workDir, sdkRepo); + await gitOperationWrapper.safeDirectory(); + } + + private async cloneSpecRepoIfNotExist() { + if (!this.context.specLink) return; + if (fs.existsSync(this.context.specRepo)) { + this.context.logger.info(`${this.context.specRepo} has already exited, and no need to clone it`); + return; + } + const gitOperationWrapper = new GitOperationWrapper('/'); + const prLinkMatch = this.context.specLink.match(/http.*\/([^\/]*\/[^\/]*)\/pull\/([0-9]+)/); + const branchLinkMatch = this.context.specLink.match(/(https.*\/[^\/]*)\/tree\/(.*)/); + const mainBranchMatch = this.context.specLink.match(/http.*\/([^\/]*\/[^\/\.]*)/); + if (prLinkMatch?.length === 3) { + const repoName = prLinkMatch[1]; + const prNumber = prLinkMatch[2]; + await gitOperationWrapper.cloneRepo(repoName, this.context.logger, this.context.specRepo); + gitOperationWrapper.changeBaseDir(this.context.specRepo); + await gitOperationWrapper.safeDirectory(); + await gitOperationWrapper.checkoutPr(prNumber); + } else if (branchLinkMatch?.length == 3) { + const repoUrl = branchLinkMatch[1]; + const branch = branchLinkMatch[2]; + await gitOperationWrapper.cloneBranch(repoUrl, branch, this.context.logger, this.context.specRepo); + gitOperationWrapper.changeBaseDir(this.context.specRepo); + await gitOperationWrapper.safeDirectory(); + } else if (mainBranchMatch?.length == 2) { + // spec repo main branch + const repoName = mainBranchMatch[1]; + await gitOperationWrapper.cloneRepo(repoName, this.context.logger, this.context.specRepo); + gitOperationWrapper.changeBaseDir(this.context.specRepo); + await gitOperationWrapper.safeDirectory(); + } else { + throw new Error(`Get invalid spec link: ${this.context.specLink}`); + } + } + public async execute() { + await this.cloneSpecRepoIfNotExist(); + await this.cloneSdkWorkBranchIfNotExist(); this.context.logger.info(`Please use vscode to connect this container.`); this.doNotExitDockerContainer(); } diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/GenerateAndBuildTask.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/GenerateAndBuildTask.ts index 566a0646174..5e02ea7f2bb 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/GenerateAndBuildTask.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/GenerateAndBuildTask.ts @@ -1,9 +1,13 @@ import { - addFileLog, GenerateAndBuildInput, + addFileLog, + GenerateAndBuildInput, GenerateAndBuildOptions, getGenerateAndBuildOutput, - getTask, removeFileLog, requireJsonc, - runScript + getTask, + removeFileLog, + requireJsonc, + runScript, + TaskResultStatus } from '@azure-tools/sdk-generation-lib'; import fs from 'fs'; import path from 'path'; @@ -40,7 +44,8 @@ export class GenerateAndBuildTask implements SDKGenerationTaskBase { repoHttpsUrl: this.context.specRepo.repoHttpsUrl, relatedReadmeMdFile: relatedReadmeMdFileRelativePath, serviceType: this.context.serviceType, - autorestConfig: this.context.autorestConfig + autorestConfig: this.context.autorestConfig, + skipGeneration: this.context.skipGeneration }; const inputJson = JSON.stringify(inputContent, undefined, 2); this.context.logger.info(`Get ${path.basename(this.context.generateAndBuildInputJsonFile)}:`); @@ -55,7 +60,7 @@ export class GenerateAndBuildTask implements SDKGenerationTaskBase { }); removeFileLog(this.context.logger, 'generateAndBuild'); this.context.taskResults['generateAndBuild'] = executeResult; - if (executeResult === 'failed') { + if (executeResult === TaskResultStatus.Failure) { throw new Error(`Execute generateAndBuild script failed.`); } if (fs.existsSync(this.context.generateAndBuildOutputJsonFile)) { @@ -65,6 +70,9 @@ export class GenerateAndBuildTask implements SDKGenerationTaskBase { const packageFolders: string[] = []; for (const p of generateAndBuildOutputJson.packages) { packageFolders.push(p.packageFolder); + if (p.result === TaskResultStatus.Failure) { + this.context.taskResults['generateAndBuild'] = TaskResultStatus.Failure; + } } this.context.packageFolders = packageFolders; } diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/MockTestTask.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/MockTestTask.ts index 38265cc73c8..fd99f9ac9e1 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/MockTestTask.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/core/tasks/MockTestTask.ts @@ -24,6 +24,7 @@ export class MockTestTask implements SDKGenerationTaskBase { } public async execute() { + if (this.context.taskResults?.['generateAndBuild'] !== TaskResultStatus.Success) return; const mockTestTask = getTask(path.join(this.context.sdkRepo, this.context.configFilePath), 'mockTest'); if (!mockTestTask) { throw new Error(`Init task is ${mockTestTask}`); diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/runMockHostCli.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/runMockHostCli.ts index 10b6c5ba181..5dec4c41fbb 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/runMockHostCli.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/runMockHostCli.ts @@ -2,6 +2,7 @@ import { initializeLogger } from '@azure-tools/sdk-generation-lib'; import { spawn } from 'child_process'; +import fs from 'fs'; import * as path from 'path'; import { Logger } from 'winston'; @@ -26,12 +27,27 @@ export function initializeDockerMockHostContext(inputParams: DockerMockHostInput return dockerMockHostContext; } -export function runMockHost() { +export async function runMockHost() { const inputParams: DockerMockHostInput & DockerCliInput = { ...dockerCliInput.getProperties(), ...dockerMockHostInput.getProperties() }; const context = initializeDockerMockHostContext(inputParams); + if (!context.specRepo) { + context.logger.log('cmdout', `Cannot start mock server because ${context.specRepo} doesn't exist.`); + return; + } + + function sleep(ms) { + return new Promise((resolve) => { + setTimeout(resolve, ms); + }); + } + + while (!fs.existsSync(context.specRepo)) { + await sleep(10000); + } + if (!context.readmeMdPath) { context.logger.log('cmdout', `Cannot get valid readme, so do not start mock server.`); return; diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerCliInput.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerCliInput.ts index 79f85450580..341f2a11f61 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerCliInput.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerCliInput.ts @@ -13,6 +13,10 @@ export class DockerCliInput { resultOutputFolder: string; dockerLogger: string; autorestConfigFilePath: string; + specLink: string; + sdkWorkBranchLink: string; + skipGeneration: boolean; + isPublicRepo: boolean; } export const dockerCliInput = convict({ @@ -77,5 +81,33 @@ export const dockerCliInput = convict({ env: 'AUTOREST_CONFIG_FILE_PATH', format: String, doc: `The absolute path to autorest configuration file. It's required when you want to input your own autorest config in generating data-plane sdk.` + }, + specLink: { + default: '', + env: 'SPEC_LINK', + arg: 'spec-link', + format: String, + doc: `The link to spec repo or spec PR.` + }, + sdkWorkBranchLink: { + default: '', + env: 'SDK_WORK_BRANCH', + arg: 'sdk-work-branch', + format: String, + doc: `The link of sdk work branch generated by pipeline.` + }, + skipGeneration: { + default: false, + env: 'SKIP_GENERATION', + arg: 'skip-generation', + format: Boolean, + doc: `Whether skip generation` + }, + isPublicRepo: { + default: true, + env: 'IS_PUBLIC_REPO', + arg: 'is-public-repo', + format: Boolean, + doc: `Whether Spec Repo is public. IT's related to authentication when running docker in pipeline. In other scenarios, it's always false` } }); diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerTaskEngineInput.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerTaskEngineInput.ts index 93849178673..d51f827e32e 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerTaskEngineInput.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/cli/dockerCli/schema/dockerTaskEngineInput.ts @@ -12,7 +12,7 @@ export class DockerTaskEngineInput { mockTestOutputJsonFile: string; headSha: string | undefined; headRef: string | undefined; - repoHttpsUrl: string; + repoHttpsUrl: string | undefined; serviceType: string; mockServerHost?: string; initTaskLog: string; @@ -78,7 +78,7 @@ export const dockerTaskEngineInput = convict({ doc: 'headRef of spec repo' }, repoHttpsUrl: { - default: 'https://github.com/Azure/azure-rest-api-specs', + default: undefined, env: 'REPO_HTTP_URL', format: String, doc: 'The http url of spec repo' diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/GitOperationWrapper.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/GitOperationWrapper.ts index 84f30f30651..9deb66b6675 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/GitOperationWrapper.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/GitOperationWrapper.ts @@ -27,6 +27,11 @@ export class GitOperationWrapper { return headRef; } + public async getRemote() { + const remote = await this.git.getConfig('remote.origin.url'); + return remote?.value?.replace('.git', ''); + } + public async safeDirectory() { await this.git.addConfig('safe.directory', this.baseDir, true, 'global'); } @@ -35,6 +40,11 @@ export class GitOperationWrapper { await this.git.raw(['config', 'core.fileMode', 'false', '--replace-all']); } + public async checkoutPr(prNumber: string) { + await this.git.raw(['fetch', 'origin', '+refs/pull/*:refs/remotes/origin/pr/*']); + await this.git.raw(['checkout', '-b', `refs/pull/${prNumber}/merge`]); + } + public async getChangedPackageDirectory(): Promise> { const changedPackageDirectories: Set = new Set(); const files = (await this.git.raw(['ls-files', '-mdo', '--exclude-standard'])).trim().split(os.EOL); @@ -49,17 +59,38 @@ export class GitOperationWrapper { return changedPackageDirectories; } - public async cloneRepo(githubRepo: string, logger: Logger) { - const child = spawn(`git`, [`clone`, `https://github.com/${githubRepo}.git`], { + public async cloneRepo(githubRepo: string, logger: Logger, repoAlias?: string) { + const additionalParams = []; + if (!!repoAlias) additionalParams.push(repoAlias); + const child = spawn(`git`, [`clone`, `https://github.com/${githubRepo}.git`, ...additionalParams], { cwd: this.baseDir, stdio: ['ignore', 'pipe', 'pipe'] }); + await this.injectListener(child, logger); + } + + private async injectListener(child: any, logger: Logger) { child.stdout.on('data', (data) => logger.log('cmdout', data.toString())); - child.stderr.on('data', (data) => logger.log('cmderr', data.toString())); + child.stderr.on('data', (data) => logger.log('cmdout', data.toString())); await new Promise((resolve) => { child.on('exit', (code, signal) => { resolve({ code, signal }); }); }); } + + public async cloneBranch(repoUrl: string, branchName: string, logger: Logger, repoAlias?: string) { + const additionalParams = []; + if (!!repoAlias) additionalParams.push(repoAlias); + const child = spawn(`git`, [`clone`, '--branch', branchName, repoUrl, ...additionalParams], { + cwd: this.baseDir, + stdio: ['ignore', 'pipe', 'pipe'] + }); + await this.injectListener(child, logger); + } + + public changeBaseDir(baseDir: string) { + this.baseDir = baseDir; + this.git = simpleGit({ baseDir: baseDir }); + } } diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/autorestConfigExtractorUtils.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/autorestConfigExtractorUtils.ts index fedca1eb4a8..e65a355b1dc 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/autorestConfigExtractorUtils.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/src/utils/autorestConfigExtractorUtils.ts @@ -1,5 +1,4 @@ import * as fs from 'fs'; -import * as os from 'os'; import * as path from 'path'; import winston from 'winston'; @@ -18,15 +17,13 @@ export function extractAutorestConfigs(autorestConfigFilePath: string, sdkRepo: if (autorestConfigs.length > 1) { logger.warn(`Docker is running in pipeline, but get autorest config for more than 1 language of sdk. So only get the first autorest config`); } - return `azure-sdk-for-${autorestConfigs[0]}`; + return `# azure-sdk-for-${autorestConfigs[0]}`.replace(/\r\n/g, '\n').replace(/\n/g, '\r\n'); } for (const autorestConfig of autorestConfigs) { let autorestFullConfig = `# azure-sdk-for-${autorestConfig}`; if (autorestFullConfig.startsWith(`# ${path.basename(sdkRepo)}`)) { logger.info(`Find autorest config for ${path.basename(sdkRepo)} in ${autorestConfigFilePath}: \n${autorestFullConfig}`); - if (os.EOL == '\n') { - autorestFullConfig = autorestFullConfig.replace(/\n/g, '\r\n'); - } + autorestFullConfig = autorestFullConfig.replace(/\r\n/g, '\n').replace(/\n/g, '\r\n'); return autorestFullConfig; } } diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/test/unit/DockerTaskEngine.test.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/test/unit/DockerTaskEngine.test.ts index 8bcc43c6af8..8e48c710704 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-cli/test/unit/DockerTaskEngine.test.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-cli/test/unit/DockerTaskEngine.test.ts @@ -19,7 +19,11 @@ describe('task engine', async () => { sdkRepo: path.join(tmpFolder, 'sdk-repo'), resultOutputFolder: path.join(tmpFolder, 'output'), dockerLogger: 'docker.log', - autorestConfigFilePath: path.join(path.resolve('.'), 'test', 'unit', 'utils', 'autorest-single-config.md') + autorestConfigFilePath: path.join(path.resolve('.'), 'test', 'unit', 'utils', 'autorest-single-config.md'), + specLink: '', + sdkWorkBranchLink: '', + skipGeneration: false, + isPublicRepo: false }); const dockerTaskEngineContext = new DockerTaskEngineContext(); await dockerTaskEngineContext.initialize(dockerContext); @@ -73,6 +77,7 @@ describe('task engine', async () => { repoHttpsUrl: 'https://github.com/Azure/azure-rest-api-specs' }; dockerTaskEngineContext.changeOwner = false; + dockerTaskEngineContext.skipGeneration = false; await dockerTaskEngineContext.runTaskEngine(); expect(existsSync(dockerTaskEngineContext.initTaskLog)).toBe(true); diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInput.ts b/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInput.ts index 732d413cd25..f19d3b233b8 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInput.ts +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInput.ts @@ -11,6 +11,7 @@ export type GenerateAndBuildInput = { relatedReadmeMdFile: string; serviceType: string; autorestConfig: string; + skipGeneration: boolean; }; export const getGenerateAndBuildInput = getTypeTransformer( diff --git a/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInputSchema.json b/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInputSchema.json index 1e235b51f12..8e93c22c73e 100644 --- a/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInputSchema.json +++ b/tools/sdk-generation-pipeline/packages/sdk-generation-lib/src/types/taskInputAndOuputSchemaTypes/GenerateAndBuildInputSchema.json @@ -31,6 +31,10 @@ "autorestConfig": { // The autorest config got from /autorest.md, which is mounted by user "type": "string" + }, + "skipGeneration": { + // Whether skip generation + "type": "boolean" } }, "required": ["specFolder", "headSha", "headRef", "repoHttpsUrl", "relatedReadmeMdFile"] diff --git a/tools/sdk-generation-pipeline/scripts/change-owner.sh b/tools/sdk-generation-pipeline/scripts/change-owner.sh index 32f5a2f9ca5..6c1ddebf43a 100644 --- a/tools/sdk-generation-pipeline/scripts/change-owner.sh +++ b/tools/sdk-generation-pipeline/scripts/change-owner.sh @@ -4,16 +4,20 @@ SPEC_REPO=/spec-repo WORK_DIR=/work-dir SDK_REPO=/sdk-repo +while [ ! -d "${SPEC_REPO}" ]; do + sleep 10 +done + if [ -d "${SPEC_REPO}" ]; then while true do if [ -f "/tmp/notExit" ]; then USER_GROUP_ID=`stat -c "%u:%g" ${SPEC_REPO}` if [ -d "${WORK_DIR}" ]; then - chown -R ${USER_GROUP_ID} ${WORK_DIR} + chown -R ${USER_GROUP_ID} ${WORK_DIR} > /dev/null 2>&1 fi if [ -d "${SDK_REPO}" ]; then - chown -R ${USER_GROUP_ID} ${SDK_REPO} + chown -R ${USER_GROUP_ID} ${SDK_REPO} > /dev/null 2>&1 fi fi sleep 5s diff --git a/tools/sdk-generation-pipeline/scripts/rerun-tasks b/tools/sdk-generation-pipeline/scripts/rerun-tasks index 43623947359..07d9811fe93 100755 --- a/tools/sdk-generation-pipeline/scripts/rerun-tasks +++ b/tools/sdk-generation-pipeline/scripts/rerun-tasks @@ -1,3 +1,3 @@ #!/bin/sh set -e -docker-cli "$@" \ No newline at end of file +docker-cli "$@" --skip-generation=true \ No newline at end of file