Major Refactoring of Azure DevOps Pipelines

[snnn]

This pull request introduces a significant refactoring of the Azure Pipelines CI/CD infrastructure. The primary goals of these changes are to:

1. Solve the problem that vcpkg/cmake version can get changed when the CI build machine image changes, which can make pipeline suddenly broken and interrupt our release process.
2. Reduce the Zip-Nuget-Java-Nodejs Packaging Pipeline's running time by changing how macOS universal2 binaries are built.
3. Add the support for RC releases for Java packages.

Key Changes:

1. Standardized Build Tool Setup (setup-build-tools.yml)

A new reusable template, setup-build-tools.yml, has been created to centralize the setup of essential build tools.

• Pinned Dependencies: This new template allows us to pin specific versions of cmake and vcpkg, which was not previously possible. This ensures a more stable and predictable build environment across all pipelines.
• Reduced Redundancy: By consolidating the setup logic into a single template, we have significantly reduced code duplication across numerous pipeline files, making them cleaner and easier to maintain.

Currently this file is only used in macOS and Windows pipelines since most Linux pipelines use docker to manage their environment.

2. Reworked macOS Universal Binary Build Process

The methodology for building macOS universal2 binaries has been fundamentally changed to improve reliability and flexibility.

• Python Packaging: The Python packaging pipeline will no longer produce universal2 wheels. Instead, it will generate separate wheels for x86_64 and arm64 architectures.
• NuGet C-API Packaging: The NuGet C-API pipeline has been updated to first build the x86_64 and arm64 binaries independently. These single-architecture binaries are then combined to create the final universal package, rather than building a universal2 package in a single pass.

The change is made mainly because:

• Building for both ARCHs in a single pass is too slow, which may take about 5 hours in the ADO machine pool.
• A lot of MLAS features are disabled when ORT is built in such a way.

3. Java Packaging and Testing Overhaul

The Download_Java_Tools stage in "Zip-Nuget-Java-Nodejs Packaging Pipeline" is deleted because it is no longer used. Previously it was added to reduce the times of downloading the same java packages again and again , which was to reduce download errors. Now we have setup a private ADO feed for this.

Besides, there are some major changes to the pipeline that:

1. MD5 and SHA1 checksum files are provided along with the java package files instead of SHA256. This is because Sonatype's official docs says MD5/SHA1 checkcums are required while the others are optional. See: https://central.sonatype.org/publish/requirements/#supply-javadoc-and-sources . Now the publishing would fail if we don't have the MD5/SHA1 checksum files.
2. The format of the checksum files is changed. Previously we used Linux's sha256sum command to generate such files, so each checksum file contains a hash value and a filename in the file content. However, it was not the expected format. Sonatype expects that the file only contains a hash value. This PR fixes the issue.
3. A few powershell scripts were rewritten in python to improve error check and robustness
4. Added the support for generating RC packages. Previously we had to manually modify the version numbers and manually do GPG sign.
5. Two new files jar-packaging.yml and setup-maven.yml were added. We will use maven to fetch dependent packages(instead of directly HTTP fetching) to improve supply chain security, because maven allows us using a private feed to do so.

4. Dockerfile Enhancements

The Dockerfiles used in the CI have been updated to use a BASEIMAGE argument. This makes them more flexible, allowing the base image to be specified at build time, which simplifies maintenance and updates. It will allow us to using different base image repos in different CI environments. In the future we will change the Github Actions to only fetch base images from public docker repos. Meanwhile, ADO packaging pipelines will continue to use private repos.

5. Improved Release Management

The run_packaging_pipelines.py script has been updated to provide more robust and explicit control over
the package versioning for different build scenarios. This clarifies the process for generating nightly,
release candidate (RC), and final release packages.

The script now handles three distinct cases for package versioning:

• Nightly Packages: For regular CI builds (e.g., on the main branch), the script triggers the packaging
  pipelines in "nightly" mode. This sets the IsReleaseBuild parameter to false and results in packages with
  a development version suffix (e.g., 1.2.3-dev-20250909-abcdef).

• Release Candidate (RC) Builds: To create a pre-release or RC build, the script is run with the
  --build-mode release flag, along with the --pre-release-suffix-string (e.g., rc) and
  --pre-release-suffix-number (e.g., 1) arguments. This sets the IsReleaseBuild parameter to true and
  passes the suffix information to the pipelines, resulting in a semantically versioned pre-release package
  (e.g., 1.2.3-rc.1).

• Final Release Builds: For a final release, the script is run with --build-mode release without any
  pre-release suffix arguments. This sets IsReleaseBuild to true, and the resulting package will have a
  clean, final version number (e.g., 1.2.3) based on the VERSION_NUMBER file.

Please note:

• Java packages still only support the second and third mode.
• Python packages only support the first and the last mode.

[github-advanced-security]

lintrunner found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[edgchen1]

should we also update this react-native-ci.yml job to use an older CMake for the MacOS build?

onnxruntime/tools/ci_build/github/azure-pipelines/templates/react-native-ci.yml

Line 70 in 2132530

- job: ReactNative_CI_iOS_build

[snnn]

should we also update this react-native-ci.yml job to use an older CMake for the MacOS build?

onnxruntime/tools/ci_build/github/azure-pipelines/templates/react-native-ci.yml

Line 70 in 2132530

- job: ReactNative_CI_iOS_build

Will address it after this PR.

[snnn]

Should be fairly easy to migrate the pipeline to GH by employing an AI agent.