Correct ONNX and Protobuf version in vcpkg build #23285

luncliff · 2025-01-08T04:07:06Z

Description

Changes vcpkg manifest and configuration file (vcpkg.json & vcpkg-configuration.json)

Update vcpkg version to https://github.com/microsoft/vcpkg/releases/tag/2024.12.16
Use protobuf 3.21.12(= v21.12) to sync with cmake/deps.txt
- Resolve [Build] Build fails with vcpkg due to an incorrect protobuf version. #22750
Add onnx to vcpkg manifest so find_package(ONNX) and find_dependency(Protobuf) can work as expected.
- Currently, It uses 1.16.2
- v1.17.0 will become available after [onnx] Update to 1.17.0 and fix openvino dependency error vcpkg#42942

However, onnx in vcpkg doesn't configure ONNX_DISABLE_STATIC_REGISTRATION build option.

[onnx] Add features to use ONNX build options vcpkg#38879
Create "cmake/vcpkg-triplets/" folder and triplet files which use VCPKG_CMAKE_CONFIGURE_OPTIONS for the option
- This requires VCPKG_OVERLAY_TRIPLETS environment variable for CI steps, which is a bit inconvenient.
  I will try to find simple way to get same result

Motivation and Context

Help [Build] Better support for vcpkg #23158
- "ONNX is not consumed from vcpkg"
- "Mismatch protobuf version. When vcpkg is enabled , we should not fetch protoc from Github which may cause version mismatches."
[onnx] Remove Protobuf_INCLUDE_DIR in ONNXConfig.cmake and use find_dependency(protobuf) vcpkg#43126
Create CMake option onnxruntime_USE_VCPKG #21348

* 3.21.12 fails in find_package(ONNX)

* microsoft/vcpkg@93570a2

* comment CLI commands for vcpkg install * triplets are forked from https://github.com/microsoft/vcpkg/tree/2024.12.16/triplets

snnn · 2025-01-08T17:32:58Z

/azp run Big Models, Linux Android Emulator QNN CI Pipeline, Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline

snnn · 2025-01-08T17:33:22Z

/azp run Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CUDA CI Pipeline

snnn · 2025-01-08T17:33:27Z

/azp run Windows GPU DML CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

azure-pipelines · 2025-01-08T17:33:30Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2025-01-08T17:33:53Z

Azure Pipelines successfully started running 5 pipeline(s).

azure-pipelines · 2025-01-08T17:33:59Z

Azure Pipelines successfully started running 7 pipeline(s).

snnn · 2025-01-08T20:26:33Z

cmake/vcpkg.json

-      "platform": "!windows"
-    },
    {
      "name": "nsync",


@luncliff , we no longer use nsync. This part can be removed.

Yes. I saw #20413
I will remove the others before working on WebAssembly build

onnxruntime/cmake/onnxruntime_webassembly.cmake

Line 229 in 4134cd9

target_compile_options(nsync_cpp PRIVATE ${SMEMORY_FLAG} -Wno-experimental)

### Description Changes vcpkg manifest and configuration file (vcpkg.json & vcpkg-configuration.json) * Update vcpkg version to https://github.com/microsoft/vcpkg/releases/tag/2024.12.16 * Use protobuf 3.21.12(= `v21.12`) to sync with [cmake/deps.txt](https://github.com/microsoft/onnxruntime/blob/main/cmake/deps.txt) * Resolve #22750 * Add `onnx` to vcpkg manifest so `find_package(ONNX)` and `find_dependency(Protobuf)` can work as expected. * Currently, It uses 1.16.2 * v1.17.0 will become available after microsoft/vcpkg#42942 However, `onnx` in vcpkg doesn't configure `ONNX_DISABLE_STATIC_REGISTRATION` build option. * microsoft/vcpkg#38879 * Create "cmake/vcpkg-triplets/" folder and triplet files which use `VCPKG_CMAKE_CONFIGURE_OPTIONS` for the option * This requires `VCPKG_OVERLAY_TRIPLETS` environment variable for CI steps, which is a bit inconvenient. I will try to find simple way to get same result ### Motivation and Context * Help #23158 * "ONNX is not consumed from vcpkg" * "Mismatch protobuf version. When vcpkg is enabled , we should not fetch protoc from Github which may cause version mismatches." * microsoft/vcpkg#43126 * #21348

snnn · 2025-01-09T01:19:52Z

@luncliff , the onnx port pulled in python3. Now we need to build python from source, for every platform. And python itself has a lot of dependencies. Can we avoid doing that?

snnn · 2025-01-09T01:20:29Z

Most developers would rather using a prebuilt python binary.

luncliff · 2025-01-09T02:47:51Z

@luncliff , the onnx port pulled in python3. Now we need to build python from source, for every platform. And python itself has a lot of dependencies. Can we avoid doing that?

Yes. There is an alternative of using python3 port. I will suggest the change to vcpkg repository soon.

snnn · 2025-01-09T03:39:38Z

Thanks!

### Description Changes vcpkg manifest and configuration file (vcpkg.json & vcpkg-configuration.json) * Update vcpkg version to https://github.com/microsoft/vcpkg/releases/tag/2024.12.16 * Use protobuf 3.21.12(= `v21.12`) to sync with [cmake/deps.txt](https://github.com/microsoft/onnxruntime/blob/main/cmake/deps.txt) * Resolve microsoft#22750 * Add `onnx` to vcpkg manifest so `find_package(ONNX)` and `find_dependency(Protobuf)` can work as expected. * Currently, It uses 1.16.2 * v1.17.0 will become available after microsoft/vcpkg#42942 However, `onnx` in vcpkg doesn't configure `ONNX_DISABLE_STATIC_REGISTRATION` build option. * microsoft/vcpkg#38879 * Create "cmake/vcpkg-triplets/" folder and triplet files which use `VCPKG_CMAKE_CONFIGURE_OPTIONS` for the option * This requires `VCPKG_OVERLAY_TRIPLETS` environment variable for CI steps, which is a bit inconvenient. I will try to find simple way to get same result ### Motivation and Context * Help microsoft#23158 * "ONNX is not consumed from vcpkg" * "Mismatch protobuf version. When vcpkg is enabled , we should not fetch protoc from Github which may cause version mismatches." * microsoft/vcpkg#43126 * microsoft#21348

### Description Changes vcpkg manifest and configuration file (vcpkg.json & vcpkg-configuration.json) * Update vcpkg version to https://github.com/microsoft/vcpkg/releases/tag/2024.12.16 * Use protobuf 3.21.12(= `v21.12`) to sync with [cmake/deps.txt](https://github.com/microsoft/onnxruntime/blob/main/cmake/deps.txt) * Resolve #22750 * Add `onnx` to vcpkg manifest so `find_package(ONNX)` and `find_dependency(Protobuf)` can work as expected. * Currently, It uses 1.16.2 * v1.17.0 will become available after microsoft/vcpkg#42942 However, `onnx` in vcpkg doesn't configure `ONNX_DISABLE_STATIC_REGISTRATION` build option. * microsoft/vcpkg#38879 * Create "cmake/vcpkg-triplets/" folder and triplet files which use `VCPKG_CMAKE_CONFIGURE_OPTIONS` for the option * This requires `VCPKG_OVERLAY_TRIPLETS` environment variable for CI steps, which is a bit inconvenient. I will try to find simple way to get same result ### Motivation and Context * Help #23158 * "ONNX is not consumed from vcpkg" * "Mismatch protobuf version. When vcpkg is enabled , we should not fetch protoc from Github which may cause version mismatches." * microsoft/vcpkg#43126 * #21348

…#23554) ### Description `python3` dependency is removed in `onnx` port of https://github.com/microsoft/vcpkg upstream. * microsoft/vcpkg#43236 * #23285 (comment) (Previous work) Removed `nsync`, and use ONNX 1.70.0+ in vcpkg.json(manifest) ### Motivation and Context * Help #23158 * #23456

…microsoft#23554) ### Description `python3` dependency is removed in `onnx` port of https://github.com/microsoft/vcpkg upstream. * microsoft/vcpkg#43236 * microsoft#23285 (comment) (Previous work) Removed `nsync`, and use ONNX 1.70.0+ in vcpkg.json(manifest) ### Motivation and Context * Help microsoft#23158 * microsoft#23456

…#23554) ### Description `python3` dependency is removed in `onnx` port of https://github.com/microsoft/vcpkg upstream. * microsoft/vcpkg#43236 * #23285 (comment) (Previous work) Removed `nsync`, and use ONNX 1.70.0+ in vcpkg.json(manifest) ### Motivation and Context * Help #23158 * #23456

### Description Changes vcpkg manifest and configuration file (vcpkg.json & vcpkg-configuration.json) * Update vcpkg version to https://github.com/microsoft/vcpkg/releases/tag/2024.12.16 * Use protobuf 3.21.12(= `v21.12`) to sync with [cmake/deps.txt](https://github.com/microsoft/onnxruntime/blob/main/cmake/deps.txt) * Resolve #22750 * Add `onnx` to vcpkg manifest so `find_package(ONNX)` and `find_dependency(Protobuf)` can work as expected. * Currently, It uses 1.16.2 * v1.17.0 will become available after microsoft/vcpkg#42942 However, `onnx` in vcpkg doesn't configure `ONNX_DISABLE_STATIC_REGISTRATION` build option. * microsoft/vcpkg#38879 * Create "cmake/vcpkg-triplets/" folder and triplet files which use `VCPKG_CMAKE_CONFIGURE_OPTIONS` for the option * This requires `VCPKG_OVERLAY_TRIPLETS` environment variable for CI steps, which is a bit inconvenient. I will try to find simple way to get same result ### Motivation and Context * Help #23158 * "ONNX is not consumed from vcpkg" * "Mismatch protobuf version. When vcpkg is enabled , we should not fetch protoc from Github which may cause version mismatches." * microsoft/vcpkg#43126 * #21348

…#23554) ### Description `python3` dependency is removed in `onnx` port of https://github.com/microsoft/vcpkg upstream. * microsoft/vcpkg#43236 * #23285 (comment) (Previous work) Removed `nsync`, and use ONNX 1.70.0+ in vcpkg.json(manifest) ### Motivation and Context * Help #23158 * #23456

luncliff added 4 commits January 6, 2025 22:23

vcpkg: use 2024.12.16, format manifest

267d283

vcpkg: use protobuf 3.21.12

6784c6a

* 3.21.12 fails in find_package(ONNX)

vcpkg: use latest commit reference

3955fff

* microsoft/vcpkg@93570a2

gh-actions: install homebrew packages

6d9e5f0

luncliff marked this pull request as draft January 8, 2025 04:07

vcpkg: provide ONNX_DISABLE_STATIC_REGISTRATION with overlay triplets

8d8dc51

* comment CLI commands for vcpkg install * triplets are forked from https://github.com/microsoft/vcpkg/tree/2024.12.16/triplets

luncliff marked this pull request as ready for review January 8, 2025 13:22

ci: use VCPKG_OVERLAY_TRIPLETS with build.py

df681aa

snnn added the build build issues; typically submitted using template label Jan 8, 2025

snnn self-requested a review January 8, 2025 18:03

snnn approved these changes Jan 8, 2025

View reviewed changes

snnn merged commit 5b9c968 into microsoft:main Jan 8, 2025
75 checks passed

snnn reviewed Jan 8, 2025

View reviewed changes

luncliff deleted the support/vcpkg-onnx branch January 9, 2025 02:47

luncliff mentioned this pull request Jan 13, 2025

[onnx,onnx-optimizier] Remove "python3" dependency and "pybind11" feature microsoft/vcpkg#43236

Merged

7 tasks

luncliff mentioned this pull request Jan 31, 2025

Use latest vcpkg commit in configuration, sync manifest with deps.txt #23554

Merged

Correct ONNX and Protobuf version in vcpkg build #23285

Correct ONNX and Protobuf version in vcpkg build #23285

Uh oh!

Conversation

luncliff commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

snnn commented Jan 8, 2025

Uh oh!

snnn commented Jan 8, 2025

Uh oh!

snnn commented Jan 8, 2025

Uh oh!

azure-pipelines bot commented Jan 8, 2025

Uh oh!

azure-pipelines bot commented Jan 8, 2025

Uh oh!

azure-pipelines bot commented Jan 8, 2025

Uh oh!

Uh oh!

snnn Jan 8, 2025

Choose a reason for hiding this comment

Uh oh!

luncliff Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snnn commented Jan 9, 2025

Uh oh!

snnn commented Jan 9, 2025

Uh oh!

luncliff commented Jan 9, 2025

Uh oh!

snnn commented Jan 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

luncliff commented Jan 8, 2025 •

edited

Loading

luncliff Jan 9, 2025 •

edited

Loading