[libtorch] Add new port #8318

luncliff · 2019-09-24T08:10:38Z

This PR adds a new port for PyTorch.
Instead of performing local build, the new port extracts prebuilt binaries.

Sources:
- https://pytorch.org/
- https://github.com/pytorch/pytorch

Changes

New: libtorch 1.2.0 (C++ part for PyTorch)

Related Issues

Port Request: Add libtorch #5209 [New port request] pytorch. #6345

Triplets

x64-windows (CPU, GPU)
x64-osx (CPU)
x64-linux (CPU, GPU)

Unavailables

*-windows-static
*-uwp

Installation

CPU Only

For windows, there are only 1 available triplet.

vcpkg install --triplet x64-windows libtorch

For Linux and OSX, simply install with …

vcpkg install libtorch

GPU (with CUDA)

Notice that CUDA support is only available for Windows/Linux

vcpkg install --triplet x64-windows libtorch[cuda]
vcpkg install --triplet x64-linux   libtorch[cuda]

Usage

Targets of the protobuf and cpuinfo sould be specified.

This is custom installation of the LibTorch for vcpkg:

    find_package(Protobuf   CONFIG REQUIRED)
    find_package(cpuinfo    CONFIG REQUIRED)
    find_package(Torch      CONFIG REQUIRED)

    target_link_libraries(main
    PRIVATE
        # from vcpkg
        protobuf::libprotobuf cpuinfo::clog cpuinfo::cpuinfo

        # from pytorch prebuilt package
        c10 torch caffe2::mkl
    )

cbezault · 2019-09-26T12:47:45Z

In general we don't delete ports. If caffe users should really really really be using this port we should turn caffe into a stub port which prints a deprecation message and redirects the user to this port by having a dependency on this one.

luncliff · 2019-09-26T12:56:19Z

@cbezault Oh, I never thought about the point. Will recover the port soon.

* Download PyTorch 1.2.0 through official url * SHA512 checksum for each platforms * Replaced packaged protobuf to that of VcPkg * use cu100 when feature 'cuda' is requested

luncliff · 2019-09-26T13:05:06Z

Just fixed the removal of caffe2 0.8.

cenit · 2019-09-27T04:34:18Z

@luncliff I also propose to leave caffe as an empty port redirecting to this one. Otherwise we would have to check for the reciprocal installation since the two might be incompatible

luncliff · 2019-09-27T05:37:51Z

@cenit Sorry, I can't get the word, 'empty'.
Is that mean we have to keep ports/caffe2/? or the installation should modify something under installed/{triplet}/{caffe2, libtorch}?

cenit · 2019-09-27T06:55:04Z

something similar to what happened to ilmbase when it got merged to openexr. Now ilmbase is empty and depends on openexr, so when you install it you are "redirected".
https://github.com/microsoft/vcpkg/tree/master/ports/ilmbase

luncliff · 2019-09-27T08:31:57Z

@cbezault @cenit Understood. Doing so is indeed easy.
However, the new port(libtorch) is 1.2 and the current caffe2 port is 0.8. I can't sure about their compatibility issue.

When I open the PR I thought they have to be replaced.
But on second thought, I have no confidence that this is the "really really really" case. If possible, I would like to leave them as is.

If I have to make the caffe2 empty, then I will do within days. Just like the ilmbase did.

cenit · 2019-09-27T12:17:40Z

Since caffe2 just became libtorch from the c++ point of view (and also the python one...), I can't see the point in keeping both. Am I missing something?

also don't be afraid about broken api/abi compatibility. There is no port that needs fixing, and vcpkg itself is very aggressive in updating libraries... so this would be just the perfect way to update an "old" caffe2 port!

cenit · 2019-09-27T13:06:53Z

for the "history" of caffe2 we also have the x-history command, so don't be afraid in removing the old portfile. There is no real reason to keep it "visible" in the master tree.

* port 'caffe2' depends 'libtorch' * port 'caffe2' became empty

luncliff · 2019-09-27T13:10:18Z

@cenit Oh, wow. Learned a new feature!

cenit · 2019-09-27T19:05:28Z

Also, i’d create an “empty” caffe port (without the 2, yes, linking to caffe2 that links to pytorch). That would be useful for newcomers and to close many issues that otherwise get opened from time to time

luncliff · 2019-09-30T01:07:50Z

I asked about the difference between caffe2 and caffe to my coworker. And his answer was they have much difference unlike their name. So I think adding caffe port in this PR is not appropriate.

Related Issues: caffe

cenit · 2019-09-30T04:53:54Z

Ok, fair enough

cenit · 2019-10-02T09:52:36Z

what are the errors on the windows platform? I tried to see also CI:14505 build but it looks empty. Might be that the force-push removed the references to the previous trials, but CI remembered them and declared port already failed. Can you please retrigger the build, just bumping the version inside CONTROL file for example, if you don't know what was wrong?
I am asking this because I'd like to see this PR merged soon and declaring failures as expected with a reason might be necessary

MVoz · 2019-10-04T16:48:29Z

install pre-built ?

error

Installing package libtorch[core,cuda]:x64-windows...
The following files are already installed in E:/tools/vcpkg/installed/x64-windows and are in conflict with libtorch:x64-windows
Installed by cpuinfo:x64-windows
    debug/lib/clog.lib
    debug/lib/cpuinfo.lib
    include/clog.h
    include/cpuinfo.h
    lib/clog.lib
    lib/cpuinfo.lib

luncliff · 2019-10-04T17:07:53Z

With the message I recognized that the cpuinfo is merged. I will adjust the part in this weekend, **as soon as possible**. My windows laptop is broken (not working) now so I have to borrow one.

…

On 5 Oct 2019, at 1:48 AM, Voskrese ***@***.***> wrote: install pre-built ? error Installing package libtorch[core,cuda]:x64-windows... The following files are already installed in E:/tools/vcpkg/installed/x64-windows and are in conflict with libtorch:x64-windows Installed by cpuinfo:x64-windows debug/lib/clog.lib debug/lib/cpuinfo.lib include/clog.h include/cpuinfo.h lib/clog.lib lib/cpuinfo.lib — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8318?email_source=notifications&email_token=ADLSYE2XPXXMUJ5MWXF6J3DQM5XW5A5CNFSM4IZ4IRCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAMHZFA#issuecomment-538475668>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADLSYE5F73FWQLSM76ZMKKTQM5XW5ANCNFSM4IZ4IRCA>.

* usage: add cpuinfo targets * portfile checks target architecture and linkage option by this update, 'x86-windows' and '*-windows-static' must fail.

* the prebuilt package can't be used for the platform :(

luncliff · 2019-10-06T06:53:00Z

Just updated the files. Now it seems like x64-windows is working well.

cenit · 2019-10-06T13:51:44Z

Just read the portfile and the initial message better now:

Instead of performing local build, the new port extracts prebuilt binaries.

I am a little bit uncomfortable with it. Is it really necessary? Whenever possible, vcpkg builds library from sources...

luncliff · 2019-10-07T12:49:37Z

Sorry for late comment, @cenit. I couldn't focus on the issue thesedays.
For the downloading behavior, I agree that the install step should build whenever possible.

However, me and my teammates wanted to ship the package (libtorch 1.2.0) and some other build results in the vcpkg. For the purpose my decision was to place those prebuilt into the installed dir.

If such portfile can't be adopted, I think this PR should be closed.
In my opinion, someone can be motivated to enhance the file if there is an existing one. I wish this libtorch port to be the case.

cenit · 2019-10-07T13:25:53Z

I can understand your position. Which is reasonable. I am not sure the mods might agree with it or not. The only pre-built ports that you can find on vcpkg are ones that are not providing the source code, and so the built artifacts are the only things available... let’s see here their stance

jasjuang · 2019-10-16T16:41:57Z

Do we have any decision on this PR? It would be amazing to have libtorch working in vcpkg.

luncliff · 2019-10-17T00:56:30Z

we are waiting for the feedback. :)

…

On Thu, Oct 17, 2019, 1:42 AM Jason Juang ***@***.***> wrote: Do we have any decision on this PR? It would be amazing to have libtorch working in vcpkg. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8318?email_source=notifications&email_token=ADLSYE574FMCDMVP36FFN3DQO476PA5CNFSM4IZ4IRCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBNFBTA#issuecomment-542789836>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLSYE2RNFQFLOCJXKXDB7LQO476PANCNFSM4IZ4IRCA> .

cbezault · 2019-10-17T11:32:48Z

@luncliff if you can expand on why you think shipping a pre-built binary is a superior choice in this case I would be interested in hearing it.

luncliff · 2019-10-17T14:06:34Z

@cbezault I don't think using a prebuilt is a kind of superior choice.
In my opinion, the pros and cons of doing so are like the following.

Pros

Reduced build time:
pytorch's build time takes too much time (more than 50min on Windows).
Bypassing dependencies:
In vcpkg, Intel MKL requires some manual installation. Which broked my team's CI builds. As a user, I wanted to avoid such ports if possible.

Cons

Possible linking issues in the future:
Especially protobuf is the module that made my team suffer from version matching. Port of protobuf and cpuinfo will make progress as time flows, but prebuilt can't handle those issues.

Considering the issue, definitely, the best choice is to build PyTorch from the source. But I think the way requires too much for using the port.

cbezault · 2019-10-17T17:20:30Z

It is unfortunate that building will take so long. It seems like the right thing for us to do is to provide a tarball of MKL like Conda does. (We could literally take from conda...).

Once binary caching gets released to the public the build time problem should mostly go away. You could also try to use the beta binary caching feature in your CI.

luncliff · 2019-10-17T23:33:56Z

I can try but it's out of my cost... I will close this for now so another person can try in the future.

…

On Fri, Oct 18, 2019, 2:20 AM Curtis J Bezault ***@***.***> wrote: It is unfortunate that building will take so long. It seems like the right thing for us to do is to provide a tarball of MKL like Conda does. (We could literally take from conda...) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8318?email_source=notifications&email_token=ADLSYE3YDRMG5GV6Z2H54SDQPCNHJA5CNFSM4IZ4IRCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQ332A#issuecomment-543276520>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADLSYE3IJ4US3NX4AOLCD7LQPCNHJANCNFSM4IZ4IRCA> .

traversaro · 2019-10-18T08:37:44Z

You could also try to use the beta binary caching feature in your CI.

That is great, I was not aware there was a plan to use those binaries outside your CI system. I thought about using those binary by lurking the CI system logs, but then I desisted as I thought it was only for internal use. Do you have some entry point on how to do that? Thanks in advance.

cbezault · 2019-10-18T16:26:28Z

It's been a minute since I've had to turn it on but I think there's a --binary-caching flag and there's definitely some sort of environment variable you can set. Note there's still some details that are not added to the hash of the packages. Notably the type of compiler used is not hashed, ports which just just check for system dependencies such as cuda are not hashed correctly, and ports which have dependencies on ports in other triplets like proj4[database] are not hashed correctly.

cbezault · 2019-10-18T16:27:31Z

Note you will also need to point vcpkg/archives to somewhere all your ci machines can reach (like a network drive)

Rastaban assigned ras0219-msft Sep 24, 2019

grdowns changed the title ~~[new port] libtorch~~ [libtorch] Add new port Sep 26, 2019

[libtorch] libtorch (PyTorch C++)

112bb84

* Download PyTorch 1.2.0 through official url * SHA512 checksum for each platforms * Replaced packaged protobuf to that of VcPkg * use cu100 when feature 'cuda' is requested

[libtorch] redirect caffe2 to libtorch

0293625

* port 'caffe2' depends 'libtorch' * port 'caffe2' became empty

luncliff added 5 commits October 5, 2019 02:09

Merge branch 'master' into port/libtorch

511dae4

[libtorch] resolve conflict with the 'cpuinfo'

bd82945

[libtorch] check linkage / target arch

e06f4b3

* usage: add cpuinfo targets * portfile checks target architecture and linkage option by this update, 'x86-windows' and '*-windows-static' must fail.

[libtorch] disable uwp triplet

d395531

* the prebuilt package can't be used for the platform :(

[libtorch] usage shows import of 'cpuinfo'

bc2d96b

luncliff closed this Oct 17, 2019

luncliff mentioned this pull request Apr 10, 2021

[libtorch] create a new port #17199

Merged

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libtorch] Add new port #8318

[libtorch] Add new port #8318

luncliff commented Sep 24, 2019 •

edited

Loading

cbezault commented Sep 26, 2019

luncliff commented Sep 26, 2019

luncliff commented Sep 26, 2019 •

edited

Loading

cenit commented Sep 27, 2019

luncliff commented Sep 27, 2019 •

edited

Loading

cenit commented Sep 27, 2019

luncliff commented Sep 27, 2019 •

edited

Loading

cenit commented Sep 27, 2019 •

edited

Loading

cenit commented Sep 27, 2019

luncliff commented Sep 27, 2019 •

edited

Loading

cenit commented Sep 27, 2019 •

edited

Loading

luncliff commented Sep 30, 2019

cenit commented Sep 30, 2019

cenit commented Oct 2, 2019

MVoz commented Oct 4, 2019

luncliff commented Oct 4, 2019 via email

luncliff commented Oct 6, 2019 •

edited

Loading

cenit commented Oct 6, 2019

luncliff commented Oct 7, 2019 •

edited

Loading

cenit commented Oct 7, 2019 •

edited

Loading

jasjuang commented Oct 16, 2019

luncliff commented Oct 17, 2019 via email

cbezault commented Oct 17, 2019

luncliff commented Oct 17, 2019 •

edited

Loading

cbezault commented Oct 17, 2019 •

edited

Loading

luncliff commented Oct 17, 2019 via email

traversaro commented Oct 18, 2019

cbezault commented Oct 18, 2019

cbezault commented Oct 18, 2019

[libtorch] Add new port #8318

[libtorch] Add new port #8318

Conversation

luncliff commented Sep 24, 2019 • edited Loading

Changes

Related Issues

Triplets

Unavailables

Installation

CPU Only

GPU (with CUDA)

Usage

cbezault commented Sep 26, 2019

luncliff commented Sep 26, 2019

luncliff commented Sep 26, 2019 • edited Loading

cenit commented Sep 27, 2019

luncliff commented Sep 27, 2019 • edited Loading

cenit commented Sep 27, 2019

luncliff commented Sep 27, 2019 • edited Loading

cenit commented Sep 27, 2019 • edited Loading

cenit commented Sep 27, 2019

luncliff commented Sep 27, 2019 • edited Loading

cenit commented Sep 27, 2019 • edited Loading

luncliff commented Sep 30, 2019

cenit commented Sep 30, 2019

cenit commented Oct 2, 2019

MVoz commented Oct 4, 2019

luncliff commented Oct 4, 2019 via email

luncliff commented Oct 6, 2019 • edited Loading

cenit commented Oct 6, 2019

luncliff commented Oct 7, 2019 • edited Loading

cenit commented Oct 7, 2019 • edited Loading

jasjuang commented Oct 16, 2019

luncliff commented Oct 17, 2019 via email

cbezault commented Oct 17, 2019

luncliff commented Oct 17, 2019 • edited Loading

Pros

Cons

cbezault commented Oct 17, 2019 • edited Loading

luncliff commented Oct 17, 2019 via email

traversaro commented Oct 18, 2019

cbezault commented Oct 18, 2019

cbezault commented Oct 18, 2019

luncliff commented Sep 24, 2019 •

edited

Loading

luncliff commented Sep 26, 2019 •

edited

Loading

luncliff commented Sep 27, 2019 •

edited

Loading

luncliff commented Sep 27, 2019 •

edited

Loading

cenit commented Sep 27, 2019 •

edited

Loading

luncliff commented Sep 27, 2019 •

edited

Loading

cenit commented Sep 27, 2019 •

edited

Loading

luncliff commented Oct 6, 2019 •

edited

Loading

luncliff commented Oct 7, 2019 •

edited

Loading

cenit commented Oct 7, 2019 •

edited

Loading

luncliff commented Oct 17, 2019 •

edited

Loading

cbezault commented Oct 17, 2019 •

edited

Loading