Fix Thrift dependency when using system Arrow by philo-he · Pull Request #10355 · facebookincubator/velox

philo-he · 2024-07-01T06:39:27Z

With this 0d80228 commit included, velox will resolve dependency by firstly
finding lib arrow from system. In this path, linking with lib thrift is lacking, which
causes thrift header not found issue.

With this pr, velox will try to find thrift lib. If it is not found, velox will build arrow
from source with thrift bundled. If found, lib arrow will be linked with it. This pr
also lets setup scripts install thrift bundled in arrow.

netlify · 2024-07-01T06:39:43Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`a547494`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/668dd7cc4128990008058520

philo-he · 2024-07-01T06:52:02Z

@assignUser, @majetideepak, @pedroerp, this is an urgent fix for CI issue. Please prioritize to review it.

CMake/FindArrow.cmake

majetideepak · 2024-07-01T09:14:10Z

@philo-he why was Thrift not installed from source in the original PR?

philo-he · 2024-07-01T09:31:18Z

@philo-he why was Thrift not installed from source in the original PR?

@majetideepak, thanks for your review. CMake only installs arrow libs, not its dependency libs. I will fix it in another pr. We can firstly land this patch to avoid blocking other pr.

BTW, just fixed your comment.

majetideepak · 2024-07-01T10:47:14Z

@philo-he a simpler workaround to unblock CI is to bundle Arrow like before.
I opened a PR here #10360
Can you help fix the broken arrow thrift dependency? Thanks!

majetideepak · 2024-07-01T11:02:08Z

My worry with installing thrift is that it might conflict with fbthrift. Let's first validate the thrift install before we enable auto for Arrow source in CI.

philo-he · 2024-07-01T11:14:09Z

My worry with installing thrift is that it might conflict with fbthrift. Let's first validate the thrift install before we enable auto for Arrow source in CI.

@majetideepak, I see. I will continue the fix for thrift dependency here. Thanks!

tigrux · 2024-07-02T00:53:18Z

Hello, if this commit, as it is, is merged then it will break the ubuntu builds.
In setup-ubuntu.sh, the package libthrift-dev needs to be removed from the list of packages that are installed.
Otherwise, the build will use the wrong thrift.

scripts/setup-centos9.sh

majetideepak

@philo-he changes look good to me. Were you able to verify the fix locally?
We can revert the CI job for Linux release with adapters to use auto for Arrow.

philo-he · 2024-07-02T14:27:55Z

Hello, if this commit, as it is, is merged then it will break the ubuntu builds. In setup-ubuntu.sh, the package libthrift-dev needs to be removed from the list of packages that are installed. Otherwise, the build will use the wrong thrift.

@tigrux, I just removed libthrift-dev from setup-ubuntu.sh. Thanks for your comment!

philo-he · 2024-07-02T14:29:45Z

@philo-he changes look good to me. Were you able to verify the fix locally? We can revert the CI job for Linux release with adapters to use auto for Arrow.

@majetideepak, yes, I have verified locally. And just reverted Arrow from BUNDLED to AUTO. Thanks!

majetideepak · 2024-07-03T01:39:42Z

Looks good to me. I would like @assignUser to make another pass as well.

philo-he · 2024-07-03T02:59:00Z

Looks good to me. I would like @assignUser to make another pass as well.

@majetideepak, I note assignUser's Github state is on vacation.
@rui-mo, do you have any comment?

rui-mo

Thanks. Looks good to me.

majetideepak

Thanks, @philo-he

philo-he · 2024-07-03T06:19:23Z

@philo-he can you address this issue as well? I want to avoid a conflict. #10378 (comment) CC: @czentgr

@majetideepak, just fixed that comment in the latest commit. Thanks!

Yohahaha · 2024-07-03T09:11:15Z

@philo-he do you verified this PR in Gluten?

philo-he · 2024-07-03T09:18:19Z

@philo-he do you verified this PR in Gluten?

@Yohahaha, just tested in my local. I'm creating a Gluten PR to see whether Gluten CI can pass. Thanks for your reminding!

philo-he · 2024-07-04T00:36:06Z

No issue is found so far.
@bikramSingh91, could you merge this pr if you have no comment? Thanks!

CMake/FindArrow.cmake

czentgr

Thank you! I suppose you should also squash the commits?

czentgr · 2024-07-09T22:00:11Z

scripts/setup-centos9.sh

+
+    # Install thrift.
+    cd _build/thrift_ep-prefix/src/thrift_ep-build
+    $SUDO cmake --install ./ --prefix /usr/local/


$SUDO isn't defined here. Don't you also need to add the definition in this script like it is done for Ubuntu (and in the functions that call cmake etc)? It isn't inherited from setup-helper-functions.sh.
On Ubuntu it does this:

SUDO="${SUDO:-"sudo --preserve-env"}"

Nit: do we want to use ${SUDO} like in the other scripts?

@czentgr, I just removed the use of ${SUDO} in this setup-centos9.sh and ${SUDO} is still not defined with sudo, consistent with other places where cmake_install is used in this script.
BTW, I find sudo is not installed in some centos-9 images used for velox CI. So just defining ${SUDO} without installing sudo can cause execution failure.

facebook-github-bot · 2024-07-10T18:16:11Z

@pedroerp has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

philo-he · 2024-07-12T00:24:07Z

@pedroerp, could you please merge this pr? The CI failure is not relevant to this pr. Thanks!

facebook-github-bot · 2024-07-12T15:13:42Z

@pedroerp merged this pull request in c3dd274.

conbench-facebook · 2024-07-12T15:39:45Z

Conbench analyzed the 1 benchmark run on commit c3dd2742.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Summary: With this facebookincubator@0d80228 commit included, velox will resolve dependency by firstly finding lib arrow from system. In this path, linking with lib thrift is lacking, which causes thrift header not found issue. With this pr, velox will try to find thrift lib. If it is not found, velox will build arrow from source with thrift bundled. If found, lib arrow will be linked with it. This pr also lets setup scripts install thrift bundled in arrow. Pull Request resolved: facebookincubator#10355 Reviewed By: bikramSingh91 Differential Revision: D59595645 Pulled By: pedroerp fbshipit-source-id: 9f26eee8347a6c650e70163554fa9b2b72f5bfac

assignUser

Thanks, this is a feasible workaround for now. 👍

assignUser · 2024-07-17T03:42:13Z

scripts/setup-ubuntu.sh

+      -DCMAKE_INSTALL_PREFIX=/usr/local \
+      -DCMAKE_BUILD_TYPE=Release \
+      -DARROW_BUILD_STATIC=ON \
+      -DThrift_SOURCE=BUNDLED


If we remove this we should be able to keep the system thrift and also use that for velox.

@assignUser, seems thrift is not explicitly installed into system in any other places. Here, with Thrift_SOURCE set to BUNDLED and the following code to install it, velox build can use the thrift after setup script is executed. Right?

Yes, nice trick installing arrows build thrift :D

…tor#10355)" This reverts commit c3dd274.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 1, 2024

philo-he mentioned this pull request Jul 1, 2024

CI Linux release with adapters is broken #10352

Closed

philo-he force-pushed the fix-thrift branch from 7d472f3 to 44b04c2 Compare July 1, 2024 07:05

philo-he changed the title ~~Fix build issue related to thrift~~ Fix CI build issue related to thrift Jul 1, 2024

majetideepak requested a review from assignUser July 1, 2024 09:09

majetideepak reviewed Jul 1, 2024

View reviewed changes

CMake/FindArrow.cmake Outdated Show resolved Hide resolved

philo-he mentioned this pull request Jul 2, 2024

[VL] Daily Update Velox Version (2024_06_30) apache/gluten#6284

Merged

philo-he changed the title ~~Fix CI build issue related to thrift~~ Fix build issue related to thrift Jul 2, 2024

majetideepak reviewed Jul 2, 2024

View reviewed changes

scripts/setup-centos9.sh Outdated Show resolved Hide resolved

majetideepak reviewed Jul 2, 2024

View reviewed changes

philo-he force-pushed the fix-thrift branch 2 times, most recently from 99365f8 to a28deb3 Compare July 2, 2024 14:19

rui-mo reviewed Jul 3, 2024

View reviewed changes

majetideepak approved these changes Jul 3, 2024

View reviewed changes

majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jul 3, 2024

majetideepak changed the title ~~Fix build issue related to thrift~~ Install bundled thrift after Arrow system install Jul 3, 2024

majetideepak changed the title ~~Install bundled thrift after Arrow system install~~ Install bundled Thrift after Arrow system install Jul 3, 2024

majetideepak removed the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jul 3, 2024

majetideepak changed the title ~~Install bundled Thrift after Arrow system install~~ Fix thrift dependency when using system Arrow Jul 3, 2024

majetideepak changed the title ~~Fix thrift dependency when using system Arrow~~ Fix Thrift dependency when using system Arrow Jul 3, 2024

majetideepak approved these changes Jul 3, 2024

View reviewed changes

majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Jul 3, 2024

majetideepak requested a review from czentgr July 3, 2024 06:27

philo-he mentioned this pull request Jul 3, 2024

[VL] Using system arrow does not trigger find thrift apache/gluten#6321

Closed

philo-he mentioned this pull request Jul 3, 2024

[VL] Set Arrow_SOURCE to AUTO to allow using system arrow libs apache/gluten#6325

Merged

philo-he force-pushed the fix-thrift branch from 83f28eb to feeadab Compare July 5, 2024 03:42

majetideepak reviewed Jul 6, 2024

View reviewed changes

CMake/FindArrow.cmake Show resolved Hide resolved

czentgr reviewed Jul 9, 2024

View reviewed changes

philo-he force-pushed the fix-thrift branch 2 times, most recently from ba9382b to 188cf14 Compare July 10, 2024 00:21

Initial

a547494

philo-he force-pushed the fix-thrift branch from 188cf14 to a547494 Compare July 10, 2024 00:37

czentgr approved these changes Jul 10, 2024

View reviewed changes

facebook-github-bot closed this in c3dd274 Jul 12, 2024

facebook-github-bot added the Merged label Jul 12, 2024

assignUser reviewed Jul 17, 2024

View reviewed changes

bdice added a commit to bdice/velox that referenced this pull request Dec 19, 2024

Revert "Fix Thrift dependency when using system Arrow (facebookincuba…

4ddb6e9

…tor#10355)" This reverts commit c3dd274.

karthikeyann pushed a commit to karthikeyann/velox that referenced this pull request Sep 2, 2025

Revert "Fix Thrift dependency when using system Arrow (facebookincuba…

dd722f3

…tor#10355)" This reverts commit c3dd274.

Conversation

philo-he commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox canceled.

Uh oh!

philo-he commented Jul 1, 2024

Uh oh!

Uh oh!

majetideepak commented Jul 1, 2024

Uh oh!

philo-he commented Jul 1, 2024

Uh oh!

majetideepak commented Jul 1, 2024

Uh oh!

majetideepak commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philo-he commented Jul 1, 2024

Uh oh!

tigrux commented Jul 2, 2024

Uh oh!

Uh oh!

majetideepak left a comment

Choose a reason for hiding this comment

Uh oh!

philo-he commented Jul 2, 2024

Uh oh!

philo-he commented Jul 2, 2024

Uh oh!

majetideepak commented Jul 3, 2024

Uh oh!

philo-he commented Jul 3, 2024

Uh oh!

rui-mo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majetideepak left a comment

Choose a reason for hiding this comment

Uh oh!

philo-he commented Jul 3, 2024

Uh oh!

Yohahaha commented Jul 3, 2024

Uh oh!

philo-he commented Jul 3, 2024

Uh oh!

philo-he commented Jul 4, 2024

Uh oh!

Uh oh!

czentgr left a comment

Choose a reason for hiding this comment

Uh oh!

czentgr Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

philo-he Jul 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jul 10, 2024

Uh oh!

philo-he commented Jul 12, 2024

Uh oh!

facebook-github-bot commented Jul 12, 2024

Uh oh!

conbench-facebook bot commented Jul 12, 2024

Uh oh!

assignUser left a comment

Choose a reason for hiding this comment

Uh oh!

assignUser Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

philo-he Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

assignUser Jul 17, 2024

philo-he commented Jul 1, 2024 •

edited

Loading

netlify bot commented Jul 1, 2024 •

edited

Loading

majetideepak commented Jul 1, 2024 •

edited

Loading

rui-mo left a comment •

edited

Loading

philo-he Jul 10, 2024 •

edited

Loading