Skip to content

[VL] Set Arrow_SOURCE to AUTO to allow using system arrow libs#6325

Merged
zhouyuan merged 1 commit intoapache:mainfrom
philo-he:fix-thrift
Jul 15, 2024
Merged

[VL] Set Arrow_SOURCE to AUTO to allow using system arrow libs#6325
zhouyuan merged 1 commit intoapache:mainfrom
philo-he:fix-thrift

Conversation

@philo-he
Copy link
Copy Markdown
Member

@philo-he philo-he commented Jul 3, 2024

What changes were proposed in this pull request?

Depends on the fix in facebookincubator/velox#10355.

How was this patch tested?

CI.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jul 3, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@Yohahaha
Copy link
Copy Markdown
Contributor

Yohahaha commented Jul 3, 2024

      set_target_properties(
        Arrow::${LIB_NAME}
        PROPERTIES IMPORTED_LOCATION ${ARROW_LIB_${LIB_NAME}}
                   INTERFACE_INCLUDE_DIRECTORIES ${ARROW_HOME}/install/include)

if use system arrow, seems gluten still use ${ARROW_HOME}/install/include as include dir.

@philo-he
Copy link
Copy Markdown
Member Author

philo-he commented Jul 4, 2024

      set_target_properties(
        Arrow::${LIB_NAME}
        PROPERTIES IMPORTED_LOCATION ${ARROW_LIB_${LIB_NAME}}
                   INTERFACE_INCLUDE_DIRECTORIES ${ARROW_HOME}/install/include)

if use system arrow, seems gluten still use ${ARROW_HOME}/install/include as include dir.

@Yohahaha, thanks for your comment! I just removed the setting for arrow include directories, as they are actually not needed in Gluten.

@Yohahaha
Copy link
Copy Markdown
Contributor

Yohahaha commented Jul 4, 2024

set(ARROW_INCLUDE_DIR "${ARROW_INSTALL_DIR}/include")

this line should also be removed.

@philo-he
Copy link
Copy Markdown
Member Author

philo-he commented Jul 4, 2024

set(ARROW_INCLUDE_DIR "${ARROW_INSTALL_DIR}/include")

this line should also be removed.

@Yohahaha, yes, just removed. Thanks!

@philo-he philo-he marked this pull request as ready for review July 13, 2024 04:12
@philo-he
Copy link
Copy Markdown
Member Author

@Yohahaha, the corresponding velox patch has been merged. Do you have any other comment on this gluten PR?

@Yohahaha
Copy link
Copy Markdown
Contributor

@Yohahaha, the corresponding velox patch has been merged. Do you have any other comment on this gluten PR?

If arrow has installed in other places, seems we still set ARROW_HOME under velox path, could you clarify?

if(${CMAKE_BUILD_TYPE} STREQUAL "Debug")
  set(ARROW_HOME
      ${VELOX_HOME}/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/
  )
else()
  set(ARROW_HOME
      ${VELOX_HOME}/_build/release/CMake/resolve_dependency_modules/arrow/arrow_ep
  )
endif()

@Yohahaha
Copy link
Copy Markdown
Contributor

@Yohahaha, the corresponding velox patch has been merged. Do you have any other comment on this gluten PR?

If arrow has installed in other places, seems we still set ARROW_HOME under velox path, could you clarify?

if(${CMAKE_BUILD_TYPE} STREQUAL "Debug")
  set(ARROW_HOME
      ${VELOX_HOME}/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/
  )
else()
  set(ARROW_HOME
      ${VELOX_HOME}/_build/release/CMake/resolve_dependency_modules/arrow/arrow_ep
  )
endif()

mayby it's not an issue, find_library should find correct arrow lib if ARROW_HOME is wrong.

@philo-he philo-he changed the title Fix thrift lib not linked issue when system arrow libs are used [VL] Set Arrow_SOURCE to AUTO to allow using system arrow libs Jul 15, 2024
@philo-he
Copy link
Copy Markdown
Member Author

@Yohahaha, thanks for your comment! The arrow home is set to let cmake firstly find arrow libs from that path. If it does not exist (because velox finds the lib from system, then it doesn't build arrow from source), cmake will find the libs from system. Though we always expect cmake in Gluten/Velox finds the libs from system now, it may be good to still keep the arrow home path for debugging (e.g., by export Arrow_SOURCE=BUNDELD, we can mandatorily use bundled arrow to exclude any issues due to system arrow).

@zhouyuan zhouyuan merged commit b93f254 into apache:main Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants