Skip to content

HADOOP-18357. Retarget solution file to VS2019 #4616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
GauthamBanasandra wants to merge 4 commits into from
Closed

HADOOP-18357. Retarget solution file to VS2019 #4616

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
33a7947
HADOOP-18357. Retarget solution file to VS2019
GauthamBanasandra Jul 23, 2022
8093b32
Run CI
GauthamBanasandra Jul 23, 2022
62f6e66
Revert "Run CI"
GauthamBanasandra Jul 23, 2022
8d02105
Undo formatting changes
GauthamBanasandra Jul 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -16,7 +15,6 @@
See the License for the specific language governing permissions and
limitations under the License.
-->

<Project DefaultTargets="CheckRequireIsal;Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Release|Win32">
Expand All @@ -39,12 +37,14 @@
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
Copy link
Member

@goiri goiri Jul 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the old number and where was it defined?

Copy link
Member Author

@GauthamBanasandra GauthamBanasandra Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the old number should've been v100. I'm not sure why it wasn't defined. Visual Studio automatically added this when I ran the command to retarget the project files.

Copy link
Member

@goiri goiri Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check if it was inherited from somewhere else?
This may trigger inconsistencies.

Copy link
Member Author

@GauthamBanasandra GauthamBanasandra Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@goiri I checked on the current trunk. There's no PlatformToolset or v100 defined anywhere -

image

image

Copy link
Member

@goiri goiri Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So how does it know which VS version to be scoped to?

Copy link
Member Author

@GauthamBanasandra GauthamBanasandra Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the retarget command from VS2019 command prompt, thus the retarget tool added the PlatformToolset version corresponding to VS2019.
The retarget command would take care of inserting the right version based on Visual Studio command prompt that we choose. Please also note that the subsequent mvn install command must be run from the same command prompt.

Copy link
Member

@goiri goiri Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we didn't have the target before; why would we add it now?

Copy link
Member Author

@GauthamBanasandra GauthamBanasandra Jul 27, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why would we add it now?

The compilation fails without the PlatformToolset when compiled with Visual Studio 2019 (I'm not sure about VS2010). I'm fine with documenting running the retarget command as a pre-requisite step prior to running the mvn install command, if we don't want to add the PlatformToolset here.

Copy link
Member

@goiri goiri Jul 27, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at HADOOP-14667
Specially: MSBuild Solution files are converted to the version of VS at build time
If this makes sense, we go with this.

Copy link
Member Author

@GauthamBanasandra GauthamBanasandra Jul 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing. Thanks for sharing this @goiri 😊. I'm able to confirm that while building the hadoop-common-project module, the vcxproj and solution files automatically get upgraded to the appropriate Visual Studio version. All thanks to

hadoop/hadoop-common-project/hadoop-common/pom.xml

Lines 844 to 923 in 95a8587

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>convert-ms-winutils</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
<arguments>
<argument>${basedir}\src\main\winutils</argument>
<argument>${project.build.directory}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>compile-ms-winutils</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>msbuild</executable>
<arguments>
<argument>${basedir}/src/main/winutils/winutils.sln</argument>
<argument>/nologo</argument>
<argument>/p:Configuration=Release</argument>
<argument>/p:OutDir=${project.build.directory}/bin/</argument>
<argument>/p:IntermediateOutputPath=${project.build.directory}/winutils/</argument>
<argument>/p:WsceConfigDir=${wsce.config.dir}</argument>
<argument>/p:WsceConfigFile=${wsce.config.file}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>convert-ms-native-dll</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
<arguments>
<argument>${basedir}\src\main\native</argument>
<argument>${project.build.directory}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>compile-ms-native-dll</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>msbuild</executable>
<arguments>
<argument>${basedir}/src/main/native/native.sln</argument>
<argument>/nologo</argument>
<argument>/p:Configuration=Release</argument>
<argument>/p:OutDir=${project.build.directory}/bin/</argument>
<argument>/p:CustomZstdPrefix=${zstd.prefix}</argument>
<argument>/p:CustomZstdLib=${zstd.lib}</argument>
<argument>/p:CustomZstdInclude=${zstd.include}</argument>
<argument>/p:RequireZstd=${require.zstd}</argument>
<argument>/p:CustomOpensslPrefix=${openssl.prefix}</argument>
<argument>/p:CustomOpensslLib=${openssl.lib}</argument>
<argument>/p:CustomOpensslInclude=${openssl.include}</argument>
<argument>/p:RequireOpenssl=${require.openssl}</argument>
<argument>/p:RequireIsal=${require.isal}</argument>
<argument>/p:CustomIsalPrefix=${isal.prefix}</argument>
<argument>/p:CustomIsalLib=${isal.lib}</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin> 

[INFO] --- exec-maven-plugin:1.3.1:exec (convert-ms-native-dll) @ hadoop-common ---
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.com
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe

Microsoft Visual Studio 2019 Version 16.11.9.
Copyright (C) Microsoft Corp. All rights reserved.
Upgrading project 'native'...
        Configuration 'Release|Win32': changing Platform Toolset to 'v142' (was 'v100').
        Configuration 'Release|x64': changing Platform Toolset to 'v142' (was 'v100').

Migration completed successfully, but some warnings were detected during migration.
For more information, see the migration report:
E:\hadoop-common-project\hadoop-common\src\main\native\UpgradeLog.htm

I'm able to fully build the trunk as well -
image

I'll abandon this PR since I'm able to build the trunk on Windows without these changes.

</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>DynamicLibrary</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
Expand Down Expand Up @@ -169,4 +169,4 @@
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>
</Project>
25 changes: 16 additions & 9 deletions hadoop-common-project/hadoop-common/src/main/native/native.vcxproj.filters
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -16,7 +15,6 @@
See the License for the specific language governing permissions and
limitations under the License.
-->

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Source Files">
Expand Down Expand Up @@ -48,19 +46,16 @@
<ClCompile Include="src\org\apache\hadoop\util\NativeCodeLoader.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\org\apache\hadoop\io\compress\lz4\lz4.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\org\apache\hadoop\io\compress\lz4\lz4hc.c">
<ClCompile Include="src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\org\apache\hadoop\io\compress\lz4\Lz4Compressor.c">
<ClCompile Include="src\org\apache\hadoop\io\compress\zlib\ZlibCompressor.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\org\apache\hadoop\io\compress\lz4\Lz4Decompressor.c">
<ClCompile Include="src\org\apache\hadoop\io\compress\zlib\ZlibDecompressor.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\org\apache\hadoop\security\JniBasedUnixGroupsMappingWin.c">
<ClCompile Include="src\org\apache\hadoop\yarn\server\nodemanager\windows_secure_container_executor.c">
<Filter>Source Files</Filter>
</ClCompile>
</ItemGroup>
Expand All @@ -86,5 +81,17 @@
<ClInclude Include="src\org_apache_hadoop.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib_ZlibCompressor.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib_ZlibDecompressor.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\org\apache\hadoop\yarn\server\nodemanager\windows_secure_container_executor.h">
<Filter>Header Files</Filter>
</ClInclude>
</ItemGroup>
</Project>
8 changes: 5 additions & 3 deletions hadoop-common-project/hadoop-common/src/main/winutils/libwinutils.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -16,7 +15,6 @@
See the License for the specific language governing permissions and
limitations under the License.
-->

<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
Expand Down Expand Up @@ -46,23 +44,27 @@
<ConfigurationType>StaticLibrary</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>StaticLibrary</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>StaticLibrary</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>StaticLibrary</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
Expand Down Expand Up @@ -188,4 +190,4 @@
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>
</Project>
8 changes: 5 additions & 3 deletions hadoop-common-project/hadoop-common/src/main/winutils/winutils.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand All @@ -16,7 +15,6 @@
See the License for the specific language governing permissions and
limitations under the License.
-->

<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
Expand Down Expand Up @@ -46,24 +44,28 @@
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
</PropertyGroup>
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
Expand Down Expand Up @@ -207,4 +209,4 @@
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>
</Project>