Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mxe] Add Windows cross-compiler support. #55

Merged
merged 1 commit into from
Jun 6, 2016

Conversation

jonpryor
Copy link
Member

@jonpryor jonpryor commented Jun 6, 2016

Certain Xamarin.Android features require that Mono be built for
Windows, e.g. the AOT compilers require a build of mono that
executes on Windows to generate the AOT native libraries.

Unfortunately, building Mono on Windows continues to be a massive
PITA. (Autotools on Windows requires Cygwin/mingw, running shell
scripts on Windows is painfully slow, it's all brittle, etc.)

To work around this pain, we instead build the Mono/Windows binaries
on OS X, via MXE, which produces a gcc-based cross-compiler
which generates Windows binaries and is executable from Unix.
This in turn requires that we have MXE, so add a
_CreateMxeToolchains target to android-toolchain.targets which
will build MXE. The installation prefix for MXE can be overridden via
the new $(AndroidMxeInstallPrefix) MSBuild property; it defaults to
$HOME/android-toolchain/mxe.

Rework the $(AndroidSupportedAbis) MSBuild property so that it
must include the "host" ABI, and add support for a new host-win64
value which will use MXE to generate 64-bit Windows binaries for
libmonosgen-2.0.dll and libMonoPosixHelper.dll.

We can't always process host-$(HostOS) because of an xbuild bug.
The scenario is that if you want to just build host-win64, the
obvious thing to do is:

cd build-tools/mono-runtimes
xbuild /p:AndroidSupportedAbis=host-win64

Alas, if host-$(HostOS) is always processed, this inexplicably
causes host-$(HostOS) to be re-rebuilt, which (1) is a waste of
time, and (2) fails -- inexplicably -- in the _BuildRuntimes target
because make(1) thinks that the configure flags have somehow changed,
which currently makes no sense at all. (When can we move to MSBuild?)

Changing $(AndroidSupportedAbis) so that host-$(HostOS) is
explicitly processed instead of implicitly processed allows working
around the above xbuild bug, as host-$(HostOS) won't be implicitly
processed on every build, but only when required.

Additionally, we add a new MSBuild task so that we can
determine of a particular process is in $PATH. This is useful
becuase listing requirements within README.md is a road to pain --
e.g. xxd(1) is required to build src/monodroid but if it's missing
it'll still build but you'll instead get a linker failure because
the monodroid_config and monodroid_machine_config symbols aren't
present. Building MXE requires that even more programs be present
within $PATH, so explicitly check for these so that useful error
messages can be generated instead of obscure ones.

Finally, a note about autotools and generating Windows native
libraries: creation of .dll files requires that an appropriate
objdump be present so it can determine if e.g. libkernel32.a is an
import library or an archive. If x86_64-w64-mingw32.static-objdump
isn't found -- e.g. because $PATH doesn't contain it -- then no .dll
files will be created, and much head scratching will occur.

To rectify this, override the OBJDUMP and DLLTOOL values when invoking
configure so that that full paths are used and $PATH use is
reduced. (Who wants x86_64-w64-mingw32.static-objdump in $PATH?)

* `host-Linux`
* `host-win64`: Cross-compile Windows 64-bit binaries from Unix.

The default value is `host-$(HostOS);armeabi-v7a`, where `$(HostOS)`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the separator here be a , instead of a ; ?

@@ -9,19 +9,23 @@
<HostOS Condition=" '$(HostOS)' == '' And '$(OS)' == 'Unix' And Exists ('/Applications') ">Darwin</HostOS>
<HostOS Condition=" '$(HostOS)' == '' And '$(OS)' == 'Unix' ">Linux</HostOS>
<HostCc Condition=" '$(HostCc)' == '' ">clang</HostCc>
<HostCxx Condition=" '$(HostCc)' == '' ">clang++</HostCxx>
<HostCxx Condition=" '$(HostCxx)' == '' ">clang++</HostCxx>
<ManagedRuntime Condition=" '$(ManagedRuntime)' == '' And '$(OS)' != 'Windows_NT' ">mono</ManagedRuntime>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest setting the default compiler names to HostCc = cc and HostCxx = c++ which on OSX are symlinked to clang and clang++, respectively, and on Linux they point to g++ and gcc. This makes the setting more "portable" since clang isn't installed by default by most Linux distributions yet.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that can be done as a separate commit.

Certain Xamarin.Android features require that Mono be built for
Windows, e.g. the [AOT compilers][aot] require a build of mono that
executes on Windows to generate the AOT native libraries.

Unfortunately, building Mono on Windows continues to be a massive
PITA. (Autotools on Windows requires Cygwin/mingw, running shell
scripts on Windows is painfully slow, it's all brittle, etc.)

To work around this pain, we instead build the Mono/Windows binaries
on OS X, via [MXE][mxe], which produces a gcc-based cross-compiler
which generates Windows binaries and is executable from Unix.
This in turn requires that we have MXE, so add a
`_CreateMxeToolchains` target to `android-toolchain.targets` which
will build MXE. The installation prefix for MXE can be overridden via
the new `$(AndroidMxeInstallPrefix)` MSBuild property; it defaults to
`$HOME/android-toolchain/mxe`.

Rework the `$(AndroidSupportedAbis)` MSBuild property so that it
must include the "host" ABI, and add support for a new `host-win64`
value which will use MXE to generate 64-bit Windows binaries for
libmonosgen-2.0.dll and libMonoPosixHelper.dll.

We can't always process `host-$(HostOS)` because of an xbuild bug.
The scenario is that if you want to just build `host-win64`, the
obvious thing to do is:

	cd build-tools/mono-runtimes
	xbuild /p:AndroidSupportedAbis=host-win64

Alas, if `host-$(HostOS)` is always processed, this inexplicably
causes `host-$(HostOS)` to be re-rebuilt, which (1) is a waste of
time, and (2) fails -- inexplicably -- in the `_BuildRuntimes` target
because make(1) thinks that the configure flags have somehow changed,
which currently makes no sense at all. (When can we move to MSBuild?)

Changing `$(AndroidSupportedAbis)` so that `host-$(HostOS)` is
explicitly processed instead of implicitly processed allows working
around the above xbuild bug, as `host-$(HostOS)` won't be implicitly
processed on every build, but only when required.

Additionally, we add a new <Which/> MSBuild task so that we can
determine if a particular program is in `$PATH`. This is useful
because listing requirements within README.md is a road to pain --
e.g. xxd(1) is required to build `src/monodroid` but if it's missing
it'll still *build* but you'll instead get a *linker* failure because
the `monodroid_config` and `monodroid_machine_config` symbols aren't
present. Building MXE requires that even more programs be present
within $PATH, so explicitly check for these so that *useful* error
messages can be generated instead of obscure ones.

Finally, a note about autotools and generating Windows native
libraries: creation of `.dll` files *requires* that an appropriate
objdump be present so it can determine if e.g. `libkernel32.a` is an
import library or an archive. If `x86_64-w64-mingw32.static-objdump`
isn't found -- e.g. because $PATH doesn't contain it -- then no `.dll`
files will be created, and much head scratching will occur.

To rectify this, override the OBJDUMP and DLLTOOL values when invoking
`configure` so that that full paths are used and `$PATH` use is
reduced.  (Who wants `x86_64-w64-mingw32.static-objdump` in `$PATH`?)

[aot]: https://developer.xamarin.com/releases/android/xamarin.android_5/xamarin.android_5.1/#AOT_Support
[mxe]: http://mxe.cc/
@grendello grendello merged commit 0c073f6 into dotnet:master Jun 6, 2016
jonpryor added a commit that referenced this pull request Jun 6, 2016
[Configuration] Change default $(HostCc), $(HostCxx) values

As [Marek mentioned in PR 55][0], the default C and C++ compiler
program names should be `cc` and `c++`, not `clang` and `clang++`,
as `cc` and `c++` are more portable across various Unixes.

[0]: #55 (comment)
grendello pushed a commit that referenced this pull request Jun 6, 2016
[Configuration] Change default $(HostCc), $(HostCxx) values

As [Marek mentioned in PR 55][0], the default C and C++ compiler
program names should be `cc` and `c++`, not `clang` and `clang++`,
as `cc` and `c++` are more portable across various Unixes.

[0]: #55 (comment)
radical pushed a commit that referenced this pull request May 8, 2018
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=42476

Bug #42476 notes that the `string.Format()` within the JNI reference
count assertions contributes to process memory allocations which the
profiler reports, and those generated `strings` are pointless because
they're not used unless the assertion fails.

This is a fair point, outside of whether or not the assertions should
be present at all...

Improve the `JniRuntime.JniObjectReferenceManager` JNI reference
assertions so that they return early if the assertion wouldn't fire.
This avoids the unnecessary string allocation.
grendello pushed a commit that referenced this pull request Aug 12, 2020
Fixes: dotnet/android-libzipsharp#64

Context: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1139578
Context: https://liquid.microsoft.com/Web/Object/Read/ms.security/Requirements/Microsoft.Security.SystemsADM.10039#guide

Changes: dotnet/android-libzipsharp@1.0.10...1.0.20

  * dotnet/android-libzipsharp@1752f95: Statically Linux libzip.so (#70)
  * dotnet/android-libzipsharp@2b1762f: Fix Dll SearchPath to include Assembly Directory. (#69)
  * dotnet/android-libzipsharp@28b4639: Merge pull request #68 from dellis1972/fixwindowssearch
  * dotnet/android-libzipsharp@fdabcda: Fix Windows to look in Assembly Directory for 32bit dll.
  * dotnet/android-libzipsharp@b332af0: Fix Native Crash on Windows. (#67)
  * dotnet/android-libzipsharp@96eb5e3: Use DefaultDllImportSearchPathsAttribute (#66)
  * dotnet/android-libzipsharp@755a42a: Bump libzip to 1.7.3 and go back to mingw (#65)
  * dotnet/android-libzipsharp@5ae5e70: Merge pull request #60 from xamarin/msvc-static-link
  * dotnet/android-libzipsharp@30ff680: Build libzip with static CRT and VS2019
  * dotnet/android-libzipsharp@bad320e: Merge pull request #59 from dellis1972/theswitcharoo
  * dotnet/android-libzipsharp@d7bc2c5: Merge pull request #58 from xamarin/optimize-winbuild
  * dotnet/android-libzipsharp@34dc213: Make 64 bit Linux native lib the default.
  * dotnet/android-libzipsharp@d3aad35: fixup! Optimize libzip build
  * dotnet/android-libzipsharp@0970a01: Optimize libzip build
  * dotnet/android-libzipsharp@d321af1: Merge pull request #57 from xamarin/fix-win32-packaging
  * dotnet/android-libzipsharp@80b739d: Fix a typo which caused 64-bit dll to be packaged for 32-bit Windows
  * dotnet/android-libzipsharp@1665db0: Bump the version (to 1.0.12), to prepare to release (#56)
  * dotnet/android-libzipsharp@dd5e939: Throw exception instead of silently failing if zip save/close fails (#54)
  * dotnet/android-libzipsharp@2df5b16: Fix enumerating zip with deleted entries (#53)
  * dotnet/android-libzipsharp@a042554: Add .editorconfig, copied from xamarin-android (#55)
  * dotnet/android-libzipsharp@a0973d4: Bump libzip to 1.6.1 (#49)

Changes: nih-at/libzip@rel-1-5-1...v1.7.3

  * Context: https://libzip.org/news/release-1.7.3.html
  * Context: https://libzip.org/news/release-1.7.2.html
  * Context: https://libzip.org/news/release-1.7.1.html
  * Context: https://libzip.org/news/release-1.7.0.html

Two primary changes of note in this xamarin/LibZipSharp bump:

 1. Bump to `libzip` 1.7.3, which contains numerious fixes and
    improvements over the previously used 1.5.1 release.

 2. Use of `DefaultDllImportSearchPathsAttribute` so that native
    library dependencies are loaded securely, i.e. w/o allowing
    "other" libraries to be loaded from unsafe directories.

Unfortunately, the `libzip` bump itself caused issues, PR #4751 and
PR #4937 each had integration tests "randomly" SIGSEGV.  This was
eventually tracked down to a bug within `libzip` itself, fixed at:

  * nih-at/libzip#202
jonpryor pushed a commit that referenced this pull request Aug 20, 2020
Fixes: dotnet/android-libzipsharp#64

Context: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1139578
Context: https://liquid.microsoft.com/Web/Object/Read/ms.security/Requirements/Microsoft.Security.SystemsADM.10039#guide

Changes: dotnet/android-libzipsharp@1.0.10...1.0.20

  * dotnet/android-libzipsharp@1752f95: Statically Linux libzip.so (#70)
  * dotnet/android-libzipsharp@2b1762f: Fix Dll SearchPath to include Assembly Directory. (#69)
  * dotnet/android-libzipsharp@28b4639: Merge pull request #68 from dellis1972/fixwindowssearch
  * dotnet/android-libzipsharp@fdabcda: Fix Windows to look in Assembly Directory for 32bit dll.
  * dotnet/android-libzipsharp@b332af0: Fix Native Crash on Windows. (#67)
  * dotnet/android-libzipsharp@96eb5e3: Use DefaultDllImportSearchPathsAttribute (#66)
  * dotnet/android-libzipsharp@755a42a: Bump libzip to 1.7.3 and go back to mingw (#65)
  * dotnet/android-libzipsharp@5ae5e70: Merge pull request #60 from xamarin/msvc-static-link
  * dotnet/android-libzipsharp@30ff680: Build libzip with static CRT and VS2019
  * dotnet/android-libzipsharp@bad320e: Merge pull request #59 from dellis1972/theswitcharoo
  * dotnet/android-libzipsharp@d7bc2c5: Merge pull request #58 from xamarin/optimize-winbuild
  * dotnet/android-libzipsharp@34dc213: Make 64 bit Linux native lib the default.
  * dotnet/android-libzipsharp@d3aad35: fixup! Optimize libzip build
  * dotnet/android-libzipsharp@0970a01: Optimize libzip build
  * dotnet/android-libzipsharp@d321af1: Merge pull request #57 from xamarin/fix-win32-packaging
  * dotnet/android-libzipsharp@80b739d: Fix a typo which caused 64-bit dll to be packaged for 32-bit Windows
  * dotnet/android-libzipsharp@1665db0: Bump the version (to 1.0.12), to prepare to release (#56)
  * dotnet/android-libzipsharp@dd5e939: Throw exception instead of silently failing if zip save/close fails (#54)
  * dotnet/android-libzipsharp@2df5b16: Fix enumerating zip with deleted entries (#53)
  * dotnet/android-libzipsharp@a042554: Add .editorconfig, copied from xamarin-android (#55)
  * dotnet/android-libzipsharp@a0973d4: Bump libzip to 1.6.1 (#49)

Changes: nih-at/libzip@rel-1-5-1...v1.7.3

  * Context: https://libzip.org/news/release-1.7.3.html
  * Context: https://libzip.org/news/release-1.7.2.html
  * Context: https://libzip.org/news/release-1.7.1.html
  * Context: https://libzip.org/news/release-1.7.0.html

Two primary changes of note in this xamarin/LibZipSharp bump:

 1. Bump to `libzip` 1.7.3, which contains numerious fixes and
    improvements over the previously used 1.5.1 release.

 2. Use of `DefaultDllImportSearchPathsAttribute` so that native
    library dependencies are loaded securely, i.e. w/o allowing
    "other" libraries to be loaded from unsafe directories.

Unfortunately, the `libzip` bump itself caused issues, PR #4751 and
PR #4937 each had integration tests "randomly" SIGSEGV.  This was
eventually tracked down to a bug within `libzip` itself, fixed at:

  * nih-at/libzip#202
@github-actions github-actions bot locked and limited conversation to collaborators Feb 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants