clang produces larger binaries than gcc for arm7

[froydnj]

We've been working on compiling Firefox for Android with clang lately and today I finally got everything to build with NDK r12 and clang today. All the necessary patches aren't committed yet, so replicating this would be a bit tricky at the moment, but the end result is that our main library, libxul.so, compiled with clang is about 10% bigger than when we use gcc (text size, as measured by size(1)). Our mobile team is quite concerned about our APK size already, and telling them that a mandated compiler switch would tack on another 3MB+ of size...well, let's just say they wouldn't be happy. 😉

This result was pretty surprising, as we were expecting similar performance and even hoping for a codesize win. I have verified that the build flags were identical in both builds (same -O options, -fno-exceptions turned on, -ffunction-sections + -Wl,--gc-sections turned on, both builds using libc++, etc.). It is entirely possible that I've missed some key clang option, though I can verify that using -funique-section-names doesn't change the result.

Judging from a quick skim of diff'd nm output, the clang build has many more symbols, especially template methods. I can't tell whether this is an effect of inlining heuristics, or of clang somehow keeping those symbols around when they're not actually used.

I apologize for not being able to provide replication instructions; I'll try to get all the necessary patches committed soon. We had a workweek last week, and that's been a bit disruptive for getting code reviewed and committed.

Is this sort of difference something that you've seen in your own testing as well, or does this sound like an outlier?

[enh]

measured across the whole platform, the switch was a wash --- some devices were a few MiB smaller, some a few MiB larger. and that's out of ~1GiB+!

i think the ChromeOS folks are investigating switching atm, so i'll see if we can out more about their experience --- libwebview_chromium.so is orders of magnitude larger than the next largest library on the system (and we take prebuilts, so they won't have been affected by the rest of the platform switching).

out of interest --- do you build for x86/x86-64 too? do you see similar results there or is that better/worse?

[froydnj]

We build for x86 as well, but I haven't measured that. I will do that and report back.

[stephenhines]

Which -O flags are you passing? Are you using -Os or just -O2? It would be really useful to know the common set of flags. One other question is are you stripping these binaries? Clang has larger debug info than GCC, but that shouldn't be a problem for shipped builds.

[froydnj]

We use -Os everywhere; the other interesting flags that we compile with are: -fno-short-enums -fno-exceptions -march=armv7-a -mthumb -mfpu=vfp -mfloat-abi=softfp -mno-unaligned-access -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-math-errno.

We link with -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -Wl,--icf=safe -Wl,--hash-style=sysv -Wl,--gc-sections.

The measurements are done with unstripped binaries, but the measurements only look at things that wouldn't get stripped away; in the below output, build-android is a build with clang and build-android-gcc is a build with gcc:

froydnj@thor:/opt/build/froydnj/build-android$ size toolkit/library/libxul.so
   text    data     bss     dec     hex filename
37104041    2331336  192928 39628305    25cae11 toolkit/library/libxul.so
froydnj@thor:/opt/build/froydnj/build-android-gcc$ size toolkit/library/libxul.so
   text    data     bss     dec     hex filename
33482353    2358544  185792 36026689    225b941 toolkit/library/libxul.so

Or, if you prefer:

froydnj@thor:/opt/build/froydnj/build-android$ readelf -SW toolkit/library/libxul.so |egrep 'Size|text'
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [12] .text             PROGBITS        003537c0 3537c0 1b78544 00  AX  0   0 64
froydnj@thor:/opt/build/froydnj/build-android-gcc$ readelf -SW toolkit/library/libxul.so |egrep 'Size|text'
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [12] .text             PROGBITS        00344140 344140 1808bd0 00  AX  0   0 64

The first measurement with size includes a bunch of other things, like read-only strings and so forth. The second measurement with readelf measures the size of the .text section straight-up, and shows a size increase of ~3.5MB, which is a ~14% increase of just executable code.

[jmgao]

Could you upload the gcc and clang-built binaries somewhere for comparison?

[froydnj]

Could you upload the gcc and clang-built binaries somewhere for comparison?

Sure. Is it a problem if I strip debug information but leave symbols? My upload speed is terrible.

[jmgao]

I think that should be fine.

[froydnj]

I've uploaded them at https://people.mozilla.org/~nfroyd/libxuls.tar.bz2 . That'll unpack into:

drwxr-xr-x froydnj/froydnj   0 2016-06-25 02:07 libxul-comparison/
-rwxr-xr-x froydnj/froydnj 69143396 2016-06-25 02:06 libxul-comparison/libxul-clang.so
-rwxr-xr-x froydnj/froydnj 66911236 2016-06-25 02:07 libxul-comparison/libxul-gcc.so

Please let me know if you need anything else.

[DanAlbert]

FYI, it sounds like the ChromeOS folks might have found something. It seems that clang is usually better or equivalent, but there are some pathological cases where it produces much larger binaries.

[stephenhines]

The ChromeOS issues are actually invalid, since they were omitting -mthumb, which isn't on by default for the clang target they are using. When they recompiled for Thumb-2, they got a 5.5% reduction vs. GCC on code size. Can you check to make sure that you are indeed using -mthumb throughout your build (and not putting -mno-thumb anywhere)?

[froydnj]

Can you check to make sure that you are indeed using -mthumb throughout your build (and not putting -mno-thumb anywhere)?

We are using -mthumb everywhere; -mno-thumb is used nowhere in our build tree. After a build, running the following command:

readelf -sW toolkit/library/libxul.so |grep FUNC |awk '$2 ~ /[048c]$/ {print}'

(that is, show me all the function symbols that have arm addresses; thumb addresses would have the lowest bit set and would match the regex [159d]$) shows only undefined symbols that will be resolved via dynamic linking, or a few symbols such as __adddf3 and the like from libgcc or compiler-rt. There's also a very few handwritten FFI stubs that don't come in thumb versions. But all of these functions only add up to ~55K or so.

Eyeballing a build build log also shows that we are passing -mthumb everywhere that matters.

[DoDoENT]

In our company, we use exactly the same compile flags for both GCC and Clang, as described in #21, yet binaries produced by Clang are still significantly larger.

[bog-dan-ro]

Any news on this matter?
I just tested Qt libs with clang (r13-beta2) and g++ (r10e) and clang stripped libs are +40% larger than g++ ones (of course using the same compile & link flags), which IMHO is way too much. Even supposing clang bins are +40% faster, which I sincerely doubt that, being +40% larger is still a big issue.

If anyone wants to test it, I can provide the steps to compile Qt to reproduce this problem.

[DanAlbert]

@pirama-arumuga-nainar: You'd looked at some of the other qt issues, could you take a look at the 40% (wat) size increase? Anything that bad probably has some pretty obvious causes...

[pirama-arumuga-nainar]

@bog-dan-ro Can you post instructions to reproduce? It'd be great if you can isolate it to one or few object files with egregious increase, and post build commands with both gcc and clang.

[bog-dan-ro]

@pirama-arumuga-nainar they are almost the same instructions as #34 (comment)

Sadly I compared the final stripped libs ...
This are the steps you need to do to build the entire Qt (some modules are less than 40% and some more):

$ git clone git://code.qt.io/qt/qtbase.git -b 5.7.1
$ cd qtbase

Final step is to configure & compile Qt, you'll need Android NDK & SDK (with API-16 installed):

Build for with clang:

$ ./configure -developer-build -release -xplatform android-clang -nomake tests -nomake examples -android-ndk /path/to/NDKr13 -android-sdk /path/to/SDK
$ make -j8 (you'll need to wait a few minutes)

The libs will be placed in libs

Build for with g++ (please use NDKr10e):

$ ./configure -developer-build -release -xplatform android-g++ -nomake tests -nomake examples -android-ndk /path/to/NDKr10e -android-sdk /path/to/SDK
$ make -j8 (you'll need to wait a few minutes)

Same as above, the libs will be placed in libs

I'll do the same thing tomorrow, maybe I'll spot the object files that are way too big.

Edit: qtbase is enough
Here are my results (after I stripped them[1]):

bogdan@zmeu:~/work/qt/qtbase/lib$ ls -l *.so
-rwxr-xr-x 1 bogdan bogdan   13556 oct 27 11:09 libQt5Concurrent.so
-rwxr-xr-x 1 bogdan bogdan 3455648 oct 27 11:09 libQt5Core.so
-rwxr-xr-x 1 bogdan bogdan 2508172 oct 27 11:09 libQt5Gui.so
-rwxr-xr-x 1 bogdan bogdan  693468 oct 27 11:09 libQt5Network.so
-rwxr-xr-x 1 bogdan bogdan  201944 oct 27 11:09 libQt5OpenGL.so
-rwxr-xr-x 1 bogdan bogdan  234704 oct 27 11:09 libQt5PrintSupport.so
-rwxr-xr-x 1 bogdan bogdan  115924 oct 27 11:09 libQt5Sql.so
-rwxr-xr-x 1 bogdan bogdan  132440 oct 27 11:09 libQt5Test.so
-rwxr-xr-x 1 bogdan bogdan 3499412 oct 27 11:09 libQt5Widgets.so
-rwxr-xr-x 1 bogdan bogdan  120008 oct 27 11:09 libQt5Xml.so

bogdan@zmeu:~/work/qt/qtbase-clang/lib$ ls -l *.so
-rwxr-xr-x 1 bogdan bogdan   14928 oct 27 11:09 libQt5Concurrent.so
-rwxr-xr-x 1 bogdan bogdan 4232656 oct 27 11:09 libQt5Core.so
-rwxr-xr-x 1 bogdan bogdan 3574704 oct 27 11:09 libQt5Gui.so
-rwxr-xr-x 1 bogdan bogdan 1096904 oct 27 11:09 libQt5Network.so
-rwxr-xr-x 1 bogdan bogdan  257608 oct 27 11:09 libQt5OpenGL.so
-rwxr-xr-x 1 bogdan bogdan  323092 oct 27 11:09 libQt5PrintSupport.so
-rwxr-xr-x 1 bogdan bogdan  188136 oct 27 11:09 libQt5Sql.so
-rwxr-xr-x 1 bogdan bogdan  176812 oct 27 11:09 libQt5Test.so
-rwxr-xr-x 1 bogdan bogdan 4968236 oct 27 11:09 libQt5Widgets.so
-rwxr-xr-x 1 bogdan bogdan  201532 oct 27 11:09 libQt5Xml.so

For instance libQt5Core.so is just ~20% larger but libQt5Sql.so is over 60% (same libQt5Xml.so)

[1] $ find *.so | xargs ~/android/android-ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-strip -s

[bog-dan-ro]

@pirama-arumuga-nainar QtXML seems the easiest to check it hs just a few .o files. You can find the .o files in qtbase/src/xml/.obj and from qtbase/src/xml you can make clean all to see the command lines that are executed.

[DanAlbert]

@bog-dan-ro: Just looking at build output, you're not using -ffunction-sections, -fdata-sections, or -Wl,--gc-sections (I've tried adding them a few times, but there seem to be some races in the build system and both make -B and make clean fail for me, and re-running configure takes ages). That would definitely be the first step in shrinking your binaries.

Another thing worth noting, I see you're using C++1z with Clang. I hope you're not using anything higher than C++11 when building with GCC, because GCC 4.9's C++14 ABI is not correct (it's still a pre-final ABI that is no longer supported by GCC or by Clang).

[bog-dan-ro]

@DanAlbert -ffunction-sections -fdata-sections does shrink the binaries size made by clang, but it shrinks also gcc bins even more :)

bogdan@zmeu:~/work/qt/qtbase/lib$ ls -l libQt5Xml.so 
-rwxr-xr-x 1 bogdan bogdan 120008 oct 27 13:55 libQt5Xml.so
bogdan@zmeu:~/work/qt/qtbase-clang/lib$ ls -l libQt5Xml.so 
-rwxr-xr-x 1 bogdan bogdan 197432 oct 27 13:54 libQt5Xml.so

As you can see the percentage is the same ~65% ...

To test it locally please apply https://paste.kde.org/pmagqlgvf and make clean all. No need to reconfigure!

[bog-dan-ro]

Sorry, the gcc bins are still the same. clang bins shrinked from 201532 to 197432, but still ~65% larger than gcc.

[pirama-arumuga-nainar]

@bog-dan-ro I am unable to build with Clang based on your instructions. The ARM triple doesn't get passed to the .c files in harfbuzz. If I do configure with '-target ..' in CFLAGS, some files that get compiled with gcc throw an error.

[bog-dan-ro]

@pirama-arumuga-nainar did you used ndkr13 for clang?
I'm on IRC (freenode) #necessitas channel, please feel free to ping me if you need any help.

[pirama-arumuga-nainar]

@froydnj Just looking at the libxul shared objects takes me down a rabbit hole. I definitely see differences in inlining in a few places. Can you share build instructions or maybe a capture of the build commands for a single module?

@bog-dan-ro Here's the list of commands I run:

git clone git://code.qt.io/qt/qt5.git -b 5.7.1 qt5-clang
cd qt5-clang
./init-repository -f --ignore-submodules qtwebkit qtwebkit-examples  qtwebengine
CFLAGS="-ffunction-sections -fdata-sections" ./configure -developer-build -release -xplatform android-clang -nomake tests -nomake examples -android-ndk <path-to-ndk-r13> -android-sdk <path-to-sdk>
make

I get the following error:

/ssd1/pirama/ndk-related/android-ndk-r13/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: .obj/harfbuzz-buffer.o: incompatible target

All the .c files in harfbuzz are targeted for x86_64.

[bog-dan-ro]

@pirama-arumuga-nainar You don't need to clone all the modules, qtbase is enough. I've updated the instructions a few days ago, please check #133 (comment) again.
Also make sure you replace /path/to/NDKr13 and /path/to/SDK with the real paths :)

[bog-dan-ro]

@pirama-arumuga-nainar please use 5.6.2 branch instead, it seems we have some problems in 5.7.1 branch (sorry for not checking it earlier).

EDIT: clean & rebuild 5.7.1 branch and everything seems ok.

[rprichard]

Firefox has switched to using Clang by default: https://bugzilla.mozilla.org/show_bug.cgi?id=1163171. Regarding code size:

https://bugzilla.mozilla.org/show_bug.cgi?id=1163171#c14:

clang 5.0 (NDK 15b) with -Oz (but turn off elfhack since elfhack throws assertion)
32754501 Jul 6 09:52 fennec-56.0a1.en-US.android-arm.apk

gcc 4.9 (NDK 15b)
32918280 Jul 6 10:19 fennec-56.0a1.en-US.android-arm.apk

clang w/ -Oz is smaller than gcc w/ -Os.

[rprichard]

Something else to look at: try fixing/reenabling the -arm-promote-constant optimization (#642).

[enh]

given that firefox has switched, should we move remaining issues to a separate Qt bug?

[DanAlbert]

The fundamental issues here are still present. Firefox switched despite this issue, not because the issue is fixed aiui.

@rprichard it looks like you still have a handful of things mentioned above that you want to investigate. Up to you if you want to split those out into separate bugs and close this one.

[DanAlbert]

Oh, never mind. Missed the update showing that -Oz outperformed GCC. There are still a couple other avenues for improving things further here though, so if we're going to close this it should be while also opening other bugs to track those.

[enh]

we should probably update https://android.googlesource.com/platform/ndk/+/master/docs/ClangMigration.md too with some of the other tips. (and make it more findable. i have to search for "Clang Migration Notes" rather than "NDK clang migration" to find it, and i never remember that. i usually end up finding it by browsing the source tree :-/ .)

actually, maybe we should just have a more general "Things that affect your binary size" page --- much of this stuff (such as -Bsymbolic or -ffunction-sections) will still be useful even in the distant future when no-one's thinking in terms of "migration".

[rprichard]

#573 is fixed now, so I think we could switch the NDK build systems' defaults back to -Oz.

[DanAlbert]

Yeah, I'll do that.

[enh]

https://android-review.googlesource.com/c/platform/ndk/+/708632 switched us back to -Oz, so this is fixed in r18?

[rprichard]

I think there's more work we could do here, but maybe we can close it. It's an open-ended bug report.

• r18 also includes my EH table size reduction (https://reviews.llvm.org/D43001)
• Clang still forces a frame pointer on arm32, whereas gcc didn't. (I saw a 2-3% reduction by adding -fomit-frame-pointer.)
• I just retested libQtCore.so with r18's Clang, and the binary is still 7.3% bigger than GCC's. (It's only 4.0% bigger if I build with -Oz -fomit-frame-pointer.)
• Looking at all the Qt solibs combined, r18's Clang binaries are 3.9% bigger than GCC's and 1.0% bigger with -Oz -fomit-frame-pointer.

---

Edit: When I tested with r18, I accidentally compiled it in a way that statically linked part of libc++ into the Qt binaries, inflating the Clang binary sizes. I'll post updated figures later.

[rprichard]

I measured Qt5 sizes again using the 5.11.1 tag and this patch, which is needed to work with the per-API linker scripts (#672). It seems that -lc++ doesn't work anymore, but Qt always links libc++ dynamically, so I changed qmake.conf to look for a particular libc++.so linker script, with either r18 or an earlier NDK:

diff --git a/mkspecs/android-clang/qmake.conf b/mkspecs/android-clang/qmake.conf
index b665000d00..b136bb0be4 100644
--- a/mkspecs/android-clang/qmake.conf
+++ b/mkspecs/android-clang/qmake.conf
@@ -40,7 +40,8 @@ QMAKE_CFLAGS += -DANDROID_HAS_WSTRING --sysroot=$$NDK_ROOT/sysroot \
 ANDROID_SOURCES_CXX_STL_LIBDIR = $$NDK_ROOT/sources/cxx-stl/llvm-libc++/libs/$$ANDROID_TARGET_ARCH
 
 ANDROID_STDCPP_PATH = $$ANDROID_SOURCES_CXX_STL_LIBDIR/libc++_shared.so
-ANDROID_CXX_STL_LIBS = -lc++
+ANDROID_CXX_STL_LIBS = $$ANDROID_SOURCES_CXX_STL_LIBDIR/libc++.so.$$replace(ANDROID_PLATFORM, "android-", "")
+!exists($$ANDROID_CXX_STL_LIBS): ANDROID_CXX_STL_LIBS = $$ANDROID_SOURCES_CXX_STL_LIBDIR/libc++.so
 
 QMAKE_CFLAGS_OPTIMIZE_SIZE = -Oz
 

I used GCC from r17b and Clang from r18 build 4855205, and I targeted either android-16 or android-21. For Clang, I added -fomit-frame-pointer to QMAKE_CFLAGS_OPTIMIZE_SIZE to match GCC's behavior. android-21 binaries are smaller because they aren't linked with libandroid_support.a. (I wouldn't think android-21 would be the default SDK minimum because AFAICT it's going to exclude 15% of active devices, but it was used above and on https://wiki.qt.io/Android.)

API 16:

	GCC	Clang	Change	Change(Pct)
libQt5Concurrent	13712	18192	4480	32.67%
libQt5Core	3759300	3816628	57328	1.52%
libQt5Gui	2864572	2856168	-8404	-0.29%
libQt5Network	611792	604324	-7468	-1.22%
libQt5OpenGL	218484	218712	228	0.10%
libQt5PrintSupport	247204	250792	3588	1.45%
libQt5Sql	128372	131564	3192	2.49%
libQt5Test	177708	183896	6188	3.48%
libQt5Widgets	3622528	3608236	-14292	-0.39%
libQt5Xml	128360	129672	1312	1.02%
TOTAL	11772032	11818184	46152	0.39%

API 21:

	GCC	Clang	Change	Change(Pct)
libQt5Concurrent	13712	18192	4480	32.67%
libQt5Core	3759300	3804328	45028	1.20%
libQt5Gui	2864572	2847976	-16596	-0.58%
libQt5Network	611792	604328	-7464	-1.22%
libQt5OpenGL	218484	218712	228	0.10%
libQt5PrintSupport	247204	246700	-504	-0.20%
libQt5Sql	128372	131564	3192	2.49%
libQt5Test	177708	183908	6200	3.49%
libQt5Widgets	3622528	3600048	-22480	-0.62%
libQt5Xml	128360	129672	1312	1.02%
TOTAL	11772032	11785428	13396	0.11%

[perracodex]

I understand the arm64-v8a binaries are larger than armeabi-v7a, but, is it normal they are all between 20% to 30% larger than the v7a version? This is what I'm getting, same code same compilation, yet 20% to 30% size difference per binary.

[enh]

armv7 has the https://en.wikipedia.org/wiki/ARM_architecture#Thumb compressed encoding so many common instructions will be 16-bit. armv8 has no compressed encoding so every instruction is 32-bit.

[stephenhines]

20-30% is within the realm of what we see for 32-bit vs. 64-bit as well, so this isn't unexpected.

[DanAlbert]

The only open question remaining here is whether -fomit-frame-pointer should be the default: #824.