|
| 1 | +This is the D8 and R8 integration specification for Xamarin.Android. |
| 2 | + |
| 3 | +# What is D8? What is R8? |
| 4 | + |
| 5 | +At a high level, here are the steps that occur during an Android |
| 6 | +application's Java compilation: |
| 7 | +- `javac` compiles Java code |
| 8 | +- `desugar` remove's the "sugar" (from Java 8 features) that are not |
| 9 | + fully supported on Android |
| 10 | +- ProGuard shrinks compiled Java code |
| 11 | +- `dx` "dexes" compiled Java code into Android [dex][dex] format. This |
| 12 | + is an alternate Java bytecode format supported by the Android |
| 13 | + platform. |
| 14 | + |
| 15 | +This process has a few issues, such as: |
| 16 | +- [proguard](https://www.guardsquare.com/en/products/proguard/manual) |
| 17 | + is made by a third party, and aimed for Java in general (not Android |
| 18 | + specific) |
| 19 | +- `dx` is slower than it _could_ be |
| 20 | + |
| 21 | +So in 2017, Google announced a "next-generation" dex compiler named |
| 22 | +[D8](https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html). |
| 23 | + |
| 24 | +- D8 is a direct replacement for `dx` |
| 25 | +- R8 is a replacement for ProGuard, that also "dexes" at the same |
| 26 | + time. If using R8, a D8 call is not needed. |
| 27 | + |
| 28 | +Both tools have support for various other Android-specifics: |
| 29 | +- Both `desugar` by default unless the `--no-desugaring` switch is |
| 30 | + specified |
| 31 | +- Both support [multidex][multidex], although `d8` does not have |
| 32 | + support for using the ProGuard rules format (the |
| 33 | + `--main-dex-rules` switch). |
| 34 | +- R8 has full support for [multidex][multidex]. |
| 35 | + |
| 36 | +Additionally, R8 is geared to be backwards compatible to ProGuard. |
| 37 | +It uses the same file format for configuration and command-line |
| 38 | +parameters as ProGuard. However, at the time of writing this, there |
| 39 | +are still several flags/features not implemented in R8 yet. |
| 40 | + |
| 41 | +For more information on how R8 compares to ProGuard, please see |
| 42 | +[this comparison from the ProGuard team](https://www.guardsquare.com/en/blog/proguard-and-r8). |
| 43 | + |
| 44 | +You can find the source for D8 and R8 at: |
| 45 | +<https://r8.googlesource.com/r8/> |
| 46 | + |
| 47 | +For reference, `d8 --help`: |
| 48 | +``` |
| 49 | +Usage: d8 [options] <input-files> |
| 50 | + where <input-files> are any combination of dex, class, zip, jar, or apk files |
| 51 | + and options are: |
| 52 | + --debug # Compile with debugging information (default). |
| 53 | + --release # Compile without debugging information. |
| 54 | + --output <file> # Output result in <outfile>. |
| 55 | + # <file> must be an existing directory or a zip file. |
| 56 | + --lib <file> # Add <file> as a library resource. |
| 57 | + --classpath <file> # Add <file> as a classpath resource. |
| 58 | + --min-api # Minimum Android API level compatibility |
| 59 | + --intermediate # Compile an intermediate result intended for later |
| 60 | + # merging. |
| 61 | + --file-per-class # Produce a separate dex file per input class |
| 62 | + --no-desugaring # Force disable desugaring. |
| 63 | + --main-dex-list <file> # List of classes to place in the primary dex file. |
| 64 | + --version # Print the version of d8. |
| 65 | + --help # Print this message. |
| 66 | +``` |
| 67 | + |
| 68 | +For reference, `r8 --help`: |
| 69 | +``` |
| 70 | +Usage: r8 [options] <input-files> |
| 71 | + where <input-files> are any combination of dex, class, zip, jar, or apk files |
| 72 | + and options are: |
| 73 | + --release # Compile without debugging information (default). |
| 74 | + --debug # Compile with debugging information. |
| 75 | + --output <file> # Output result in <file>. |
| 76 | + # <file> must be an existing directory or a zip file. |
| 77 | + --lib <file> # Add <file> as a library resource. |
| 78 | + --min-api # Minimum Android API level compatibility. |
| 79 | + --pg-conf <file> # Proguard configuration <file>. |
| 80 | + --pg-map-output <file> # Output the resulting name and line mapping to <file>. |
| 81 | + --no-tree-shaking # Force disable tree shaking of unreachable classes. |
| 82 | + --no-minification # Force disable minification of names. |
| 83 | + --no-desugaring # Force disable desugaring. |
| 84 | + --main-dex-rules <file> # Proguard keep rules for classes to place in the |
| 85 | + # primary dex file. |
| 86 | + --main-dex-list <file> # List of classes to place in the primary dex file. |
| 87 | + --main-dex-list-output <file> # Output the full main-dex list in <file>. |
| 88 | + --version # Print the version of r8. |
| 89 | + --help # Print this message. |
| 90 | +``` |
| 91 | + |
| 92 | +# What did Xamarin.Android do *before* D8/R8? |
| 93 | + |
| 94 | +In other words, what is currently happening *before* we introduce D8/R8 support? |
| 95 | + |
| 96 | +1. The [Javac](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Javac.cs) |
| 97 | + MSBuild task compiles `*.java` files to a `classes.zip` file. |
| 98 | +2. The [Desugar](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Desugar.cs) |
| 99 | + MSBuild task "desugars" using `desugar_deploy.jar` if |
| 100 | + `$(AndroidEnableDesugar)` is `True`. |
| 101 | +3. The [Proguard](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Proguard.cs) |
| 102 | + MSBuild task shrinks the compiled Java code if |
| 103 | + `$(AndroidEnableProguard)` is `True`. Developers may also supply |
| 104 | + custom proguard configuration files via `ProguardConfiguration` |
| 105 | + build items. |
| 106 | +4. The [CreateMultiDexMainDexClassList](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/CreateMultiDexMainDexClassList.cs) |
| 107 | + MSBuild task runs `proguard` to generate a final, combined |
| 108 | + `multidex.keep` file if `$(AndroidEnableMultiDex)` is `True`. |
| 109 | + Developers can also supply custom `multidex.keep` files via |
| 110 | + `MultiDexMainDexList` build items. |
| 111 | +5. The [CompileToDalvik](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/CompileToDalvik.cs) |
| 112 | + MSBuild task runs `dx.jar` to generate a final `classes.dex` file |
| 113 | + in `$(IntermediateOutputPath)android\bin`. If `multidex` is |
| 114 | + enabled, a `classes2.dex` (and potentially more) are also generated |
| 115 | + in this location. |
| 116 | + |
| 117 | +# What does this process look like with D8 / R8 enabled? |
| 118 | + |
| 119 | +Xamarin.Android now has two new MSBuild tasks: `<R8/>` and `<D8/>`. |
| 120 | + |
| 121 | +1. The [Javac](https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Tasks/Javac.cs) |
| 122 | + MSBuild task will remain unchanged. |
| 123 | +2. `D8` will run if `$(AndroidEnableMultiDex)` is `False`, |
| 124 | + `$(AndroidLinkTool)` is not `r8`, and "desugar" by default. |
| 125 | +3. Otherwise, `R8` will run if `$(AndroidEnableMultiDex)` is `True` or |
| 126 | + `$(AndroidLinkTool)` is `r8` and will also "desugar" by default. |
| 127 | + |
| 128 | +So in addition to be being faster in general (if Google's claims are |
| 129 | +true), we will be calling a *single* command line tool to produce dex |
| 130 | +files! |
| 131 | + |
| 132 | +# So how do developers use it? What are sensible MSBuild property defaults? |
| 133 | + |
| 134 | +Currently, a `csproj` file might have the following properties: |
| 135 | +```xml |
| 136 | +<Project> |
| 137 | + <PropertyGroup> |
| 138 | + <AndroidEnableProguard>True</AndroidEnableProguard> |
| 139 | + <AndroidEnableMultiDex>True</AndroidEnableMultiDex> |
| 140 | + <AndroidEnableDesugar>True</AndroidEnableDesugar> |
| 141 | + </PropertyGroup> |
| 142 | +</Project> |
| 143 | +``` |
| 144 | + |
| 145 | +To enable the new behavior, we have introduced two new enum-style |
| 146 | +properties: |
| 147 | +- `$(AndroidDexTool)` - supports `dx` or `d8` |
| 148 | +- `$(AndroidLinkTool)` - supports `proguard` or `r8` |
| 149 | + |
| 150 | +But for an existing project, a developer could opt-in to the new |
| 151 | +behavior with two properties: |
| 152 | +```xml |
| 153 | +<Project> |
| 154 | + <PropertyGroup> |
| 155 | + <AndroidEnableProguard>True</AndroidEnableProguard> |
| 156 | + <AndroidEnableMultiDex>True</AndroidEnableMultiDex> |
| 157 | + <AndroidEnableDesugar>True</AndroidEnableDesugar> |
| 158 | + <!--New properties--> |
| 159 | + <AndroidDexTool>d8</AndroidDexTool> |
| 160 | + <AndroidLinkTool>r8</AndroidLinkTool> |
| 161 | + </PropertyGroup> |
| 162 | +</Project> |
| 163 | +``` |
| 164 | + |
| 165 | +There should be two new MSBuild properties to configure here, because: |
| 166 | +- You could use `D8` in combination with `proguard`, as `R8` is not |
| 167 | + "feature complete" in comparison to `proguard`. |
| 168 | +- You may not want to use code shrinking at all, but still use `D8` |
| 169 | + instead of `dx`. |
| 170 | +- You shouldn't be able to use `dx` in combination with `R8`, it |
| 171 | + doesn't make sense. |
| 172 | +- Developers should be able to use the existing properties for |
| 173 | + enabling code shrinking, `multidex`, and `desugar`. |
| 174 | + |
| 175 | +Our reasonable defaults would be: |
| 176 | +- If `AndroidDexTool` is omitted, `dx` and `CompileToDalvik` |
| 177 | + should be used. Until D8/R8 integration is deemed stable and enabled |
| 178 | + by default. |
| 179 | +- If `AndroidDexTool` is `d8` and `AndroidEnableDesugar` is |
| 180 | + omitted, `AndroidEnableDesugar` should be enabled. |
| 181 | +- If `AndroidLinkTool` is omitted and `AndroidEnableProguard` is |
| 182 | + `true`, we should default `AndroidLinkTool` to `proguard`. |
| 183 | + |
| 184 | +MSBuild properties default to: |
| 185 | +```xml |
| 186 | +<AndroidDexTool Condition=" '$(AndroidDexTool)' == '' ">dx</AndroidDexTool> |
| 187 | +<!--NOTE: $(AndroidLinkTool) would be blank if code shrinking is not used at all--> |
| 188 | +<AndroidLinkTool Condition=" '$(AndroidLinkTool)' == '' And '$(AndroidEnableProguard)' == 'True' ">proguard</AndroidLinkTool> |
| 189 | +<AndroidEnableDesugar Condition=" '$(AndroidEnableDesugar)' == '' And ('$(AndroidDexTool)' == 'd8' Or '$(AndroidLinkTool)' == 'r8') ">True</AndroidEnableDesugar> |
| 190 | +``` |
| 191 | + |
| 192 | +If a user specifies combinations of properties: |
| 193 | +- `AndroidDexTool` = `d8` and `AndroidEnableProguard` = `True` |
| 194 | + - `AndroidLinkTool` will get set to `proguard` |
| 195 | +- `AndroidDexTool` = `dx` and `AndroidLinkTool` = `r8` |
| 196 | + - This combination doesn't really *make sense*, but we don't need to |
| 197 | + do anything: only `R8` will be called because it dexes and shrinks |
| 198 | + at the same time. |
| 199 | +- `AndroidEnableDesugar` is enabled when omitted, if either `d8` or |
| 200 | + `r8` are used |
| 201 | + |
| 202 | +For new projects that want to use D8/R8, code shrinking, and |
| 203 | +`multidex`, it would make sense to specify: |
| 204 | +```xml |
| 205 | +<Project> |
| 206 | + <PropertyGroup> |
| 207 | + <AndroidEnableMultiDex>True</AndroidEnableMultiDex> |
| 208 | + <AndroidDexTool>d8</AndroidDexTool> |
| 209 | + <AndroidLinkTool>r8</AndroidLinkTool> |
| 210 | + </PropertyGroup> |
| 211 | +</Project> |
| 212 | +``` |
| 213 | + |
| 214 | +# Additional D8 / R8 settings? |
| 215 | + |
| 216 | +`--debug` or `--release` needs to be explicitly specified for both D8 |
| 217 | +and R8. We use the [AndroidIncludeDebugSymbols][debug_symbols] |
| 218 | +property for this. |
| 219 | + |
| 220 | +`$(AndroidD8ExtraArguments)` and `$(AndroidR8ExtraArguments)` can be |
| 221 | +used to explicitly pass additional flags to D8 and R8. |
| 222 | + |
| 223 | +# How are we compiling / shipping D8 and R8? |
| 224 | + |
| 225 | +We have added a submodule to `xamarin-android` for |
| 226 | +[r8](https://r8.googlesource.com/r8/). It will be pinned to a commit |
| 227 | +with a reasonable release tag, such as `1.2.50` for now. |
| 228 | + |
| 229 | +To build r8, we have to: |
| 230 | +- Download and unzip a tool named [depot_tools][depot_tools] from the |
| 231 | + Chromium project |
| 232 | +- Put the path to `depot_tools` in `$PATH` |
| 233 | +- Run `gclient` so it will download/bootstrap gradle, python, and |
| 234 | + other tools |
| 235 | +- Run `python tools\gradle.py d8 r8` to compile `d8.jar` and `r8.jar` |
| 236 | +- We will need to ship `d8.jar` and `r8.jar` in our installers, |
| 237 | + similar to how we are shipping `desugar_deploy.jar` |
| 238 | + |
| 239 | +# Performance Comparison |
| 240 | + |
| 241 | +| MSBuild Target | Options Enabled | Time | APK size (bytes) | dex size (bytes) | |
| 242 | +| --- | --- | ---: | ---: | ---: | |
| 243 | +| _CompileToDalvikWithDx | n/a | 11074ms | 13378157 | 3894720 | |
| 244 | +| _CompileToDalvikWithD8 | d8, (desugar enabled) | 8543ms | 13124205 | 3314064 | |
| 245 | +| _CompileToDalvikWithD8 | d8, (desugar disabled) | 9550ms | 13124205 | 3314064 | |
| 246 | +| _CompileToDalvikWithDx | multi-dex | 15632ms | 13390498 | 3916496 | |
| 247 | +| _CompileToDalvikWithD8 | d8, multi-dex | 25979ms | 13054626 | 3264096 | |
| 248 | +| _CompileToDalvikWithDx | proguard | 11903ms | 12804717 | 2446964 | |
| 249 | +| _CompileToDalvikWithD8 | d8, r8 | 13799ms | 12513901 | 1835588 | |
| 250 | +| _CompileToDalvikWithDx | multi-dex, proguard | 17279ms | 12804770 | 2449512 | |
| 251 | +| _CompileToDalvikWithD8 | d8, multi-dex, r8 | 13792ms | 12513954 | 1837588 | |
| 252 | + |
| 253 | +_NOTE: desugar is enabled by default with d8/r8_ |
| 254 | + |
| 255 | +I timed this builds with [this script][powershell_script], with a "Hello World" Xamarin.Forms app. Build logs here: [d8andr8.zip][d8andr8_zip] |
| 256 | + |
| 257 | +One can draw their own conclusions on which options are faster, better, smaller. |
| 258 | + |
| 259 | +Some of my thoughts: |
| 260 | +- Default options for d8 and r8 seem to be faster? |
| 261 | +- Disabling `desugar` is slower? |
| 262 | +- Enabling `multi-dex` makes the dex file larger, because new classes are required. The app wasn't large enough to warrant a `classes2.dex`. |
| 263 | +- `d8` does not support multi-dex, and so choosing `d8` + `multi-dex` actually runs `r8` with `--no-tree-shaking --no-minification`. These options are _slower_? |
| 264 | + |
| 265 | +[dex]: https://source.android.com/devices/tech/dalvik/dalvik-bytecode |
| 266 | +[multidex]: https://developer.android.com/studio/build/multidex |
| 267 | +[debug_symbols]: https://github.com/xamarin/xamarin-android/blob/221a2190ebb3aaec9ecd9b1cf8f7f6174c43153a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets#L315-L336 |
| 268 | +[depot_tools]: http://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html |
| 269 | +[powershell_script]: https://github.com/jonathanpeppers/HelloWorld/blob/39e2854f6ca39c0941fb8bd6f2a16d8b7663003e/build.ps1 |
| 270 | +[d8andr8_zip]: https://github.com/xamarin/xamarin-android/files/2470385/d8andr8.zip |
0 commit comments