Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[all OS versions] Android NDK is updated to r22. #2420

Closed
9 tasks done
AlenaSviridenko opened this issue Jan 11, 2021 · 29 comments
Closed
9 tasks done

[all OS versions] Android NDK is updated to r22. #2420

AlenaSviridenko opened this issue Jan 11, 2021 · 29 comments

Comments

@AlenaSviridenko
Copy link
Contributor

AlenaSviridenko commented Jan 11, 2021

Breaking changes
Default Android NDK was updated to r22.

Target date
Images deployment starts January, 11 and will take 3-4 days.

The motivation for the changes
We provide only the latest NDK by default due to space concerns

Possible impact
If your workflow uses features that were removed from r22 — your build will be broken. Please, refer to NDR r22 changelog for more information.

Virtual environments affected

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Ubuntu 20.04
  • macOS 10.13
  • macOS 10.14
  • macOS 10.15
  • macOS 11.0
  • Windows Server 2016 R2
  • Windows Server 2019

Mitigation ways

  • Ubuntu: sudo ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;<exact_version>"
  • macOS: ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;<exact_version>"
  • Windows: $env:ANDROID_HOME/tools/bin/sdkmanager --install "ndk;<exact_version>"

If the time is critical for you, consider using actions/cache to win some installation time.

@carstenhag
Copy link

Hello there! We use Azure DevOps pipelines with the macOS 10.15 image to build our Android apps.

Yesterday we wanted to build releases of our apps, but we got gradle errors back that the NDK version was not available. We googled and found this issue :). We then tried out your specified mitigation way, but that unfortunately did not work.

We tried out all of these commands, but they all failed:

sudo ${ANDROID_HOME}/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} --install "ndk;22.0.7026061"
sudo ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;22.0.7026061"
export JAVA_OPTS='-XX:+IgnoreUnrecognizedVMOptions --add-modules java.se.ee'
sudo ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;22.0.7026061"

Error:

Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema
	at com.android.repository.api.SchemaModule$SchemaModuleVersion.<init>(SchemaModule.java:156)
	at com.android.repository.api.SchemaModule.<init>(SchemaModule.java:75)
	at com.android.sdklib.repository.AndroidSdkHandler.<clinit>(AndroidSdkHandler.java:81)
	at com.android.sdklib.tool.sdkmanager.SdkManagerCli.main(SdkManagerCli.java:73)
	at com.android.sdklib.tool.sdkmanager.SdkManagerCli.main(SdkManagerCli.java:48)
Caused by: java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlSchema
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 5 more

Could you please tell us when the migration will be finished, or how we can mitigate the issue? Seems like today is the last day of migration, but the rollout progress is only at 25%. Thanks in advance!

@maxim-lobanov
Copy link
Contributor

Hello @carstenhag, Could you please provide image version from your last successful build and first failed build? (Image version can be found in the logs of first step of build).
What exact version of NDK do you need to use in your build?
The previous image version (20201212.1) contained NDK 21.3.6528147. The new image version (20210110.1) contains NDK 22.0.7026061.
If your build started to fail after image upgrade, I guess you need to install NDK 21.3.6528147 in runtime, not 22.0.7026061 (if I understand you correctly)

@ystefinko
Copy link

ystefinko commented Jan 15, 2021

ndk-bundle now don't contain directory called : platforms. We faced issue with gradle:
Configure project :app
NDK is missing a "platforms" directory.
If you are using NDK, verify the ndk.dir is set to a valid NDK directory. It is currently set to /usr/local/lib/android/sdk/ndk-bundle.
If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning.

Issue happened first time yesterday afternoon and continues today. Can you please support ?
Job url: https://dev.azure.com/heremaps/here-data-sdk/_build/results?buildId=6038&view=logs&j=73abdb92-db38-5d21-0cd3-fc834455f885&t=539f8d8b-d758-576c-ec15-b41b94d5cb7f

@carstenhag
Copy link

carstenhag commented Jan 15, 2021

Hello there! We have been using 21.3.6528147 for some months now. Our builds failed with that (because we got a new image), the logs said that we should use 22.0.7026061, so we did so and we committed that.

But now we seem to always get old images again which don't have 22.0.7026061 so I wanted to manually install it :D

So basically, some images were 20201212.1, some 20210110.1, as expected as the rollout is not finished. Just the mitigation way provided does not work, which is bad.

@dentuzhik
Copy link

dentuzhik commented Jan 15, 2021

We just hit the NDK mismatch on our CI, and while I understand that you have to keep them updated, is there any way to announce it through some other channels?
It hit our releases quite unexpectedly and it would be nice to know in advance 🙇🏼

I guess now with discussions available, it could have been a "Discussion" which one could watch?
image

Or is there maybe a way to "freeze" the version of virtual environments so upgrades are in the hands of consumer?

@carstenhag
Copy link

Same here. I know that Azure DevOps does announce brownouts of image and have seen those before. Would have been very good to announce this as well.

@maxim-lobanov
Copy link
Contributor

@ystefinko, could you please try mitigation step above? Adding sudo ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;21.3.6528147" as a first step of your build?

@carstenhag , sorry for inconvenience, deployment should be finished on Monday.
I think you have two options:

  • reverting your app to NDK 21.3.6528147 and use mitigation step sudo ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;21.3.6528147"
  • leave on 22.0.7026061 and manually install 22.0.7026061. As soon as new image is fully deployed, installation time will be reduced to zero since NDK will be already available.

@carstenhag
Copy link

carstenhag commented Jan 15, 2021

@maxim-lobanov unfortunately the suggested line does not work (I had posted the errors we got at #2420 (comment)).

(We have queued our release pipeline 30 times today and our two releases did now go through, seems like 2 out of 30 times it did get the new image)

@maxim-lobanov
Copy link
Contributor

@dentuzhik , have you tried to subscribe to releases?

Regular Weekly Rhythm can be followed by watching Releases. Pre-release is created when deployment is started. As soon as deployment is finished, it is converted to release.

@carstenhag , sorry for inconvenience. Usually deployment of macOS takes less 2 days but this week we have stuck with multiple environment issues.

@carstenhag
Copy link

Maybe we are talking past eachother, I will try to be more specific :). It is not a problem that the rollout takes some time.

For us it is a problem, that the mitigation does not work. We would have been happy if the mitigation would have installed the ndk version.

@Kerizer
Copy link

Kerizer commented Jan 15, 2021

For us root cause was in bug with Android gradle plugin - https://issuetracker.google.com/issues/144111441
With 4.1.1 version it's possible to completely remove hardcoded ndkVersion and let plugin to resolve it automatically

@maxim-lobanov
Copy link
Contributor

@carstenhag , oh, just realized an issue.
Try without sudo ${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;21.3.6528147". sudo is needed only for Ubuntu images. Will fix the first post.

@carstenhag
Copy link

Thanks, that did work (I used ${ANDROID_HOME}/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} --install "ndk;22.0.7026061" to be precise, but I guess --sdk_root can be omitted)

@maxim-lobanov
Copy link
Contributor

I have logged bug for an issue with sudo: #2454
sdkmanager refers incorrect Java version via sudo. It is reason why initial command fails on macOS.

@al-cheb
Copy link
Contributor

al-cheb commented Jan 15, 2021

This behavior is by design for sudo:
-E' The -E (preserve environment) option indicates to the security policy that the user wishes to preserve their existing environment variables. The security policy may return an error if the -E option is specified and the user does not have permission to preserve the environment.

you should use flag -E to pass env variables, because JAVA_HOME is missing:

Mac-1610725225646:~ runner$ sudo java -version
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.2+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode, sharing)

Mac-1610725225646:~ runner$ sudo -E java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.275-b01, mixed mode)

unset JAVA_HOME:

Mac-1610725225646:~ runner$ unset JAVA_HOME
Mac-1610725225646:~ runner$ java -version
openjdk version "14.0.2" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.2+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode, sharing)

@ystefinko
Copy link

@maxim-lobanov
I tried out all of these commands, but they all failed:

sudo ${ANDROID_HOME}/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} --install "ndk;22.0.7026061"
sudo ${ANDROID_HOME}/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} --install "ndk;21.3.6528147"
unset ANDROID_NDK_HOME
Gradle 5.1.1, Android gradle plugin 3.4.0(tried 3.4.3).
Our android example app still not compiling: https://dev.azure.com/heremaps/here-data-sdk/_build?definitionId=3&_a=summary
We need urgent support , as our development became blocked by changes on your side.

@carstenhag
Copy link

carstenhag commented Jan 18, 2021

@ystefinko As maxim wrote at #2420 (comment), leave out the sudo, then it works. I also confirmed that this works some comments ago...

${ANDROID_HOME}/tools/bin/sdkmanager --sdk_root=${ANDROID_HOME} --install "ndk;22.0.7026061" will install the ndk version 22.0.7026061. If you need any other version, specify that in its place.

@ystefinko
Copy link

@carstenhag and @maxim-lobanov issue still exist. I ran it without sudo too.

@dmitry-shibanov
Copy link
Contributor

Hello @ystefinko. As a solution you can try

  • setup build gradle to the 4.1.0 : classpath 'com.android.tools.build:gradle:4.1.0'
  • update gradle version to 6.5.1: distributionUrl=https\://services.gradle.org/distributions/gradle-6.5.1-all.zip
  • Update build.sh part to this:
${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;21.3.6528147" --sdk_root=${ANDROID_HOME} > /dev/null
export ANDROID_NDK_HOME=$ANDROID_HOME/ndk/21.3.6528147

mkdir -p build && cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE="$ANDROID_HOME/ndk/21.3.6528147/build/cmake/android.toolchain.cmake" \
	-DANDROID_PLATFORM=android-28 \
	-DANDROID_NDK="$ANDROID_HOME/ndk/21.3.6528147" \
	-DANDROID_ABI=arm64-v8a \
	-DOLP_SDK_ENABLE_TESTING=NO \
	-DOLP_SDK_BUILD_EXAMPLES=ON

@ystefinko
Copy link

@dmitry-shibanov hello!
Thanks for advice!

All your suggestions helped. Build passed.
Minor issues detected:

  1. Gradle detected incorrect version of NDK -
    > Configure project :app
    2021-01-19T11:16:48.6826915Z Checking the license for package NDK (Side by side) 21.1.6352462 in /usr/local/lib/android/sdk/licenses
    2021-01-19T11:16:48.6827494Z License for package NDK (Side by side) 21.1.6352462 accepted.
    2021-01-19T11:16:48.6828028Z Preparing "Install NDK (Side by side) 21.1.6352462 (revision: 21.1.6352462)".
    2021-01-19T11:18:09.0800388Z "Install NDK (Side by side) 21.1.6352462 (revision: 21.1.6352462)" ready.
    2021-01-19T11:18:09.0801346Z Installing NDK (Side by side) 21.1.6352462 in /usr/local/lib/android/sdk/ndk/21.1.6352462
    2021-01-19T11:18:09.0801881Z "Install NDK (Side by side) 21.1.6352462 (revision: 21.1.6352462)" complete.
    2021-01-19T11:18:09.3817717Z "Install NDK (Side by side) 21.1.6352462 (revision: 21.1.6352462)" finished.

  2. $GRADLE_HOME variable is available but I can't use it. Seems like PATH is incorrect, because we can't access any file there. This is minor as we are sill using gradlew.

@AlenaSviridenko AlenaSviridenko unpinned this issue Jan 19, 2021
@AlenaSviridenko AlenaSviridenko pinned this issue Jan 19, 2021
@sashaweiss
Copy link

This broke our CI as well - would love for there to be an announcement channel for stuff like this. Quick note that the docs here for the macos-10.15 installed software (and probably in the lists for other images) don't show this as updated, noticed when I went googling for a solution.

ystefinko added a commit to heremaps/here-data-sdk-cpp that referenced this issue Jan 20, 2021
Azure updated their pre-installed NDK from 21 to 22.
Also NDK 22 don't include `platforms` directory.
Examples were not compiling due to error:
"NDK is missing a "platforms" directory"
Fixed by installing NDK 21 in job and using it for
further compilations.
Discussed in:
actions/runner-images#2420

Also gradle and android-gradle-plugin were updated.

Relates-To: OLPEDGE-2443

Signed-off-by: Yaroslav Stefinko <[email protected]>
ystefinko added a commit to heremaps/here-data-sdk-cpp that referenced this issue Jan 20, 2021
Azure updated their pre-installed NDK from 21 to 22.
Also NDK 22 don't include `platforms` directory.
Examples were not compiling due to error:
"NDK is missing a "platforms" directory"
Fixed by installing NDK 21 in job and using it for
further compilations.
Discussed in:
actions/runner-images#2420

Also gradle and android-gradle-plugin were updated.

Relates-To: OLPEDGE-2443

Signed-off-by: Yaroslav Stefinko <[email protected]>
@AlenaSviridenko
Copy link
Contributor Author

AlenaSviridenko commented Jan 20, 2021

Hi all,
we started working on rolling back images due to huge impact. It can take 1-2 days.

We will revisit the strategy of NDK support, and will rollout image with the fix next week.
Sorry for inconveniences and thank you for your patience.

@carstenhag
Copy link

carstenhag commented Jan 20, 2021

@sashaweiss About the readme being out of date: The READMEs only get updated after the rollout is 100% through. No clue where I saw it, but a contributor mentioned this

@miketimofeev
Copy link
Contributor

@carstenhag @sashaweiss the proper link to the readme can be found in the set up job step
image

@sashaweiss
Copy link

@miketimofeev thanks for the tip! I'll use that next time, that'll be handy.

@typeoneerror
Copy link

Earlier in the week I bumped to 22.x and a few actions have run fine.

This morning...

No version of NDK matched the requested version 22.0.7026061. Versions available locally: 21.3.6528147

Was this rolled back in ubuntu-latest?

@maxim-lobanov
Copy link
Contributor

Unfortunately, yes. We have got a lot of reports from customers about breaking builds with NDK 22.x. Also we have got feedback from Xamarin.Android team - NDK 22 is not supported ye. We have to roll images back and freeze NDK 21 on images for a while. A bit more context in #2481 (comment)

jonpryor pushed a commit to dotnet/android-tools that referenced this issue Jan 20, 2021
Context: actions/runner-images#2420
Context: dotnet/android#5499
Context: dotnet/android#5526
Context: android/ndk#1427
Context: https://developer.android.com/studio/command-line/variables#envar

Xamarin.Android is not (yet) compatible with the recently released
Android NDK r22 version.  Azure build images have recently rolled out
an update which includes NDK r22 and, thus, it breaks builds for
customers using any form of Xamarin.Android AOT build.

In an attempt to detect broken/incompatible NDK versions as well as
to select the "best one", this commit adds code to scan the known NDK
locations in search of the preferred version.

Given an `AndroidSdkInfo` creation of:

	var info = new AndroidSdkInfo (logger:logger,
	        androidSdkPath: sdkPath,
	        androidNdkPath: ndkPath,
	        javaSdkPath: javaPath);
	var usedNdkPath = info.AndroidNdkPath;

If `ndkPath` is not `null` and otherwise valid, then `usedNdkPath`
is `ndkPath`.

If `ndkPath` is `null` or is otherwise invalid (missing `ndk-stack`,
etc.), then we search for an `info.AndroidNdkPath` value as follows:

 1. If `androidSdkPath` is not `null` and valid, then we check for
    Android SDK-relative NDK locations, in:

      * `{androidSdkPath}/ndk/*`
      * `{androidSdkPath}/ndk-bundle`

    For each found SDK-relative NDK directory, we filter out NDKs for
    which we cannot determine the package version, as well as those
    which are "too old" (< `MinimumCompatibleNDKMajorVersion`) or
    "too new" (> `MaximumCompatibleNDKMajorVersion`), currently r22.

    We prefer the NDK location with the highest version number. 

 2. If `androidSdkPath` is not `null` and valid and if there are no
    Android SDK-relative NDK locations, then we use the user-selected
    "preferred NDK location".  See also
    `AndroidSdkInfo.SetPreferredAndroidNdkPath()`.

 3. If `androidSdkPath` is not `null` and valid and if the preferred
    NDK location isn't set or is invalid, then we check directories
    specified in `$PATH`, and use the directory which contains
    `ndk-stack`.

 4. If `androidSdkPath` is not `null` and valid and `$PATH` didn't
    contain `ndk-stack`, then we continue looking for NDK locations
    within the Android SDK locations specified by the `$ANDROID_HOME`
    and `$ANDROID_SDK_ROOT` environment variables.
    As with (1), these likewise look for e.g. `${ANDROID_HOME}/ndk/*`
    or `${ANDROID_SDK_ROOT}/ndk-bundle` directories and select the
    NDK with the highest supported version.

 5. If `androidSdkPath` is `null`, then *first* we try to find a
    valid Android SDK directory, using on Unix:

     a. The preferred Android SDK directory; see also
        `AndroidSdkInfo.SetPreferredAndroidSdkPath().

     b. The `$ANDROID_HOME` and `ANDROID_SDK_ROOT`
        environment variables.

     c. Directories within `$PATH` that contain `adb`.

    Once an Android SDK is found, steps (1)…(4) are performed.

In (1) and (4), we now look for the Android SDK packages containing
the NDK.  There are two kinds of such packages:

  * `ndk-bundle` is the older package which allows for installation of
    only one NDK inside the SDK directory
  * `ndk/*` is a newer package which allows for installation of several
    NDK versions in parallel.  Each subdirectory of `ndk` is an `X.Y.Z`
    version number of the NDK.

In each of these directories we look for the `source.properties` file
from which we then extract the NDK version and then we sort thus
discovered NDK instances using their version as the key, in the
descending order.  The latest compatible (currently: less than 22 and
more than 15) version is selected and its path returned to the caller.
jonpryor pushed a commit to dotnet/android-tools that referenced this issue Jan 20, 2021
Context: actions/runner-images#2420
Context: dotnet/android#5499
Context: dotnet/android#5526
Context: android/ndk#1427
Context: https://developer.android.com/studio/command-line/variables#envar

Xamarin.Android is not (yet) compatible with the recently released
Android NDK r22 version.  Azure build images have recently rolled out
an update which includes NDK r22 and, thus, it breaks builds for
customers using any form of Xamarin.Android AOT build.

In an attempt to detect broken/incompatible NDK versions as well as
to select the "best one", this commit adds code to scan the known NDK
locations in search of the preferred version.

Given an `AndroidSdkInfo` creation of:

	var info = new AndroidSdkInfo (logger:logger,
	        androidSdkPath: sdkPath,
	        androidNdkPath: ndkPath,
	        javaSdkPath: javaPath);
	var usedNdkPath = info.AndroidNdkPath;

If `ndkPath` is not `null` and otherwise valid, then `usedNdkPath`
is `ndkPath`.

If `ndkPath` is `null` or is otherwise invalid (missing `ndk-stack`,
etc.), then we search for an `info.AndroidNdkPath` value as follows:

 1. If `androidSdkPath` is not `null` and valid, then we check for
    Android SDK-relative NDK locations, in:

      * `{androidSdkPath}/ndk/*`
      * `{androidSdkPath}/ndk-bundle`

    For each found SDK-relative NDK directory, we filter out NDKs for
    which we cannot determine the package version, as well as those
    which are "too old" (< `MinimumCompatibleNDKMajorVersion`) or
    "too new" (> `MaximumCompatibleNDKMajorVersion`), currently r22.

    We prefer the NDK location with the highest version number. 

 2. If `androidSdkPath` is not `null` and valid and if there are no
    Android SDK-relative NDK locations, then we use the user-selected
    "preferred NDK location".  See also
    `AndroidSdkInfo.SetPreferredAndroidNdkPath()`.

 3. If `androidSdkPath` is not `null` and valid and if the preferred
    NDK location isn't set or is invalid, then we check directories
    specified in `$PATH`, and use the directory which contains
    `ndk-stack`.

 4. If `androidSdkPath` is not `null` and valid and `$PATH` didn't
    contain `ndk-stack`, then we continue looking for NDK locations
    within the Android SDK locations specified by the `$ANDROID_HOME`
    and `$ANDROID_SDK_ROOT` environment variables.
    As with (1), these likewise look for e.g. `${ANDROID_HOME}/ndk/*`
    or `${ANDROID_SDK_ROOT}/ndk-bundle` directories and select the
    NDK with the highest supported version.

 5. If `androidSdkPath` is `null`, then *first* we try to find a
    valid Android SDK directory, using on Unix:

     a. The preferred Android SDK directory; see also
        `AndroidSdkInfo.SetPreferredAndroidSdkPath().

     b. The `$ANDROID_HOME` and `ANDROID_SDK_ROOT`
        environment variables.

     c. Directories within `$PATH` that contain `adb`.

    Once an Android SDK is found, steps (1)…(4) are performed.

In (1) and (4), we now look for the Android SDK packages containing
the NDK.  There are two kinds of such packages:

  * `ndk-bundle` is the older package which allows for installation of
    only one NDK inside the SDK directory
  * `ndk/*` is a newer package which allows for installation of several
    NDK versions in parallel.  Each subdirectory of `ndk` is an `X.Y.Z`
    version number of the NDK.

In each of these directories we look for the `source.properties` file
from which we then extract the NDK version and then we sort thus
discovered NDK instances using their version as the key, in the
descending order.  The latest compatible (currently: less than 22 and
more than 15) version is selected and its path returned to the caller.
@AlenaSviridenko
Copy link
Contributor Author

Closing this issue, because we reverted image with r22 and will deploy this week with default to r21.

@amq
Copy link

amq commented Oct 1, 2021

Here we go again #4144

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests